CC-BY

Using Advene to bridge the gap between users and ontologies in movie annotation

Olivier Aubert - LS2N - Nantes Université - www.olivieraubert.net
WICED 2022

Outline

  • AdA project - context and challenges
  • Advene platform
  • Ontology integration
  • Data visualization

AdA project

https://projectada.github.io/

  • Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
  • Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
  • Interdisciplinary collaboration: Cinepoietics (Media Studies, Freie Universität Berlin), HPI Potsdam (semantic web, video analysis)

Challenges and needs

Challenges

  • Developing a methodological framework for movie analysis, with quantitative support
  • Semantic video-annotations
  • Exploration of automation possibilities (feature extraction)

Needs

  • Provide a 2-way bridge between users and semantic data
  • Optimize the manual annotation process
  • Build a ground truth for future automation

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

  • Accompany active reading of AV documents through annotations
  • Create, user, share analyses of AV documents as hypervideos
  • Started in 2002 with Yannick Prié and P.A. Champin
  • Free Software (GPL) cross-platform desktop application (Python/Gtk/Gstreamer)
  • www.advene.org

Principle

Sorry, your browser does not support SVG.

Features

  • User defined structures and visualisations (templates, rules…)
  • A package is a set of (annotations, annotation structure, views, queries) independent from the video
  • Generic data model. Use of text-based microsyntax.
    • import/export filters
    • no strict constraints
  • Flexible and extensible (through templates and plugins)
  • Offline work

Advene and the AdA project

advene_ada.png

Principle

Sorry, your browser does not support SVG.

Semantic web extensions

  • WebAnnotation/JSON-LD import and export
  • OWL importer
    • Convert OWL schema into Advene structures
  • OWL-derived metadata preserved
    • used in Checker framework
    • used for GUI shortcuts
    • used in WebAnnotation export

Produced ontology and data

https://projectada.github.io/

  • 8 annotation levels (Segmentation, Camera, Montage, Image Composition, Acoustics…)
  • 78 annotation types
  • 566671 annotations over 7 movies, 69 documentaries and many TV reports
  • CC-SA 3.0

ontology_graph.png

AdA ontology model

Feature-extraction plugins

  • Using Advene generic feature-extraction framework or API import filters
  • Principle
    • define a gstreamer audio/video decoding pipeline, such as videoconvert ! videoscale ! video/x-raw,width=160,pixel-aspect-ratio=(fraction)1/1,format=GRAY8
    • define a process_frame method, called for each frame
    • produce annotations as python dicts

Example plugin outputs

advene-ada-color-motion.png

Visualisation

  • Multiple visualisations are already present in Advene: timeline, table, transcript…
  • Advene embeds a webserver and a template language interpreter, able to generate and serve X/HTML documents
    • Agile testbed for experiments

Web-based timeline

visualization

ada_timeline.png

One possible visualisation, overview of annotations for one single movie

Perspectives

blocking.png

advene_blocking.png

Advene in other contexts - import plugin for Blocking notation (Rémi Ronfard)

Conclusion

  • Advene in the AdA project
    • Not just a tool, but an instrument integrated in the scholar process
    • Flexible and extensible
    • Plugin-based architecture
    • Standard frameworks/APIs for feature extraction, data checking, data import/export…
  • The AdA video annotation dataset is available online

Thank you for your attention

Any question?

@Olivier_Aubert
www.olivieraubert.net
www.advene.org
https://projectada.github.io/