Using Advene to bridge the gap between users and ontologies in movie annotation

Olivier Aubert - LS2N - Nantes Université - www.olivieraubert.net

WICED 2022

Outline

• AdA project - context and challenges
• Advene platform
• Ontology integration
• Data visualization

AdA project

https://projectada.github.io/

• Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
• Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
• Interdisciplinary collaboration: Cinepoietics (Media Studies, Freie Universität Berlin), HPI Potsdam (semantic web, video analysis)

Challenges and needs

Challenges

• Developing a methodological framework for movie analysis, with quantitative support
• Semantic video-annotations
• Exploration of automation possibilities (feature extraction)

Needs

• Provide a 2-way bridge between users and semantic data
• Optimize the manual annotation process
• Build a ground truth for future automation

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

• Accompany active reading of AV documents through annotations
• Create, user, share analyses of AV documents as hypervideos
• Started in 2002 with Yannick Prié and P.A. Champin
• Free Software (GPL) cross-platform desktop application (Python/Gtk/Gstreamer)
• www.advene.org

Principle

Features

• User defined structures and visualisations (templates, rules…)
• A package is a set of (annotations, annotation structure, views, queries) independent from the video
• Generic data model. Use of text-based microsyntax.
  • import/export filters
  • no strict constraints
• Flexible and extensible (through templates and plugins)
• Offline work

Advene and the AdA project

Principle

Semantic web extensions

• WebAnnotation/JSON-LD import and export
• OWL importer
  • Convert OWL schema into Advene structures
• OWL-derived metadata preserved
  • used in Checker framework
  • used for GUI shortcuts
  • used in WebAnnotation export

Produced ontology and data

https://projectada.github.io/

Feature-extraction plugins

• Using Advene generic feature-extraction framework or API import filters
• Principle
  • define a gstreamer audio/video decoding pipeline, such as videoconvert ! videoscale ! video/x-raw,width=160,pixel-aspect-ratio=(fraction)1/1,format=GRAY8
  • define a process_frame method, called for each frame
  • produce annotations as python dicts

Example plugin outputs

Visualisation

• Multiple visualisations are already present in Advene: timeline, table, transcript…
• Advene embeds a webserver and a template language interpreter, able to generate and serve X/HTML documents
  • Agile testbed for experiments

Web-based timeline

visualization

One possible visualisation, overview of annotations for one single movie

Perspectives

Advene in other contexts - import plugin for Blocking notation (Rémi Ronfard)

Conclusion

• Advene in the AdA project
  • Not just a tool, but an instrument integrated in the scholar process
  • Flexible and extensible
  • Plugin-based architecture
  • Standard frameworks/APIs for feature extraction, data checking, data import/export…
• The AdA video annotation dataset is available online

Thank you for your attention

Any question?

@Olivier_Aubert
www.olivieraubert.net
www.advene.org
https://projectada.github.io/