Using Advene to bridge the gap between users and ontologies in movie annotation
Olivier Aubert - LS2N - Nantes Université - www.olivieraubert.net
WICED 2022
Outline
- AdA project - context and challenges
- Advene platform
- Ontology integration
- Data visualization
AdA project
- Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
- Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
- Interdisciplinary collaboration: Cinepoietics (Media Studies, Freie Universität Berlin), HPI Potsdam (semantic web, video analysis)
Challenges and needs
Challenges
- Developing a methodological framework for movie analysis, with quantitative support
- Semantic video-annotations
- Exploration of automation possibilities (feature extraction)
Needs
- Provide a 2-way bridge between users and semantic data
- Optimize the manual annotation process
- Build a ground truth for future automation
Advene project
Annotate Digital Video, Exchange on the NEt
to be added to something or become a part of it, though inessential (Webster, 1913)
- Accompany active reading of AV documents through annotations
- Create, user, share analyses of AV documents as hypervideos
- Started in 2002 with Yannick Prié and P.A. Champin
- Free Software (GPL) cross-platform desktop application (Python/Gtk/Gstreamer)
- www.advene.org
Principle
Features
- User defined structures and visualisations (templates, rules…)
- A package is a set of (annotations, annotation structure, views, queries) independent from the video
- Generic data model. Use of text-based microsyntax.
- import/export filters
- no strict constraints
- Flexible and extensible (through templates and plugins)
- Offline work
Advene and the AdA project
Principle
Semantic web extensions
- WebAnnotation/JSON-LD import and export
- OWL importer
- Convert OWL schema into Advene structures
- OWL-derived metadata preserved
- used in Checker framework
- used for GUI shortcuts
- used in WebAnnotation export
Produced ontology and data
- 8 annotation levels (Segmentation, Camera, Montage, Image Composition, Acoustics…)
- 78 annotation types
- 566671 annotations over 7 movies, 69 documentaries and many TV reports
- CC-SA 3.0
AdA ontology model
Feature-extraction plugins
- Using Advene generic feature-extraction framework or API import filters
- Principle
- define a gstreamer audio/video decoding pipeline, such as
videoconvert ! videoscale ! video/x-raw,width=160,pixel-aspect-ratio=(fraction)1/1,format=GRAY8 - define a
process_framemethod, called for each frame - produce annotations as python dicts
- define a gstreamer audio/video decoding pipeline, such as
Example plugin outputs
Visualisation
- Multiple visualisations are already present in Advene: timeline, table, transcript…
- Advene embeds a webserver and a template language interpreter, able
to generate and serve X/HTML documents
- Agile testbed for experiments
Web-based timeline
One possible visualisation, overview of annotations for one single movie
Perspectives
Advene in other contexts - import plugin for Blocking notation (Rémi Ronfard)
Conclusion
- Advene in the AdA project
- Not just a tool, but an instrument integrated in the scholar process
- Flexible and extensible
- Plugin-based architecture
- Standard frameworks/APIs for feature extraction, data checking, data import/export…
- The AdA video annotation dataset is available online
Thank you for your attention
Any question?
@Olivier_Aubert
www.olivieraubert.net
www.advene.org
https://projectada.github.io/
1
Using Advene to bridge the gap between users and ontologies in movie annotation
Olivier Aubert - LS2N - Nantes Université - www.olivieraubert.net
WICED 2022