Using Advene to accompany research in AudioVisual Digital Humanities

Olivier Aubert - www.olivieraubert.net

FOSDEM 2020 - 01/02/2020

Plan

• The Advene project
• Use cases in Digital Humanities
  • REMIND method
  • AdA project
• Conclusion

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

• Accompany active reading of AV documents through annotations
• Create, user, share analyses of AV documents as hypervideos
• Started in 2002 with Yannick Prié and P.A. Champin
• Free Software (GPL) cross-platform desktop application (Python/Gtk/Gstreamer)
• www.advene.org

Interface

Principle

Scientific challenges

• Knowledge engineering
• Document engineering
• HCI
• Dataviz
• Activity traces

Features

• User defined structures and visualizations (templates, rules…)
• Preserve structured annotations as long as possible in the workflow
• A package is a set of (annotations, annotation structure, views, queries) independent from the video
• Generic data model. Use of text-based microsyntax.
  • import filters
  • no strict constraints
• Flexible and extensible (through templates and plugins)
• Offline work

2 use cases in Digital Humanities

• Interview analysis in museology : REMIND
• Systematic quantitative analysis and semantic annotation in media studies : AdA project

REMIND method

Goal: study museum visitors' perception through video-based autoconfrontation

Example: Museographie

Analysis process

• Transcribe interview (note-taking view)
• Identify hexadic signs (from course-of-action methodology)
• Use relations to express courses of experience
• Generate visualizations through templates
• Write analyzes
• Publish on the web

REMIND method

Exploratory structuration / Publication of analyses as hypervideos

www.museographie.fr

AdA project

www.ada.cinepoetics.fu-berlin.de

• Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
• Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
• Interdisciplinary collaboration: Cinepoietics (Media Studies, Freie Universität Berlin), HPI Potsdam (semantic web, video analysis)

Challenges and needs

Challenges

• Developing a methodological framework for analysis, with quantitative support
• Annotation content semantization
• Exploration of automation possibilities

Needs

• Build a ground truth for future automation
• Optimize the manual annotation process
• Provide a bridge between users and semantic data

Application

Process

• Exploratory modelling of an ontology through Advene schemas definition
• Conversion of initial Advene schemas into bootstrap ontology
• Conversion of OWL ontology into Advene schema
  • convert properties (URIs) into "local" vocabularies (folksonomies)
  • preserve URI/vocabulary mapping through metadata
• Non-blocking data validation feedback
• Export of information into an extended WebAnnotation model

Principle

Produced ontology (HPI)

ada.filmontology.org

• 8 annotation levels: Segmentation, Camera, Montage, Image Composition, Acoustics, Language, Bodily expressivity, Motifs
• 78 annotation types
• (currently) 86000 annotations over 5 movies and many TV reports
• CC-SA 3.0

Data visualization

visualization

Advene developments for the AdA project 1/2

HCI

• Update code for adapting to new systems (esp. Python3/Gtk3/Gstreamer1)
• Performance improvements
• Constraint vocabularies support
• Data visualization

Advene developments for the AdA project 2/2

• Improvements of fusion/import features for collaboration
• Semantic web interoperability
  • Ontology import
  • RDF export (WebAnnotation + extensions)
• Integration of a concept-detection server from HPI

Conclusion

• Advene - a useful tool for AV Digital Humanities project
  • Able to export/import from multiple other models/formats
  • Flexible and extensible
  • Supports the whole scholar workflow from exploration to publication
• Available support for custom developments and training - contact me
• Many ideas, looking for funding

Thanks for you attention

Any question?

@Olivier_Aubert
www.olivieraubert.net
www.advene.org

Appendix: Active Reading