Video annotation with Advene

Sorry, your browser does not support SVG.

Olivier Aubert -
@UniBielefeld Digital Academy - 22/09/2022

Where do I speak from?

  • Visiting Assistant Professor (Nantes University) / Consultant in Knowledge Engineering
  • Research themes:
    Digital Humanities - Knowledge engineering - Document engineering - Audiovisual annotation - Hypervideos - Data visualization - Activity traces - User-centered systems
  • Lead architect and developer for video-annotation solutions: Advene, COCoNotes, Visu…


  • Some definitions: annotations, active reading…
  • Advene usage
  • Illustrating examples
  • Conclusion

Annotation in research

Annotation - one of the fundations of scholarship work


Page of the Codex Oxoniensis Clarkianus 39 (Clarke Plato). Dialogue Gorgias. Public Domain

Annotation definition

  • annotation - information (free text, constrained vocabulary, structured data, image, graphic…) linked to a document through an anchor
  • video annotation - anchor = (spatio)temporal fragment of a video, defined by a begin and end timecode

WebAnnotation definition

annotation - information body (free text, constrained vocabulary, structured data, image, graphic…) linked to a target document through a selector

Sorry, your browser does not support SVG.

video annotation - selector = (spatio)temporal fragment of a video, defined by a MediaFragment URI

Video specificities

  • Temporal nature: impossible to get a global/overview of the whole document itself. We have to use representations/surrogates.
  • No obvious/trivial segmentation (as word/sentence/paragraph/page in text)
  • Cognitive complexity: the time of the media can conflict with the time of the reflection and the time of annotation.

Various video annotation communities

media studies
Advene - VIAN - Mediascope - Lignes de Temps

music studies
Travis, MusiqueLab, Transcribe

Anvil - ELAN - Exmeralda - Transcriber

behavioral research
BORIS - Transana - VCode - Noldus/Observer - NVivo

Amara - AegisSub

COCoNotes - Travis-Go - Lacuna stories - Mediathread

social annotations
Vialogues - YiNote - Reclipped

vatic - LabelMe

LongoMatch - Powerchalk - CoachLogic - SportsCode

Active reading

Sorry, your browser does not support SVG.

Advene project

Annotate Digital Video, Exchange on the NEt

to be added to something or become a part of it, though inessential (Webster, 1913)

Create, use, exchange analyses of audiovisual documents as hypervideos

  • Started 2002 with Yannick Prié and P.A. Champin
  • Free software (GPL) cross-platform desktop app (Linux/Windows/Mac)

Scientific motivation

  • Knowledge engineering
  • Document engineering
  • Human-Computer Interfaces
  • Data visualisation
  • Activity traces


Sorry, your browser does not support SVG.




  • Ability to work offline
  • User-defined structure and visualisations (templates, rules…)
  • Package as bundle (annotations, annotation structure, views, queries) independent from video
  • Simple data model
    • import filters
    • no strict constraints
  • Flexible and extensible (through plugins)

Advene in practice

What will we obtain?

Example package - Source video - template package


Typical process

  • Transcribe a video or audio (Note Taking tools, or automatic processing)
  • Define analysis categories (in the Timeline)
  • Categorise annotations (in the Timeline)
  • Navigate/search annotated video
  • Generate visualisations

General interface

  • Flexible interface
  • Different views used according to the task
  • 1 default configuration (workspace) saved in the package
  • Possibility to save named workspaces (view configurations)


What/how to transcribe?

  • Annotating is not only defining content.
  • Importance of segmentation of temporal/opaque content
  • Difference between time point and interval/annotation
    • Single points (cursory reading, points of interest)
    • Coarse segmentation (chapters, sequences)
    • Precise segmentation in some cases (subtitles/transcription…)

Transcribe a video/audio


Basic shortcuts of the Note Taking view:

  • Control-space: play/pause
  • Control-left: 2 seconds rewind
  • Control-Enter: insert a timestamp mark in the transcription


Convert notes to annotations

Notes are just text with timestamp marks in it. Not yet annotations handled by the platform.


Other possibilities

  • ActiveBookmark view
    • Insert keyboard shortcut
  • Create annotation from the timeline
    • Right-click / Create annotation
    • Control-click in the timeline background
  • Toolbar buttons:
    • Create Text annotation
    • Create Graphical annotation

Extract information


Categorise information




Generate visualisations


Application examples

  • Conference recording navigation: Tim Berners Lee
  • Movie analysis: Nosferatu, Il caso Mattei
  • Interview analysis: Museographie
  • Semantic annotation: AdA project
  • Lab experiment analysis
  • Accessibility for blind users

Example: Nosferatu

Hypervideo publishing


Example: Nosferatu nightmarish scenes

Graphical annotation


Example: Il caso Mattei

Work with Livia Giunti for her PhD about digital tools and time representation in movies

  • Importance of digital analysis tools
  • Comparison of different tools
  • Study of annotation structuring
  • Definition of views dedicated to temporality (shot duration bar
  • graph, chronologic vs diegetic time)

Example: Il caso Mattei

Dedicated datavisualization


Example: Museographie

Objective: study visitor's perception of museum exhibits through analysis of visitor video interviews


  • visitors interviewed while replaying a recording of their visit with subjective camera
  • 40+ annotated videos
  • online publication of views and raw data
  • dedicated visualization
  • automated tasks (via plugins)

Example: Museographie

Analysis process

  • Transcribe interview (note-taking view)
  • Identify hexadic signs (from course-of-action methodology)
  • Use relations to express courses of experience
  • Generate visualizations through templates
  • Write analyzes
  • Publish on the web

Example: Museographie

Exploratory structuring
Analysis publication as custom hypervideo


Exemple: AdA project

  • Affektrhetoriken des Audiovisuellen - Audio-visual rhetorics of affect
  • Systematic-empirical study of staging patterns in audiovisual representations of the 2007 financial crisis
  • Interdisciplinary collaboration: Cinepoietics (Media Studies, Freie Universität Berlin), HPI Potsdam (semantic web, video analysis)

AdA project - Challenges and needs


  • Developing a methodological framework for movie analysis, with quantitative support
  • Semantic video-annotations
  • Exploration of automation possibilities (feature extraction)


  • Provide a 2-way bridge between users and semantic data
  • Optimize the manual annotation process
  • Build a ground truth for future automation

AdA project - Advene screenshot



Sorry, your browser does not support SVG.

Semantic web extensions

  • WebAnnotation/JSON-LD import and export
  • OWL importer
    • Convert OWL schema into Advene structures
  • OWL-derived metadata preserved
    • used in Checker framework
    • used for GUI shortcuts
    • used in WebAnnotation export

Produced ontology and data

  • 8 annotation levels (Segmentation, Camera, Montage, Image Composition, Acoustics…)
  • 78 annotation types
  • 566671 annotations over 7 movies, 69 documentaries and many TV reports
  • CC-SA 3.0


AdA ontology model

Feature-extraction plugins

  • Using Advene generic feature-extraction framework or API import filters
  • Principle
    • define a gstreamer audio/video decoding pipeline, such as videoconvert ! videoscale ! video/x-raw,width=160,pixel-aspect-ratio=(fraction)1/1,format=GRAY8
    • define a process_frame method, called for each frame
    • produce annotations as python dicts

Example plugin outputs



  • Multiple visualisations are already present in Advene: timeline, table, transcript…
  • Advene embeds a webserver and a template language interpreter, able to generate and serve X/HTML documents
    • Agile testbed for experiments

Web-based timeline



One possible visualisation, overview of annotations for one single movie

Example: Lab experiment analysis

A Case Study of Object and Occlusion Management on the eLabBench, a Mixed Physical/Digital Tabletop (Aurélien Tabard - University Lyon 1/CNRS)

  • Graphical annotation of video capture of a laboratory bench
  • Study of user behaviour through most used bench areas

Example: Lab experiment analysis


Example: Lab experiment analysis

Sorry, your browser does not support SVG.

Example output

Quantitative processing of graphical annotations


Example: accessibility

Dynamic annotation rendering through a Braille table



  • Video annotation is a complex task
    • Genericity vs. simplicity
    • Automation vs. specificity
  • Some guiding principles
    • Flexible data structure
    • Time segmentation/navigation
    • Various visualizations
    • Allow non-consistent data (but warn about them)
  • Annotations are precious
    • need for interoperability in order not to lose data
    • how much can you trust online platforms?

Thank you!

Your questions and remarks are welcome