Media Streams
Video annotation and editing system
By Marc Davis, Brian Williams, and Golan Levin

 

  

[Personal note: Media Streams was conceived by Marc Davis and developed between 1991 and 1997 at the Machine Understanding Group of the MIT Media Laboratory and at Interval Research Corporation by Marc Davis, Brian Williams, and Golan Levin [myself]. I was involved from the project's inception as the principal designer of its iconic visual language and as a contributing designer of its interface and interaction.]

Media Streams is a system for annotating, retrieving, repurposing, and automatically assembling digital video. It uses a stream-based, semantic representation of video content with an iconic visual language interface of hierarchically structured, composable, and searchable primitives. Media Streams addresses problems of annotation convergence and human-computer communication by creating a standardized, computationally readable and writable visual language for representing consensual interpretations of video content.

The Media Streams video content annotation system uses a vocabulary of more than 6000 icons to represent the characters, objects, behaviors and settings of the broadcast universe. Because they are combined in a grammar with a syntax and semantics—permitting meaningful combinations numbering in the hundreds of millions—the Media Streams icons are not merely an iconography but a true visual language. This generative and searchable language supports gestalt information visualization, quick recognition and browsing of annotations, the potential for global use, and the representation of semantic, relational and temporal video content. Creating the Media Streams lexicon involved knowlege-engineering a sensible relational hierarchy of thousands of concepts, and inventing a consistent and readable set of recombinable sub-iconic graphic elements.

The Media Streams iconic annotation language has numerous advantages over traditional keyword annotation systems, including its ability to describe relations between descriptions, its ability to clearly render overlapping and contained actions, its ability to refer more directly to the intrinsic visual qualities of the video medium, and its ability to serve as a "consensus" language for multimedia professionals.

More information on Media Streams can be found here.

   Detailed images and descriptions of the Media Streams user interface and visual language can be accessed from the links below:
  To the Media Timeline [Link] The Media Streams Media Timeline, on which iconic annotations of video are temporally indexed. Each stream in the Media Timeline contains annotations about a unique aspect of video content, such as settings, characters, objects, actions, camera motions, etcetera.
  To the Icon Space [Link] The Media Streams Icon Space, an atemporal, hierarchically-indexed "dictionary" of iconic descriptors. The Icon Space incorporates utilities for icon construction and search.
  To examples of icons [Link] A small but representative selection of icons from the Media Streams visual annotation lexicon. In the interest of space, the icons displayed here have not been arranged to reflect the grammars or knowledge hierarchies into which they are ordinarily structured.
 

A Media Streams bibliography:

Davis, Marc. "Media Streams: An Iconic Visual Language for Video Annotation." Telektronikk 4.93 (1993a): 59-71.

Davis, Marc. "Media Streams: An Iconic Visual Language for Video Annotation." In: Proceedings of 1993 IEEE Symposium on Visual Languages in Bergen, Norway , IEEE Computer Society Press, 196-202, 1993b.

Davis, Marc. "Knowledge Representation for Video." In: Proceedings of Twelfth National Conference on Artificial Intelligence (AAAI-94) in Seattle, Washington , AAAI Press, 120-127, 1994.

Davis, Marc. "Media Streams: An Iconic Visual Language for Video Representation." In: Readings in Human-Computer Interaction: Toward the Year 2000, ed. Ronald M. Baecker, Jonathan Grudin, William A. S. Buxton, and Saul Greenberg. 854-866. 2nd ed., San Francisco: Morgan Kaufmann Publishers, Inc., 1995.

Davis, Marc. "Media Streams: Representing Video for Retrieval and Repurposing." Ph.D. Thesis, Massachusetts Institute of Technology, 1995.

Levin, Golan. "Media Streams 3.0: User's Guide and Manual." Internal document. Palo Alto, California: Interval Research Corporation, 1994. Also included as an appendix in Marc Davis' Ph.D. Thesis (see above).