Advancing Multimedia on the Web The development of SMIL 2.0 and what comes next SBMidia 2001 Florianopolis Brazil Patrick Schmitz cogit@ludicrum.org http://www.ludicrum.org 8:55 AM PT Oct 15, 2001 Title slide for talk at SBMIDIA 2001 in Florianopolis Brazil. bg.jpg Overview of tutorial What is SMIL? Basic Timing and Synchronization Animation and Time Manipulations Integrating SMIL Timing with XML languages Looking ahead What is SMIL? Synchronized Multimedia Integration Language Declarative language for synchronized multimedia Based upon XML - Extensible Markup Language SMIL 1.0 Released from W3C June 98 No HTML/CSS integration SMIL 2.0 Released August 2001 XHTML+SMIL profile approaching last call What is SMIL 2? SMIL 1 vs. SMIL 2 Lots of new support in modules Multiple language profiles Modules of SMIL Timing and Sync are the core Media, Animation, Transitions Layout: big difference between SMIL, XHTML+SMIL Misc (but useful) modules Why a language vs. an API? Authors are not programmers How tools support authoring Round trip and exchange support Iterative authoring Script alternative usually primitive Stateful, brittle, expensive Language does support a DOM What is Timing and Synchronization? Document choreography When things begin, how long they last Relationships: holding things together Grouping, constraining, repeating, filling Semantic model Time graph for document Dynamic, not static Dependency relationships Use cases Slide shows - like this one! Audio with synchronized text Music synced to lyrics Digital Talking Books Animation with time manipulations TV and Radio enhancement: timed annotations
How does it work? Background concepts Basic syntax Hierarchic time (tree model) Sync-arcs and time dependents Mixing scheduled and interactive time Handling multiple begin and end conditions Managing runtime synchronization Background Concepts Arithmetic with time End-point exclusive intervals Indirection and relative time Times are computed relative to other times Resolved and unresolved time Basic Timing Syntax Timing attributes begin, dur, end, repeat Time containers (grouping) Parallel and sequence groups Exclusive groups and interrupt semantics 
<t:par begin="1s" repeatDur="indefinite">
  <t:audio src="media/talk.mp3" syncBehavior="locked" />
  <p timeContainer="seq" >
    <span dur="2.9s" >
        SMIL Timing syntax consists of a set of </span><br>
    <span dur="3s"  >
        <em>attributes</em> for controlling the behavior of media, </span><br>
    <span dur="2s"  >
        and several types of <em>time containers</em> </span><br>
    <span dur="4s"  >
        that group media together for coordinated presentation. </span>
  </p>
</t:par>

SMIL Timing syntax consists of a set of
attributes for controlling the behavior of media,
and several types of time containers
that group media together for coordinated presentation.

Hierarchic Time Compare to scene graph Each time container transforms time. Transformed time is basis for all descendents Each element sees local time Element simple time Element active time Can convert from one time space to another Sync-arcs & time dependents What is a Sync-arc? When one element defines a begin or end time relative to the begin or end of another element.
img begin="foo.begin" end="bar.end" ... Creates an additional 'arc' in the timegraph Defines a time dependency 'Short' and 'long' sync arcs Define timing, not runtime sync! Scheduled vs. Event timing SMIL 1 supported scheduled timing SMIL 2 adds interaction begin="foo.click" end="img3.load" Leverages concept of unresolved time Exists within scheduled model Converting event times to schedule Interactive timing and dynamism Propagating changes <code>begin</code> and <code> end</code> Lists Lists of times for begin and end Different 'ways' or 'rules' Multiple begin/end times Define intervals for element Controlled with restart attribute Multiple begins and cyclic timing 
<p id="red"   begin="0; green.end-2s" dur="4s" .../>
<p id="blue"  begin="red.end-2s"      dur="4s" .../>
<p id="green" begin="blue.end-2s"     dur="4s" .../>

Using Cyclic timing


Red

Blue

Green


Using Repeat timing


Red

Blue

Green

Controlling runtime synchronization behavior Internet delivery of media is unreliable Media may not begin when scheduled Streaming media may hiccup or slew Documents are not always monolithic SMIL 2 defines sync scope and sync behavior Includes support to 'own' the clock SMIL Animation Animation defined: the time-based manipulation of some object property Animation elements support common general cases animate supports generic interpolation of scalar values animateMotion abstracts object motion animateColor abstracts color interpolation set supports simplest discrete animation cases Animation elements timed like media SMIL Animation model Each element has a single target targetElement references the element attributeName references the specific property Composition model controls interaction among animations activation (begin) time determines priority each animation can override or add to lower priority animations Animations can be cumulative (recursive) SMIL Time Manipulations 4 additional attributes applied to timed elements speed, accelerate and decelerate, autoReverse Often uesd to emulate common physical phenomena Gravity, momentum, etc. Actually control the pace of time 
<h1 style="position:absolute; top:0; left:0" timeContainer="par" >
  Bounce
   <t:animateMotion by="0,100" dur="3s" accelerate="1" autoReverse="true"
                    repeatCount="indefinite" />
   <t:animateMotion by="500,0" additive="sum" dur="18s" repeatCount="indefinite" />
   <t:animateColor attributeName="color" from="white" to="gold"
      dur="3s" accelerate="1" autoReverse="true" repeatCount="indefinite" />
 </h1>

Bounce

SMIL Time Manipulations - syntax animate speed='2.5' ... Attribute value is a multiple of normal play speed animate accelerate='0.5' ... Attribute value is a proportion of simple duration accelerate and decelerate do not affect the duration animate autoReverse="true" Play forward then backward SMIL Time Manipulations and structured time Time manipulations can be applied to time containers Very useful for animation reuse Allows simplified control over complex animations Time is modified for entire subtree Only slightly complicates arithmetic Spec includes pseudo-code algorithms SMIL Time Manipulations issues Fallback semantics for media Not all media renderers can handle this Time model maintained, but media plays best effort Not appropriate for all presentations! Timing Integration Motivation for a common model Requirements Examples: HTML+SMIL, SVG Approaches to Integration Future work and applications Motivation Common authoring semantics Leverage SMIL in HTML/CSS content Provide Time model for XML documents Well-defined syntax and semantics One clock for the whole document Synchronize HTML to TV, Radio, etc. Requirements Need common, straight-forward authoring models Need flexible approach to syntax Sometimes want inline syntax Sometimes want to model like style Sometimes need a separate document Need to be able to mix approaches Integration example: XHTML+SMIL XHTML+SMIL lets Web developers: Control DHTML properties along a timeline Integrate media as part of their page description Synchronize media elements and actions in the page Reduces dependency on scripting as a way of controlling animation of properties Integration example: SVG Adds SMIL Animation elements Target CSS and SVG properties No direct integration of SMIL timing with SVG elements Cannot simply apply transform over time Simplified time graph No time containers Does have sync arcs, event timing and multiple begin/end values No time manipulations Demos Simple HTML demos Expanding ad images Business presentations XHTML+SMIL Timing and Media Markup Media elements: video, audio, et al. Timing elements: par, seq, excl timeContainer attribute Timing attributes: begin, end, et al. Applied to most HTML content Includes event-based/interactive declaration Animation support CSS properties, motion, effects, etc. What does <code>begin</code> mean for <code>div</code> or <code>strong</code>? timeAction controls semantics of adding timing to HTML elements intrinsic: defined for phrasal and presentation elements, reverts to visibility for text, div, etc. Schedules media. display, visibility control style style controls inline style (CSS/XSL) class adds class name to XML class property Approaches to Integration Inline syntax Attributes added to language elements Approach used in SMIL, HTML+SMIL Styled Timing CSS or XSL stylesheets used to apply timing to a language Timesheets Separate timing from both content and presentation style Inline Syntax approach Easy to understand, easy to author Generalized, extensible semantics with timeAction Better when document structure aligned to timing structure May also be used to override or augment styled timing Requires aligned model of timing properties Styled Timing approach Useful when document structure aligns closely with timing structure Example: sequence of highlights on list CSS changes required to manipulate timing properties CSS3 may include SMIL module XSLT provides interesting alternative General need: filter chain model Universal cascade and view model Issues with Styled Timing Must preclude feedback loops If timing controls style, and style redefines timing, what should happen? Possible solution: lock timing properties when applying timeAction Alternative solution: Static application of timing properties using XSLT Specifying the side-effects When does the effect of timing ripple through CSS, XSL, etc.??? General problem common to animation Timesheets approach Abstracts timing away from content and presentation style SMIL timing, timeAction, no media Animation elements? Useful when: document structure and timing structure do not align synchronization spans multiple documents inline syntax impractical or illegal there are Copyright restrictions Digital Talking Books Issue: Interpreting multiple references to an element Multiple orthogonal timeActions are easy: just do all of them Multiple instances problematic Synthesize elements? (No!) Conclusion: model as animation Requires property-based model for timeAction Simplified animation "sandwich" Use activation priority, no composition Other Issues How to combine Timesheets with other approaches? Inline and Styled Timing define a cascade Inline markup overrules Styled Timing Timesheets define an additive model Do we need composition tools to combine with other approaches? Defining sync among documents ITV model of HTML synced to TV Digital Talking Books Integration conclusions Common model based upon timeAction is essential Cascade rules combine Styled and Inline Timing approaches Models timing as properties a la CSS Timesheets layer, rather than override Timesheets can specify multiple actions, using animation semantics W3C must define filter chain Future work & Applications Formalize Timesheets Collaborating with Philips, CWI. Potential customers DTB, eBooks, multimodal documents Named Timespaces (my recent work at MSR) Defines sync and interrupt semantics Synchronizes XHTML+SMIL to broadcast television, DVD and CD content Tool for accessible multimedia? Looking ahead - Timing and Fragmentation Intro to XML Fragment Interchange Fragmentation and Annotation on the Semantic Web Issues/Questions for SMIL Issues for compound documents XML Fragment Interchange Allows one document to use a portion of another document XPointer, XPath specify fragment Fragment Context Information provides intelligent context Represents fragment as part of larger structure Includes any necessary(!) semantic context Can allow client to access full source document Not the same as XInclude Fragmentation and Annotation on the Semantic Web Annotations to a document refer to (a fragment of) the document Segmentation is a form of annotation Smart representation uses pointers, not copies Annotations are documents! Content authoring will move to annotation Semantic web depends upon intelligent fragment references Issues/Questions for SMIL What will it mean to use fragments of timed documents? Should the entire (sparse) timegraph be included? Can we allow (and patch) 'dangling' sync-arc references? How to define timed annotations of timed documents? Can timing and animation be defined that will (better) survive document fragmentation (and transformation)? Issues for compound documents How to deal with ID clash? We need a concept of ID-spaces, a la namespaces Namespace tree allows cross-document references Including/referencing node is bridge point Useful for script/code, for timing, etc. Issues for compound documents Need to represent fragment references as structured document in the DOM Makes structured ID-spaces possible Has precedent in DOM for frames Requires smarter User Agents to deal with accessibility, etc. Resources Specs available at http://www.w3.org/TR/smil20 Early implementation in IE 5.5/IE6 (HTML+TIME) IE docs/demos at msdn.microsoft.com SMIL Animation in SVG viewers Time model tutorial available: research.microsoft.com/scripts/pubs/view.asp?TR_ID=MSR-TR-2001-01