SlideShare a Scribd company logo

Multimedia Data Navigation and the Semantic Web (SemTech 2006)

•Download as PPT, PDF•
•
Bradley Allen
Bradley Allen

The document describes a system for faceted navigation of multimedia content using semantic web technologies. It discusses using ontologies expressed in RDF(S) and OWL to represent metadata, BBC rush footage used as a case study, and visual facets for color, texture and combinations that were generated through MPEG-7 feature extraction and self-organizing map clustering. The system allows retrieval of clips and shots based on textual and visual facet filtering of the RDF represented multimedia data.

Related slideshows

NISO DCMI Webinar bibframe-20130123
NISO DCMI Webinar bibframe-20130123NISO DCMI Webinar bibframe-20130123
NISO DCMI Webinar bibframe-20130123
 
NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...
NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...
NISO/DCMI September 25 Webinar: Implementing Linked Data in Developing Countr...
 
Better Search With Structured Knowledge
Better Search With Structured KnowledgeBetter Search With Structured Knowledge
Better Search With Structured Knowledge
 
1 of 32
Multimedia Data Navigation and the Semantic Web Valery A. Petrushin and Bradley P. Allen
Outline About the authors Faceted Navigation Semantic Web Techniques RDF(S) Dublin Core SKOS TGM LSCOM, SMIL & MPEG-7 Case Study: BBC Rushes Implementation BBC Rushes Navigator Metadata representation Architecture User interface Future work Contact Information Demo
About the Authors Valery A. Petrushin, Ph.D. Sr. Researcher, Accenture Technology Labs Semantics of programming languages Multimedia data mining, analysis, annotation and retrieval Georgia Tech, Glushkov Institute for Cybernetics Bradley P. Allen Founder and CTO Siderean Software, Inc. Semantic-based navigation, Web personalization services, case-based reasoning Former founder and CTO of Limbex Corp. and TriVida Corp. Carnegie-Mellon University
Faceted navigation Facets are metadata properties whose ranges form a near-orthogonal set of controlled vocabularies Creator: “Dickens, Charles” Subject: Arsenic, Antimony Location: World > U.S. > California > Venice Facets form a frame of reference for information overview, access and discovery Other properties serve as landmarks and cues Faceted navigation uses facets to provide end user access and discovery in the context of large collections of semi-structured information
Faceted Navigation Built Using Semantic Web Standards Define/reuse ontologies expressed in RDF(S)/OWL Classes for defining instances and controlled vocabularies Properties for facets and additional asset metadata attributes Import/transform aggregated instance metadata into an RDF representation Resources referred to via URIs Content and controlled vocabularies Write application profiles in terms of RDF
Building Faceted Navigation Applications … then represented as instances of concepts in ontologies and tagged using controlled vocabularies… … then application profiles are created… … that define navigation services for user applications Metadata is aggregated… Term Event Person Place Text Application Profiles
Semantic Web Technology RDF(S) – Resource Description Framework (Schema) Dublin Core SKOS – Simple Knowledge Organization System TGM-I & II – Thesaurus for Graphic Materials LSCOM – Large Scale Concept Ontology for Multimedia SMIL – Synchronized Multimedia Integration Language MPEG-7 – Multimedia Content Description Interface
RDF (S) RDF (S) - Resource Description Framework (Schema) http://www.w3.org/RDF/ http://www.w3.org/TR/rdf-schema/ language for representing metadata about Web resources Triple : subject – predicate -- > object Example: <?xml version=&quot;1.0&quot;?> <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:contact=&quot;http://www.w3.org/2000/10/swap/pim/contact#&quot;> <contact:Person rdf:about=&quot;http://www.accenture.com/techlabs/VAP/contact#me&quot;> <contact:fullName>Valery A. Petrushin</contact:fullName> <contact:mailbox rdf:resource=&quot;mailto:valery.a.petrushin@accenture.com&quot;/> </contact:Person> </rdf:RDF>
Dublin Core (DC) Dublin Core http://dublincore.org/documents/ vocabulary for describing documents (title, creator, subject, description, publisher, contributor, date, type, format, identifier, source, language, relation, coverage, rights) Example: <?xml version=&quot;1.0&quot;?> <!DOCTYPE rdf:RDF PUBLIC &quot;-//DUBLIN CORE//DCMES DTD 2002/07/31//EN&quot; &quot;http://dublincore.org/documents/2002/07/31/dcmes-xml/dcmes-xml-dtd.dtd&quot;> <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot;> <rdf:Description rdf:about=&quot;http://www.accenture/techlabs/Petrushin&quot;> <dc:title> Multimedia Data Mining and Knowledge Discovery</dc:title> <dc:creator> Valery A. Petrushin </dc:creator > <dc:publisher>Springer Verlag</dc:publisher> </rdf:Description> </rdf:RDF>
SKOS SKOS – Simple Knowledge Organization System http://www.w3.org/2004/02/skos/ model for expressing structure and content of concept schemes (thesauri, taxonomies, etc.) Specifies concepts, collections of concepts and relations between concepts (broader, narrower, related) Example: <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:skos=&quot;http://www.w3.org/2004/02/skos/core#&quot;> <rdf:Description rdf:about=&quot;http://www.example.com/concepts#people&quot;> <skos:broader rdf:resource=&quot;http://www.example.com/concepts#mammals&quot;/> <skos:narrower rdf:resource=&quot;http://www.example.com/concepts#children&quot;/> <skos:narrower rdf:resource=&quot;http://www.example.com/concepts#adults&quot;/> </rdf:Description> </rdf:RDF>
TGM – I & II TGM – Thesaurus for Graphic Materials (The Library of Congress) TGM-I – Subject Terms (6,300) http://www.loc.gov/rr/print/tgm1/toc.html TGM-II – Genre and Physical Characteristic Headings (600) http://www.loc.gov/rr/print/tgm2/ Example: TGM-I: Term: Sand Narrower Term: Quicksand Related Term: Dunes, Sand sculpture, Sandpaintings TGM-II: Term: Aerial views Public Note: Views from a high vantage point. Used For: Air views, Balloon views, Views, Aerial Broader Term: Views Narrower Term: Aerial photographs Related Term: Bird's-eye views, Panoramic views
LSCOM, SMIL & MPEG-7 LSCOM – Large Scale Concept Ontology for Multimedia http://www.acemedia.org/aceMedia/files/multimedia_ontology/presentations_1st_meeting/arda.pdf SMIL – Synchronized Multimedia Integration Language http://www.w3.org/TR/REC-smil/ Simple language for representing multiple synchronized media streams MPEG-7 – Multimedia Content Description Interface http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm Advanced language for representing multimedia content ISO Standard
Case Study: BBC Rushes Rushes are raw footage … with a promise to turn into golden nuggets of stockshots TRECVID 2005 Video Retrieval Competition at NIST http://www-nlpir.nist.gov/projects/trecvid/ Problem: create a system that helps a TV program maker compose a video using current clips and rushes Data Statistics: Duration: 49.3 hours Content: Clips about vacation and travel 4 issues of “Summer Holiday” (~ 2 hours) BBC One News (30’) + fragment (~3’)
BBC Rushes: Data Statistics - 1 Statistics: clip level 615 clips (308 development + 307 test sets) Duration (mm:ss) : Minimal / Maximal - 00:03.48 / 47:11 Mean / Median – 04:49 / 02:25 Std - 06:02.73 Keywords: Different keywords / Occurrences – 1036 / 4908 Mean / Median – 7.98 / 7 Minimal / Maximal – 0 / 34
BBC Rushes: Data Statistics - 2 Statistics: shot level Number of shots 10,064 Shot duration (mm:ss) Minimal - 0:00.04 Maximal – 22:45.16 Mean – 0:17.51 Median – 0:09.74 Std - 0:33.97 Number of key frames Total: 39,132 Median per shot: 2 Mean per shot: 3.8 Maximal: 377 Minimal: 1
BBC Rushes: representation Ontologies RDFS, Dublin Core, SKOS Controlled vocabularies TGM-1 (reflecting Light Scale Concept Ontology for Multimedia), ISO8601 (temporal hierarchy of dates), MPEG-7 (visual features) Instances trecvid:Shot, trecvid:Clip Application profile Retrieve instances of type trecvid:Clip Textual facets: dc:title (clip title), dc:subject (keywords), dc:creator (director), dcterms:created (production date), dcterms:issued (show date), dc:extent (duration) Retrieve instances of type trecvid:Shot Visual facets: dc:subject with values skos:narrower than trecvid:color, trecvid:texture and trecvid:colorplustexture Textual facets through reference to containing clip
Ontology Schema dcterms: partOf dc: title dc: creator dc: subject dc: subject dc: created skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader VISUAL TEXTUAL Clip Shot KeyFrame Color Texture Color+Texture Title Creator Subject Date
BBC Rushes: visual facets Facets: color, texture, [shape] + combinations Color, texture, color+texture To build facets Extract features (MPEG-7): Color: dominantColor(24), colorStructure (256) , colorLayout (12) Texture: edgeHistogram (80), homogenousTexture (60) SOM Clustering of keyframes Select as a visual “word” the closest keyframe to node centroid Represent keyframes as SKOS concepts, centroids as skos:broader of cluster members Example: SOM for color 35x28 (=980 nodes)
Self-organizing Maps SOM = Kohonen NN = Topology-preserving map Unsupervised learning (Clustering + Visualization) X = { x i } , x i  R d - input data M = { m k } , m k  R d - prototype vectors (codebook) = neurons on 1D or 2D grid Training: 1. Start with random m k 2. For x i find best-matching unit (BMU) m c 3. Update prototype vectors in neighborhood where is the neighborhood kernel is radius at time t Two phases: rough and fine tuning
BBC Rushes: RDF subgraph Chilli_peppers v159_001.wmv v159.mpg “ michelle jones” 2000-03-01 dc:subject dc:creator dcterms:partOf dc:created dc:subject color#26547 f000000000.jpg skos:broader skos:broader 2000 2000-03 Hot_peppers Peppers Year skos:broader skos:broader skos:broader skos:broader “ thailand, chiang mai/chillis” dc:title Color skos:broader
BBC Rushes: RDF/XML serialization <trecvid:Clip rdf:about=&quot;http://swvideo.techlabs.accenture.com/v159.mpg&quot;> <rdf:type rdf:resource=&quot;&dctype;MovingImage&quot; /> <dc:title>thailand, chiang mai/chillis</dc:title> <dcterms:extent>202200</dcterms:extent> <dc:creator>michelle jones</dc:creator> <dc:identifier>mrs320354</dc:identifier> <dcterms:created rdf:resource=&quot;tag:siderean.com,1752-09-14:2000-03-01&quot; /> <dcterms:issued rdf:resource=&quot;tag:siderean.com,1752-09-14:2000-07-18&quot; /> <dc:subject rdf:resource=&quot;&trecvid;thailand&quot; /> <dc:subject rdf:resource=&quot;&trecvid;chiang_mai&quot; /> <dc:subject rdf:resource=&quot;&trecvid;chillis&quot; /> <dc:subject rdf:resource=&quot;&trecvid;peppers&quot; /> <dc:subject rdf:resource=&quot;&trecvid;chilli_peppers&quot; /> <dc:subject rdf:resource=&quot;&trecvid;vegetables&quot; /> <dc:subject rdf:resource=&quot;&trecvid;markets&quot; /> <dc:subject rdf:resource=&quot;&trecvid;street_markets&quot; /> <dc:subject rdf:resource=&quot;&trecvid;food_markets&quot; /> <dc:subject rdf:resource=&quot;&trecvid;food&quot; /> <dc:subject rdf:resource=&quot;&trecvid;herbs&quot; /> <dc:relation>http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg </dc:relation> </trecvid:Clip> <skos:Concept rdf:about=&quot;&trecvid;chilli_peppers&quot;> <skos:broader rdf:resource=&quot;&tgm1;Hot_peppers&quot;/> <skos:prefLabel>chilli peppers</skos:prefLabel> </skos:Concept> <skos:Concept rdf:about='tag:siderean.com,1752-09-14:2000-03-01'> <skos:prefLabel>2000-03-01</skos:prefLabel> <skos:broader rdf:resource='tag:siderean.com,1752-09-14:2000-03'/> </skos:Concept> <trecvid:Shot rdf:about=&quot;http://swvideo.techlabs.accenture.com/shotsWMV/v159_001.wmv&quot;> <rdf:type rdf:resource=&quot;&dctype;MovingImage&quot; /> <dcterms:isPartOf rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.mpg&quot; /> <dcterms:extent>21000</dcterms:extent> <dc:relation>http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg</dc:relation> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000240.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000280.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000001440.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000003120.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000005440.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000009680.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000011520.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000012040.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000013800.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000014800.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000015120.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000016760.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000018280.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000019360.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000021000.jpg&quot;/> </trecvid:Shot> <skos:Concept rdf:about=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot;> <skos:broader rdf:resource=&quot;http://swvideo.techlabs.accenture.com/color#26547&quot; /> <skos:prefSymbol rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot; /> </skos:Concept> <skos:Concept rdf:about=&quot;http://swvideo.techlabs.accenture.com/color#26547&quot;> <skos:broader rdf:resource=&quot;&trecvid;color&quot; /> <skos:prefSymbol rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v289.fset/f000048880.jpg&quot; /> </skos:Concept>
BBC Rushes Navigator: Architecture AJAX client in Firefox XRBR query XRBR response BBC Rushes RDF http://www.siderean.com/bbcrush/bbcrush.jsp (with Firefox 1.5) Metadata Aggregator Metadata Store Navigation Web Services
Lessons Learned Data preparation Robust shot boundary detection Careful selection of keyframes Motion based Salient object based Filtering redundant keyframes Using group-of-frames (GOF) features Concept recognition/propagation Propagate keywords from clip to shots Recognize concepts from visual data Probabilistic reasoning Derive concepts from data (data mining) + labeling
Summary Methodology of Multimedia Data Representation Semantic Web Technology Multimedia Data Mining Prototype of Multimedia Retrieval System BBC Rushes Web-based Interface using AJAX
Future work More facets Shape + combinations Geographical location More Interfaces Map of the world for browsing places Hierarchy of SOM for browsing clips and shots More Tools Tagging tool for creating and managing metadata Tools for creating video databases (shot extraction, feature extraction, clustering, classification of events, etc.) Tools for creating audio-video compositions (TV programs, commercials, etc.)
BBC Rushes Navigator: Navigation with LSCOM
BBC Rushes Navigator: Hierarchical Drill-down on People Facet
BBC Rushes Navigator: Faceted View of All Shots
BBC Rushes Navigator: Searching by Subject
BBC Rushes Navigator: Searching by Color, Playlist composition
BBC Rushes Navigator: Drill-down using Subject and Color
Contact Information Valery A. Petrushin [email_address] Bradley P. Allen [email_address]

More Related Content

Multimedia Data Navigation and the Semantic Web (SemTech 2006)

  • 1. Multimedia Data Navigation and the Semantic Web Valery A. Petrushin and Bradley P. Allen
  • 2. Outline About the authors Faceted Navigation Semantic Web Techniques RDF(S) Dublin Core SKOS TGM LSCOM, SMIL & MPEG-7 Case Study: BBC Rushes Implementation BBC Rushes Navigator Metadata representation Architecture User interface Future work Contact Information Demo
  • 3. About the Authors Valery A. Petrushin, Ph.D. Sr. Researcher, Accenture Technology Labs Semantics of programming languages Multimedia data mining, analysis, annotation and retrieval Georgia Tech, Glushkov Institute for Cybernetics Bradley P. Allen Founder and CTO Siderean Software, Inc. Semantic-based navigation, Web personalization services, case-based reasoning Former founder and CTO of Limbex Corp. and TriVida Corp. Carnegie-Mellon University
  • 4. Faceted navigation Facets are metadata properties whose ranges form a near-orthogonal set of controlled vocabularies Creator: “Dickens, Charles” Subject: Arsenic, Antimony Location: World > U.S. > California > Venice Facets form a frame of reference for information overview, access and discovery Other properties serve as landmarks and cues Faceted navigation uses facets to provide end user access and discovery in the context of large collections of semi-structured information
  • 5. Faceted Navigation Built Using Semantic Web Standards Define/reuse ontologies expressed in RDF(S)/OWL Classes for defining instances and controlled vocabularies Properties for facets and additional asset metadata attributes Import/transform aggregated instance metadata into an RDF representation Resources referred to via URIs Content and controlled vocabularies Write application profiles in terms of RDF
  • 6. Building Faceted Navigation Applications … then represented as instances of concepts in ontologies and tagged using controlled vocabularies… … then application profiles are created… … that define navigation services for user applications Metadata is aggregated… Term Event Person Place Text Application Profiles
  • 7. Semantic Web Technology RDF(S) – Resource Description Framework (Schema) Dublin Core SKOS – Simple Knowledge Organization System TGM-I & II – Thesaurus for Graphic Materials LSCOM – Large Scale Concept Ontology for Multimedia SMIL – Synchronized Multimedia Integration Language MPEG-7 – Multimedia Content Description Interface
  • 8. RDF (S) RDF (S) - Resource Description Framework (Schema) http://www.w3.org/RDF/ http://www.w3.org/TR/rdf-schema/ language for representing metadata about Web resources Triple : subject – predicate -- > object Example: <?xml version=&quot;1.0&quot;?> <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:contact=&quot;http://www.w3.org/2000/10/swap/pim/contact#&quot;> <contact:Person rdf:about=&quot;http://www.accenture.com/techlabs/VAP/contact#me&quot;> <contact:fullName>Valery A. Petrushin</contact:fullName> <contact:mailbox rdf:resource=&quot;mailto:valery.a.petrushin@accenture.com&quot;/> </contact:Person> </rdf:RDF>
  • 9. Dublin Core (DC) Dublin Core http://dublincore.org/documents/ vocabulary for describing documents (title, creator, subject, description, publisher, contributor, date, type, format, identifier, source, language, relation, coverage, rights) Example: <?xml version=&quot;1.0&quot;?> <!DOCTYPE rdf:RDF PUBLIC &quot;-//DUBLIN CORE//DCMES DTD 2002/07/31//EN&quot; &quot;http://dublincore.org/documents/2002/07/31/dcmes-xml/dcmes-xml-dtd.dtd&quot;> <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:dc=&quot;http://purl.org/dc/elements/1.1/&quot;> <rdf:Description rdf:about=&quot;http://www.accenture/techlabs/Petrushin&quot;> <dc:title> Multimedia Data Mining and Knowledge Discovery</dc:title> <dc:creator> Valery A. Petrushin </dc:creator > <dc:publisher>Springer Verlag</dc:publisher> </rdf:Description> </rdf:RDF>
  • 10. SKOS SKOS – Simple Knowledge Organization System http://www.w3.org/2004/02/skos/ model for expressing structure and content of concept schemes (thesauri, taxonomies, etc.) Specifies concepts, collections of concepts and relations between concepts (broader, narrower, related) Example: <rdf:RDF xmlns:rdf=&quot;http://www.w3.org/1999/02/22-rdf-syntax-ns#&quot; xmlns:skos=&quot;http://www.w3.org/2004/02/skos/core#&quot;> <rdf:Description rdf:about=&quot;http://www.example.com/concepts#people&quot;> <skos:broader rdf:resource=&quot;http://www.example.com/concepts#mammals&quot;/> <skos:narrower rdf:resource=&quot;http://www.example.com/concepts#children&quot;/> <skos:narrower rdf:resource=&quot;http://www.example.com/concepts#adults&quot;/> </rdf:Description> </rdf:RDF>
  • 11. TGM – I & II TGM – Thesaurus for Graphic Materials (The Library of Congress) TGM-I – Subject Terms (6,300) http://www.loc.gov/rr/print/tgm1/toc.html TGM-II – Genre and Physical Characteristic Headings (600) http://www.loc.gov/rr/print/tgm2/ Example: TGM-I: Term: Sand Narrower Term: Quicksand Related Term: Dunes, Sand sculpture, Sandpaintings TGM-II: Term: Aerial views Public Note: Views from a high vantage point. Used For: Air views, Balloon views, Views, Aerial Broader Term: Views Narrower Term: Aerial photographs Related Term: Bird's-eye views, Panoramic views
  • 12. LSCOM, SMIL & MPEG-7 LSCOM – Large Scale Concept Ontology for Multimedia http://www.acemedia.org/aceMedia/files/multimedia_ontology/presentations_1st_meeting/arda.pdf SMIL – Synchronized Multimedia Integration Language http://www.w3.org/TR/REC-smil/ Simple language for representing multiple synchronized media streams MPEG-7 – Multimedia Content Description Interface http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm Advanced language for representing multimedia content ISO Standard
  • 13. Case Study: BBC Rushes Rushes are raw footage … with a promise to turn into golden nuggets of stockshots TRECVID 2005 Video Retrieval Competition at NIST http://www-nlpir.nist.gov/projects/trecvid/ Problem: create a system that helps a TV program maker compose a video using current clips and rushes Data Statistics: Duration: 49.3 hours Content: Clips about vacation and travel 4 issues of “Summer Holiday” (~ 2 hours) BBC One News (30’) + fragment (~3’)
  • 14. BBC Rushes: Data Statistics - 1 Statistics: clip level 615 clips (308 development + 307 test sets) Duration (mm:ss) : Minimal / Maximal - 00:03.48 / 47:11 Mean / Median – 04:49 / 02:25 Std - 06:02.73 Keywords: Different keywords / Occurrences – 1036 / 4908 Mean / Median – 7.98 / 7 Minimal / Maximal – 0 / 34
  • 15. BBC Rushes: Data Statistics - 2 Statistics: shot level Number of shots 10,064 Shot duration (mm:ss) Minimal - 0:00.04 Maximal – 22:45.16 Mean – 0:17.51 Median – 0:09.74 Std - 0:33.97 Number of key frames Total: 39,132 Median per shot: 2 Mean per shot: 3.8 Maximal: 377 Minimal: 1
  • 16. BBC Rushes: representation Ontologies RDFS, Dublin Core, SKOS Controlled vocabularies TGM-1 (reflecting Light Scale Concept Ontology for Multimedia), ISO8601 (temporal hierarchy of dates), MPEG-7 (visual features) Instances trecvid:Shot, trecvid:Clip Application profile Retrieve instances of type trecvid:Clip Textual facets: dc:title (clip title), dc:subject (keywords), dc:creator (director), dcterms:created (production date), dcterms:issued (show date), dc:extent (duration) Retrieve instances of type trecvid:Shot Visual facets: dc:subject with values skos:narrower than trecvid:color, trecvid:texture and trecvid:colorplustexture Textual facets through reference to containing clip
  • 17. Ontology Schema dcterms: partOf dc: title dc: creator dc: subject dc: subject dc: created skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader skos: broader VISUAL TEXTUAL Clip Shot KeyFrame Color Texture Color+Texture Title Creator Subject Date
  • 18. BBC Rushes: visual facets Facets: color, texture, [shape] + combinations Color, texture, color+texture To build facets Extract features (MPEG-7): Color: dominantColor(24), colorStructure (256) , colorLayout (12) Texture: edgeHistogram (80), homogenousTexture (60) SOM Clustering of keyframes Select as a visual “word” the closest keyframe to node centroid Represent keyframes as SKOS concepts, centroids as skos:broader of cluster members Example: SOM for color 35x28 (=980 nodes)
  • 19. Self-organizing Maps SOM = Kohonen NN = Topology-preserving map Unsupervised learning (Clustering + Visualization) X = { x i } , x i  R d - input data M = { m k } , m k  R d - prototype vectors (codebook) = neurons on 1D or 2D grid Training: 1. Start with random m k 2. For x i find best-matching unit (BMU) m c 3. Update prototype vectors in neighborhood where is the neighborhood kernel is radius at time t Two phases: rough and fine tuning
  • 20. BBC Rushes: RDF subgraph Chilli_peppers v159_001.wmv v159.mpg “ michelle jones” 2000-03-01 dc:subject dc:creator dcterms:partOf dc:created dc:subject color#26547 f000000000.jpg skos:broader skos:broader 2000 2000-03 Hot_peppers Peppers Year skos:broader skos:broader skos:broader skos:broader “ thailand, chiang mai/chillis” dc:title Color skos:broader
  • 21. BBC Rushes: RDF/XML serialization <trecvid:Clip rdf:about=&quot;http://swvideo.techlabs.accenture.com/v159.mpg&quot;> <rdf:type rdf:resource=&quot;&dctype;MovingImage&quot; /> <dc:title>thailand, chiang mai/chillis</dc:title> <dcterms:extent>202200</dcterms:extent> <dc:creator>michelle jones</dc:creator> <dc:identifier>mrs320354</dc:identifier> <dcterms:created rdf:resource=&quot;tag:siderean.com,1752-09-14:2000-03-01&quot; /> <dcterms:issued rdf:resource=&quot;tag:siderean.com,1752-09-14:2000-07-18&quot; /> <dc:subject rdf:resource=&quot;&trecvid;thailand&quot; /> <dc:subject rdf:resource=&quot;&trecvid;chiang_mai&quot; /> <dc:subject rdf:resource=&quot;&trecvid;chillis&quot; /> <dc:subject rdf:resource=&quot;&trecvid;peppers&quot; /> <dc:subject rdf:resource=&quot;&trecvid;chilli_peppers&quot; /> <dc:subject rdf:resource=&quot;&trecvid;vegetables&quot; /> <dc:subject rdf:resource=&quot;&trecvid;markets&quot; /> <dc:subject rdf:resource=&quot;&trecvid;street_markets&quot; /> <dc:subject rdf:resource=&quot;&trecvid;food_markets&quot; /> <dc:subject rdf:resource=&quot;&trecvid;food&quot; /> <dc:subject rdf:resource=&quot;&trecvid;herbs&quot; /> <dc:relation>http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg </dc:relation> </trecvid:Clip> <skos:Concept rdf:about=&quot;&trecvid;chilli_peppers&quot;> <skos:broader rdf:resource=&quot;&tgm1;Hot_peppers&quot;/> <skos:prefLabel>chilli peppers</skos:prefLabel> </skos:Concept> <skos:Concept rdf:about='tag:siderean.com,1752-09-14:2000-03-01'> <skos:prefLabel>2000-03-01</skos:prefLabel> <skos:broader rdf:resource='tag:siderean.com,1752-09-14:2000-03'/> </skos:Concept> <trecvid:Shot rdf:about=&quot;http://swvideo.techlabs.accenture.com/shotsWMV/v159_001.wmv&quot;> <rdf:type rdf:resource=&quot;&dctype;MovingImage&quot; /> <dcterms:isPartOf rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.mpg&quot; /> <dcterms:extent>21000</dcterms:extent> <dc:relation>http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg</dc:relation> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000240.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000280.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000001440.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000003120.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000005440.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000009680.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000011520.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000012040.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000013800.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000014800.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000015120.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000016760.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000018280.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000019360.jpg&quot;/> <dc:subject rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000021000.jpg&quot;/> </trecvid:Shot> <skos:Concept rdf:about=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot;> <skos:broader rdf:resource=&quot;http://swvideo.techlabs.accenture.com/color#26547&quot; /> <skos:prefSymbol rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v159.fset/f000000000.jpg&quot; /> </skos:Concept> <skos:Concept rdf:about=&quot;http://swvideo.techlabs.accenture.com/color#26547&quot;> <skos:broader rdf:resource=&quot;&trecvid;color&quot; /> <skos:prefSymbol rdf:resource=&quot;http://swvideo.techlabs.accenture.com/v289.fset/f000048880.jpg&quot; /> </skos:Concept>
  • 22. BBC Rushes Navigator: Architecture AJAX client in Firefox XRBR query XRBR response BBC Rushes RDF http://www.siderean.com/bbcrush/bbcrush.jsp (with Firefox 1.5) Metadata Aggregator Metadata Store Navigation Web Services
  • 23. Lessons Learned Data preparation Robust shot boundary detection Careful selection of keyframes Motion based Salient object based Filtering redundant keyframes Using group-of-frames (GOF) features Concept recognition/propagation Propagate keywords from clip to shots Recognize concepts from visual data Probabilistic reasoning Derive concepts from data (data mining) + labeling
  • 24. Summary Methodology of Multimedia Data Representation Semantic Web Technology Multimedia Data Mining Prototype of Multimedia Retrieval System BBC Rushes Web-based Interface using AJAX
  • 25. Future work More facets Shape + combinations Geographical location More Interfaces Map of the world for browsing places Hierarchy of SOM for browsing clips and shots More Tools Tagging tool for creating and managing metadata Tools for creating video databases (shot extraction, feature extraction, clustering, classification of events, etc.) Tools for creating audio-video compositions (TV programs, commercials, etc.)
  • 26. BBC Rushes Navigator: Navigation with LSCOM
  • 27. BBC Rushes Navigator: Hierarchical Drill-down on People Facet
  • 28. BBC Rushes Navigator: Faceted View of All Shots
  • 29. BBC Rushes Navigator: Searching by Subject
  • 30. BBC Rushes Navigator: Searching by Color, Playlist composition
  • 31. BBC Rushes Navigator: Drill-down using Subject and Color
  • 32. Contact Information Valery A. Petrushin [email_address] Bradley P. Allen [email_address]