Let's have a closer look at media abstractions. How can we capture the characterization of a variety of media types in one common media abstraction.

A media abstraction has the following components:

• state : smallest chunk of media data
• feature : any object in a state
• attributes : characteristics of objects
• feature extraction map : to identify content
• relations : to capture state-dependent information
• (inter)relations between 'states' or chunks

However, before giving some examples, we must note that the feature extraction map does not need to provide information about the content of a chunk of media data automatically. It may well be a hand-coded annotation.

Our first example is an image database.

states: { pic1.gif,...,picn.gif } 

features: names of people 

extraction: find people in pictures 

relations: left-of, ...

In an image database it does not make much sense to speak about relations between 'states' or chunks of media data, that is the images. %5

For our next example though, video databases, it does make sense to speak about such relations, since it allows us to talk about scenes as sequences of frames.

states:  set of frames 

features:  persons and objects

extraction:  gives features per frame 

relations:  frame-dependent and frame-independent information

inter-state relation:  specifies sequences of frames

Now, with this definition of media abstractions, we can define a simple multimedia database, simply as a finite set M of media abstractions In order to deal with the problems of synonymy and inheritance, we can define a structured multimedia database that supports:

• equivalence relations: to deal with synonymy
• partial ordering : to deal with inheritance
• query relaxation : to please the user

Query Language

Having media abstractions, what would a query language for such a database look like? We may extend SQL with special functions as indicated below:

Type: object  |->  type 

Object With Features:  f |-> { o |  object o contains  f }  

Object With Features And Attributes:  (f,a,v) |-> { o |  o contains f with  a=v }  

 Features In Object:  o |-> { f | o  contains  f }  

 Features And Attributes In Object:  o |-> { (f,a,v) | o  contains  f  with  a=v }

Having such functions we can characterize an extension of SQL, which has been dubbed SMDS(Switched Multimegabit Data Services)-SQL in Multimedia Database, as follows:

SELECT -- media entities

• m : if m is not a continuous media object
• m:[i,j] -- m is continuous, i,j integers (segments)
• m.a -- m is media entity, a is attribute

FROM

• <media><source><m>

WHERE

• term IN funcall

As an example, look at the following SMDS-SQL snippet.

SELECT M

FROM   smds source1 M

WHERE  Type(M) = Image AND

M IN Object With Feature ("Dennis") AND

M IN Object With Feature ("Jane") AND

left("Jane","Dennis",M)

Note that M is a relation in the image database media abstraction, which contains one or more images that depict Jane to the left of Dennis. Now, did they exchange the briefcase, or did they not?

When we do not have a uniform representation, but a hybrid representation for our multimedia data instead, we need to be able to: express queries in specialized language, and to perform operations (joins) between SMDS and non-SMDS data.

• express queries in specialized language
• perform operations (joins) between SMDS and non-SMDS data

Our variant of SQL, dubbed HM-SQL, differs from SMDS-SQL in two respects: function calls are annotated with media source, and queries to non-SMDS data may be embedded.

• function calls are annotated with media source
• queries to non-SMDS data may be embedded

As a final example, look at the following snippet:

SELECT M

FROM smds video1, videodb video2

WHERE M IN smds: Object With Feature ("Dennis") AND

M IN videodb:Video With Object("Dennis")

In this example, we are collecting all video fragments with Dennis in it, irrespective of where that fragment comes from, an (smds) database or another (video) database.