FIELD OF INVENTION
The present invention relates to a system and method for automatically
categorizing recorded and reproduced multimedia data based on location
metadata.
The system of the present invention can be incorporated in an apparatus that
produces or records multimedia data (sound, images, video) such as a camera, a
video camera, a motion picture camera, a video recorder (i.e., camcorder]), a
digital still camera, multifunctional cellular camera having multimedia recording
capabilities, or multifunctional portable multimedia player with sound recording
facility.
BACKGROUND OF THE INVENTION
The concept of integrating camera metadata with low-level vision cues for photo
classification is known. Document Boutell M. and Jiebo Luo, 2004 "Photo
classification by integrating image content and camera metadata", International

Conference on Pattern Recognition, Aug. 23-26, Vol. 4, pp.901-904 and
document Boutell M. and Jiebo Luo, 2005 "Beyond pixels: Exploiting camera
metadata for photo classification", Pattern Recognition, Vol. 38, pp. 935-946
describe the problems Indoor-Outdoor classification, sunset detection, and
manmade-natural classification of manmade photos.
Document Jiebo Luo and Boutell M., 2005 "Method for semantic scene
classification using camera metadata and content-based cues", US Patent
Application Publication, 0105776A1, May 19 describes the problem of semantic
classification of consumer photos into indoor, outdoor - sunset, picnic, and
beach. The metadata features used are exposure time, flash fired, subject
distance, and focal length. Low-level semantic cues considered are color and
texture.
Document Hang Liu, 2006, "Method and apparatus for automatically attaching a
location indicator to produced, recorded, and reproduced images", US Patent
Application Publication, 0114336, June 1 describes a method and apparatus for
automatically attaching a location indicator to images. This document makes
basis of the present invention. The attached location will be automatically
displayed by the image reproduction device.

Northcutt John and Sony Ericsson mobile communication AB, 2005, "Location
status indicator for mobile phones", International Application Published under the
PCT, WIPO, July 7 describes a method and apparatus for location status indicator
for mobile phones. The document describe a method of presenting location data
representing a mobile phone's current location to a mobile phone user via the
mobile phone display. The location information is ironically displayed and
periodically updated. The present invention can work in synergy with the
location status indicator of this document.
Document Eastman Kodak Company, 2007, "Location based Image Classification
with Map Segmentation", US Patent Application Publication, US2007.0115373
describes a method and system to cluster the captured multimedia records based
on capture locations, wherein capture locations are identified by segmenting a
map stored as a metadata in each image record. The method has a
disadvantage of segmenting a map before clustering capture records. Further,
map segmentation algorithms are very time consuming and makes the invention
difficult to implement in low processing power devices like cameras and cell
phones.

Hewagamage, K.P.; Hirakawa, M., 2000, "Augmented Album: Situation
Dependent System for a Personal Digital Video / Image Collection", International
Conference on Multimedia and Expo, ICME, IEEE, Vol. 1, Pages 323-326
describes an Augmented Album, an application developed to demonstrate how
user situations can be used to provide an easy-to-use and easy-to-remember
interface for the management and retrieval of digital pictures that consist of both
digital video clips and still images. In this system, contextual information such as
the location, time, and user events, are captured when a picture is taken. It
represents the meaning of the picture as well as its content information to some
extent, and thus benefits us to retrieve image / video clips. At the same time,
the contextual information could be used to achieve a more realistic organization
of those pictures on a computer system. However, it did not address the
creation of directory tree structure and recommendor system, which are prime
embodiments of the present invention.

SUMMARY OF THE INVENTION
The main object of the present invention is to provide a method and system for
auto-categorization based on location metadata and sub-categorization into
multimedia data type and provide a recommendor system based on location,
time, date, and media types.
The invention first clusters and thus, auto-categorizes the recorded and
reproduced multimedia data according to location indication attached with the
multimedia data. To auto-categorize multimedia data based on location
indication, the invention utilizes a progressive constructive clustering procedure,
which is extremely fast to build into the low processing power devices like mobile
phones, cameras, and portable multimedia players. Multimedia data are further
sub-categorized into image, video, and audio categories within each location
category. This auto-categorized multimedia data will be stored in the two level
of directory structure where in the first level is directory names same as location
metadata clustered and identified by the clustering algorithm. Number of
directories will be automatically identified by the categories (i.e., total number of
locations over which the multimedia data has been recorded). Within each
location directory, the second level of directories will be created as per
multimedia data, which is essentially classifying image, audio, and video data.

Based on the auto-categorized and sub-categorized multimedia data, the
invention then develops a recommendor system for retrieving the multimedia
data on the basis of location categories and type of multimedia data. A
recommendor system extracts all the locations identified by progressive
constructive clustering procedure and provides fast and easy retrieval of
multimedia databased on type of data (audio, video, image), location categories,
time, and date.
In a preferred embodiment the present invention provides a method for
automatically categorizing recorded and reproduced multimedia data
independent of any user intervention, comprising the steps of: attaching a
location indicator metadata to multimedia data stored in internal or external
memory of a PC or a multimedia recording devices; categorizing the data with
location indicator metadata by a progressive constructive clustering procedure;
sub-categorizing the data into image, video and audio categories; identifying
total number of location categories and multimedia files in each category;

creating a two-level directory tree based on location metadata and multimedia
file types for storing the auto- categorized and sub-categorized multimedia data;
building a recommendor system for fast and easy retrieval of the multimedia
data on the basis of location category and type of data; and generating a slide
show for playing in a multimedia player.
BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS
The invention can now be described in detail with the help of the figures of the
accompanying drawings in which
Figure 1 shows in flow diagram form the method of the present invention.
Figure 2 shows an example directory and subdirectory structure
created out of location metadata and multimedia file
types using the present invention.

DETAILED DESCRIPTION
The present invention utilizes location metadata attached with multimedia data
(image, audio, video) for digital content management or solving in one way the
problem of managing large scale multimedia data. Here large scale means,
management of digital contents containing millions of entries of multiple file
formats for audio, video, and image data.
Attaching a location metadata in recorded images has been described by Hang
liu. The method of automatically attaching a location indicator to produced,
recorded and reproduced images can be easily extended to other multimedia
data, e.g., audio and video. Mobile phones not equipped with prior art [4] can
access the location status indicator for mobile phones described by Northcutt
John and attach the location metadata to recorded multimedia data.
In general, most multimedia recording and reproduction device encode metadata
in the header of the file. Headers are specific to the type of the multimedia data
and do not limit the present invention application. For example, most of the

digital still cameras store image metadata in the header of the Exif file. The
present invention can be implemented in hardware or firmware and can also be
produced as an application to be used on personal computers for digital content
management based on location metadata. In any case, whether to use or not
the invention application for digital content management is solely the choice of
the end user.
As shown in Figure 1 of the drawings, the method steps of the present invention
can be described under the following two parts, which makeup both the
embodiment of the invention in its best form.
I. Auto-categorization based on location metadata. This embodiment is
either an application running in multimedia reproduction devices like
personal computers, digital albums, USB or Ethernet monitors, or an
embedded application built into multimedia recording devices like
cameras, digital cameras, camcorders, and mobile phones.

(i) For image database stored in personal computer or memory
(internal or external) of multimedia recording device do the first
level of categorization based on location metadata. There can
be n number of images to categorize, where n can be in the
range of millions. The clustering steps are described as follows:
a. Parse the location metadata of the very first multimedia file stored
in the memory and store the location information in a string Si.
Initialize the category counter C=1. Initialize the array counter
for number of multimedia files in each category Nc and make N(l)
= 1. Store the string representing the name of the multimedia file
in structure MFile.
b. Take on multimedia file, which is not processed by an application,
parse the location metadata and store it in the string Snew
Compare the string Snew with all the stored songs S1 ,SC. If
comparison result with all of the strings S1,.....Sc is FALSE, i.e., string
Snew is not equal to any of the strings S1, ,SC, go to step d.

c. If comparison result of string Snew with an arbitrary location string
Sk where Sk  {S1,...,Sc} is TRUE, the multimedia file with the
location string Snew is equal to multimedia file with location string Sk
(or in other words, both the multimedia files are recorded at the
same location). Increase N(c) by 1. Update MFile structure.
d. Take multimedia file with location string Snew as a new category.
Increase C by 1. Update Mfile structure to point to next category.
(ii) After execution of algorithms described in steps a, b, c and d of 1 above
the system identifies.
• Total number of location categories in C.
• Number of multimedia files in each location category c=1,....,C in
N(c).
• String for each location category Sc, C=1,...., C.
• Structure MFile containing the name strings of multimedia file in
each location category.

(iii) Create a directory tree with root directory name as SAMSUNG_DCM.
• Create C subdirectories with name of 1st directory as a string S1, 2nd
directory as a string S2, , cth directory as a string Sc.
• Parse the structure MFile for each category. Classify the
multimedia files within each category into either of audio, image, or
video category.
• Create one to three directories (based on the availability of
multimedia files) Audio Files, Image Files, and Video Files and copy
the appropriate multimedia files within each category to these
subdirectories.
(iv) If new multimedia sample(s) arrives, automatically classify into one of
the directory as per its location metadata and subdirectory as per type
of the multimedia data (i.e., audio, image, video).
• If new multimedia sample(s) already belong to one of the location
category, it classifies to the matched location category and then sub
classifies into the audio, image, or video category. If multimedia type
category does not exist for the new sample(s), automatically creates new
multimedia type category and classifies the sample(s).

• If new multimedia sample(s) does not belong to any of the location
category, it creates new location category and classifies the sample(s).
II. Recommendor system built based on the embodiment described in part I
above for efficient search and retrieval of multimedia files. This
embodiment is either an application running in multimedia reproduction
devices like personal computers, digital albums, USB or Ethernet monitors,
or an embedded application built into multimedia recording devices like
cameras, digital cameras, camcorders, and mobile phones.
1. Provide SEARCH and STATICS options for user interaction.
2. Build user level statistics for summary view of multimedia files in the
STATISTICS option. This contain number of locations, list of locations
where multimedia files have been recorded, number of multimedia files at
each location, and number of audio, image, and video files at each
location.

3. Initialize array counters NL and ND to maintain the log of search hits of
location and date.
4. In the SEARCH option present.
A. Location selection option with names of locations available in pull
down menu. Default location taken as the location of the last
search.
B. Multimedia file type selection option with Audio, Image, and Video
available in pull down menu. Default file type taken as the file type
of the last search.
C. Date selection option with
i. user entry in DD/MM/YYYY format and
ii. display of a calendar and select DD, MM, YYYY from there
on with default date taken as the date of last search.

D. Update array counters NL and ND.
5. Search the multimedia files as per the search criteria given by the user
and list the results.
6. Based on the weekly or monthly (as per user preference) status of array
counters NL and ND, publish the most preferred location and date in the
STATISTICS over the period.
7. Check whether user wants to create a slide show out of Images and Video
files retrieved through search options.
8. Generate slide show with user selected transition effects and theme tunes.
9. Update recommendor system after new sample(s) comes in.
The system of the present invention can be built into Samsung multimedia
recording devices before their shipment.

WE CLAIM
1. A method for automatically categorizing recorded and reproduced
multimedia data independent of any user intervention, comprising the
steps of:
(a) attaching a location indicator metadata to multimedia data
stored in internal or external memory of a PC or a
multimedia recording devices;
(b) categorizing the data with location indicator metadata by a
progressive constructive clustering procedure;
(c) sub-categorizing the data into image, video and audio
categories;
(d) identifying total number of location categories and
multimedia files in each category;
(e) creating a two-level directory tree based on location
metadata and multimedia file types for storing the auto-
categorized and sub-categorized multimedia data;
(f) building a recommendor system for fast and easy retrieval
of the multimedia data on the basis of location category
and type of data; and
(g) generating a slide show for playing in a multimedia player.

2. The method as claimed in claim 1, wherein said clustering step comprises
the steps of:
(a) parsing the location metadata of first storing the location
information in a string S1, initializing the category counter
C=1 and the array counter for number of multimedia files
in each category Nc making N(1) = 1 and storing the string
representing the name of the multimedia file in structure
MFile;
(b) taking one multimedia file that is not processed by an
application, parsing the location metadata and storing it in
a string Snew, comparing the string Snew with all stored data
or songs S1, Sc);
(c) when comparison result with all of the strings S1 Sc
is FALSE, i.e., string Snew is not equal to any of the strings
S1, ,Sc going to step (e);
(d) when comparison result of string Snew with an arbitrary
location string Sk where Sk e{S1,, Sc}, is TRUE, the
multimedia file with the location string Snew is equal to

multimedia file with location string Sk (or in other words,
both the multimedia files are recorded at the same location);
and
(e) taking multimedia file with location string Snew as a new
category.
3. The method as claimed in claim 1, wherein creating a two level directory
tree comprises:
- creating C subdirectories with name of 1st directory as a string S1,
2nd directory as a string S2, Cth directory as a string Sc.
- parsing the structure MFile for each category. Classifying the
multimedia files within each category into either of audio, image, or
video category.
- creating one to three directories (based on the availability of
multimedia files) Audio Files, Image Files, and Video Files and copy
the appropriate multimedia files within each category to these
subdirectories.

4. The method as claimed in claim 1, wherein multimedia samples when
received are classified into one of the directory as per its location
metadata and subdirectory as per type of the multimedia data (i.e., audio,
image, video).
5. The method as claimed in claim 4, wherein if new multimedia sample(s)
already belong to one of the location category, it classifies to the matched
location category and then sub classifies into the audio, image, or video
category. If multimedia type category does not exist for the new
sample(s), automatically creates new multimedia type category and
classifies the sample(s).
6. The method as claimed in claim 4, wherein if new multimedia sample(s)
does not belong to any of the location category, it creates new location
category and classifies the sample(s).

7. The method as claimed in claim 1, wherein said recommendor system is
an application running in multimedia reproduction devices like personal
computers, digital albums, USB or Ethernet monitors, or an embedded
application built into multimedia recording devices like cameras, digital
cameras, camcorders, and mobile phones.
8. A method for automatically categorizing recorded and reproduced
multimedia data independent of any user intervention, substantially as
herein described and illustrated in the figures of the accompanying
drawings.

The main object of the present invention is to provide a method and system for auto-categorization based on location metadata and sub-categorization into multimedia data type and provide a recommendor system based on location,
time, date, and media types. The invention first clusters and thus, auto-categorizes the recorded and reproduced multimedia data according to location indication attached with the multimedia data. To auto-categorize multimedia data based on location indication, the invention utilizes a progressive constructive clustering procedure,
which is extremely fast to build into the low processing power devices like mobile phones, cameras, and portable multimedia players. Multimedia data are further sub-categorized into image, video, and audio categories within each location category. This auto-categorized multimedia data will be stored in the two level
of directory structure where in the first level is directory names same as location metadata clustered and identified by the clustering algorithm. Number of directories will be automatically identified by the categories (i.e., total number of locations over which the multimedia data has been recorded). Within each
location directory, the second level of directories will be created as per multimedia data, which is essentially classifying image, audio, and video data. Based on the auto-categorized and sub-categorized multimedia data, the invention then develops a recommendor system for retrieving the multimedia data on the basis of location categories and type of multimedia data. A recommendor system extracts all the locations identified by progressive constructive clustering procedure and provides fast and easy retrieval of multimedia databased on type of data (audio, video, image), location categories,
time, and date. In a preferred embodiment the present invention provides a method for
automatically categorizing recorded and reproduced multimedia data independent of any user intervention, comprising the steps of: attaching a location indicator metadata to multimedia data stored in internal or external
memory of a PC or a multimedia recording devices; categorizing the data with location indicator metadata by a progressive constructive clustering procedure; sub-categorizing the data into image, video and audio categories; identifying total number of location categories and multimedia files in each category; creating a two-level directory tree based on location metadata and multimedia file types for storing the auto- categorized and sub-categorized multimedia data; building a recommendor system for fast and easy retrieval of the multimedia
data on the basis of location category and type of data; and generating a slide show for playing in a multimedia player.