FORM 2
THE PATENTS ACT, 1970 (39 of 1970)
&
THE PATENT RULES, 2003
COMPLETE SPECIFICATION
{See Section 10 and Rule 13)
Title of invention:
SYSTEM AND METHOD FOR CREATING A MULTIMEDIA BOOK WITH
CONTEXTUAL RE-USE OF EXISTING MULTIMEDIA CONTENTS
APPLICANT:
TATA Consultancy Services Limited A company Incorporated in India under The Companies Act, 1956
Having address:
Nirmal Building, 9th Floor,
Nariman Point, Mumbai 400021,
Maharashtra, India
The following specification particularly describes the invention and the manner in which it is to be performed.

FIELD OF THE INVENTION:
The present invention relates to the field of multimedia. In particular this invention relates to the creation of Multimedia books with contextual re-use of existing multimedia contents, its storage and linear or non-linear navigation.
BACKGROUND OF THE INVENTION:
Multimedia books are collections of computer readable documents containing text, graphics, sound, video or animation etc. These documents include what is called hypertext and hypermedia. Hypertext refers to words (or even pictures) in the document with a link to other documents or pages in the same document. Mutually linked images, digital video and sound are similarly called hypermedia. By combining these elements we can get bundled products that are:
• Efficient - you may replace information you read in a textbook with information you can see and hear.
• Direct - you may get information using the best medium, such as voice or animations.
• Personal and interactive - y ou may control the flow of information and look for information in a non-linear or an event-driven manner. The information accessed depends on individual choices and will not be constrained to sequential turning of pages.
Use of multimedia gives us a new dimension of designing the learning products (books) compared to the existing ones which are redundant and static. Compared to a standard textbook one may illustrate processes rather than stationary depictions. Moreover, continuous linkage between multimedia content often deals with time or space dependent phenomena, absent in a standard textbook with only static image content.
The development of information and communication technology has evolved new concepts like "open learning", "virtual schools" , "virtual classrooms" , "the digital university" and "distributed teaching". It is expected that the use of multimedia books will be important in dissemination of educational content in such emerging concepts.

Some of the inventions belonging to this domain known to us are described below:
US Patent number 7624337 discloses a method and system are provided for tagging, indexing, searching, retrieving, manipulating, and editing video images on a wide area network such as the Internet. A first set of method is provided for enabling users to add bookmarks to multimedia files, such as movies, and audio files, such as music. The multimedia bookmark facilitates the searching of portions or segments of multimedia files, particularly when used in conjunction with a search engine.
US Patent application 20100004944 discloses a computer implemented method and system for compiling and publishing an online book in an online collaborative environment. In this application the system takes user input like subjects, contents, syllabus, out line of books for creation of the online book. Further the users can also discuss, review, annotation and comment on the multimedia data and print book or provide online copy.
US Patent application 20100161653 discloses a software interface that increases the efficiency for reading, abstracting of information and locating material within an electronic publication, pursuant to which author content and/or researcher content may be quickly searched, accessed and stored on an academic tool utilizing e-book technology.
US Patent application 20100005379 describes the methods and apparatus which are provided for efficient and effective loading of media in a multi-media presentation, such that a user may be able to view and/or edit the presentation. In one embodiment, a method for loading media comprises providing a maximum number of layers per page of a collage document, the collage document including at least one page having a plurality of layers, and each layer being associated with a media object.
Synchronized Multimedia Integration Language (SMIL) 1.0 Specification, (http://www.w3.org/TR/REC-SlMlL) by the World Wide Web Consortium (W3C) provides means for integrating a set of independent multimedia objects into a synchronized multimedia presentation. Using SMIL, an author can:
1. describe the temporal behavior of the presentation
2. describe the layout of the presentation on a screen
3. associate hyperlinks with media objects

The LimSee3, a multimedia authoring tool based on template mechanism and customizable interface, {http://limsee3.gforge.inria.fr/public-site/about.html) illustrates that how to improve the reusability not only with template definitions but also by the homogeneous structuring of documents.
Some of the shortcomings of the state of the art mentioned above are:
• Absence of a model for building a reusable media content library and repurposing of contents in multiple contexts.
• Absence of personalization by the author by the way of synchronized integration of audio-visual and textual articulation with a multimedia presentation.
• Absence of facility for a viewer to personalize the contents with personal annotations in multimedia format for later synchronized viewing, an operation that is similar to and an extension of margin-writing on a paper book by a reader.
In order to address the above mentioned lacunae, the proposed invention provides a system and method for annotation of multimedia content and synchronized presentation of multimedia contents in custom spatial layouts and temporal order. The present invention also enables the user to create a 'Multimedia Book', where information is presented in multimedia format and organized in chapters using customized spatial layout and temporal order. The contents can be in different media forms such as images, video snippets, interactive flash animations, sketches, hand-written notes, presentation slides, documents in Portable Document Format (PDF) and can be obtained from a reusable content library or the author's local collection.
Moreover, a 'Multimedia Book' can be personalized with an author's articulation in different media forms, such as speech, video, sketches, handwritten and typed text where the articulation synchronizes with the custom content being presented. Further, a viewer can make personal notes in different media formats, such as speech, video, sketches, handwritten text and typed text, which is played back in a synchronized manner with the contents of the book.
Other features and advantages of the present invention will be explained in the following description of the invention having reference to the appended drawings.

OBJECTS OF THE INVENTION:
The primary object of the invention is to provide a system and method for creation of a multimedia book with contextual re-use of existing multimedia contents.
Another object of the invention is to provide a system and method that enables linear or nonlinear navigation of the created Multimedia book.
Another object of the invention is to provide a system and method that is capable of ingesting multimedia contents in a content repository, annotate them at different levels of granularity and to establish their ownership. .
Another object of the invention is to provide a method capable of searching in the content repository to locate appropriate contents or appropriate parts of the contents through browsing and searching.
Another object of the invention is to provide a method for searching only on annotated parts of multimedia content rather than searching the entire multimedia content.
Another object of the invention is to provide a system and method that enables authoring a Multimedia Book containing chapters, where the chapters comprise multimedia contents placed coherently in a spatial layout and synchronized in time using table of contents, wherein the contents can be selected either from the content repository or author's local collection.
Another object of the invention is to provide a system and method to create book layouts and chapter layouts in the multimedia book and ingest them in the layout library.
Another object of the invention is to provide a system and method to track and account for reuse of multimedia objects in one or more multimedia books,
Yet another object of the invention is to provide a method that enables recording an articulation and its synchronization with a Multimedia Book at a designated place holder. The

articulation can be in the form of, but not restricted to, speech, video, handwritten or typed text.
Yet another object of the invention is to enable playing a Multimedia Book through an interactive browser that permits non-linear navigation through the book using table of contents.
Yet another object of the invention is to create a multimedia book which is author driven and results in enhanced viewer experience.
Yet another object of the invention is to create a multimedia book that can be personalized where viewer can make personal notes in different formats.
SUMMARY OF THE INVENTION:
The present invention provides a system and method for creating multimedia books with contextual re-use of existing multimedia contents, its storage and a browser for its linear or non-linear navigation. The said system and method ingest multimedia contents from the content repository; annotate them at different levels of granularity for establishing their ownership, thereby the user can search in the content repository to locate appropriate contents or appropriate parts of the contents through browsing and searching. The proposed invention provides a system comprising the following components.
1. A multimedia content repository comprising physical storage device, processing device, input-output devices, file system and database system, and connected to the Internet, to store the multimedia data, metadata and index.
2. A multimedia book library comprising physical storage device, processing device, file system and database system, and connected to the Internet, to store multimedia books together with metadata, catalog and index tables.
3. A user interface:
a. to annotate multimedia contents at different levels of granularity and ingest them in the content repository;

b. to search and to browse through the content repository and select
appropriate parts of multimedia contents;
c. to create book layouts and chapter layouts and ingest them in the layout
library. The book layout portrays the spatial and temporal arrangements of
the sections in the book like, but not restricted to, foreword, preface, prolog,
table of contents, epilog. The chapter layout portrays the spatial layout for a
chapter;
d. to author a multimedia book by combining contents from content library and
from author's local collection. The interface allows the selection of desired
book and chapter layouts from the layout library. The contents can be placed
coherently in a spatial layout of each chapter and synchronized in time. The
space layout of the chapter shows the spatial arrangement of the contents
and can be chosen from a layout library or can be created by an author while
time layout portrays the timing aspects;
e. to facilitate the recording of articulation of the author in the form of video,
hand sketches, markings, speech and handwritten or typed text;
f. To facilitate publishing a Multimedia Book, i.e. to ingest a book, together with
necessary metadata, to the Multimedia Book Library, where it is appropriately
cataloged and indexed.
to navigate through the multimedia book. The interface facilitates searching for desired chapter in the book, linear or non-linear navigation and marking personal notes in the book.
BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS
The foregoing summary, as well as the following detailed description of preferred embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the invention, there is shown in the drawings example constructions of the invention; however, the invention is not limited to the specific system and method disclosed in the drawings:
Figure 1 shows multimedia content creation and search system according to a first embodiment of the invention.
Figure 2 shows multimedia book authoring system according to a second embodiment of the

invention.
Figure 3 shows multimedia book navigation system according to a third embodiment of the invention.
DETAIL DESCRIPTION OF THE INVENTION
Some embodiments of this invention, illustrating its features, will now be discussed in detail. The words "comprising," "having," "containing," and "including," and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.
It must also be noted that as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Although any methods, and systems similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present invention, the preferred methods, and systems are now described.
Figure 1 shows multimedia content creation and content search system according to a first embodiment of the invention. Creator authentication module 101 enables content creators to create their accounts and log in to the content creation system. The authentication into the system allows content creators to ingest multimedia contents and annotate them that in turn enable to catalog the contents as well as maintain their corresponding ownerships. The ingestion 102 module enables an author to ingest multimedia contents in content repository 107. The module provides a user interface for selecting and ingesting multimedia contents like still images, video snippets, interactive flash animations, sketches, hand-written notes, presentation slides, PDF documents, etc.
Segmentation 103 module comprises software programs for segmentation of media data along with extraction of media features. The media contents can be segmented either spatially (for example, different objects J regions in a still image) or temporally (for example, different scenes in a video) by using different segmentation criteria and methods, such as segments of fixed sizes and/or durations, identified image regions or video shots, silence or

speaker transition in speech, etc. depending on specific embodiment of the invention. The media instances can either be physically segmented, i.e. new media instances created or logically segmented, i.e. segments are marked on the original media instance depending on the specific embodiment of the invention.
The feature extraction 104 module extracts various media features of the segmented contents. The media features are typically, but not restricted to, low level media features recommended in MPEG-7 standards, for example, color features, texture features, shape features, for images and speech descriptors, genre descriptors for audio and motion descriptors, genre descriptors, object descriptors along with image features for videos depending on specific embodiment of the invention.
The annotation 105 module enables an author to annotate ingested contents and the content segments at different levels of granularity. For example, a video can have scene level, event level and object-level annotations, an image can have object level annotations and so on depending on specific embodiment of the invention. Further this module (105) also enables capturing of geographical identification metadata that allows geo-tagging of the contents which can help an author to find location specific contents.
The metadata generated by modules 102, 103, 104, and 105 can be of different types, for example spatial and temporal segment description, extracted media features for the media instance or its segments, annotations for the media instance or its segments created interactively by the authors, geo-tagging data, etc. or a combination thereof, depending on specific embodiment of the invention. The module can be realized as one processing element or several processing elements with different audio-visual media processing capabilities depending on the specific embodiment of the invention. The generated metadata is stored in different ways, such as in XML documents following MPEG-7 schema, depending on specific embodiment of the invention.
The semantic indexing module 106 does indexing of contents, which have been ingested into the system, using keywords used in the annotations. The indexing is done using, but not limited to, balanced trees, B/B+ trees, hashing, etc. depending on specific embodiment of the invention.

The multimedia content repository 107 comprises physical storage device, processing device, input-output devices, file system and database system to store all the ingested multimedia contents, corresponding metadata, layout library and index. The repository is made accessible through the internet for remote access by content creators and authors. The repository is a centralized or a distributed system depending on specific embodiment of the invention.
Figure 2 shows multimedia book authoring system according to a second embodiment of the invention. The author authentication 201 module enables authors to create accounts and log in to the book authoring system. The authentication into the system allows authors to create and store books online and establish their ownership. Create book 202 module enables an author to record metadata for the book for example, but not restricted to, the author's name, biography, title of the book, topic and broader subject of the book, and its fitment for any particular audience.
A layout library 204 comprises physical storage device, server, file system and database system to store and retrieve different layouts, typically but not restricted to book layouts, cover layouts and chapter layouts, which are required to create a multimedia book. The library is connected to the Internet to support remote access by the users. The book layout contains information of the spatial and temporal structures for the different sections of a multimedia book, typically but not restricted to, cover, credits, body and glossary and their temporal sequence. The body of the book contains several chapters. The chapter and cover layouts contain information about the spatial positioning of multimedia contents in the chapter or on the cover in the form of regions where each region is the placeholder for one or more multimedia contents. The regions can be separate or can be overlapping depending on specific embodiment of the invention. The layouts are saved in XML or some other format depending on specific embodiment of the invention.
An author can use existing book, chapter or cover layouts during authoring a multimedia book. Additionally, an author or a content creator can create one or more new book, chapter or cover layouts using module 205 and store them in the library with ownership information, either for his private use or for use by other authors.
Select book layout 203 module enables an author to browse through the available book

layouts in the layout library 204 and select the appropriate layout. Alternatively, an author can create new book and chapter layouts. ToC creation module 206 facilitates creation of Table of Contents for the multimedia book, The ToC (Table of Contents) comprises several ToC elements each of which conveys the contents of a chapter in the book and number of ToC elements equals number of chapters in the book. A ToC element enables easy localization of desired chapter in the book and is typically made up of text or sketches created by the author, an image or a video thumbnail from the content library or the author's local machine, or a combination thereof, depending on specific embodiment of the invention. Select chapter layout module 207 enables an author to browse through the chapter layouts in the layout library 204 and select the appropriate layout. All the chapters will have a single layout or a separate layout can be selected for each chapter depending on specific embodiment of the invention. Add content module 208 enables an author to add multimedia contents in the chapter.
The module provides an interface to browse or search through the multimedia content repository 107 or from the author's local collection 209 or both, depending on specific embodiment of the invention. The module 208 also provides an interface for browsing through a content library and searching of desired content through content type, keywords or sample media. Specific embodiments of the invention can also support ontology based navigation and semantic search. Temporal organization module 210 allows an author to arrange the contents added to the chapter on the timeline of the chapter. For any region in the spatial layout, the start-time and duration of content can be selected with a constraint that they do not overlap. In addition, media start time can be selected for continuous media, like audio and video, so that, in effect, a part of the content is included.
Module 211 provides an interface to an author to record articulation. It comprises equipments, such as still or video camera, microphone, digital pen, and such other devices that enables recording of the articulation in different forms, for example but not limited to, speech recording, video recording, typed text and free hand sketches, or a combination thereof, that may or may not overlay with the existing contents, depending on specific embodiment of the invention.
The recordings of the articulation are integrated in the designated placeholders, may or may not overlay with the other contents of the chapter. The articulations can be recorded and

integrated with the complete book or desired chapters or appropriate parts of the chapters depending on specific embodiment of the invention. Module 212 facilitates creation of a multimedia glossary of the book by automatically extracting a list of terms associated with specific media objects and integrating them in the designated location in the book, The module uses keywords from the textual annotations of various granules of the chapters in the book. Module 213 enables an author to create the cover for the book, create credit section, and integrate them along with the body of the book.
The book cover comprises a video, an image, textual description, speech, animation or combination of them depending on specific embodiment of the invention. The module 213 generates the credit section for the book by extracting the author information from author authentication module 201 and also extracts information owners of the contents used in the book from content metadata that is stored in content repository 107.
The complete Multimedia Book comprises a book cover, included multimedia contents in the book chapters, a ToC, glossary, credits and a script that contains information to display the contents according to the created spatial layout and synchronized according to the created time-line. The script can be generated in, but not limited to, SMIL or interactive flash depending on specific embodiment of the invention. The book thus created is stored in the book library 214 which comprises physical storage, file system and a server to store and retrieve the created multimedia books. The storage is connected over the Internet for remote access by the users. The module.213 also does cataloging of books using metadata captured through module 202. In addition, the module indexes the created multimedia books using textual keywords or media examples like images used in the chapters or both depending on specific embodiment of the invention.
Figure 3 shows the multimedia book browsing system according to a third embodiment of the invention. The user authentication module 301 enables users to create accounts and log in to the book browsing and navigation system. Module 302 allows a user to browse the book library 214 using the catalog and the index tables generated in module 213. The module 302 facilitates searching of books using textual keywords or media examples or both based on specific embodiment of the invention. The module 303 provides a user with a system that allows navigating through the book. A user can navigate through the book online where the contents actually reside in the book library 214 or can be downloaded on the local machine

as a complete package and browsed offline depending on specific embodiment of the invention.
A user can navigate through the book on laptop or desktop or a mobile device depending on specific embodiment of the invention. The module 304 enables a user to add personal notes to the book at different levels of granularity. For example, notes can be added at scene level in videos or at the object levels in videos and images, etc. depending on specific embodiment of the invention. The personal notes can be in the form of text, speech, overlaid markings or sketch depending on specific embodiment of the invention. The notes are played back synchronously with the book when the latter is played back again. The notes can be marked as private or public by the user. A private note is accessible to the user alone, while a public note is accessible to other users also. The notes are saved either on the user's local machine or on the server depending on specific embodiment of the invention.
While considerable emphasis has been placed herein on the particular features of this invention, it will be appreciated that various modifications can be made, and that many changes can be made in the preferred embodiments without departing from the principles of the invention. These and other modifications in the nature of the invention or the preferred embodiments will be apparent to those skilled in the art from the disclosure herein, whereby it is to be distinctly understood that the foregoing descriptive matter is to be interpreted merely as illustrative of the invention and not as a limitation.

WE CLAIM:
1. A system for creating a Multimedia Book characterized in being capable of contextual re-use of existing multimedia contents, the said system comprising of; multimedia content repository; user input-interface means;
processing module means having program instructions stored in memory of the system that are configured to cause the processor to:
a. segment the stored multimedia contents;
b. extract the media features of the segmented contents;
c. enable a user to annotate segmented contents;
d. enable semantic indexing of media contents; and
user-output interface means to present the user with such multimedia book with said integrated contents.
2. The system of Claim 1, wherein the multimedia data, metadata and index are stored in multimedia content repository.
3. The system of Claim 2, wherein the multimedia content repository further comprises of physical storage device, processing device, input-output devices, file system and database system to store all the ingested contents, corresponding meta data, layout library and index.
4. The system of Claim 1, wherein the user input-interface means enables the user to annotate multimedia contents, to search and browse through the content repository and to create book and chapter layouts.
5. The system of Claim 1, wherein the media contents can be segmented either spatially or temporally.
6. The system of Claim 1, wherein the multimedia contents are annotated at different levels of granularity after ingestion in the content repository.

7. The system of Claim 1, wherein the indexing is done using, but not limited to balanced trees, B/B+ trees or hashing.
8. The system of Claim 1, wherein the created multimedia book is adapted according to layouts pre-stored in the layout library.
9. The system of Claim 1, wherein the desired chapter can be navigated in linear or Non-linear manner.
10. A method for creating a multimedia book characterized in having contextual re-use of existing multimedia contents wherein the said method comprises the processor implemented steps of:
a. storing multimedia data, metadata and index in a multimedia content
repository;
b. providing user-input interface for accepting a query for segmentation of
stored multimedia data;
c. extracting the media features of the segmented contents;
d. enabling an author to annotate such segmented contents;
e. automatic indexing of media contents; and
f. providing user-output interface to present the user such multimedia book
with said integrated contents.
11. The method of Claim 10, wherein multimedia contents are annotated at different levels of granularity after ingestion in the content repository.
12. The method of Claim 10, wherein the user input-interface means enables the user to annotate multimedia contents, to search and browse through the content repository and to create book and chapter layouts.
13. The method of Claim 10, wherein the multimedia contents can be segmented either spatially or temporally.
14. The method of Claim 10, wherein the indexing is done using, but not limited to balanced trees, B/B+ trees or hashing.

15. The method of Claim 10, wherein the desired chapter can be navigated in linear or Non-linear manner.