FIELD OF THE INVENTION:
[0001] The present disclosure relates to a technique that allows an untagged multimedia
data to be managed. More specifically, present disclosure relates to technique that allows a
user/system created multimedia data (i.e. not tagged at the time of creation) to be tagged and stored
5 and retrieved using natural language query.
BACKGROUND OF THE INVENTION:
[0002] Emojis, stickers or emoticons are one of the most common means of conversing
these days. If one needs to express his feelings, mood or needs to present his views quickly, they
10 may simply need to look out for an appropriate emoji/sticker/emoticon. In order to retrieve the
correct emoji/sticker/emoticon the user may have to look out for the tagging that has been attached
with the emoji/sticker/emoticon at the time of creation, which is either done manually or is done
by system such as in case of few basic emoji/sticker/emoticon prestored with the system.
[0003] At present there are numerous techniques available that allow the user/system to
15 create emoji/sticker/emoticon swiftly. However, in most of these cases the emoji/sticker/emoticon
created by the user/system are left untagged after use and they keep lying in the database as such.
Moreover, a user trying to retrieve such an emoji/sticker/emoticon from the database is unable to
do so as there is no tag attached to it. Furthermore, even if there is a basic tagged attached to any
such user/system generated emoji/sticker/emoticon, it is completely difficult to retrieve such
20 emoji/sticker/emoticon from the database, if the user does not have any idea about the tag and in
general uses natural language query to search for such emoji/sticker/emoticon.
[0004] Thus, there exist a need for the technology that not only allows the user/system
generated emoji/sticker/emoticon to be tagged automatically, but simultaneously allows the user
to retrieve such emoji/sticker/emoticon on the basis of natural language query.
25
SUMMARY OF THE INVENTION:
[0005] The present disclosure overcomes one or more shortcomings of the prior art and
provides additional advantages discussed throughout the present disclosure. Additional features
and advantages are realized through the techniques of the present disclosure. Other embodiments
30 and aspects of the disclosure are described in detail herein and are considered a part of the claimed
disclosure.
3
[0006] In one non-limiting embodiment of the present disclosure is disclosed is a method
for managing untagged multimedia data. Said method comprising the steps of receiving untagged
multimedia data from one or more user devices, identifying at least one of: one or more objects
present inside the multimedia data and text present inside the multimedia data. The method further
5 comprises indexing the multimedia data based on the at least one of the identified one or object
and text present inside the multimedia data and retrieving the one or more indexed multimedia
data based on natural language query received from the one or more user devices.
[0007] In yet another non-limiting embodiment of the present disclosure, the method
further discloses that the step of identifying comprises: (i). extracting the one or more objects
10 present inside the multimedia data, (ii). classifying the extracted one or more objects by matching
with at least one real-world object, (iii). generating a first confidence score based on said matching,
wherein the one or more identified object is classified as said real-world object, if the first
confidence score is above a pre-determined threshold and (iv). tagging the multimedia data based
on said classification.
15 [0008] In still another non-limiting embodiment of the present disclosure said method
discloses combining the tagged data with the extracted keywords and indexing said multimedia
data under one or more categories within a database.
[0009] In yet another non-limiting embodiment of the present disclosure the method
discloses that retrieving the one or more indexed multimedia data based on the natural language
20 query comprises (i). extracting one or more keywords from text of the natural language query,
wherein the one or more keywords are extracted by parsing the natural language query based on a
set of pre-defined parameters and (ii). semantically searching said database for the one or more
extracted keywords and (iii). identifying the one or more indexed multimedia data from the
database, based on said searching.
25 [0010] In still another non-limiting embodiment of the present disclosure recites
generating a second confidence score, in response to semantically searching said database for the
one or more extracted keywords; and identifying the one or more indexed multimedia data from
the database, if the second confidence score is above a predetermined threshold.
[0011] In another non-limiting embodiment of the present disclosure recites a system to
30 manage untagged multimedia data. In an embodiment, said system comprises one or more user
devices, a network interface and at least one server operatively coupled to the one or more user
4
devices through the network interface. In said embodiment, said server further comprise a
transceiver configured to receive untagged multimedia data from the one or more user devices and
an identification unit configured to identify at least one of: one or more objects present inside the
multimedia data and text present inside the multimedia data. Said system further comprises an
5 indexing unit configured to index the multimedia data based on the at least one of the identified
one or object and text present inside the multimedia data and at least one processor configured to
retrieve the one or more indexed multimedia data based on natural language query received from
the one or more user devices, from a database.
[0012] In yet another non-limiting embodiment the present disclosure recites that said
10 processor in combination with the identification unit is further configured to (i). extract the one or
more objects present inside the multimedia data, (ii). classify the extracted one or more objects by
matching with at least one real-world object, (iii). generate a first confidence score based on said
matching, wherein the one or more identified object is classified as said real-world object, if the
first confidence score is above a pre-determined threshold and (iv). tag the multimedia data based
15 on said classification.
[0013] In still another non-limiting embodiment the present disclosure recites that the
processor in combination with the indexing unit is further configured to extract one or more
keywords from text of the natural language query, wherein the one or more keywords are extracted
by parsing the natural language query based on a set of pre-defined parameters. Further, said
20 disclosure recites semantically search said database for the one or more extracted keywords and
identify the one or more indexed multimedia data from the database, based on said searching.
[0014] In yet another non-limiting embodiment of the present disclosure recites that said
processor is further configured to combine the tagged data with the extracted keywords and index
said multimedia data under one or more categories within the database.
25 [0015] In still another non-limiting embodiment of the present disclosure recites that said
processor is further configured to generate a second confidence score, in response to semantical
search performed on said database for the one or more extracted keywords; and identify the one or
more indexed multimedia data from the database, if the second confidence score is above a
predetermined threshold.
30 [0016] In another non-limiting embodiment of the present disclosure recites a server
configured to manage untagged multimedia data. In an embodiment said server comprising a first
5
module to receive untagged multimedia data from the one or more user devices and a second
module to identify at least one of: one or more objects present inside the multimedia data and text
present inside the multimedia data. Said server further comprising a third module to index the
multimedia data based on the at least one of the identified one or object and text present inside the
5 multimedia data and a fourth module to retrieve the one or more indexed multimedia data based
on natural language query received from the one or more user devices, from a database.
[0017] The foregoing summary is illustrative only and is not intended to be in any way
limiting. In addition to the illustrative aspects, embodiments, and features described above, further
aspects, embodiments, and features will become apparent by reference to the drawings and the
10 following detailed description.
OBJECTS OF THE INVENTION:
[0018] The main object of the present invention is to allow a user/system generated
emoji/sticker/emoticon to be searched using natural language query.
15 [0019] Another main object of the present invention is to automatically tag the user/system
generated emoji/sticker/emoticon.
[0020] Yet another object of the present invention is to allow an untagged, preexisting
emoji/sticker/emoticon to be searched using natural language query.
[0021] Still another object of the present invention is to prevent a user from creating a new
20 emoji/sticker/emoticon for an already existing one.
[0022] Yet another object of the present invention is to avoid the system from being
overcrowded due to redundant emoji/sticker/emoticon.
[0023] Still another object of the present invention is to enhance user experience regarding
using and creating emoji/sticker/emoticon.
25
BRIEF DESCRIPTION OF DRAWINGS:
[0024] The accompanying drawings, which are incorporated in and constitute a part of this
disclosure, illustrate exemplary embodiments and, together with the description, serve to explain
the disclosed embodiments. In the figures, the left-most digit(s) of a reference number identifies
30 the figure in which the reference number first appears. The same numbers are used throughout the
figures to reference like features and components. Some embodiments of system and/or methods
6
in accordance with embodiments of the present subject matter are now described, by way of
example only, and with reference to the accompanying figures, in which:
[0025] Fig. 1 illustrates a system, by way of block diagram, configured (i). to tag a
user/system generated multimedia data and (ii) to search/reterive the user/system generated
5 multimedia data using natural language query, according to various embodiments.
[0026] Figure 2 illustrates a server, by way of block diagram, configured to (i). to tag a
user/system generated multimedia data and (ii) to search/reterive the user/system generated
multimedia data using natural language query, according to various embodiments.
[0027] Figures 3A-3H discloses the functioning of said system by way of an example,
10 according to various embodiments.
[0028] Figure 4A discloses a method, by way of flowchart, for tagging user/system
generated emoji/sticker/emoticon, according to various embodiments.
[0029] Figure 4B discloses a method, by way of flowchart, for searching user/system
generated emoji/sticker/emoticon using natural language query, according to various
15 embodiments.
[0030] Figure 5 discloses a server, by way of block diagram, illustrating various means
required to carry out the present invention, according to various embodiments.
[0031] It should be appreciated by those skilled in the art that any block diagrams herein
represent conceptual views of illustrative systems embodying the principles of the present subject
20 matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition
diagrams, pseudo code, and the like represent various processes which may be substantially
represented in computer readable medium and executed by a computer or processor, whether or
not such computer or processor is explicitly shown.
25 DETAILED DESCRIPTION OF DRAWINGS:
[0032] In the present document, the word "exemplary" is used herein to mean "serving as
an example, instance, or illustration." Any embodiment or implementation of the present subject
matter described herein as "exemplary" is not necessarily to be construed as preferred or
advantageous over other embodiments.
30 [0033] While the disclosure is susceptible to various modifications and alternative forms,
specific embodiment thereof has been shown by way of example in the drawings and will be
7
described in detail below. It should be understood, however that it is not intended to limit the
disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all
modifications, equivalents, and alternative falling within the scope of the disclosure.
[0034] The terms “comprises”, “comprising”, “include(s)”, or any other variations thereof,
5 are intended to cover a non-exclusive inclusion, such that a setup, system or method that comprises
a list of components or steps does not include only those components or steps but may include
other components or steps not expressly listed or inherent to such setup or system or method. In
other words, one or more elements in a system or apparatus proceeded by “comprises… a” does
not, without more constraints, preclude the existence of other elements or additional elements in
10 the system or apparatus.
[0035] Embodiments of the present disclosure relates to a system and a method for
allowing a user/system generated emoji/sticker/emoticon to be tagged automatically and retrieving
such emoji/sticker/emoticon using natural language query. In particular, present disclosure recites
the steps of receiving untagged multimedia data from one or more user devices, identifying at least
15 one of: one or more objects present inside the multimedia data and text present inside the
multimedia data, indexing the multimedia data based on the at least one of the identified one or
object and text present inside the multimedia data and retrieving the one or more indexed
multimedia data based on natural language query received from the one or more user devices.
[0036] In the following detailed description of the embodiments of the disclosure,
20 reference is made to the accompanying drawings that form a part hereof, and in which are shown
by way of illustration specific embodiments in which the disclosure may be practiced. These
embodiments are described in sufficient detail to enable those skilled in the art to practice the
disclosure, and it is to be understood that other embodiments may be utilized and that changes may
be made without departing from the scope of the present disclosure. The following description is,
25 therefore, not to be taken in a limiting sense.
[0037] The present disclosure provides for an improvement in the conventional art and
proposes a method and a system for not only tagging an untagged multimedia data such as
sticker/emoji/emoticon/Animoji, but also to allow the user to retrieve relevant said multimedia
data based from a natural language query.
30 [0038] In the present document some of the terms may be used repeatedly throughout the
disclosure. For clarity said terms are illustrated below:
8
[0039] Emoji in context of the present application may be defined as a set of graphical
symbols or a simple pictorial representation that represents an idea or concept, independent of any
language and specific words or phrases. In particular, emoji may be used to convey one’s thoughts
and emotions through a messaging platform without any bar of language. Further, the term emoji
5 or emoticon may mean more or less same in the context of the present application and may be used
interchangeably throughout the disclosure, without departing from the scope of the present
application.
[0040] Sticker in context of the present application may relate to an illustration which is
available or may be designed (using various application) to be placed on or added to a message. In
10 simple words sticker is an elaborate emoticon, developed to allow more depth and breadth of
expression than what is possible by means of ‘emojis’ or ‘emoticons’. Stickers are generally used,
on digital media platforms, to quickly and simply convey an emotion or thought. In some
embodiments, the stickers may be animated, derived from cartoon-like characters or real-life
peoples etc. and are often intended to be witty, cute, irreverent or creative, but in a canned kind of
15 way. In some embodiments, stickers may also be designed to represent real-world events in more
interactive and fascinating form to be shared between users on various multimedia messaging
platforms.
[0041] Avatar in context of the present application relates to graphical representation of a
user, user’s image/selfie or the user's character. Thus, it may be said that an avatar may be
20 configured to represent emotion/expression/feeling of the user by means of an image converted
into avatar capturing such emotion/expression/feelings by various facial expressions or added
objects such as heart, kisses etc. Further, it is to be appreciated that an avatar may take either a
two-dimensional form as an icon on platform such as messaging/chat platforms and or a threedimensional form such as in virtual environment. Further, the term avatar, profile picture, user pic
25 means same in context of the present application and may be used interchangeably throughout the
disclosure without departing from the scope of the present application.
[0042] Coming to figure 1, figure 1 describes a block diagram of a system 100. The system
100 comprises a plurality of user terminals 102a…102n, a network 104 and a server 106. In an
embodiment, the plurality of the user terminals/devices 102a..102n and the server 106 may remain
30 in communication with each other via the network 104, The network 104 may be any local or wide
area network.
9
[0043] The user devices 102a…102n may be a mobile phone, a tablet, an iPAD, a
computer, a laptop, etc. The user terminal 102a…102n may include a memory, one or more
processors coupled to the memory and a display. The user terminals 102a…102n may also include
one or more cameras to take pictures or to allow users to take their selfies. In an exemplary
5 embodiment, it is to be appreciated that said user devices 102a…102n may include other essential
components required to generate a multimedia data in accordance with the embodiments of the
present invention and the same are not explained for the sake of brevity. In another exemplary
embodiment, the user device 102a…102n discussed in figure 1 may use one or more known
techniques for creating said multimedia data and is not restricted to any one specific technique.
10 [0044] In another aspect, the present invention disclose that said user device 102a…102n
may include a mobile widget or a web platform installed therein that may allow the user of the
user device 102a…102n to generate multimedia data that he wishes to exchange/share with other
users. It is to be appreciated, for generating said multimedia data the user device 102a..102n may
remain connected to the server 106 by means of the mobile widget or the web platform. In an
15 exemplary embodiment, the multimedia data or the multimedia content discussed in the present
disclosure may relate to sticker/emoji/emoticon/Animoji or any other similar data generated and
used by the user to express/shared his/her fillings with other users on same web platform or even
on other similar web platforms. In an exemplary embodiment, the term multimedia data,
multimedia content, emoji, sticker, emoticon, avatar etc. may mean same in context of said
20 invention and thus may be used interchangeably.
[0045] Once, the said multimedia data is generated, the user device 102a…102n allow the
user to generally tag such data so that it can be stored for future use. However, in most of the
cases, the user of the user device 102a..102n forgets to tag such multimedia data and the said
multimedia data keeps lying untagged and is of no use to other users. Figure 2 below, explains in
25 detail the functioning of server 106 in detail to overcome such problems and achieve the desired
objective of the invention.
[0046] Figure 2 illustrates server 200 that allows a user/system generated multimedia data
such as emoji/sticker/emoticon to be tagged automatically and retrieve such
emoji/sticker/emoticon in response to receiving a natural language query from a user. In an
30 embodiment, the server 200 may include one or more essential components required to achieve
the desired objective of the present invention. Further, some of these essential
10
elements/components required to achieve the objective of the present invention are described in
below paragraphs. However, in addition to said elements/components, the server 200 may include
one or more known essential means required for tagging, storing and retrieving the
emoji/sticker/emoticon and the same are not disclosed, for the sake of brevity, in the specification.
5 Furthermore, in another embodiment server 200 may remain operatively connected to one or more
third party servers (not shown) for carrying out the desired objective of the present invention.
[0047] As shown in figure 2, the server 200 includes a transceiver 202 configured to
receive at least one of: untagged multimedia data or a user query in natural language from the one
or more user devices 102a..102n. In an aspect, it is evident from figure 1 that the server 200
10 remains connected to the plurality of user devices 102a…102n for exchange of information/data
through the transceiver 202. In one exemplary embodiment, the transceiver 202 may be configured
to share necessary content with the user devices 102a..102n, through the web platform or mobile
widget, essential for creating such multimedia content. However, such an aspect is well known in
the art and same is not explained for the sake of brevity. In a first exemplary embodiment, the
15 transceiver 202 may be configured to receive untagged multimedia data from the one or more user
devices 102a..102n, which is essential for carrying out said invention and is explained in detail
below.
[0048] Figure 2 further discloses that said server 200 includes an identification unit 204.
In an aspect, said identification unit 204 may remain operatively connected to the transceiver 202.
20 Further, in an aspect, the identification unit 204 may include at least one of an object identification
unit 204A and a text identification unit 204B. In an exemplary embodiment, the object
identification unit 204A and a text identification unit 204B may be clubbed as single unit (as shown
in figure 2). In another exemplary embodiment, the object identification unit 204A and a text
identification unit 204B may be separate units attached to each other (not shown).
25 [0049] The key function of the identification unit 204 is to identify at least one of: one or
more objects present inside the multimedia data and text present inside the multimedia data. In
order to perform said functionality, said server 200 may further include a processor 206 operatively
connected to the identification unit 204. In an aspect of the present invention, the identification
unit 204 in combination with the processor 204 is configured to extract at least the one or more
30 objects present inside the multimedia data and or text included in the multimedia data. It is to be
appreciated that the object identification unit 204A may use one of the several known ways to
11
identify an object/text present in the emoji/sticker/emoticon. Further, the object identification unit
204A may be configured to extract the identified object/text from the emoji/sticker/emoticon based
on contextual and/or visual information.
[0050] Further, in an aspect of the present invention, to make use of the extracted object
5 or text, the identification unit 204 in combination with the processing unit 204 may be configured
to classify the extracted one or more objects by matching them with at least one real-world object.
In an embodiment, to classify the one or more extracted objects with the real-world objects the
processor 206, in combination with identification unit 204, may be configured to match these
objects with the images of pre-stored objects present in memory 208 of the server 200. In another
10 embodiment, to classify the one or more extracted objects with the real-world objects the processor
206, in combination with identification unit 204, may be configured to match these objects with
the images of similar objects available at third party servers (not shown).
[0051] Upon classifying said extracted objects, the processor 206 in combination with the
identification unit 204 may be configured to generate a first confidence score based on said
15 matching. In particular, processor 206 may be configured to identify the extracted objects as the
one or more real-world objects, if the first confidence score is above a pre-determined threshold.
In one exemplary embodiment, the processor 206 may be configured to identify the extracted
objects as the one or more real-world objects, if the first confidence score is above 60%. It is to
be noted that said example is just for the sake of explanation and is not to be construed in any
20 limiting sense.
[0052] The processor 206 in combination with the identification unit 204 is further
configured to tag said multimedia data based on said classification. It is to be appreciated that if
the processor 206 identifies more than one objects in the multimedia data and the first confidence
score for one or more of these objects is found to be above the predetermined threshold then, said
25 multimedia data is tagged for each such respective objects. Thus, in this way one multimedia data
capturing multiple objects may be tagged for one or more such objects present within it. The
processor 206 may use one of several known techniques for generating a confidence score on the
basis of object matching and is not restricted to any one such technique.
[0053] Further as shown in figure 2, said server 200 may also include an indexing unit 210
30 operatively connected to the processor 206 and the identification unit 204. In an essential aspect,
once the multimedia data is tagged based on object classification (as discussed above), the indexing
12
unit 210 may be configured to index the multimedia data based on the at least one of the identified
one or object and text (identified previously) present inside the multimedia data. The indexed
multimedia data is then stored in the memory 208 for future use. In an exemplary embodiment,
the indexing of said multimedia data is essential as it makes the retrieval of stored multimedia
5 content easy in future by any user.
[0054] In second embodiment, the present disclosure recites the functionality of the server 200 in
combination with user device 102a….102n for retrieval of said stored multimedia content from a
query received from a user in natural language. In said aspect, the transceiver 202 of the server
200 may be configured to receive the query from any user of the user device 102a…102n in natural
10 language.
[0055] Further in respect of said embodiment, the processor 206 of the server 200 in
combination with the indexing unit 210 may be configured to extract one or more keywords from
text of the natural language query received from the user device 102a..102n. In an aspect, the
indexing unit 210 may be configured to extract the one or more relevant keywords from the natural
15 language query by parsing the received natural language query based on a set of pre-defined
parameters. In an example, the pre-defined parameters for extracting keywords from natural
language query may include at least one of name, object, place, event, gesture and like similar
parameters.
[0056] Upon indexing said relevant keywords, from the natural language query, the
20 processor 206 is configured to semantically search said memory/database 208 for the one or more
extracted keywords. The processor 206 is further configured to identify the one or more indexed
multimedia data from the database 208, based on said searching. In particular, the processor 206
may be configured to search the indexed database 208 for the extracted keywords and identify one
or more pre-stored multimedia data/content, which was stored in the first embodiment of the
25 invention. Thus, it would correct to say that in the second embodiment of the invention, the
processor 206 is configured to retrieve the one or more indexed multimedia data based on natural
language query received from the one or more user devices 102a…102n, from a database 208.
[0057] In an aspect, the database /memory/storage unit/repository 208 may be located
inside or outside the system 200. However, the system 200 shown in figure 1 shows the storage
30 unit 208 as a part of the system 100. Further, it is to be appreciated that the processor 206 may be
further configured to generate a second confidence score, in response to semantical search
13
performed on said database 208 for the one or more extracted keywords and identify the one or
more indexed multimedia data from the database only if the second confidence score is above a
predetermined threshold.
[0058] In another exemplary embodiment, said system 200 may further discloses having a
5 user interface unit or I/O interface unit 212. The user interface unit 212 may allow a user (not
shown) to directly enter a query to search for a relevant prestored (including user/system created)
emoji/sticker/emoticon, without the need of user devices 102a…102n. The uniqueness of the
system 200 lies in the fact that it allows the user to retrieve relevant emoji/sticker/emoticon even
if the query entered by the user is natural language query. In particular, the processing unit 206 of
10 the system 200 processes the user query to identify the intent and value of the query. In an
exemplary embodiment, the processing unit 206 may then take help of the indexing unit 210 to
locate at least one emoji/sticker/emoticon relevant to the user query and share the same with the
user, as discussed in second embodiment i.e. above paragraphs of the disclosure.
[0059] Coming to figures 3, it may be appreciated that figure 3 discloses the functioning
15 of server 200 by way of an example. In particular, Figures 3A-3D, recites the functioning of the
first embodiment of the present invention which is tagging an untagged multimedia data based on
object identification, by way of an example.
[0060] In an exemplary embodiment, figure 3A discloses an emoji/sticker/emoticon 300
created by a user (not shown) expressing his feelings for ISRO’s Chandrayan-II Mission and shares
20 the same with his friend but leaves it untagged, the other user who has no idea of such
emoji/sticker/emoticon 300 being created may never be able to use it. The system 200 disclosed
in figure 2 of the present application not only tags such an emoji/sticker/emoticon 300 but also
allows any user to retrieve any such emoji/sticker/emoticon 300, even by using natural language
query.
25 [0061] In first exemplary embodiment, figure 3A, discloses the emoji/sticker/emoticon
300 created by a user (not shown) expressing his feelings for ISRO’s Chandrayan-II Mission using
an Indian flag, an ISRO rocket and a thumps up smiley in the foreground and leaves it untagged.
In such an aspect, the server 200, receives said emoji/sticker/emoticon 300 from the user device
102a..102n, via a transceiver 202. The received emoji/sticker/emoticon 300 (as shown in figure
30 3A) is then processed by identification unit 204. The identification unit 204 is configured to detect
objects that are visually seen in the emoji/sticker/emoticon 300, as shown in figure 3B. For
14
example, in the present case the identification unit 204 may be configured to detect objects like
Moon, Indian flag, Rocket and thumps up smiley visually represented in the
emoji/sticker/emoticon 300 and extract the same from the emoji/sticker/emoticon 330, as shown
in figure 3B.
5 [0062] Further, after the identification unit 204 has extracted the one or more objects
present inside the multimedia data as shown in figure 3B, the processor 206 in combination with
identification unit 204 is configured to classify the extracted one or more objects (i.e. moon, ISRO
rocket, Indian flag and smiley thumbs up) by matching with at least one real-world objects. In an
aspect, the processor 206 may be configured to match the extracted one or more objects (i.e. moon,
10 ISRO rocket, Indian flag and smiley thumbs up) with images of real-world objects present inside
the memory 208, as shown in figure 3C.
[0063] Further, figure 3C shows that the processor 206 is further configured to generate a
first confidence score based on said matching, wherein the one or more identified object is
classified as said real-world object, if the first confidence score is above a pre-determined
15 threshold. In an exemplary aspect, figure 3C shows a first exemplary confidence score (in
percentage) generated against matching for each of the extracted object.
[0064] If the confidence score for each identified object is found to be above the predetermined threshold, the processor 206 tags the multimedia data 300 based on said classification.
In an exemplary aspect, the processor 206 may be configured to tag the multimedia data under
20 more than one classification on basis on matching of score extracted objects with the real-world
objects.
[0065] Further as shown in figure 3D, the identification unit 204 may then share the
emoji/sticker/emoticon 300 along with extracted information with the indexing unit 210. The
indexing unit 210 may be configured to index the received emoji/sticker/emoticon 300 at least on
25 the basis of one of object, component, keyword, content and like traits identified in the
emoji/sticker/emoticon/image 300. Further, said emoji/sticker/emoticon/image may then be
stored, by the storage unit 208, on the basis of said segregation. In said embodiment, the indexing
unit 210 may be configured to index said emoji/sticker/emoticon/image 300 under categories such
as wishing luck, moon, spaceship, Chandrayan-II and shall store the same in the database 208.
30 [0066] In second exemplary embodiment of the present invention which relates to
retrieving an emoji/sticker/emoticon/image 300 from the database 208 on the basis of query
15
received from a user in natural languages is disclosed in figures 3E-3H.
[0067] To understand the same, let’s consider an example, where any other user wants to
use an emoji/sticker/emoticon/image 300 in context of India’s Chandrayan-II mission, he/she may
simply type a query in the user device 102a…102n. In an aspect of the present invention, as shown
5 in figure 3E, the query may be a simple or a natural language query. For example, said user may
type a query such as “I want to wish best of luck to ISRO for their Chandrayan-II mission". When
such a query is received by the processor 206 of the server 200, as shown in figure 3F, the processor
206 processes said user query to identify the intent and value of the query.
[0068] In particular, the processor 206 in combination with the indexing unit 210 may be
10 configured to extract one or more keywords (represented by K1, K2 and K3) from text of the
natural language query, as shown in figure 3F. It is to be noted that the one or more keywords are
extracted by parsing the natural language query based on a set of pre-defined parameters, as
discussed above. In said example, the processor 206 may be configured to extract keywords like
“I want to wish best of luck to ISRO for their Chandrayan-II mission". Further, as shown in figure 3G,
15 the processor 206 in combination with the indexing unit 210 may be configured semantically
search said database 208 for said one or more extracted keywords.
[0069] After searching the database 208, the processor 206 in combination with the
indexing unit 210 may be configured to identify the one or more indexed multimedia data from
the database 208, based on said searching, as shown in figure 3H. In particular, the processor 206
20 of the server 200 may be configured to identify the intent and value of the query and retrieves one
or more relevant emoji/sticker/emoticon stored in the storage unit 208 and share the same with the
user, as shown in figure 3H. For example, in the present case the server 200 may be configured to
provide emoji/sticker/emoticon, that was created by another user expressing his feelings for
ISRO’s Chandrayan-II Mission using an Indian flag, an ISRO rocket and a thumps up smiley, to
25 said user as relevant result.
[0070] Moving on to figure 4 that disclose the method steps of the first embodiment i.e.
tagging an untagged multimedia data by means of Figure 4A and method steps of the second
embodiment i.e. how to retrieve such multimedia data using a natural language query by means in
figure 4B.
30 [0071] Figure 4A discloses a method 400 for tagging a user/system generated
emoji/sticker/emoticon. Said method discloses at step 402 receiving, from a user device
16
102a..102n untagged user/system created emoji/sticker/emoticon (i.e. untagged multimedia data),
at the server 200. The method 400 further discloses, at step 404 identifying at least one of: one or
more objects present inside the multimedia data and text present inside the multimedia data. In
particular, server 200 may be configured to use the identification unit 204 for identifying the object
5 that may be visually seen in the created emoji/sticker/emoticon.
[0072] In particular, though not exclusively disclose in step 404, the method 400 further
discloses that the step 404 of identifying further comprises extracting the one or more objects
present inside the multimedia data and classifying the extracted one or more objects by matching
with at least one real-world object. The step 404 of method 400 further includes the steps of
10 generating a first confidence score based on said matching, wherein the one or more identified
object is classified as said real-world object, if the first confidence score is above a pre-determined
threshold and tagging the multimedia data based on said classification.
[0073] The method 400 then moves to step 406 which discloses indexing the multimedia
data based on the at least one of the identified one or object and text present inside the multimedia
15 data. In particular, at step 406, said method 400 discloses the step of indexing the created
emoji/sticker/emoticon on the basis of at least one of object, keyword, content, component a like
traits identified from the user/system created emoji/sticker/emoticon. Said method further
discloses, at step 408, discloses storing the said tagged emoji/sticker/emoticon in the storage unit
208 for future use.
20 [0074] Figure 4B discloses a method 400 for allowing a user to enter a search query in
order to retrieve a suitable emoji/sticker/emoticon from the storage unit 208. Most importantly,
said method allows the processor 206 of the present application to parse said query into vital
keyword and retrieve relevant results with the help of indexer unit 210. In particular, at step 410,
said method discloses receiving by the user device 102a…102n, a natural language query. At step
25 412, said method discloses identifying intent and value of the query.
[0075] Further, at step 412, method 400, discloses retrieving the one or more indexed
multimedia data based on natural language query received from the one or more user devices
102a..102n. In an exemplary embodiment, the method 400 at step 412 i.e. retrieving the one or
more indexed multimedia data based on the natural language query further comprises the steps of
30 extracting one or more keywords from text of the natural language query, wherein the one or more
keywords are extracted by parsing the natural language query based on a set of pre-defined
17
parameters. As a subsequent step it discloses the step of semantically searching said database for
the one or more extracted keywords and identifying the one or more indexed multimedia data from
the database, based on said searching.
[0076] Finally, at step 416, the method 400 discloses outputting relevant results to the user
5 for their use. It is to be appreciated that the method 400 of figure 4B further discloses generating
a second confidence score, in response to semantically searching said database 208 for the one or
more extracted keywords; and identifying the one or more indexed multimedia data from the
database only if the second confidence score is above a predetermined threshold.
[0077] Further, Figure 5 illustrates a block diagram 500 of a server 502 for managing
10 untagged multimedia data according to an embodiment of the present disclosure. In an exemplary
embodiment, the server 502 may comprise modules 504. In some embodiments, the data stored in
the memory 208 (not shown) may be processed by the modules 504 of the server 502. The modules
504 may be stored within the memory 208 in form of software instructions. In another example,
the modules 504 communicatively coupled to the processing unit 206, and may also be present
15 outside the memory 208, and implemented as hardware.
[0078] In some embodiments, the modules 504 may include, for example, a first module
504a, a second module 504b, a third module 504cand a fourth module 504d. The modules 504a,
504b, 504c and 504d may be configured to perform various miscellaneous functionalities of the
server 502. It will be appreciated that such aforementioned modules 504a, 504b, 504c and 504d
20 may be represented as a single module or a combination of different modules.
[0079] The first module 504a may configured to receive untagged multimedia data from
the one or more user devices 102a..102n. Further, the second module 504b may configured to
identify at least one of: one or more objects present inside the multimedia data and text present
inside the multimedia data. Specifically, the second module 504b in combination with one or more
25 other modules 502a, 502c and 502d may be configured to (i). extract the one or more objects
present inside the multimedia data, (ii). classify the extracted one or more objects by matching
with at least one real-world object, (iii). generate a first confidence score based on said matching,
wherein the one or more identified object is classified as said real-world object, if the first
confidence score is above a pre-determined threshold and (iv). tag the multimedia data based on
30 said classification.
[0080] Further the third module 504c may be configured to index the multimedia data
18
based on the at least one of the identified one or object and text present inside the multimedia data.
At last, the fourth module 504d may be configured retrieve the one or more indexed multimedia
data based on natural language query received from the one or more user devices, from a database.
To achieve so, the fourth module 504d in combination with one or more other modules 502a, 502b
5 and 502c may be configured to extract one or more keywords from text of the natural language
query, wherein the one or more keywords are extracted by parsing the natural language query
based on a set of pre-defined parameters and then semantically search said database 208 for the
one or more extracted keywords. Once the searching is completed, the fourth module 504d is
configured to identify the one or more indexed multimedia data from the database 208, based on
10 said searching.
[0081] Further, it may be appreciated that the fourth module 504d is configured to generate
a second confidence score, in response to semantical search performed on said database 208 for
the one or more extracted keywords and identify the one or more indexed multimedia data from
the database, if the second confidence score is above a predetermined threshold.
15 [0082] The illustrated steps are set out to explain the exemplary embodiments shown, and
it should be anticipated that ongoing technological development will change the manner in which
particular functions are performed. These examples are presented herein for purposes of
illustration, and not limitation. Further, the boundaries of the functional building blocks have been
arbitrarily defined herein for the convenience of the description. Alternative boundaries can be
20 defined so long as the specified functions and relationships thereof are appropriately performed.
[0083] Those of skill would further appreciate that the various illustrative logical blocks,
units, means, modules, circuits, and algorithm steps described in connection with the aspects
disclosed herein may be implemented as electronic hardware, computer software, or combinations
of both. To clearly illustrate this interchangeability of hardware and software, various illustrative
25 components, blocks, modules, circuits, and steps have been described above generally in terms of
their functionality. Whether such functionality is implemented as hardware or software depends
upon the particular application and design constraints imposed on the overall system. Skilled
artisans may implement the described functionality in varying ways for each particular application,
but such implementation decisions should not be interpreted as causing a departure from the scope
30 of the present disclosure.
[0084] As used in this disclosure, the terms “means”, “module”, “unit”, and the like are
19
intended to refer to a computer-related entity, either hardware, a combination of hardware and
software, software, or software in execution. For example, “means” may be, but is not limited to
being, a process running on a processor, a processor, an object, an executable, a thread of
execution, a program, and/or a computer. By way of illustration, both an application running on a
5 server and the server can be a system. One or more components may reside within a process and/or
thread of execution and a component may be localized on one computer and/or distributed between
two or more computers.
[0085] Various aspects will be presented in terms of systems that may include a number
of units/components, means, modules, and the like. It is to be understood and appreciated that the
10 various systems may include additional components/units, modules, etc. and/or may not include
all of the components, modules, means etc. discussed in connection with the figures. A
combination of these approaches may also be used.
[0086] In addition, the various illustrative logical blocks, units, modules, and means
described in connection with the aspects disclosed herein may be implemented or performed with
15 a general purpose processor, a digital signal processor (DSP), an application specific integrated
circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device,
discrete gate or transistor logic, discrete hardware components, or any combination thereof
designed to perform the functions described herein. A general purpose processor may be a
microprocessor, but in the alternative, the processor may be any conventional processor, controller,
20 microcontroller, or state machine. A processor may also be implemented as a combination of
computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of
microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such
configuration.
[0087] Although the present invention has been described in considerable detail with
25 reference to figures and certain preferred embodiments thereof, other versions are possible.
Therefore, the spirit and scope of the present invention should not be limited to the description of
the preferred versions contained herein.

We Claim:
1. A method for managing untagged multimedia data, said method comprising:
receiving untagged multimedia data from one or more user devices;
identifying at least one of: one or more objects present inside the multimedia data and text
5 present inside the multimedia data;
indexing the multimedia data based on the at least one of the identified one or object and
text present inside the multimedia data; and
retrieving the one or more indexed multimedia data based on natural language query
received from the one or more user devices.
10
2. The method as claimed in claim 1, wherein the step of identifying further comprises:
a. extracting the one or more objects present inside the multimedia data;
b. classifying the extracted one or more objects by matching with at least one realworld object;
15 c. generating a first confidence score based on said matching, wherein the one or more
identified object is classified as said real-world object, if the first confidence score
is above a pre-determined threshold; and
d. tagging the multimedia data based on said classification.
20 3. The method as claimed in claim 1, further comprises combining the tagged data with the
extracted keywords and indexing said multimedia data under one or more categories within
a database.
4. The method as claimed in claim 1, wherein retrieving the one or more indexed multimedia
25 data based on the natural language query comprises:
a. extracting one or more keywords from text of the natural language query, wherein
the one or more keywords are extracted by parsing the natural language query based
on a set of pre-defined parameters; and
b. semantically searching said database for the one or more extracted keywords; and
30 c. identifying the one or more indexed multimedia data from the database, based on
said searching.
22
5. The method as claimed in claim 1, further comprise generating a second confidence score,
in response to semantically searching said database for the one or more extracted keywords;
and identifying the one or more indexed multimedia data from the database, if the second
5 confidence score is above a predetermined threshold.
6. A system to manage untagged multimedia data, said system comprising:
one or more user devices;
a network interface; and
10 at least one server operatively coupled to the one or more user devices through the
network interface, wherein said server further comprising:
a transceiver configured to receive untagged multimedia data from the one or more
user devices;
an identification unit configured to identify at least one of: one or more objects
15 present inside the multimedia data and text present inside the multimedia data;
an indexing unit configured to index the multimedia data based on the at least one
of the identified one or object and text present inside the multimedia data; and
at least one processor configured to retrieve the one or more indexed multimedia
data based on natural language query received from the one or more user devices, from a
20 database.
7. The system as claimed in claim 6, wherein the processor in combination with the
identification unit is further configured to:
a. extract the one or more objects present inside the multimedia data;
25 b. classify the extracted one or more objects by matching with at least one real-world
object;
c. generate a first confidence score based on said matching, wherein the one or more
identified object is classified as said real-world object, if the first confidence score
is above a pre-determined threshold; and
30 d. tag the multimedia data based on said classification.
23
8. The system as claimed in claim 6, wherein the processor in combination with the indexing
unit is further configured to:
a. extract one or more keywords from text of the natural language query, wherein the
one or more keywords are extracted by parsing the natural language query based
5 on a set of pre-defined parameters; and
b. semantically search said database for the one or more extracted keywords; and
c. identify the one or more indexed multimedia data from the database, based on said
searching.
10 9. The system as claimed in claim 1, wherein the processor is further configured to combine
the tagged data with the extracted keywords and index said multimedia data under one or
more categories within the database.
10. The system as claimed in claim 1, wherein the processor is further configured to generate
15 a second confidence score, in response to semantical search performed on said database for
the one or more extracted keywords; and identify the one or more indexed multimedia data
from the database, if the second confidence score is above a predetermined threshold.