FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
1. Title of the invention: IMAGE SEARCHING AND MINING
2. Applicant(s)
NAME
NATIONALITY
ADDRESS
TATA CONSULTANCY
SERVICES LIMITED
Indian
Nirmal Building, 9th Floor, Nariman Point,
Mumbai-400021, Maharashtra, India
3. Preamble to the description
COMPLETE SPECIFICATION
The following specification particularly describes the invention and the manner in which it is to be performed.
1
PD003597IN-SC
TECHNICAL FIELD
[0001]
The present subject matter relates, in general, to searching and, in particular, to systems and methods for image searching.
BACKGROUND
[0002]
Conventionally, searching for any subject in the internet is usually performed by determining appropriate keywords or key phrases and providing the keywords or the key phrases, referred to as a search string, to a search engine. The search engine retrieves variety of documents, henceforth referred to as search results, such as web pages, portable document format (pdf) files, spreadsheets, images, multimedia files, text documents, rich text format documents, relevant to the search string based on metadata associated with the search results. The metadata of the documents may include title of the document, words or phrases used to describe the documents and words or phrases used very frequently in the body of the document, etc. Further various conventionally known algorithms are used to rank or order the search results in order or relevance or based on user preferences and past usage history of the user making it easy for the user to locate the information the user is looking for.
[0003]
However, in certain cases, the user may come across an image which the user may not be able to aptly describe as a search string. Further, in many cases on seeing the image of a product, the user may develop an interest in the product, but may not be able to recognize the brand or the model number of the product. In such cases a wild or unstructured search for the category of product in a search engine would result in a large number of search results. Further, filtering out the relevant information from such a large number of search results may involve a lot of time and efforts. Furthermore, this may be inconvenient for the user as well, who after a while may also lose interest in the product.
SUMMARY
[0004]
This summary is provided to introduce concepts related to systems and methods for image searching, and the concepts are further described below in the detailed description. This summary is neither intended to identify essential features of the claimed subject matter nor is it intended for use in determining or limiting the scope of the claimed subject matter.
2
[0005]
In one implementation, the method of image searching comprises receiving an image from a client device and processing the received image for generation of one or more layers wherein each of the one or more layers includes one or more objects. The method further comprises detecting at least one logo on the generated one or more layers based on logo detection parameters and determining the extent of match between the one or more object and a reference product by comparing the one or more object with reference product associated with a brand identified by the detected logo based on object attributes.
BRIEF DESCRIPTION OF THE DRAWINGS
[0006]
The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the drawings to reference like features and components.
[0007]
Fig. 1a illustrates a network environment implementing an image search system, in accordance with an embodiment of the present subject matter.
[0008]
Fig. 1b illustrates an implementation of an image data repository, in accordance with an embodiment of the present subject matter.
[0009]
Fig. 2 illustrates a method of image searching, in accordance with an implementation of the present subject matter.
DETAILED DESCRIPTION
[00010]
Systems and methods for image searching are described herein. The systems and methods can be implemented in a variety of computing systems. Examples of such computing systems include, but are not restricted to, mainframe computers, workstations, personal computers, desktop computers, minicomputers, servers, multiprocessor systems, laptops, and the like.
[00011]
Conventional search engines require a user to enter a key word or a key phrase pertaining to the subject the user wants to retrieve information about. However, in certain scenarios, the user may not be able to determine the appropriate key words or key phrases which are relevant to the subject. For example, say a first user likes a certain chronograph
3
worn by a second user. However, in case the first user is unable to identify the brand name or the model number of the chronograph, the first user may have to do a general search on chronographs. A general search for chronograph on online resources such as search engines or e-commerce portals may result in a large number of results. Filtering of these results to extract the details pertaining to the required chronograph may be a time and resource consuming task. Even, if the first user is able to identify the brand of the chronograph, choosing the chronograph which sparked his interest from a large number, usually running into hundreds, of models manufactured by the brand is often a tedious and cumbersome task. In the above example, an image search engine which is configured to receive an image of the certain chronograph and search for similar or same images and provide the user with information associated with the similar or same images would help the user in locating the information he is looking for. For example, a user may like a particular model of Titan™ chronograph. However a general search on chronographs would return a large number of search results from which filtering out the information or documents pertaining to the said Titan™ chronograph would be a difficult task. Even a search on Titan™ chronographs would return a high number of results, running into hundreds, from which determining the relevant documents pertaining to the particular model of Titan™ chronograph would be a cumbersome and time consuming task
[00012]
In another scenario, a rare disease may be detected by a unique pattern in magnetic resonance imaging (MRI) scan or a computer tomography (CT) scan or electrocardiogram (ECG) scan. In such a scenario, attempts to identify the disease by entering keywords such as symptoms present in a patient to identify the disease may have limited success. However, comparing the MRI scan or a CT scan or ECG scan of the patient to a reference MRI scan or a CT scan or ECG scan indicating the rare disease would help in diagnosis of the disease of the patient. If the scan results of the patient are same or similar to the reference scans of a particular disease, the disease of the patient may get detected at an early stage and the diagnosis process of the patient could be started and carried out accordingly. Yet in another scenario, a surveillance camera may be used to monitor the security of a closed area such as a showroom. In such a case the current image captured by the
4
surveillance camera may be compared with a reference image to detect any unaccounted for activity, such as intrusion and raise an alarm accordingly.
[00013]
Generally, techniques of image search or image comparison include pixel by pixel comparison, which often results in a high processing load on a computing device performing the image comparison. This reduces the throughput of the computing system and accordingly the computing system may take longer than expected to provide the search results thereby diminishing user experience. Further, pixel by pixel matching technique sometimes result in the omission of relevant results due to differences in light intensity, difference in angle at which the photograph was taken, differences of color balance, hue and saturation, etc. Thus, the conventional techniques of image searching are often processor intensive and produce inaccurate results.
[00014]
The present subject matter describes systems and methods for image searching. In one implementation, the systems and methods for image searching are being used for identification of a product. It should be appreciated by those skilled in the art that though the systems and methods for image searching are described in the context of identification of a product, the same should not be construed as a limitation. For example, the systems and methods for image searching and mining may be implemented for various other purposes, such as in surveillance and in medical diagnosis albeit modifications as would be known by those skilled in the art.
[00015]
In one implementation, the method of image search includes receiving an image for processing so as to identify a product of a users interest, henceforth referred to as the product of interest in the image. The image may be received from various sources, such as picture taken by a camera, a mobile communication device, and uploaded to the image search engine. In an example, the image is processed to determine the quality of the image based on one or more quality attributes, such as color balance, hue, saturation, light intensity, contrast, and brightness. Various operations involving variation of image parameters, such as removal of red eyes, normalization of light intensity, corrections of color balance, hue, and saturation may be performed so as to enhance the quality of the received image before determining the quality of the received image. In one implementation, if the image is very dark or the 5
[00016]
If the received image is determined to be above or of the threshold quality the received image is further processed for determining various objects in the image. In one implementation, layering is performed on the received image. The layering process breaks the image into multiple layers such that each layer contains one object. Each of the layers is then further processed for determining unique identification mark, such as logos, trademark, signature, and emblems, on any of the object. For the sake of clarity, the term logo shall be understood to include trademarks, signature, emblems, symbols or any other unique identifiers used by brands to tag products associated with the brands. In one example, the logo is identified based on various logo detection parameters, such as contrasting texture of the logo with respect to the surrounding portion of the image, content depicted in different font or style or color, colorful or distinctive design of the logo, and specific or characteristic shape of the logo.
[00017]
If the logo is identified, all reference products tagged with the brand associated with the identified logo are taken as references and compared with the object of interest, which is analogous to the product of interest, present in the layers based on various object attributes, such as colour, texture, shape, and size. If the extent of match, i.e., the similarity, between the object of interest and any reference product tagged with the brand exceeds a pre-defined threshold, say 75%, the said product is deemed to be same or similar as the object on interest and various details pertaining to the matched reference product is presented to the user.
[00018]
In one implementation, the match between the object of interest and the reference product is done by comparing object attributes, such as texture or pattern of the
6
object of interest and the reference product. Further, the comparison between the object of interest and the reference product may also be done by varying colors. For example, in one implementation, the most prominent colors of the object of interest and the reference product may be compared. In another example, in case of a multi colored object of interest, a certain number, say five, of the most prominent colors of the object and the reference product may be compared. In one implementation, the prominence of the colors are determined based on various factors such as area occupied, color level light intensity of the colour, brightness of a color, etc.
[00019]
Additionally or alternately, object attributes, such as the shape and size of the object of interest are compared with the object attributes of a reference product to determine the extent of match. In one example, the object attributes, such as the shape and the size, may be assigned a priority based an implementation of the described image search technique. In one implementation the object attribute shape is given priority over the object attribute size as size is dependent on the distance from which the image of the object was captured. In another implementation, the size of the object of interest is estimated by relative sizes of the surrounding objects. As mentioned earlier, if the extent of similarity between the object of interest and the reference product exceeds a pre-defined threshold, the reference product is determined to match the object of interest.
[00020]
In case of a logo not being identified on the object of interest or the extent of similarity between the object of interest and the reference product being below the pre-defined threshold, the objects in various layers are converted into single line diagrams so as to ascertain the basic shape of the object. In one implementation, the single line diagram is generated based on sharp changes in light intensity. The single line diagram so generated is further processed to extract discontinuous sub-objects which may be part of an object. In one implementation, the sub-objects may be determined by varying various sub-object generation parameters such as the light intensity. The various sub-objects so generated are cropped and extracted. Typically, in cropping one larger object is cut and split into a plurality of smaller objects. In many situations, one or more large objects form a single layer. These large objects may not fit into any conventional shape such as sphere, rectangle, and triangle. The larger
7
objects may also include several smaller objects which appear to be attached to the larger objects. For example, in an image having the front view of an aircraft, the four wings and the tail would appear to be stuck on to the basic oval shape body of the aircraft. Hence, in said example, the wings and the tail would be cropped and designated as sub-objects.
[00021]
The extracted and cropped sub-objects are then compared with various categories of the reference products for identification based on one or more sub-object parameters such as shape, relative size, and relative position with regard to each other. Once a potential category of the reference products has been identified, the extracted and cropped sub objects are compared with various reference products classified under the potential category of the reference products based on other object attributes such as color, texture, and pattern. In case of the extent of similarity between the shapes, patterns of the sub-objects and a reference product exceeds a pre-defined threshold, say 80%, the reference product is considered to be a match with the object of interest and the same is provided to the user.
[00022]
In another embodiment of the present subject matter, the efficiency of the image search engine is further optimized by implementing a database of reference products in form of a hierarchical or a tree structure. In said implementation, one primary table would contain all base images which are indicative of the basic shape of each category of a reference product such as a ball, bat, car, mobile phone, and laptop. Each of the base images would have a secondary table storing all the variations of the base image. For example, one base image might be a ball and the secondary table would contain various types of ball, such as a tennis ball, football, volleyball, basket ball, cricket ball, golf ball. Another base image might be a racquet and the secondary table would contain various types of racquets such as a badminton racquet, a tennis racquet. Splitting the database into multiple tables so as to have a hierarchical or a tree like structure reduces the number of comparisons that need to be performed to match an object with a reference image. For example, consider a single database having eight items, like, tennis ball, football, volleyball, basket ball, cricket ball, golf ball, badminton racquet, a tennis racquet. In such a database, in order identify an image of a tennis racquet eight comparisons may be required as against four comparisons as per the present subject matter, i.e., two comparisons to identify the basic shape of the racquet and then two
8
more comparisons to identify the tennis racquet. Further, it will be understood that in case of a large number of images, such a database may help reduce computational time and resources.
[00023]
Thus, the systems and methods for image searching provide for reduction in computational time and resources and enhance user experience. Further, the systems and methods for image searching increase the accuracy of search results and enhance the chances of matching a product of interest with a reference product. These and other advantages of the present subject matter would be described in greater detail in conjunction with the following figures. While aspects of described systems and methods for the selection based purchasing can be implemented in any number of different computing systems, environments, and/or configurations, the embodiments are described in the context of the following exemplary system(s).
[00024]
Fig. 1a illustrates a network environment 100 implementing an image search system 102, according to an embodiment of the present subject matter. In said embodiment, the network environment 100 includes the image search system 102 configured to receive images from various sources, for example a digital camera, a mobile communication device with a camera, a website, etc. In one implementation, the image search system 102 may be included within an existing web portal or a search engine. The image search system 102 may be implemented in a variety of computing systems such as a laptop computer, a desktop computer, a notebook, a workstation, a mainframe computer, a server, a network server and the like. It will be understood that the image search system 102 may be accessed by users through one or more client devices 104 or applications residing on client devices 104, such as search applications, image comparison applications. Examples of the client devices 104 include, but are not limited to, a portable computer 104-1, a mobile phone 104-2, a handheld device 104-3, a workstation 104-N, etc. Further, the client devices 104 may also be used as sources of the image. As shown in the figure, such client devices 104 are communicatively coupled to the image search system 102 through a network 106 for facilitating one or more users to connect to the image search system 102.
[00025]
The network 106 may be a wireless network, wired network or a combination thereof. The network 106 can be implemented as one of the different types of networks, such
9
as intranet, local area network (LAN), wide area network (WAN), the internet, and such. The network 106 may either be a dedicated network or a shared network, which represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), etc., to communicate with each other. Further, the network 106 may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, etc.
[00026]
In one implementation, the image search system 102 includes a processor(s) 108, input-output (I/O) interface(s) 110, and a memory 112. The processor(s) 108 are electronically coupled to the memory 112. The processor(s) 108 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) 108 are configured to fetch and execute computer-readable instructions stored in the memory 112.
[00027]
The I/O interface(s) 110 may include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, etc., allowing the image search system 102 to interact with the client devices 104. Further, the I/O interface(s) 110 may enable the image search system 102 to communicate with other computing devices, such as web servers and external data servers (not shown in figure). The I/O interface(s) 110 can facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example LAN, cable, etc., and wireless networks such as WLAN, cellular, or satellite. The I/O interface(s) 110 may include one or more ports for connecting the image search system 102 to a number of devices to or to another server.
[00028]
The memory 112 can include any computer-readable medium known in the art including, for example, volatile memory (e.g., RAM), and/or non-volatile memory (e.g., EPROM, flash memory, etc.). In one embodiment, the memory 112 includes module(s) 114 and program data 116. The module(s) 114 further include an image analysis module 118, a search module 120, an optimization module 122, and other module(s) 124. It will be appreciated that such modules may be represented as a single module or a combination of
10
different modules. Additionally, the memory 112 further includes data 116 that serves, amongst other things, as a repository for storing data fetched processed, received and generated by one or more of the module(s) 114. The data 116 includes, for example, rules data 126, analysis data 128, and other data 130. Additionally, the aforementioned data can be organized using data models, such as relational or hierarchical data models.
[00029]
In one implementation, the network environment 100 also includes an image data repository 132, which stores various reference images and product details associated with the reference products. The product details include, for example, technical specifications of the reference products, features of the reference products, price at which various vendors are offering to sell the reference product, and address of the vendors. Though the image search system 102 has been shown to external to the image data repository 132, it should be appreciated by those skilled in the art, that in other implementations, the image data repository 132 may be an integral part of the image search system 102 or the image data repository 132 may be connected to the image search system 102 through a network such as the network 106.
[00030]
In operation, the image search system 102 receives an image from the client device 104 for processing. For example, a user may use an application running on a mobile communication device or a web portal, or an application running on a digital camera to upload an image. The image analysis module 118 determines the quality of the received image. In one implementation, the image analysis module 118 implements various conventionally known image editing and image enhancement techniques involving variation of one or more image parameters, such as removal of red eyes, normalization of light intensity, corrections of color balance, and hue and saturation, so as to correct imperfections in the image thereby enhancing the quality of the image. If the image analysis module 118 determined the received image is determined to be below a certain threshold quality, the image analysis module 118 generates an alert notifying the user of the sub-standard quality of the image by various means such as an error code, a message, an audio-visual notification, etc. For example, in one embodiment, the image analysis module 118 may generate a message 11
stating “Error Code 101- Bad Image Quality” notifying the user of the poor quality of the image.
[00031]
If the image analysis module 118 determines the received image to be above or of the threshold quality, the image analysis module 118 further processes the received image for determining various objects in the image. It will be understood by those skilled in the art that the received image may include multiple objects. In one implementation, the image analysis module 118 uses a conventional image editing process known as layering to split the received image into one or more layers such that each layer contains at least ?one object.
[00032]
For example, the user may be interested in obtaining various product details associated with a digital wall clock placed on another user’s workstation. The received image, as uploaded or transmitted by the user, would include, in addition of the digital clock, various items of the other user such as the other user’s stationery, the other user’s mobile phone, the other user’s laptop, the other user’s desk organizer, etc. The image analysis module 118 on determining the received image to be above a threshold quality breaks the received image into multiple layers. For example, a first layer would contain the other user’s stationery, a second layer would contain the other user’s mobile phone, a third layer would contain the other user’s digital clock, and a fourth layer would contain the other user’s desk organizer and so on.
[00033]
The image analysis module 118 then analyzes each of the layers for determining logos on the object in which the user has expressed interest, henceforth referred to as the object of interest. In one implementation, the image analysis module 118 identifies the logo based on various logo detection parameters such as contrasting texture of the logo with respect to the surrounding portion of the layer, content depicted in a different font or style or color, colorful or distinctive design of the logo, specific or characteristic shape of the logo etc.
[00034]
If the logo is identified, the search module 120 retrieves one or more reference products tagged with the brand associated with the identified logo from the image data repository 132 and tags the retrieved reference products to be potentially the object of interest. In one embodiment, the image analysis module 118 determines one or more object attributes
12
of the object on interest such as colour, texture, shape, size. Based on one or more object selection parameters, the search module 120 tries to match one or more tagged reference products with the object of interest and computes the extent of match between a tagged reference product and the object of interest in various formats, such as fraction and percentage.
[00035]
In one example, the search module 120 is configured to determine the extent of match between the object of interest and any reference product by comparing the object attributes such as the texture or pattern of the object of interest and the reference product. The search module 120 is further configured to compare the object of interest and the reference product by varying colors. In one implementation, the search module 120 is configured to compare the object of interest and the reference product based on the most prominent colors of the object of interest and the reference product. In another example, in case of a multi colored object of interest the search module 120 is configured to compare the object of interest and the reference product based on a certain number, say five, of the most prominent colors of the object and the reference product. In said implementation, the object of interest and the reference product may determine the prominence of the colors based on various factors such as area occupied, and by color level light intensity of each color.
[00036]
If the extent of match, i.e. the similarity, between the object of interest and any reference product tagged with the brand exceeds a pre-defined threshold, say 75%, the search module 120 is configured to tag the reference product to be same or similar as the object of interest. In one embodiment, the search module 120 is configured to retrieve various product details pertaining to the said reference product from the image data repository 132 and present the retrieved product details to the user.
[00037]
Moreover, in another embodiment, the search module 120 is further configured to determine the extent of match by comparing the object of interest and the reference products based on the shape and size of the object of interest and the reference product. In one implementation the search module 120 is configured to assign higher weightage to the object selection parameter shape as compared to the object selection parameter size as size is dependent on the distance from which the image of the object was captured. In another
13
implementation, the image analysis module 118 is configured to estimate the size of the object of interest by comparing with relative sizes of the surrounding objects. As mentioned earlier, the search module 120 is configured to compute the extent of similarity between the object of interest and the reference product is based on various object selection parameters of the object of interest. If the extent of similarity between the object of interest and the reference product exceeds a pre-defined threshold, the object of interest and the reference product are determined to be a match.
[00038]
In case of a logo not being identified on the object of interest or the extent of similarity between the object of interest and the reference product being below the pre-defined threshold, the image analysis module 118 is configured to convert the object in various layers into single line diagrams so as to determine the basic shape of the object. In one implementation, image analysis module 118 is configured to generate the single line diagram is based on sharp changes in light intensity. The image analysis module 118 further processed the generated single line diagram to extract discontinuous sub-objects which may be part of a single object. In one implementation, the sub-objects may be determined by the image analysis module 118 by varying various parameters such as the light intensity.
[00039]
The search module 120 then extracts and crops the sub-objects and compares the sub-objects with various categories of reference products for identification based on one or more sub-object parameters, such as shape, relative size, relative position, with regard to each other, etc. For example, if the object of interest is a car which could not be identified, the object of interest would be processed for extraction of discontinuous sub objects based on light intensity factors. The various sub objects would be, for example wheels, windshield, etc. The search module 120 would determine the relative positioning of the sub objects, such as the wheels, windshield, which would then be used to identify the object of interest, i.e., a car. On identification of a potential category of reference products by the search module 120, the extracted and cropped sub objects are compared with various reference products classified under the potential category of reference products based on other object attributes, such as color, texture. In case of the extent of similarity between the shape pattern of the sub-objects
14
and a reference product exceeding a pre-defined threshold, say 80%, the reference product is considered to be a match with the object of interest and the same is presented to the user.
[00040]
In another embodiment of the present subject matter, the optimization module 122 is configured to optimize the operation of the image search engine 102 and decrease the response time or throughput of the image search system 102. In one implementation, the efficiency of the image search system 102 is further optimized by implementing the image data repository 132 in form of a hierarchical or a tree structure, which is described in detail later in the specification with respect to Fig 1b. In said tree structure, one primary table would contain all base images and each of the base images would be linked to a secondary table storing all the variations of the base image. In one implementation, the optimization module 122 orders the primary tables in descending order of frequency of successful hit. A successful hit of a primary table is considered to occur when the search module 120 determines there is a match between the object of interest and a reference product stored in that primary table and the same is stored as analysis data 128. For example, if retrieval of the analysis data 128 shows the objects of interest was found in primary table A 3 times, in primary table B 7 times and in primary table C 5 times, then based on this analysis data 128, the optimization module 122 then orders the comparison of the subsequent object of interest in the order of primary table B, primary table C and then primary table A.
[00041]
In another implementation, the optimization module 122, in order to improve search results by reducing the probability of missing out relevant reference products, may tag two primary tables to be closely associated with each other based on various parameters, such as shape ……... For example, circular reference products and oval reference products may be marked to be closely associated with each other thus reducing the probability of missing out relevant reference products.
[00042]
Thus the image search system 102 facilitates fast, efficient searching of images stored in the image data repository 132, resulting in better performance and improved throughput of the image search engine 102.
15
[00043]
Fig. 1b shows the image data repository 132, in accordance with an embodiment of the present subject matter. In the said embodiment, the image data repository 132 is implemented in a hierarchical or a tree structure. In said implementation, the image data repository 132 includes one or more primary tables 134-1, 134-2, …. , 134-N collectively referred to as primary tables 134 and individually referred to as primary table 134. Each of the primary tables 134 correspond to a basic shape or basic category of products. For example, the primary table 134-1 may correspond to the basic shape bat, the primary table 134-2 may correspond to the basic shape ball and so on.
[00044]
Further, each of the primary tables 134 is linked to one or more secondary tables. In one implementation, the primary table 134-1 is linked to secondary tables 136-1, 136-2, ….. 136-N, the primary table 134-2 is linked to secondary tables 138-1, 138-2, ….. 138-N, the primary table 134-N is linked to secondary tables 140-1, 140-2, ….. 140-N. Each of the secondary tables 136, 138, 140 contains specific sub-categories of products represented by the base images. For example, if the primary table 134-1 corresponds to the basic shape of a bat, the secondary tables 136 may have various sub categories of bat, such as cricket bat, baseball bat and so on. Though the image data repository 132 is shown to have two levels of data abstraction, it should be appreciated by those skilled in the art, that in other embodiments, the image data repository may have any number of levels of data abstraction. For example, if the secondary table 136-1 corresponds to cricket bat, a number of tertiary tables (not shown in the figure) may be linked to the secondary table 136-1, such that each tertiary table corresponds to a particular type of cricket bat, such as an English willow bat, a Kashmir willow bat, and a lightweight carbon handle bat. In said implementation, each of the tertiary table may be linked to one or more quaternary table which would correspond to models of the particular type of cricket bat manufactured by one or more cricket bat manufacturers.
[00045]
Implementing the image data repository 132 in a hierarchical or tree structure reduces the number of comparisons that need to be performed to match an object of interest with a reference product. For example, conventional method of implementing a single database storing eight items, i.e., tennis ball, football, volleyball, basket ball, cricket ball, golf 16
ball, badminton racquet, a tennis racquet, would require eight comparisons for identifying an image of the object of interest, say tennis racquet. In the proposed implementation, the primary table 134-1 would store the primary shape of a racquet and the primary table 134-2 would store the primary shape of ball. Further, the secondary tables 136 would store various types of racquet, i.e. a tennis racquet, a badminton racquet, whereas the secondary table 138 would store the various types of balls i.e. tennis ball, football, volleyball, basket ball, cricket ball, golf ball. In said implementation, there would be initially two comparisons to identify the basic shape of the racquet and then two more comparisons to identify the tennis racquet. Hence, the number of comparisons is now four as opposed to eight earlier. Thus, owing to hierarchical structure of the image data repository 132, the number of comparisons is reduced to a greater extent when there are a large number of base images and associated variations.
[00046]
The image search system 102 may be used for various applications such as selection based purchasing. In one implementation the user uses a mobile client device 104 to capture an image of the product of interest. The product of interest can be any commodity or consumer items that the user may be interested in purchasing. The product may be displayed in a retail environment, such as a showroom, a mall or a store, or may be used by another user. Once the image is captured, the captured image can be transmitted to the image search system 102. On receiving the image transmitted by the user, the image search system 102 processes the received image to determine various object attributes or characteristics that may be associated with the product of interest. Based on the determined attributes, products which are same or similar to the product of interest are determined by the search module 120. The similar products can be products that are on sale by one or more other vendors or retail outlets. Besides the similar products, other product details associated with the similar products can be displayed. In one implementation, the product details may indicate the prices, features, specifications, proximity of the locations where the similar product is available with respect to the users present locations, and so on. Based on the availability of the similar products, the purchaser would be in a position to make an informed decision and proceed with the purchase of product of interest from the present location, or from vendors or retail outlets that are in proximity to the above locations.
17
[00047]
Thus the user is made aware of various details pertaining to the product which would help him to make an informed decision as to whether he wants to purchase a product or not. In another implementation, the user may connect to his friends through a social forum or platform to seek their opinion on the product or read product reviews and based on the opinions may decide to purchase the product.
[00048]
In addition to making a choice as to what product of interest to purchase, the purchaser would eventually have to pay for the commodity that has been selected for purchase. The payment procedure would involve a waiting time at the billing counter.
[00049]
In one implementation, the client device 104 of the user, such as a smart phone, is configured to read a unique identifier placed on a product by the shopping complex or the store or the mall or the showroom, collectively referred to as the local vendor so as to uniquely identify a product. The unique identifier may be a local vendor specific customized tag, a unique alpha-numeric string, a barcode, a radio frequency identification (RFID) tag, etc.
[00050]
In operation, the user would scan the unique identifier of the product of interest. On completion of reading or scanning the unique identifiers of the products, the user may initiate a transaction to complete payment for the purchased products. In one implementation, the user may utilize the payment gateway of a financial institution, in which the user has an account, to complete the transaction and make payment for the purchased products. On the successful completion of the payment, indicated by generation of a unique transaction ID, the products are marked as sold in the records maintained by the local vendor which facilitate the user to pass through the theft prevention systems of the local vendor without raising a theft alert. In one implementation, the transaction ID may be in the form of one or more tags, such as , , wherein vendor ID is an alphanumeric string uniquely identifying a vendor. Thus the user does not need to stand in a queue at the billing counter of the local vendor and can pay for the products using the client device 104.
[00051]
Fig. 2 illustrates an exemplary method 200 for image searching, according to an embodiment of the present subject matter. The method 200 may be described in the general
18
context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types. The method 200 may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communication network. In a distributed computing environment, computer executable instructions may be located in both local and remote computer storage media, including memory storage devices.
[00052]
The order in which the method 200 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method 200 or alternative methods. Additionally, individual blocks may be deleted from the method 200 without departing from the spirit and scope of the subject matter described herein. Furthermore, the method 200 can be implemented in any suitable hardware, software, firmware, or combination thereof. The method 200 is presently provided for image searching.
[00053]
At block 202, an image is received for image search. The image may be received from various sources such as a client device 104, a digital camera, a video broadcast. Further, a user may also upload the image using an image search portal. As shown in block 204, the quality of the image is checked against a pre-defined threshold. In one implementation, the image analysis module 118 is configured to determine the quality of the received image. In said implementation, the image analysis module 118 uses various conventionally known image editing and image enhancement techniques, for example removal of red eyes, normalization of light intensity, corrections of color balance, and hue and saturation, so as to correct imperfections in the image and to enhance the quality of the image.
[00054]
If at block 204, it is determined that the quality of image is below the threshold quality (“No” branch of block 204), block 136 branches to block 206. At block 206, an error message is generated notifying a user, who initiated the search, of the poor quality of the image. In one implementation, the image analysis module 118 is configured to generate the alert notifying the user of the poor quality of the image. The alert may be of various types such as an error code, a message, an audio-visual notification, etc.
19
[00055]
However, at block 204, it is determined that the quality of image is above or of the threshold quality, (‘Yes’ branch of block 204), block 204 branches to block 208. At block 208, the received image is further processed to generate one or more layers. In one implementation, the image analysis module 118 further processes the received image for determining various objects in the image. In said implementation, the image analysis module 118 uses a conventional image editing process known as layering to break the received image into one or more layer such that each layer contains one object.
[00056]
As illustrated in block 210, an object of interest is identified. In one implementation, the user may be prompted to identify the layer on which the object of interest is present. In another implementation, the search module 120 is configured to designate the most prominent object as the object of interest. At block 212, it is determined whether there is any logo on the object of interest. In one embodiment, the image analysis module 118 is configured to analyze each of the layers for determining logos on the object of interest. The image analysis module 118 may be configured to identify the logo based on various logo detection parameters, such as contrasting texture of the logo with respect to the surrounding portion of the layer, content depicted in a different font or style or color, colorful or distinctive design of the logo, and specific or characteristic shape of the logo.
[00057]
If at block 212, the logo is detected on the object of interest (‘Yes’ branch of block 212, method 200 proceeds to block 214, where, the object of interest is compared with various reference images of the reference products associated with the logo. In one implementation, on identification of the logo, the search module 120 is configured to retrieve all reference products/images tagged with the identified logo from the image data repository 132 and tags the retrieved reference products to be potentially the object of interest. At block 216, the extent of match between the object of interest and the tagged reference products is determined. In one embodiment, the image analysis module 118 is configured to determine various object selection parameters of the object of interest, such as colour, texture, shape, and size. On the basis of one or more object selection parameters, the search module 120 is configured to match one or more tagged reference products with the object of interest and
20
compute the extent of match between a tagged reference product and the object of interest in various formats, such as a fraction, and percentage.
[00058]
At block 218, for each of the tagged reference product, it is determined if the extent of match between the object of interest and the tagged reference product is greater than a threshold match value. In one embodiment, if the extent of match, i.e., the similarity, between the object of interest and any reference product tagged with the brand exceeds a pre-defined threshold, say 75%, the search module 120 is configured to tag the said product to be same or similar as the object of interest.
[00059]
If at block 218, the extent of match between the object of interest and the tagged reference product is greater than a threshold (‘Yes’ branch of block 218, method 200 proceeds to block 220. At block 220, the results of the images search are presented to the user, i.e., all the tagged reference products being similar or same to the object of interest are provided as the results. For example, in one embodiment, the search module 120 is configured to retrieve various product details pertaining to the said reference product from the image data repository 132 and present the retrieved product details to the user. In another implementation, the user may be prompted to select any of the reference products to view the various product details pertaining to the reference product.
[00060]
In case of a logo not being identified on the object of interest (‘No’ branch of block 212) or the extent of similarity between the object of interest and the reference product being below the pre-defined threshold (‘No’ branch of block 218), the received image is further processed to generate various sub-objects of the object of interest as shown at block 222. In one implementation, the image analysis module 118 is configured to convert the objects in various layers into single line diagram so as to determine the basic shape of the object. In said implementation, image analysis module 118 is further configured to generate the single line diagram based on sharp changes in the light intensity. The image analysis module 118 further processes the generated single line diagram to extract sub-objects, which may be part of a single object. Further, the sub-objects may also be determined by the image analysis module 118 by varying various sub-object parameters such as the light intensity.
21
[00061]
As depicted in block 224, the object of interest is identified based on the at least one sub object parameter. In one implementation, the search module 120 is configured to extract and crop the sub-objects. The search module 120 is further configured to compare the sub-objects with various categories of reference products for identification based on at least one sub-object parameter like shape, relative size, relative position with regard to each other, etc. On identification of a potential category of reference products, which is similar to the object of interest, by the search module 120, the extracted and cropped sub objects are compared with various reference products classified under the potential category of reference products based on other object attributes, such as color, and texture.
[00062]
At block 226, it is determined, if the identification of the object of interest is successful. For example, if the extent of similarity between the shape pattern of the sub-objects and a reference product exceeding a pre-defined threshold, say 80%, the reference product is considered to be a match with the object of interest. If at block 226, the object of interest is successfully identified (‘Yes’ branch of block 218, method 200 proceeds to block 220, where successful results of the image search pertaining to the object of interest are presented to the user as depicted in block 220. However, if at block 226, the object of interest could not be identified (‘No’ branch of block 218, method 200 proceeds to block 228. At block 228, a notification is generated for the user informing him of the same.
[00063]
Although implementations for searching of images have been described in language specific to structural features and/or methods, it is to be understood that the appended claims are not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as exemplary implementations for image searching.
22
I/We Claim:
1.
A computer implemented method for image searching, the method comprising:
processing an image for generation of one or more layers associated with the image, wherein each of the one or more layers includes at least one objects;
detecting presence of at least one logo on the generated one or more layers based on logo detection parameters; and
determining the an extent of match between the each of the one or more objects and a reference product by comparing the one or more object with reference product associated with a brand identified by the detected logo based on object attributes.
2.
The method as claimed in claim 1 further comprises ascertaining whether the quality of the image is of a threshold quality, based on one or more quality attributes.
3.
The method as claimed in claim 1 further comprises modifying one or more image parameters associated with the received image to enhance quality of the received image
4.
The method as claimed in claim 1 further comprising providing product details associated with the reference product on the extent of match exceeding a pre-defined threshold.
5.
The method as claimed in claim 1 further comprising generating a plurality of sub-objects of the one or more objects based in part on light intensity factors.
6.
The method as claimed in claim 5 further comprising identifying the one or more objects based on sub-object parameters of the plurality of sub-objects.
7.
An image search system (102) comprising:
a processor (108); and
a memory (112) coupled to the processor (108), the memory (112) comprising an image analysis module (118) configured to
process an image to generate one or more layers, wherein each generated layer includes one object; and
identify a logo in the one or more layers based on logo detection parameters.
23
8.
The image search system (102) as claimed in claim 7, wherein the image analysis module (118) is further configured to ascertain the quality of the image based on quality attributes.
9.
The image search system (102) as claimed in claim 7, wherein the image analysis module (118) is further configured to enhance the quality of the image by variation of at least one image parameter associated with the image.
10.
The image search system (102) as claimed in claim 7 further comprising a search module (120) configured to retrieve at least one reference image of each of one or more reference products, wherein the at least one reference image is one of same or similar to the object based on object attributes.
11.
The image search system (102) as claimed in claim 7 further comprising an optimization module (122) configured to order data stored in an image data repository (132) based in part on the frequency of successful hits.
12.
The image search system (102) as claimed in claim 7, wherein the image search system (102) is coupled to an image data repository (132) configured to store at least one of reference images and product details pertaining to reference products in a hierarchical structure.
13.
A computer-readable medium having embodied thereon a computer program for executing a method comprising:
processing an image for generation of one or more layers wherein each of the one or more layers includes one or more objects;
detecting at least one logo on the generated one or more layers based on logo detection parameters; and
determining the extent of match between the one or more object and a reference product by comparing the one or more object with reference product associated with a brand identified by the detected logo based on object attributes.