Specification
BACKGROUND
The invention relates generally to digital images. More specifically, the invention relates to a method, system and computer program product for identifying user interface components and characters from captured digital images.
Captured digital images are sequences of images recorded by a screen capture tool from an application running on desktop. The screen capture tool may be an event based schedule capture tool that recognizes and stores the sequences of images in a desktop application. The desktop application may include a standalone and a web based application. The sequences of images are stored in a standard format, such as. Bitmap Image file format (BMP), Joint Photographic Experts group (JPEG), and Graphics Interchange Format (GIF). Further, the content in sequences of images includes characters, user interface components, graphics and other pictorial components.
Semantic content in captured digital images from desktop applications is a source of information that can be used to acquire knowledge and develop a conceptual knowledge base of an application. The semantic content includes content in the fomri of text, user interface components, events and their relationships with each other. The semantic content can be used for learning and mapping user interface components and characters from images on to the underlying business domains. The developed knowledge base is used for user interface design analysis and link to workflow analysis, and other business process re-engineering applications.
Various techniques are used for detection of characters from images. These techniques involve binarization, text detection and segmentation. Binarization simplifies image data by converting the images into binary pixels. Various methods for binarizing use different algorithms, based on thresholds. The segmentation step segments a character block by using various algorithms, based on connected pixels and other homogeneity criteria.
One or more of the techniques above are used to identify characters from images. The techniques above do not, however, address identification of user interface components and characters from a captured digital image of an application njnning on desktop. Further, the algorithms used for binarization and segmentation are complex.
In light of the foregoing, there is a need for a method and system for identifying user interface components and characters from a captured digital image from a desktop application. Further, the method and system need to use simple algorithms of binarization and segmentation.
SUMMARY
An object of the invention is to provide a method, system and computer program product for identifying user interface components and characters from a captured digital image of a desktop application.
Another object of the invention is to provide a method for identifying user interface components and characters from a captured digital image for business workflow engineering.
To achieve the objectives mentioned above, the invention provides a method, system and computer program product for identifying user interface components and characters from a captured digital image of a desktop application. The captured digital image is converted into a grey scale image. The grey scale image is binarized by converting it into a plurality of binary pixels. Thereafter, the plurality of binary pixels is segmented into one or more segments by marking rectangular boundaries around one or more clusters of connected binary pixels. The one or more components are recognized from the one or more segments by using one or more neural network algorithms, which have been trained offline, based on a set of character image vectors and shape vectors. The offline training of neural network algorithms results in generation of weights for each of the neural network algorithms used during recognition. Subsequently, the recognized one or more components are stored in a metadata format.
The invention described above provides identification of user interface components and characters which helps in business process re-engineering. The information obtained through identification of the user interface components and characters helps in learning the mapping of text and user interface data to underiying business domain. Further, user interface design linking with business workflow is used to study business behavioral analysis. This facilitates building of knowledge base and workflow/process automation to create knowledge based solutions. Furthermore, user interface components information can be used for creating a semi-functional prototype of an existing application.
BRIEF DESCRIPTION OF THE DRAWINGS
Various embodiments of the invention will hereinafter be described in conjunction with the appended drawings, provided to illustrate and not to limit the invention, wherein like designations denote like elements, and in which:
FIG. 1 is a flowchart illustrating a method for identifying one or more components from a captured digital image of an application, in accordance with an embodiment of the invention;
FIG. 2 is a flowchart illustrating a method for identifying one or more components from a captured digital image of an application, in accordance with another embodiment of the invention;
FIG. 3 is a flowchart illustrating a method for offline training of one or more neural network algorithms, in accordance with an embodiment of the invention;
FIG. 4 is a flowchart illustrating a method for recognizing one or more components using the one or more neural network algorithms, in accordance with an embodiment of the invention;
FIG. 5 is a block diagram of a system for identifying one or more components from a captured digital image of an application, in accordance with an embodiment of the invention;
FIG. 6 is a block diagram of a system for identifying one or more components from a captured digital image of an application, in accordance with another embodiment of the invention; and
FIG. 7 is a block diagram of a recognition module for recognizing the one or more components from the captured digital image, in accordance with an embodiment of the invention.
DETAILED DESCRIPTION OF THE DRAWINGS
The invention provides a method, system and computer program product for identifying components from a captured digital image of an application running on desktop. The captured digital image contains various components, which include user interface components and characters. Examples of user interface components include a text box, a list box, a table cell, a combo box, a list scroll box, a check box, a radio box, and a button. Characters include title of the user interface components and other text present in the captured digital image. The captured digital image is recorded using a screen capturing tool for identifying the components contained in it. The screen capture tool may be an event based schedule capture tool that capture and stores the sequences of images in an application. In various embodiments of the invention, the application may be a desktop standalone application, a web based application and the like. The sequences of images may be stored in standard fomriats known in the art, such as, Bitmap Image file format (BMP), Joint Photographic Experts group (JPEG), and Graphics Interchange Format (GIF). These sequences are used for identification of components, as described with the help of the forthcoming embodiments.
FIG. 1 is a flowchart illustrating a method for identifying one or more components from a captured digital image of an application, in accordance with an embodiment of the invention.
At step 102, the captured digital image is binarized. The binarization of the captured digital image includes conversion of the captured digital image into a binary image. In various embodiments of the invention, the captured digital image includes multi-colored pixels. Each of the multi-colored pixels of the captured digital image is converted into binary pixels of black and white during binarization. Binarization facilitates identification of the user interface components and characters from the captured digital image. In various embodiments of the invention, the conversion is based on contrast values of the multi-colored pixels. In an embodiment of the invention, different contrast-threshold values are used for user interface components and characters. The contrast value of each pixel in the captured digital image is compared with the contrast-threshold value for user interface components and characters. In an embodiment of the invention.
if the contrast value of a pixel, in the case of the text being darker than the background, is greater than the contrast-threshold value, the color of the pixel is converted into black. Otherwise, the pixel color is converted into white. In another embodiment of the invention, if the contrast value of a pixel, in the case of the text being lighter than the background, is greater than the contrast-threshold value, the pixel color is converted into white. Otherwise, the pixel color is converted into black.
In an embodiment of the invention, the contrast-threshold values may be adapted to the contrast level of the captured digital image. In another embodiment of the invention, the contrast-threshold values can be set manually by a user. In another embodiment of the invention, the contrast-threshold value is detennined by empirical experiments. Further, the contrast-threshold value is based on the type of the captured digital image.
At step 104, the binary image is segmented into segments of binary pixels. Segmentation is performed by marking rectangular boundaries around various clusters of connected pixels. In an embodiment of the invention, segmentation is based on a rectangular region-growing technique that is already known in the art. In this technique, a region-growing technique algorithm increases the size of each segment in the forward, backward, upward and downward directions. The algorithm detects the pixels in the clusters of connected pixels by traversing from the left to the right and the top to the bottom directions, to mark the left, top, right and bottom boundaries of the segments. During the process of traversing, the first-identified black pixel in a cluster is mari
Documents
Orders
| Section |
Controller |
Decision Date |
|
|
|
Application Documents
| # |
Name |
Date |
| 1 |
1521-CHE-2008 FORM-18 06-10-2009.pdf |
2009-10-06 |
| 1 |
1521-CHE-2008_EXAMREPORT.pdf |
2016-07-02 |
| 2 |
1521-CHE-2008 FORM-13 28-10-2009.pdf |
2009-10-28 |
| 2 |
1521-CHE-2008-Abstract-110516.pdf |
2016-05-13 |
| 3 |
1521-CHE-2008-Claims-110516.pdf |
2016-05-13 |
| 3 |
1521-CHE-2008 POWER OF ATTORNEY 12-01-2011.pdf |
2011-01-12 |
| 4 |
1521-CHE-2008-Drawing-110516.pdf |
2016-05-13 |
| 4 |
1521-che-2008 form-13 12-01-2011.pdf |
2011-01-12 |
| 5 |
1521-CHE-2008-Examination Report Reply Recieved-110516.pdf |
2016-05-13 |
| 5 |
1521-che-2008 form-1 12-01-2011.pdf |
2011-01-12 |
| 6 |
1521-CHE-2008-Form 1-110516.pdf |
2016-05-13 |
| 6 |
1521-che-2008 form-5.pdf |
2011-09-03 |
| 7 |
1521-CHE-2008-Other Patent Document-110516.pdf |
2016-05-13 |
| 7 |
1521-che-2008 form-3.pdf |
2011-09-03 |
| 8 |
1521-che-2008 form-1.pdf |
2011-09-03 |
| 8 |
1521-CHE-2008 AMENDED PAGES OF SPECIFICATION 03-06-2015.pdf |
2015-06-03 |
| 9 |
1521-CHE-2008 CORRESPONDENCE OTHERS 03-06-2015.pdf |
2015-06-03 |
| 9 |
1521-che-2008 drawings.pdf |
2011-09-03 |
| 10 |
1521-CHE-2008 FORM-1 03-06-2015.pdf |
2015-06-03 |
| 10 |
1521-che-2008 description (complete).pdf |
2011-09-03 |
| 11 |
1521-CHE-2008 FORM-13 03-06-2015.pdf |
2015-06-03 |
| 11 |
1521-che-2008 correspondence-others.pdf |
2011-09-03 |
| 12 |
1521-che-2008 abstract.pdf |
2011-09-03 |
| 12 |
1521-che-2008 claims.pdf |
2011-09-03 |
| 13 |
1521-che-2008 abstract.pdf |
2011-09-03 |
| 13 |
1521-che-2008 claims.pdf |
2011-09-03 |
| 14 |
1521-CHE-2008 FORM-13 03-06-2015.pdf |
2015-06-03 |
| 14 |
1521-che-2008 correspondence-others.pdf |
2011-09-03 |
| 15 |
1521-CHE-2008 FORM-1 03-06-2015.pdf |
2015-06-03 |
| 15 |
1521-che-2008 description (complete).pdf |
2011-09-03 |
| 16 |
1521-CHE-2008 CORRESPONDENCE OTHERS 03-06-2015.pdf |
2015-06-03 |
| 16 |
1521-che-2008 drawings.pdf |
2011-09-03 |
| 17 |
1521-che-2008 form-1.pdf |
2011-09-03 |
| 17 |
1521-CHE-2008 AMENDED PAGES OF SPECIFICATION 03-06-2015.pdf |
2015-06-03 |
| 18 |
1521-CHE-2008-Other Patent Document-110516.pdf |
2016-05-13 |
| 18 |
1521-che-2008 form-3.pdf |
2011-09-03 |
| 19 |
1521-CHE-2008-Form 1-110516.pdf |
2016-05-13 |
| 19 |
1521-che-2008 form-5.pdf |
2011-09-03 |
| 20 |
1521-CHE-2008-Examination Report Reply Recieved-110516.pdf |
2016-05-13 |
| 20 |
1521-che-2008 form-1 12-01-2011.pdf |
2011-01-12 |
| 21 |
1521-CHE-2008-Drawing-110516.pdf |
2016-05-13 |
| 21 |
1521-che-2008 form-13 12-01-2011.pdf |
2011-01-12 |
| 22 |
1521-CHE-2008-Claims-110516.pdf |
2016-05-13 |
| 22 |
1521-CHE-2008 POWER OF ATTORNEY 12-01-2011.pdf |
2011-01-12 |
| 23 |
1521-CHE-2008-Abstract-110516.pdf |
2016-05-13 |
| 23 |
1521-CHE-2008 FORM-13 28-10-2009.pdf |
2009-10-28 |
| 24 |
1521-CHE-2008_EXAMREPORT.pdf |
2016-07-02 |
| 24 |
1521-CHE-2008 FORM-18 06-10-2009.pdf |
2009-10-06 |