Sign In to Follow Application
View All Documents & Correspondence

Method And System For Frame Stitching Based Image Construction In An Indoor Environment

Abstract: This disclosure relates generally to a method and a system for frame stitching based image construction for an indoor environment. The method enables constructing an image of a scene by stitching a plurality of key frames identified from a plurality of image frames captured by a mobile imaging device. The method overcomes multiple challenges posed by the indoor environment, effectively providing clean stitching of the key frames to construct the image of the scene. The method provides image stitching approach that combines visual data from the mobile imaging device and an inertial sensor from an Inertial Measurement Unit (IMU) mounted on the mobile imaging device, with a feedback for error correction to generate robust stitching outputs in indoor scenario.

Get Free WhatsApp Updates!
Notices, Deadlines & Correspondence

Patent Information

Application #
Filing Date
30 April 2018
Publication Number
44/2019
Publication Type
INA
Invention Field
COMPUTER SCIENCE
Status
Email
ip@legasis.in
Parent Application
Patent Number
Legal Status
Grant Date
2024-01-15
Renewal Date

Applicants

Tata Consultancy Services Limited
Nirmal Building, 9th Floor, Nariman Point, Mumbai - 400021, Maharashtra, India

Inventors

1. GUBBI LAKSHMINARASIMHA, Jayavardhana Rama
Tata Consultancy Services Limited, Gopalan Global Axis SEZ "H" Block, No. 152 (Sy No. 147,157 & 158), Hoody Village, Bangalore - 560066, Karnataka, India
2. RAMASWAMY, Akshaya
Tata Consultancy Services Limited, Gopalan Global Axis SEZ "H" Block, No. 152 (Sy No. 147,157 & 158), Hoody Village, Bangalore - 560066, Karnataka, India
3. RAJ, Rishin
Tata Consultancy Services Limited, Gopalan Global Axis SEZ "H" Block, No. 152 (Sy No. 147,157 & 158), Hoody Village, Bangalore - 560066, Karnataka, India
4. PURUSHOTHAMAN, Balamuralidhar
Tata Consultancy Services Limited, Gopalan Global Axis SEZ "H" Block, No. 152 (Sy No. 147,157 & 158), Hoody Village, Bangalore - 560066, Karnataka, India

Specification

Claims:1. A processor implemented frame stitching based image construction method, the method comprising: receiving (302), a sequence of a plurality of image frames from a mobile imaging device, wherein the plurality of image frames correspond to a scene of interest in an indoor environment; a positional data of the mobile imaging device from an Inertial Measurement Unit (IMU) of the mobile imaging device, wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames; and a plurality of device parameters of the mobile imaging device, wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV, and frame dimensions of the plurality image frames; selecting (304) a set of key frames from the plurality of image frames, wherein the set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame, wherein each key frame from the set of key frames provides a maximum average of luminance and variance; stitching (306) the set of key frames to construct an image of the scene of interest, wherein the stitching comprises: selecting one key frame and a corresponding consecutive key frame from the set of key frames, wherein the selected key frame is the initial key frame during first iteration of stitching the set of key frames; determining a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization during image registration, wherein elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter derived from a change in IMU positional data received for each key frame and the plurality of device parameters; generating a refined warping matrix from the warping matrix to perform an image intensity based registration refinement; estimating a constraint feature based warping matrix to stitch the selected key frame and the corresponding consecutive key frame, wherein the constraint feature based warping matrix is estimated from the refined warping matrix and a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame, wherein the extracted plurality of features provide key point correspondence between the selected key frame and the corresponding consecutive key frame; and constructing (308) an image of the scene of interest by iterating the stitching of each the key frame of the set of keyframes and the corresponding consecutive key frame of each key frame , in sequence. 2. The processor implemented method as claimed in claim 1, wherein the method further comprises applying a selective approach with feedback for the frame stitching based image construction method, where the selective approach with feedback comprises: choosing (402) one of: the refined warping matrix as a final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features which provide key point correspondence between the key frame and the consecutive key frame are below a key point threshold; and the constrained feature based warping matrix as the final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features providing key point correspondence between the key frame and the consecutive key frame and are above the key point threshold; modifying (404) the horizontal translation parameter and the vertical translation parameter of the warping matrix based on the final warping matrix and a previous warping matrix to create a modified warping matrix, wherein the previous warping matrix is the warping matrix created during frame stitching of preceding frame; stitching (406) one key frame and the corresponding consecutive keyframe based on the modified warping matrix; and constructing (408) the image of the scene of interest by iterating the stitching of each key frame from the set of keyframes and the corresponding consecutive key frame, in sequence. 3. The processor implemented method as claimed in claim 1, wherein generating the refined warping matrix from the warping matrix comprises applying a gradient decent optimizer and minimization of a mean square error (MSE) metric with transformation set to translation. 4. The processor implemented method as claimed in claim 1, wherein the constraint feature based warping matrix is defined by a rotation constraint, a refined warping matrix based horizontal translation constraint, and a refined warping matrix based vertical translation constraint set using the refined warping matrix, wherein the constraint feature based warped matrix provides refinement over the refined warping matrix. 5. An image construction system (102) for frame stitching based image construction, the image construction system (102) comprises: a processor (202); an Input/Output (I/O) interface (206); a memory (204), the memory comprising a frame stitching module (212); wherein the frame stitching module (212) is configured to: receive, a sequence of a plurality of image frames from a mobile imaging device (104), wherein the plurality of image frames correspond to a scene of interest in an indoor environment; a positional data of the mobile imaging device (104) from an Inertial Measurement Unit (IMU) of the mobile imaging device (104), wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames; and a plurality of device parameters of the mobile imaging device (104), wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV and frame dimensions of the plurality image frames; select a set of key frames from the plurality of image frames, wherein the set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame, wherein each key frame from the set of key frames provides a maximum average of luminance and variance over a predefined interval defined by number of image frames; stitch the set of key frames to construct an image of the scene of interest, wherein to stitch the key frames the frame stitching module (212) is configured to: select one key frame and a corresponding consecutive key frame from the set of key frames, wherein the selected key frame is the initial key frame during first iteration of stitching the set of key frames; determine a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization during image registration, wherein elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter derived from a change in IMU positional data received for each key frame and the plurality of device parameters; generate a refined warping matrix from the warping matrix to perform an image intensity based registration refinement; estimate a constraint feature based warping matrix to stitch the selected keyframe and the corresponding consecutive key frame, wherein the constraint feature based warping matrix is estimated from the refined warping matrix and a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame, wherein the extracted plurality of features provide key point correspondence between the key frame and the consecutive key frame; and construct an image of the scene of interest by iterating the stitching of each key frame from the set of key frames and the corresponding consecutive key frame of each key frame, , in sequence. 6. The image construction system (102) as claimed in claim 5, wherein the frame stitching module (212) is configured to apply a selective approach with feedback for the frame stitching based image construction method, wherein to apply the selective approach with feedback the frame stitching module (212) is configured to: choose one of: the refined warping matrix as a final warping matrix for stitching the one key frame and the corresponding consecutive keyframes, if the extracted plurality of features which provide key point correspondence between the key frame and the consecutive key frame, are below a key point threshold; and the constrained feature based warping matrix as the final warping matrix for stitching one key frame and the corresponding consecutive keyframe, if the extracted plurality of features providing key point correspondence between the key frame and the consecutive key frame are above the key point threshold; modify the horizontal translation parameter and the vertical translation parameter of the warping matrix based on the final warping matrix and a previous warping matrix to create a modified warping matrix, wherein the previous warping matrix is the warping matrix created during frame stitching of preceding frame; stitch one key frame and the corresponding consecutive keyframe based on the modified warping matrix; and construct the image of the scene of interest by iterating the stitching of each key frame from the set of keyframes and the corresponding consecutive key frame, in sequence. 7. The image construction system (102) as claimed in claim 5, wherein the frame stitching module (212) is configured to generate the refined warping matrix from the warping matrix comprises applying a gradient decent optimizer and minimization of a mean square error (MSE) metric with transformation set to translation. 8. The image construction system (102) as claimed in claim 5, wherein the constraint feature based warping matrix is defined by a rotation constraint, a refined warping matrix based horizontal translation constraint and a refined warping matrix based vertical translation constraint set using the refined warping matrix, wherein the constraint feature based warped matrix provides refinement over the refined warping matrix. , Description:FORM 2 THE PATENTS ACT, 1970 (39 of 1970) & THE PATENT RULES, 2003 COMPLETE SPECIFICATION (See Section 10 and Rule 13) Title of invention: METHOD AND SYSTEM FOR FRAME STITCHING BASED IMAGE CONSTRUCTION IN AN INDOOR ENVIRONMENT Applicant Tata Consultancy Services Limited A company Incorporated in India under the Companies Act, 1956 Having address: Nirmal Building, 9th floor, Nariman point, Mumbai 400021, Maharashtra, India The following specification particularly describes the invention and the manner in which it is to be performed. TECHNICAL FIELD The disclosure herein generally relates to image processing, and, more particularly, to frame stitching based image construction for an indoor environment. BACKGROUND Mobile imaging devices such as Unmanned Aerial Vehicles (UAVs) or the like, are being widely used for multitude of tasks where human involvement may be a challenge or may provide low efficiency. Typically, in indoor environments which are widely spread across larger areas often with limited lightening conditions and difficulty in ease of access to locations of the indoor environment, imaging devices are popular choice to scan and monitor the indoor environments. Typical examples of such indoor environments include industry environment, warehouse environment, shopping malls and the like. Monitoring and inspection task of the mobile imaging device, for example the UAV, requires the UAV to capture image of a scene of interest. Once the scene is captured as a series of image frames, image processing is performed on the image frames to extract information. To capture maximum details or information of the scene, images are captured from closer range. Specific details of interest may include information of a product stocked in warehouse such as barcode of a product and the like. However, close capture of images leads to loss of context, also referred as global context, associated with the scene of interest. Recovery of the global context is possible through video stitching or frame stitching, wherein key frames from a sequence of image frames in a video being captured are stitched to construct image of the scene of interest. Existing methods for key frame selection mostly focus on overlap criteria for consecutive key frame selection, however such selection criteria may compromise on the quality of the key frame, effectively reducing quality of the constructed image. Existing methods stitch the key frames based on features extracted from the key frames. However, these existing methods work well, provided the scene is planar, the images are taken with appropriate placement of the mobile imaging device and they are static. The indoor environments however often pose multiple challenges. Firstly, there is often loss of Global Positioning System (GPS) signal and the indoor environment may have poor and varying lighting conditions. Secondly, for applications such as asset monitoring, video (of an object / area of interest) is required to be captured from distances very close to an object of interest, hence the global context is lost. Thirdly, between consecutive frames, feature points repeat as the area or scene of interest being monitored has similar objects stacked one above another. Thus, the existing video stitching approaches when applied for indoor environments may have limitations affecting clean stitching between the key frames, which in turn introduces errors in an image constructed for the scene of interest by stitching the key frames. SUMMARY Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one embodiment, a method for frame stitching based image construction is disclosed. The method comprises receiving a sequence of a plurality of image frames from a mobile imaging device, wherein the plurality of image frames correspond to a scene of interest in an indoor environment. The method further comprises receiving a positional data of the mobile imaging device from an Inertial Measurement Unit (IMU) of the mobile imaging device, wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames. The method further comprises receiving a plurality of device parameters of the mobile imaging device, wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV and frame dimensions of the plurality image frames. Further, the method comprises selecting a set of key frames from the plurality of image frames. The set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame. Each key frame from the set of key frames provides a maximum average of luminance and variance over a predefined interval. Further, more the method comprises stitching the set of key frames to construct an image of the scene of interest. The stitching comprises selecting one key frame and a corresponding consecutive key frame from the set of key frames, wherein the selected key frame is the initial key frame during first iteration of stitching the set of key frames. Further, determining a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization during image registration. Elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter and are derived from a change in IMU positional data received for each key frame and the plurality of device parameters, wherein IMU positional data correspond to an accelerometer and gyroscope information of the IMU. Further, the stitching comprises generating a refined warping matrix from the warping matrix to perform an image intensity based registration refinement. Furthermore the stitching comprises estimating a constraint feature based warping matrix to stitch the selected key frame and the corresponding consecutive key frame. The estimation is based on the refined warping matrix and a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame. The extracted plurality of features provide key point correspondence between the key frame and the consecutive key frame. Further, more the method comprises constructing an image of the scene of interest by iterating the stitching of each key frame from the set of key frames and the corresponding consecutive key frame of each key frame in sequence. In yet another embodiment, an image construction system for frame stitching based image construction. The image construction system comprises a processor, an Input/Output (I/O) interface, a memory. The memory comprises a frame stitching module configured to receive a sequence of a plurality of image frames from a mobile imaging device, wherein the plurality of image frames correspond to a scene of interest in an indoor environment. Further, receive positional data of the mobile imaging device from an Inertial Measurement Unit (IMU) of the mobile imaging device, wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames. Further, receive a plurality of device parameters of the mobile imaging device, wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV and frame dimensions of the plurality image frames. The frame stitching module is further configured to select a set of key frames from the plurality of image frames, wherein the set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame. Each key frame from the set of key frames provides a maximum average of luminance and variance over a predefined interval defined by number of image frames. The frame stitching module is further configured to stitch the set of key frames to construct an image of the scene of interest. To stitch the key frames the frame stitching module is configured to select one key frame and a corresponding consecutive key frame from the set of key frames, wherein the key frame is the initial key frame during first iteration of stitching the set of key frames. Further, determine a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization for image registration, wherein elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter derived from a change in IMU positional data received for each key frame and the device parameters. The IMU positional data correspond to an accelerometer and gyroscope information of the IMU. Further, generate a refined warping matrix from the warping matrix to perform an image intensity based registration refinement. The frame stitching module is further configured to estimate a constraint feature based warping matrix to stitch the key frame and the consecutive key frame from the refined warping matrix and a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame. The extracted plurality of features provide key point correspondence between the key frame and the consecutive key frame. Furthermore, the frame stitching module is configured to construct an image of the scene of interest by iterating the stitching of each key frame from the set of keyframes and the corresponding consecutive key of each keyframe, in sequence. In yet another aspect, a non-transitory computer readable medium is provided. The non-transitory computer-readable medium stores instructions which, when executed by a hardware processor, cause the hardware processor to perform actions comprising receiving a sequence of a plurality of image frames from a mobile imaging device, wherein the plurality of image frames correspond to a scene of interest in an indoor environment. The actions further comprise receiving positional data of the mobile imaging device from an Inertial Measurement Unit (IMU) of the mobile imaging device, wherein the IMU provides change in the positional data corresponding to each image frame among the plurality of image frames. The actions further comprise receiving a plurality of device parameters of the mobile imaging device, wherein the plurality of device parameters comprise a horizontal field of view (FoV), a vertical FoV and frame dimensions of the plurality image frames. The actions further comprise selecting a set of key frames from the plurality of image frames. The set of key frames comprises an initial key frame identified based on an initial key frame criteria and key frames placed at a predefined distance interval starting from the initial key frame. Each key frame from the set of key frames provides a maximum average of luminance and variance over a predefined interval defined by number of image frames. Further, more the actions comprise stitching the set of key frames to construct an image of the scene of interest. The stitching comprises selecting one key frame and a corresponding consecutive key frame from the set of key frames, wherein the key frame is the initial key frame during first iteration of stitching the set of key frames. Further, determining a warping matrix between the selected key frame and the corresponding consecutive key frame for initialization for image registration. Elements of the warping matrix comprise a horizontal translation parameter and a vertical translation parameter are derived from the change in IMU positional data received for each key frame and the device parameters, wherein IMU positional data correspond to an accelerometer and gyroscope information of the IMU. Further, the stitching comprises generating a refined warping matrix from the warping matrix to perform an image intensity based registration refinement. Furthermore the stitching comprises estimating a constraint feature based warping matrix to stitch the selected frame and the corresponding consecutive key frame from the refined warping matrix. The estimation is based on a plurality of features extracted from a constrained space of the selected key frame and the corresponding consecutive key frame. The extracted plurality of features provide key point correspondence between the key frame and the consecutive key frame. Furthermore, the actions comprise constructing an image of the scene of interest by iterating the stitching of each key frame and the corresponding consecutive key frame of each keyframe, in sequence. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles: FIG. 1 illustrates an exemplary environment implementing frame stitching based image construction system for an indoor environment to construct image of a scene of interest from a plurality of frames of the scene captured by a mobile imaging device, according to some embodiments of the present disclosure. FIG. 2 is a functional block diagram of the image construction system of FIG.1, according to some embodiments of the present disclosure. FIG. 3 is a flow diagram illustrating a method for frame stitching based image construction for the indoor environment to construct image of the scene of interest from the plurality of frames of the scene captured by the mobile imaging device, in accordance with some embodiments of the present disclosure. FIG. 4 is a flow diagram illustrating a method for the frame stitching based image construction based on a selective approach with feedback, in accordance with some embodiments of the present disclosure. FIG. 5 is an example illustrating selection of a plurality of key frames from the plurality of image frames captured by the mobile imaging device for image construction of the scene of interest, in accordance with some embodiments of the present disclosure. FIG. 6a (a) and (b) illustrates parameters used for computation of a horizontal translational parameter and a vertical translational parameter for a warping matrix, in accordance with some embodiments of the present disclosure. FIG. 6b and 6c is an example illustrating intensity based refinement on Inertial Measurement Unit (IMU) initialization, in accordance with some embodiments of the present disclosure. FIG. 6d compares stitching of two key frames using the IMU based initialization and after image-intensity based refinement, in accordance with some embodiments of the present disclosure. FIG. 7 is schematic illustration of extraction of features from a constrained space of a key frame for frame stitching based image construction, in accordance with some embodiments of the present disclosure. FIG. 8 illustrates an example of constructed image with and without using selective feedback approach, in accordance with some embodiments of the present disclosure. FIG. 9 illustrates comparison of existing and proposed frame stitching approach, in accordance with some embodiments of the present disclosure. DETAILED DESCRIPTION OF EMBODIMENTS Exemplary embodiments are described with reference to the accompanying drawings. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Wherever convenient, the same reference numbers are used throughout the drawings to refer to the same or like parts. While examples and features of disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. It is intended that the following detailed description be considered as exemplary only, with the true scope and spirit being indicated by the following claims. Referring now to the drawings, and more particularly to FIG. 1 through 9, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method. The embodiments herein provide a method and a system for frame stitching based image construction for an indoor environment. The method enables constructing an image of a scene of interest (scene) in the indoor environment by stitching a plurality of key frames identified from a plurality of image frames (frames) of the scene. The plurality of image frames are captured by a mobile imaging device while traversing a path to cover the entire scene. The method and the system can be applied to the indoor environment such as a warehouse, a large mart and the like and overcomes multiple challenges posed by the indoor environment, effectively providing clean or smooth stitching of the key frames to construct the image of the scene, as compared to the existing frame stitching methods. The term clean or smooth refers to reduced demarcation at boundary of each key frame. The method provides image stitching approach that combines data from a visual sensor, such as the mobile imaging device and an inertial sensor such as an Inertial Measurement Unit (IMU) mounted on the mobile imaging device, with a feedback for error correction to generate robust stitching outputs in indoor scenario. The method enables handling of the issues of sparse feature points or lack of feature point correspondence in a given scene by extracting key frames from the video, and intelligently combining the positional data available from the onboard IMU and visual features from the plurality of image frames captured by the mobile imaging device, to stitch the key frames together. FIG. 1 illustrates an exemplary environment 100 implementing frame stitching based an image construction system 102 for the indoor environment to construct image of a scene of interest (scene) 106 from a plurality of frames of the scene captured by a mobile imaging device 104, according to some embodiments of the present disclosure. The environment 100 depicts the scene of interest 106, alternatively referred as scene106, in the indoor environment. As depicted, the scene106 can be captured by the mobile imaging device 104, while traversing a predefined path. The entire scene 106 is captured in form of a video comprising sequence of image frames, alternatively referred as a plurality of image frames (f1-fn). The path traversed by the mobile imaging device 104, for example, may be a raster scan path or the like which covers the entire scene 106. In an embodiment, the image construction system 102 may be implemented internally within a computing device or may be externally coupled to the computing device. In an embodiment, the image construction system 102 can be configured to receive the video, as sequence of the plurality of image frames, transmitted by the mobile imaging device 104. Transmission and reception of the image frames (f1-fn) between the mobile imaging device and the image construction system can be over a wired or a wireless network 108, or a combination thereof. In an example, the network 108 can be implemented as a computer network, as one of the different types of networks, such as virtual private network (VPN), intranet, local area network (LAN), wide area network (WAN), the internet, and such. The network 108 may either be a dedicated network or a shared network, which represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), and Wireless Application Protocol (WAP), to communicate with each other. Further, the network 108 may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices. The network devices within the network 108 may interact with the image construction system 102 through communication links. In an embodiment, the computing device, which implements the image construction system 102 can be a workstation, a mainframe computer, a personal digital assistant, a general purpose server, a network server or the like. Further, on receiving the plurality of image frames f1-fn, the image construction system 102 can be configured to identify or select a set of key frames (k1-kp, p 0:8; Mfea (2; 2) > 0:8 (4) j(Mfea (1; 3) - Mint(1; 3))j < 300 (5) j(Mfea (2; 3) - Mint(2; 3))j < 300 (6) Equations in 4 put a constraint on the rotation in the image, and equations 5 and 6 limit the horizontal translation and vertical translation to within 300 pixels from the corresponding values in Mint achieved using Aint. This set of conditions ensures Mfea is a second level refinement of Mint, and at the same time is robust to the erroneous point correspondences between key frames. At step 308, the method allows the frame stitching module 212 to construct the image of the scene of interest by iterating the stitching of the key frame and the consecutive key frame for all the key frames from the set of key frames, in sequence. FIG. 4 is a flow diagram illustrating a method 400 for the selective approach with feedback (Asel) for the frame stitching based image construction, in accordance with some embodiments of the present disclosure. In an embodiment, if the extracted plurality of features, which provide key point correspondence between the key frame and the consecutive key frame are below the key point threshold then at step 402 , the method 400 allows the frame stitching module to choose the refined warping matrix as a final warping matrix for frame stitching, if the extracted plurality of features which provide key point correspondence between the key frame and the consecutive key frame are below a key point threshold, else if the key correspondence are above the key point threshold choose the constrained warping matrix, as the final warping matrix for stitching the two key frames At step 404, the method 400 allows the frame stitching module 212 to modify the horizontal translation parameter and the vertical translation parameter of the warping matrix based on the final warping matrix and a previous warping matrix to create a modified warping matrix. The previous warping matrix is the warping matrix created during frame stitching of preceding frame. At step 406, the method 400 allows the frame stitching module 212 to stitch the key frame and the consecutive key frame based on the modified warping matrix. At step 408, the method 400 allows the frame stitching module 212 to construct the image of the scene of interest 106 by iterating the stitching of the key frame and the consecutive key frame for all the key frames from the set of key frames, in sequence. The constrained feature based approach intelligently leverages the coarse level transformation extracted from the IMU data to choose the best warp matrix computed by an image key point matching technique. A major issue arises when there are not enough key point matches found. For example, when the mobile imaging device 104 captured videos for an indoor industrial scenario, this approach has limitations in many cases, due to poor quality of images and redundancy introduced by similar looking boxes. To address, this issue, the selective approach (Asel), that chooses between Aint and Afea. When there are not enough key point matches between two key frames using Afea (key correspondence below the key point threshold), then Aint is used. Relying solely on IMU results in error propagation, as seen in FIG. 8 (a). For the selective approach to work well, an error correction by introducing a feedback step is performed. With the feedback loop, each time two key frames are stitched using Afea or Aint, a final warping matrix Mfinal can be used to correct the IMU based initialization Mimu. This is implement by introducing a correction term to the IMU-based translation of equation 2. t’x = tx + ?(Mfinal (1,3) - Mimu(1,3)) (7) t’y = ty + ?(Mfinal (2,3) - Mimu(2,3)) (8) By including this feedback, the error propagation is restricted to a great extent, resulting in a robust approach for stitching any kind of images. The result of Asel is shown in FIG. 8 (c). This output can be compared with FIG 8 (a and b), which are outputs of stitching the same set of images using other approaches described (Aint and Afea). FIG. 9 illustrates comparison of existing and proposed frame stitching approach, in accordance with some embodiments of the present disclosure. The frame stitching based image construction proposed is compared with APAP stitching approach the warehouse data to get the shelf stitched images. FIG. 9 shows the outputs of (Asel) in FIG 9 (b) and the APAP in FIG. 9(a) for four shelves. For each shelf, four to seven key frames are extracted for stitching. It can observed that the APAP performs well when the scene is mostly planar. In highly non planar scenes, it fails to find to a suitable warping. The frame stitching based image construction proposed can be seen to perform reasonably well even in these scenarios. For quantitative evaluation, peak signal to noise ratio (PSNR) parameter and Structural similarity index (SSIM) are computed. Table 1 displays the values of these metrics for the APAP, the traditional SIFT matching approach and the frame stitching based image construction with and without feedback (as proposed in method 300 and method 400). The results demonstrate that the frame stitching based image construction with and without feedback (as proposed in method 300 and method 400) achieves higher stitched image quality compared to the other stitching algorithms. Also, when the method 300 and the method 400 is implemented in MATLAB, for each shelf the time taken to compute the stitching is approximately 40 seconds, indicating faster computation. Table1 below provides quantitative comparison of the method with APAP algorithm for video stitching of six shelves. Table 1: Approach PSNR SSIM APAP 15.2462 0.5099 SIFT 15.5675 0.5228 Method 330 and method 400 (Asel) 17.3452 0.6783 The method and the image construction system disclosed herein even though is particularly explained in conjunction with the mobile imaging device that captures image frames traversing defined path to cover the entire scene of interest, it can be applied to a scenario having multiple static cameras (imaging devices) provided the camera positions and camera parameters are known, and the frames captured by the different cameras have sufficient overlap. Minor modifications to warping matrix (Mimu) for initialization may be required, still being within the scope of the disclosed method and image construction system. The written description describes the subject matter herein to enable any person skilled in the art to make and use the embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope of the claims if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language of the claims. It is to be understood that the scope of the protection is extended to such a program and in addition to a computer-readable means having a message therein; such computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means can include both hardware means and software means. The method embodiments described herein could be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs. The embodiments herein can comprise hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. The functions performed by various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The illustrated steps of method 300 and method 400 are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments. Also, the words “comprising,” “having,” “containing,” and “including,” and other similar forms are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items. It must also be noted that as used herein and in the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media. It is intended that the disclosure and examples be considered as exemplary only, with a true scope and spirit of disclosed embodiments being indicated by the following claims.

Documents

Application Documents

# Name Date
1 201821016198-STATEMENTOFUNDERTAKING(FORM3) [30-04-2018(online)].pdf 2018-04-30
2 201821016198-REQUESTFOREXAMINATION(FORM-18) [30-04-2018(online)].pdf 2018-04-30
3 201821016198-FORM18 [30-04-2018(online)].pdf 2018-04-30
4 201821016198-FORM1 [30-04-2018(online)].pdf 2018-04-30
6 201821016198-DRAWINGS [30-04-2018(online)].pdf 2018-04-30
7 201821016198-COMPLETESPECIFICATION [30-04-2018(online)].pdf 2018-04-30
8 201821016198-FORM-26 [22-05-2018(online)].pdf 2018-05-22
9 201821016198-Proof of Right (MANDATORY) [23-05-2018(online)].pdf 2018-05-23
10 Abstract1.jpg 2018-08-11
11 201821016198-ORIGINAL UNDER RULE 6 (1A)-300518.pdf 2018-08-11
12 201821016198-REQUEST FOR CERTIFIED COPY [05-02-2019(online)].pdf 2019-02-05
13 201821016198-CORRESPONDENCE(IPO)-(CERTIFIED COPY)-(7-2-2019).pdf 2019-02-11
14 201821016198-FORM 3 [03-10-2019(online)].pdf 2019-10-03
15 201821016198-OTHERS [29-07-2021(online)].pdf 2021-07-29
16 201821016198-FER_SER_REPLY [29-07-2021(online)].pdf 2021-07-29
17 201821016198-COMPLETE SPECIFICATION [29-07-2021(online)].pdf 2021-07-29
18 201821016198-CLAIMS [29-07-2021(online)].pdf 2021-07-29
19 201821016198-FER.pdf 2021-10-18
20 201821016198-US(14)-HearingNotice-(HearingDate-28-12-2023).pdf 2023-11-24
21 201821016198-FORM-26 [22-12-2023(online)].pdf 2023-12-22
22 201821016198-FORM-26 [22-12-2023(online)]-1.pdf 2023-12-22
23 201821016198-Correspondence to notify the Controller [22-12-2023(online)].pdf 2023-12-22
24 201821016198-Correspondence to notify the Controller [27-12-2023(online)].pdf 2023-12-27
25 201821016198-Written submissions and relevant documents [09-01-2024(online)].pdf 2024-01-09
26 201821016198-RELEVANT DOCUMENTS [09-01-2024(online)].pdf 2024-01-09
27 201821016198-PETITION UNDER RULE 137 [09-01-2024(online)].pdf 2024-01-09
28 201821016198-PatentCertificate15-01-2024.pdf 2024-01-15
29 201821016198-IntimationOfGrant15-01-2024.pdf 2024-01-15

Search Strategy

1 searchE_08-12-2020.pdf

ERegister / Renewals

3rd: 15 Apr 2024

From 30/04/2020 - To 30/04/2021

4th: 15 Apr 2024

From 30/04/2021 - To 30/04/2022

5th: 15 Apr 2024

From 30/04/2022 - To 30/04/2023

6th: 15 Apr 2024

From 30/04/2023 - To 30/04/2024

7th: 15 Apr 2024

From 30/04/2024 - To 30/04/2025

8th: 18 Mar 2025

From 30/04/2025 - To 30/04/2026