DESC:FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(Section 10 and Rule 13)

TITLE OF THE INVENTION:
SYSTEM AND METHOD FOR ASSESSING PERFORMANCE OF A DEPTH CAMERA

APPLICANT
e-Infochips Private Limited, an Indian company of the address Building No. 2, Aaryan Corporate Park Near Shilaj Railway Crossing, Thaltej-Shilaj Road, Ahmedabad Gujarat 380054, India

The following specification particularly describes the invention and the manner in which it is to be performed:

FIELD OF INVENTION
[1] The present invention relates to a depth camera. In particular, the present invention relates to a system for assessing performance of a depth camera for a target application.
BACKGROUND OF INVENTION
[2] Depth Cameras have been a trending point of discussion in research fields. The depth cameras are used to capture a scenario to obtain its depth for example, distance of an object from a device in terms of points to render a 3D environment. Depth cameras find use in a plurality of applications like gesture recognition, motion tracking, inspection and monitoring, 3D reconstruction, obstacle detection in new-age autonomous vehicles, picking and placing products in warehouses etc. due to their ability to measure/estimate depth of an object.
[3] The efficiency and accuracy of a depth camera depends upon the calculation of 3D points in an environment. However, a depth camera may have potential depth error sources at each and every pixel. The depth errors are majorly caused due to transparent, semi-transparent and scattering media. Further, different applications, may require different accuracy of a depth camera.
[4] Thus, there arises a need for a system and method for determining performance characteristics of a depth camera for a specific application.
SUMMARY OF INVENTION
[5] Particular embodiments of the present disclosure are described herein below with reference to the accompanying drawings; however, it is to be understood that the disclosed embodiments are mere examples of the disclosure, which may be embodied in various forms. Well-known functions or constructions are not described in detail to avoid obscuring the present disclosure in unnecessary detail. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present disclosure in virtually any appropriately detailed structure.
[6] The present disclosure relates to a system and a method for assessing performance of a depth camera. In an embodiment, the system includes an imaging unit including a computing unit communicatively coupled to a depth camera, and an assessment unit including a processor communicatively coupled to the computing unit. The processor is configured to obtain an identifier of the depth camera and a target application for the depth camera. The processor is further configured to determine an imaging set-up and a performance characteristic associated with the target application, wherein the imaging set-up includes a desired position and a desired orientation of the depth camera, a desired set of objects, desired positions of the desired set of objects, and desired surrounding conditions. The processor is further configured to communicate a set of instructions to be executed corresponding to the imaging set-up to the computing unit, wherein the set of instructions comprise instructions to capture one or more images of the desired set of objects using the depth camera. The processor is further configured to receive, from the computing unit, point clouds or depth maps corresponding to the one or more images captured by the depth camera based upon the set of instructions. The processor is further configured to calculate a performance parameter based upon the point clouds or the depth maps corresponding to the one or more images, the performance parameter being indicative of the performance characteristic. The processor is further configured to compare the performance parameter with a threshold. The processor is further configured to provide a recommendation to the user indicating usability of the depth camera for the target application based upon the comparison. Based upon the set of instructions, the computing unit is configured to control the depth camera to capture the one or more image, record the point clouds or the depth maps corresponding to the one or more images, and communicate the point clouds or the depth maps corresponding to the one or more images to the processor.
[7] In an embodiment, the method includes obtaining, by a processor of an assessment unit, an identifier of a depth camera and a target application for the depth camera. The method further includes determining, by the processor, an imaging set-up and a performance characteristic associated with the target application, wherein the imaging set-up includes a desired position and a desired orientation of the depth camera, a desired set of objects, desired positions of the desired set of objects, and desired surrounding conditions. The method further includes communicating, by the processor, a set of instructions to be executed corresponding to the imaging set-up to a computing unit of an imaging unit, wherein the set of instructions comprise instructions to capture one or more images of the desired set of objects using the depth camera. The computing unit is communicatively coupled to the depth camera. The method further includes receiving, by the processor from the computing unit, point clouds or depth maps corresponding to the one or more images captured by the depth camera based upon the set of instructions. The method further includes calculating, by the processor, a performance parameter based at least upon the point clouds or the depth maps corresponding to the one or more images, the performance parameter being indicative of the performance characteristic. The method further includes comparing, by the processor, the performance parameter with a threshold. The method further includes providing, by the processor, a recommendation to the user indicating usability of the depth camera for the target application based upon the comparison.
BRIEF DESCRIPTION OF DRAWINGS
[8] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the apportioned drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the disclosure is not limited to specific methods and instrumentality disclosed herein. Moreover, those in the art will understand that the drawings are not to scale.
[9] Fig. 1A depicts a system 100 for assessing performance of a depth camera, according to an embodiment of the present disclosure.
[10] Fig. 1B depicts a flow chart of a method 150 for assessing performance of a depth camera, according to an embodiment of the present disclosure.
[11] Fig. 2A depicts an exemplary imaging setup 200 to capture images of objects using a depth camera to assess performance of a depth camera for an application, according to an embodiment of the present disclosure.
[12] Fig. 2B depicts an exemplary imaging setup 300 to capture images of objects using a depth camera to assess performance of the depth camera for an application, according to an embodiment of the present disclosure.
[13] Fig. 2C depicts an exemplary imaging setup 400 to capture images of objects using a depth camera to assess performance of the depth camera for an application, according to an embodiment of the present disclosure.
[14] Fig. 2D depicts an exemplary imaging setup 500 to capture images of objects using a depth camera to assess performance of the depth camera for an application, according to an embodiment of the present disclosure.
[15] Fig. 2E depicts an exemplary imaging setup 600 to capture images of objects using a depth camera to assess performance of the depth camera, according to an embodiment of the present disclosure.
[16] Fig. 2F depicts an exemplary imaging setup 700 to capture images of objects using a depth camera to assess performance of the depth camera, according to an embodiment of the present disclosure.
[17] Fig. 3 depicts a flow chart of a method 800 for assessing a spatial accuracy of a depth camera, according to an embodiment of the present disclosure.
[18] Fig. 4 depicts a flow chart of a method 1000 for assessing a temporal accuracy of a depth camera, according to an embodiment of the present disclosure.
[19] Fig. 5 depicts a flow chart of a method 1200 for assessing a motion impact on a depth camera, according to an embodiment of the present disclosure.
[20] Fig. 6 depicts a flow chart of a method 1400 for assessing impact of lighting conditions on a depth camera, according to an embodiment of the present disclosure.
[21] Fig. 7 depicts a flow chart of a method 1600 for assessing impact of shiny and transparent surfaces on a depth camera, according to an embodiment of the present disclosure.
[22] Fig. 8 depicts a flow chart of a method 1800 for assessing object separation capability of a depth camera, according to an embodiment of the present disclosure.
[23] Fig. 9 depicts a flow chart of a method 2000 for assessing depth quality of various object shapes and structures of a depth camera, according to an embodiment of the present disclosure.
[24] Fig. 10 depicts a flow chart of a method 2200 for assessing a bin picking scenario of a depth camera, according to an embodiment of the present disclosure.
[25] Fig. 11 depicts a flow chart of a method 2400 for assessing SLAM and navigation of a depth camera, according to an embodiment of the present disclosure.
[26] Fig. 12 depicts a flow chart of a method 2600 for assessing measurement estimation of object surfaces by a depth camera, according to an embodiment of the present disclosure.
[27] Fig. 13 depicts a flow chart of a method 2800 for assessing estimation of a moving object with respect to a depth camera, according to an embodiment of the present disclosure.
[28] Fig. 14 depicts a flow chart of a method 3000 for assessing counting of objects with respect to a depth camera, according to an embodiment of the present disclosure.
DETAILED DESCRIPTION OF ACCOMPANYING DRAWINGS
[29] Prior to describing the disclosure in detail, definitions of certain words or phrases used throughout this patent document will be defined: the terms "include" and "comprise", as well as derivatives thereof, mean inclusion without limitation; the term "or" is inclusive, meaning and/or; the phrases "coupled with" and "associated therewith", as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have a property of, or the like. Definitions of certain words and phrases are provided throughout this patent document, and those of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior as well as future uses of such defined words and phrases.
[30] Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment, but mean “one or more but not all embodiments” unless expressly specified otherwise. The terms “including,” “comprising,” “having,” and variations thereof mean “including but not limited to” unless expressly specified otherwise. An enumerated listing of items does not imply that any or all of the items are mutually exclusive and/or mutually inclusive, unless expressly specified otherwise. The terms “a,” “an,” and “the” also refer to “one or more” unless expressly specified otherwise.
[31] Although the operations of exemplary embodiments of the disclosed method may be described in a particular, sequential order for convenient presentation, it should be understood that the disclosed embodiments can encompass an order of operations other than the particular, sequential order disclosed. For example, operations described sequentially may in some cases be rearranged or performed concurrently. Further, descriptions and disclosures provided in association with one particular embodiment are not limited to that embodiment, and may be applied to any embodiment disclosed herein. Moreover, for the sake of simplicity, the attached figures may not show the various ways in which the disclosed system, method, and apparatus can be used in combination with other systems, methods, and apparatuses.
[32] The embodiments are described below with reference to block diagrams and/or data flow illustrations of methods, apparatus, systems, and computer program products. It should be understood that each block of the block diagrams and/or data flow illustrations, respectively, may be implemented in part by computer program instructions, e.g., as logical steps or operations executing on a processor in a computing system. These computer program instructions may be loaded onto a computer, such as a special purpose computer or other programmable data processing apparatus to produce a specifically-configured machine, such that the instructions which execute on the computer or other programmable data processing apparatus implement the functions specified in the data flow illustrations or blocks.
[33] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the functionality specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions that execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the data flow illustrations or blocks.
[34] Accordingly, blocks of the block diagrams and data flow illustrations support various combinations for performing the specified functions, combinations of operations for performing the specified functions and program instructions for performing the specified functions. It should also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions. Further, applications, software programs or computer readable instructions may be referred to as components or modules. Applications may be hardwired or hardcoded in hardware or take the form of software executing on a general-purpose computer such that when the software is loaded into and/or executed by the computer, the computer becomes an apparatus for practicing the disclosure, or they are available via a web service. Applications may also be downloaded in whole or in part through the use of a software development kit or a toolkit that enables the creation and implementation of the present disclosure. In this specification, these implementations, or any other form that the disclosure may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the disclosure.
[35] Furthermore, the described features, advantages, and characteristics of the embodiments may be combined in any suitable manner. One skilled in the relevant art will recognize that the embodiments may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments. These features and advantages of the embodiments will become more fully apparent from the following description and apportioned claims, or may be learned by the practice of embodiments as set forth hereinafter.
[36] Embodiments of the present disclosure relate to an assessment system (or system) for measuring and assessing the performance of a depth camera. The system determines an appropriate imaging set-up including a set of objects, based upon a target application for which the performance of the depth camera is to be assessed. Further, once the imaging set-up is determined, the system is configured to execute a plurality of instructions to capture one or more images of the set of objects and records point clouds and/or depth maps of the one or more images. The assessment system thereafter calculates one or more performance parameters indicating performance characteristics of the depth camera based upon the point clouds and/or the depth maps recorded in the selected imaging set-up for the target application. The assessment system thereafter compares the one or more performance parameters with respective thresholds and provides an assessment and/or a recommendation to a user indicating the usability/suitability of the depth camera for the target application. The thresholds may be determined based upon simulated point cloud/depth data for the target application.
[37] The proposed system presents several advantages. Since the system determines a suitable imaging set-up based upon the target application, the system enables the user to assess the performance of the depth camera in a more realistic environment in contrast to conventional systems where only a generic performance of a depth camera is provided. Consequently, the proposed system provides a more accurate performance assessment of a given depth camera for a specific application. Further, the system enables the user to compare performance of two depth cameras for various applications, thereby providing a more systematic and holistic comparison of the depth cameras. The system can also save the performance characteristics for multiple cameras under different imaging-set ups for different applications. As a result, the system also has a capability to recommend a depth camera from among multiple depth cameras for a specific application or a use case. The system can also generate standardized datasets (using various depth cameras and various imaging set-ups), which can be utilized by researchers in other relevant domains.
[38] Fig. 1A illustrates a system 100 for assessing performance of a depth camera, according to an embodiment of the present disclosure. The system 100 includes, without limitation, an assessment unit 10, an imaging unit 20 and a user device 30. The assessment unit 10 includes a processor 10a and a memory 10b. The imaging unit 20 includes a computing unit 20a. The processor 10a is communicatively coupled to the computing unit 20a.
[39] In an exemplary embodiment, the processor 10a is configured to obtain an identifier of a depth camera to be assessed and an application for which the depth camera is to be assessed (hereinafter a target application). In an embodiment, the processor 10a obtains the identifier of the depth camera and/or the target application from a user of the system 100. The user may enter the identifier of the depth camera and the target application via a user interface presented on the user device 30. The identifier of the depth camera includes, without limitation, a model number, a product identification number, a product name, etc. The target application includes, without limitation, bin picking, slam and navigation, measurement estimation of object surfaces, speed estimation of a moving object, counting of objects, etc. in different fields such as, autonomous vehicles, robotics, warehouses, medical fields, IoT devices, etc.
[40] Upon receiving the unique identifier of the depth camera and the target application, the processor 10a is configured to determine an imaging set-up and one or more performance characteristic associated with the target application. The performance characteristics include, without limitation, spatial accuracy, temporal accuracy, performance under varying light conditions, motion impact, depth analysis of shiny and transparent objects, depth analysis of various object shapes and structures, object separation capability, etc. The processor 10a may also be configured to determine one or more performance parameters indicative of the performance characteristics. The imaging set-up includes a desired set of objects, desired positions of the set of objects, desired surrounding conditions, a desired position and a desired orientation of the depth camera, etc. The one or more performance characteristics of the depth camera to be assessed include, without limitation, efficiency, accuracy, precision, reliability, processing rate, etc. of the depth camera. In an embodiment, the one or more performance parameters are statistical parameters, for example, a mean, a median, a mode, a standard deviation, a mean absolute error, a root mean square error, etc. In an embodiment, the one or more performance parameters includes counting accuracy of the number of objects counted by the depth camera. The imaging set-up and the performance characteristics are carefully designed based upon realistic scenarios and conditions that are likely to be encountered by the depth camera (e.g., varying lighting conditions, objects of different shapes, sizes and textures, motion of the depth camera and/or objects, and so on) for a given application and hence, may differ for different applications.
[41] The desired surrounding conditions include without limitation, desired lighting conditions, desired type of room, desired type of carpet/floor, desired position or desired speed of static or moving vehicle, etc. The desired lighting conditions include different intensity of lights for example, (0-30) lux, (300-500) lux, (2000-4000) lux surrounding the depth camera/ambient light, etc. The type of room includes, without limitation, an empty room having a flat wall with a logo on it, a room with a white non-shiny curtain, a room with a black non-shiny curtain, etc. The different types of carpets/floors include, without limitation, non-shiny black floor, non-shiny white floor, aluminum shiny floor etc. The static or moving vehicle includes, without limitation, an autonomous mobile robot (AMR).
[42] The processor 10a is configured to select a set of instructions to be executed corresponding to the imaging set-up and the one or more performance parameters determined for the target application. The processor 10a is further configured to communicate details about the determined imaging set-up, the set of instructions to the computing unit 20a of the imaging unit 20. In an embodiment, the processor 10a determines the imaging set-up, the one or more performance characteristics, the one or more performance parameters and selects the set of instructions based upon a look-up table mapping a plurality of applications with corresponding imaging set-ups, performance characteristics, performance parameters and sets of instructions. One example of such a look-up table is provided later.
[43] The imaging unit 20 includes, without limitation, a depth camera, a mounting device, a motorized camera slider, one or more objects (collectively, objects), one or more lighting devices (collectively, lighting devices), one or more carpets, one or more curtains, a mobile robot, an autonomous mobile robot (AMR) etc. The depth camera, the mounting device, the motorized camera slider mechanism and the objects may be individually referred to as devices of the imaging unit 20. The depth camera is communicatively coupled to the computing unit 20a.
[44] The depth camera includes a camera capable to measuring depth of a scenario. The depth camera may include an active depth sensor or a passive depth sensor. In the active depth sensor, a source illuminates an object using a light signal, for example, a laser light or an infrared (IR) signal and the reflection is captured by a receiver. The actual distance of the object is measured by calculating the time taken by the light signal from the source to the receiver (i.e., the round-trip time). Examples of the active depth sensors include Light Detection and Ranging (LiDAR) sensors, Time of Flight (ToF) sensors, etc. The passive depth sensors typically include two imaging sensors, for example, a left and a right imaging sensor, to capture respective images of the object. The distance of the object is calculated using pixel disparity between the images captured by the two imaging sensors. Stereoscopic cameras are an example of the passive depth sensors. Cameras having other types of depth sensing techniques that provide depth information of the object can also be used without deviating from the scope of the present disclosure.
[45] The mounting device is an apparatus for mounting the depth camera. It includes multiple legs to support a surface on which the depth camera is placed. The height of the surface may be adjustable. Further details of the mounting device are provided later.
[46] The motorized camera slider mechanism is used to move the depth camera to a desired location/position.
[47] The characteristics of the objects included in the imaging unit 20 include, without limitation, a transparent surface like a glass, a reflective surface like a metal, a shape (e.g., cylindrical, cubical, conical, round, flat, etc.), a size between 2 cm to 100 cm (say, a diameter between 2 cm to 100 cm, a length between 2 cm to 100 cm), surface texture (e.g., smooth surface, dimpled surface, etc.), etc. Exemplary objects include a traffic cone, a chair, a ball, a paper roll, a vase, a wine glass, a metal bowl, a cuboid box, a pyramid crate, a flowerpot, a curtain, a bin, an aluminum cover, a mirror, a glass, a golf ball, a display device, etc. The desired set of objects includes one or more of these objects.
[48] The computing unit 20a is configured to execute the set of instructions sent by the processor 10a of the assessment unit 10. In an embodiment, the set of instructions includes instructions to capture one or more images of the set of objects using the depth camera. The set of instructions may further include one or more of: instructions associated with the surrounding conditions, instructions associated with desired positions of the desired set of objects, instructions associated with the desired orientation and the desired position of the depth camera, etc.
[49] Based upon the set of instructions, the computing unit 20a is configured to control various devices of the imaging unit 20 to realize the imaging set-up. For example, based upon the set of instructions, the computing unit 20a is configured to send a control signal to a controller which is interfaced with a lighting device, to change intensity of the light emitted by the lighting device according to the desired lighting conditions, send a control signal to a controller, which is interfaced with an object of the desired set of objects, to change its position to a corresponding desired position, send a control signal to controller, which is interfaced with the mounting device to control an orientation of the depth camera to the desired orientation, send a control signal to a controller interfaced with a motorized camera slider to change a position of the depth camera to the desired position and so on.
[50] The computing unit 20a is configured to control the depth camera to capture the one or more images at a desired time, record point clouds and/or depth maps corresponding to the one or more images captured by the depth camera and associated timestamps, and communicate the recorded point clouds and/or depth maps and the timestamps to the processor 10a based upon the set of instructions.
[51] The computing unit 20a may be a microcontroller, a microprocessor, an application specific integrated circuit (ASIC), etc. The computing unit 20a is communicatively coupled with the depth camera using, for example, a universal serial bus (USB) interface, a network interface (e.g., an interface of a local area network, a Bluetooth interface, a Wi-Fi, etc.). The computing unit 20a may be coupled to controllers of various devices via suitable interfaces, for example, the controller of the motorized cameral slider can be coupled via a USB or microUSB interface, a controller of the mobile robot can be coupled via a Wi-Fi interface, the controller of the lighting device can be coupled via a USB interface and so on. The imaging unit 20 may also include a memory (not shown) to store the set of instructions received from the processor 10a. The computing unit 20a retrieves the set of instructions from the memory for execution.
[52] The processor 10a is configured to receive the point clouds and/or depth maps and the associated timestamps from the computing unit 20a. The processor 10a is configured to calculate the one or more performance parameters based at least upon the point clouds and/or depth maps corresponding to the one or more image. The processor 10a is configured to compare the one or more performance parameters with respective thresholds and provide a recommendation to the user indicating the usability of the depth camera for the target application. In an embodiment, the thresholds are determined based upon simulated, expected and/or ideal performance of the depth camera for the target application. For example, the thresholds may be values of the one or more performance parameters calculated using simulated, expected and/or ideal performance. In an embodiment, the thresholds may be the simulated point clouds and/or depth maps. As a result, the processor 10a compares the point clouds and/or depth maps corresponding to the one or more images with the simulated point clouds and depth maps. In an embodiment, the thresholds may be the point clouds and/or depth maps or the performance parameters calculated for a different depth camera. This enables the processor 10a to compare the performance of the two depth cameras under identical imaging set-ups and provide a comparison and an associated recommendation to the user. The processor 10a may also be configured to display the results on the user device 30 for the user’s references and/or further action.
[53] The memory 10b of the assessment unit 10 may store various data. The stored data may include one or more identifiers for a plurality of depth cameras, a look-up table mapping a plurality of applications with corresponding imaging set-ups, performance characteristics, performance parameters and sets of instructions, the data received from the computing unit 20a, calculated performance parameters for a given depth camera, historical trends of performance characteristics of various depth cameras, etc. The memory 10b may be any suitable computer readable storage medium. The memory 10b may be, for example, but not limited to, volatile memory, such as static random-access memory (SRAM) and dynamic random-access memory (DRAM), and/or a non-volatile memory, such as read-only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, the assessment unit 10 may be communicatively coupled with a database for storing all or part of the data of the assessment unit 10. The database may be of any type such as relational database, non-relational database, object-orientated database, a cloud-storage database, etc.
[54] The processor 10a may be a general-purpose computer, a special-purpose computer, a server, a microprocessor, an application specific integrated circuit (ASIC), etc., or any other suitable computing device.
[55] The user may access various functionalities of the assessment unit 10 using the user device 30. The user device 30 may include, without limitation, a personal computer, a laptop, a notebook, a handheld computer, a set-top box, a PDA, a mobile phone and the like. The user device 30 includes a processor, a memory one or more input devices (keyboard, mouse, touchpad, etc.) allowing the user to provide input to the assessment unit 10, and one or more displays allowing the user to view various data and/or results of the assessment unit 10. In an embodiment, a suitable application (for example, a web application, a mobile application, a local application or the like) may be installed on the user device 30, which enables the user to interact with the assessment unit 10. In an embodiment, the user device 30 may interact with the assessment unit 10 via a web browser installed on the user device 30. One or more user interfaces may be provided to the user on the user device 30 to provide one or more inputs to the assessment unit 10 and/or view results of the assessment unit 10.
[56] The assessment unit 10 and the user device 30 may communicate with each other over a network 40. The network 40 may be a wired network, a wireless network or combination thereof. Examples of the network 40 include, without limitation, a local area network (LAN), a wide area network (WAN), Internet, Wi-Fi, cellular network (e.g., 2G, 3G, 4G, 5G, etc.) and the like. The assessment unit 10 and the user device 30 may include suitable hardware and software for communicating over the network 40.
[57] The assessment unit 10 and the imaging unit 20 may communicate with each other over a network 60. The network 60 may be a wired network, a wireless network or combination thereof. Examples of the network 60 include, without limitation, a local area network (LAN), a wide area network (WAN), Internet, Wi-Fi, cellular network (e.g., 2G, 3G, 4G, 5G, etc.) and the like. The assessment unit 10 and the imaging unit 20 may include suitable hardware and software for communicating over the network 60. The network 40 and the network 60 may be the same network or different networks.
[58] In an example deployment scenario, the assessment unit 10 is implemented on a server. The server may reside locally (e.g., within the same premises where the imaging unit 20 is deployed) or may be a remote server accessible over the network 40 and/or the network 60 (e.g., a cloud server). In another example deployment scenario, the assessment unit 10 may be co-located with the imaging unit 20. For example, the assessment unit 10 may be implemented on a general-purpose computer interfaced with the computing unit 20a via a suitable communication interface or the processor 10a and the computing unit 20a may be the same. In yet another deployment scenario, the assessment unit 10 may be implemented on the user device 30, when the processor of the user device 30 has sufficient processing capabilities. In yet another deployment scenario, the assessment unit 10 may be deployed as ‘software-as-a-service’. Other deployment scenarios are also contemplated herein.
[59] Fig. 1B illustrates a flowchart of a method 150 for assessing performance of a depth camera in accordance with an embodiment of the present disclosure. At step 102, an identifier of the depth camera and an application for which the depth camera is to be assessed (hereinafter a target application) are obtained by the processor 10a of the assessment unit 10. The identifier of the depth camera includes, without limitation, one or more of: a model number, a product identification number, a product name, etc. Further, the target application includes without limitation, monitoring, tracking, inspecting, counting objects, surface measurements etc. in different fields such as autonomous vehicles, robotics, warehouses, medical fields, IoT devices etc. The user may provide identifier of the depth camera and the target application through a corresponding user interface provided on the user device 30.
[60] At step 104, upon receiving the identifier of the depth camera and the target application, a necessary imaging set-up and one or more performance characteristics to be assessed for the target application are determined by the processor 10a. The imaging set-up includes a desired set of objects, desired positions of the set of objects, desired surrounding conditions, a desired position and a desired orientation of the depth camera, etc. A set of instructions to be executed corresponding to the imaging set-up are also selected by the processor 10a. In an embodiment, the processor 10a determines the imaging set-up, the one or more performance characteristics, the one or more performance parameters and selects the set of instructions based upon a look-up table mapping a plurality of applications with corresponding imaging set-ups, performance characteristics, performance parameters and sets of instructions. The memory 10b of the assessment unit 10 may include the look-up table. One such look-up table is provided in Table 1 below. It should be appreciated that the table below is merely exemplary and should not be considered as limiting.
Application Performance characteristic Performance parameter Imaging set-up Instructions
Measurement of object surfaces Accuracy Mean
Error between measured and actual dimensions Camera height: 47mm;
Camera angle: 39-degree downward;
Camera distance: 75 – 92 cm;
Objects: traffic cone, paper roll, box;
Floor type: white non-shiny;
Lighting: 380 – 440 lumens; Instructions to adjust lighting value;
Instructions to control camera distance, height and angle;
Instructions to capture 10 images
SLAM and Navigation Accuracy of pose estimation Root mean square error Room type: closed;
Object: flower pots in corners of the room;
Floor type: shiny/non-shiny;
Wall type: white non-shiny;
Lighting: 380 – 440 lumens;
Camera: mount on an AMR Instructions to navigate the AMR through a desired trajectory;
Instructions to capture images at each pre-defined interval;

Table 1
[61] At step 106, the processor 10a communicates the set of instructions to be executed corresponding to the imaging setup to the computing unit 20a of the imaging unit 20. In an embodiment, the set of instructions includes instructions to capture one or more images of the set of objects using the depth camera (e.g., the number of images to be taken, time criteria at which the images are to be taken, etc.). The set of instructions may also include instructions related to moving required device(s) to desired position(s), controlling device(s) for the necessary condition (e.g., controlling intensity of lights, choosing an appropriate room, closing curtains, etc.), placement of the desired set of objects, etc. For example, the set of instructions may further include one or more of: instructions associated with the surrounding conditions, instructions associated with desired positions of the desired set of objects, instructions associated with the desired orientation and the desired position of the depth camera, etc. Thus, the set of instructions enable the computing unit 20a to realize the imaging set-up. The set of instructions may also include instructions related to pre- and post-processing to be performed on recorded data. The processor 10a may also communicate information related to the imaging set-up and the performance parameters to the computing unit 20a.
[62] The computing unit 20a runs the set of instructions with the imaging set-up to obtain an output. The computing unit 20a realizes the imaging set-up by manipulating and/or configuring necessary devices of the imaging unit 20 and surrounding conditions for example, adjusting conditions of lighting, the position, height, orientation and the movement of the depth camera, placement of the desired set of objects, movement of the static and/or mobile robot, adjusting curtains/carpets, etc. In an embodiment, the computing unit 20a realizes the imaging set-up by sending appropriate control signal(s) to respective controllers based upon the set of instructions as explained earlier. In an embodiment, the computing unit 20a may realize the imaging set-up automatically based upon the set of instructions. In another embodiment, the imaging set-up may be realized at least partially through human assistance. For example, the processor 10a (or the computing unit 20a) may display the details of the imaging set-up to the user on the user device 30. The details may include the set of desired objects, their desired positions, the desired position and the desired orientation of the depth camera, the desired surrounding conditions (such as, lighting conditions, carpets, type of room, texture of curtains), etc. The user may then select and place the desired objects as displayed whereas the computing unit 20a may adjust the mounting device automatically to have a desired position, height and orientation of the depth camera.
[63] Further, by executing the set of instructions, the computing unit 20a controls the depth camera placed on the mounting device to take one or more images of the desired set of objects placed in front of it (at a desired time) and record the point clouds and/or depth maps corresponding to the images. In an embodiment, the computing unit 20a records the point clouds and/or depth maps and associated timestamps in corresponding bag files. Further, the set of instructions enable the computing unit 20a to control the depth camera by placing it at pre-defined distances and record the point clouds of an object placed in front of it. The computing unit 20a communicates the recorded data to the processor 10a. The recorded data may include the bag files (containing the point clouds and/or depth maps), timestamp, etc. Based upon the set of instructions, the computing unit 20a may also performs post-processing on the recorded data. Examples of post-processing include, without limitation, extracting required data from the bag files, removing noise, sorting the data, etc. Based upon the set of instructions, the computing unit 20a communicates the recorded data to the processor 10a.
[64] The processor 10a receives the recorded data. At step 108, the processor 10a calculates one or more performance parameters based at least upon the point clouds and/or the depth maps. The one or more performance parameters may be one or more statistical parameters. The statistical parameters include, without limitation, mean, standard deviation, mean square error, etc. A few exemplary embodiments of realizing the imaging set-ups and calculating the one or more performance parameters for different performance characteristics and/or different applications are explained with respect to Figs. 3 – 14.
[65] At step 110, the processor 10a compares the calculated performance parameters with respective thresholds. The thresholds may be determined using ideal/expected/simulated performance values associated with the target application. The thresholds and/or the ideal/expected/simulated performance values may be pre-stored in the memory 10b of the assessment unit 10 and/or a database coupled to the assessment unit 10. The ideal or simulated performance values may be obtained by calculating values of the performance parameters using a simulated point cloud/depth maps determined based upon a simulated imaging set-up corresponding to imaging set-up (using for example, digital twin technology).
[66] At step 112, the processor 10a provides a recommendation to the user indicating usability/suitability of the depth camera for the target application based upon the comparison. For example, the processor 10a may recommend a given depth camera for counting objects application if the counting accuracy is greater than or equal to a corresponding threshold value. In another example, the processor 10a may recommend a given depth camera for SLAM and navigation application if a root mean square error (RMSE) calculated based upon optimized poses and ground truth poses is less than a corresponding threshold. In an embodiment, the processor 10a may display the points clouds, the calculated values of the one or more parameters, corresponding thresholds, the recommendation, etc. on the user device 30.
[67] As disclosed above, the imaging setup and performance parameters to be calculated vary from application to application. A few exemplary embodiments of imaging set-ups have been described below.
[68] Fig. 2A depicts an exemplary imaging setup 200 in accordance with a first embodiment of the present disclosure. The imaging setup 200 relates to performance characteristics such as spatial accuracy, and temporal accuracy of a depth camera for various applications. The imaging setup 200 comprises a depth camera 220, a measuring scale 202, a mounting device 208 for the depth camera 220, and a motorized camera slider 210 mounted on metal rods to move the mounting device 208 from one end to another end.
[69] The measuring scale 202 is used to plot the pre-defined distance from where the image has to be taken by the depth camera 220.
[70] The mounting device 208 is a multi-legged camera mounting device. The mounting device 208 comprises a mounting platform assembly at its top and base at its bottom. The top end of the mounting platform assembly is adapted to fix a camera or an image capturing device thereon, and the angle of the mounting platform assembly can be adjusted. The base includes support legs. The platform and base are assembled together through adjusting means. The mounting device 208 can be made up of materials without any limitation like plastic, aluminum, steel, polymer, wood, carbon fiber etc. The height of the mounting device 208 can be adjusted as per the requirements for capturing the image. In an embodiment, the mounting device 208 is a tripod. In the depicted embodiment, the mounting device 208 includes two tripods with each tripod placed close to opposite walls as shown. A metal rod extends between the two tripods.
[71] The motorized camera slider 210 moves the depth camera 220 from one end to another. In an embodiment, the computing unit 20a sends at least one control signal to a controller coupled to the motorized camera slider 210 to drive the motorized camera slider 210 and thereby, the depth camera 220 to a desired location based upon the set of instructions.
[72] The depth camera 220 is used to capture images of the objects present in the imaging setup 200.
[73] Fig. 2B depicts an exemplary imaging setup 300 in accordance with a second embodiment of the present disclosure. The imaging setup 300 relates to performance characteristics such as determining depth accuracy of a depth camera for shiny and transparent surfaces, depth quality of the depth camera for various object shapes and structures, and impact of lighting conditions for various applications. The imaging setup 300 comprises a depth camera 320, a mounting device 308 for the depth camera 320, a non-shiny floor (or carpet) 310, a glass 314, a display device 316, and a metal bowl 318.
[74] The non-shiny carpet 310 may be made up of materials such as without any limitation polyester, silk, rayon, cotton, wool, etc. The glass 314 may be of any shape. In an embodiment, the glass 314 is made of a transparent material, such as, glass, crystal glass, transparent plastic, etc. The display device 316 includes a light emitting element. The display device 316 may have any suitable shape and size. The metal bowl 318 can be made up of materials such as without any limitation stainless steel, aluminum, copper, brass, iron, etc.
[75] Fig. 2C depicts an exemplary imaging setup 400 in accordance with a third embodiment of the present disclosure. The imaging setup 400 relates to performance characteristics such as depth accuracy of a depth camera during motion impact for an application such as SLAM and navigation. The imaging setup 400 includes a closed room 450, flowerpots 404, and a depth camera 420.
[76] The closed room 450 has a floor and four walls. The floor and the walls may have any color or texture. The flowerpots 404 may have any suitable shape and dimensions. The flowerpots 404 can be made up of materials including without any limitation glass, porcelain, metal, ceramic, plastic, etc. In an embodiment, the flowerpots 404 may be placed along one or more walls or at any suitable location within the closed room 450. Though the depth camera 420 has been shown to be held by a user in the depicted embodiment, the depth camera 420 may be mounted on a mounting device (not shown) or an autonomous moving robot (AMR) (not shown) to vary the position of the depth camera 420 as described earlier.
[77] Fig. 2D depicts an exemplary imaging setup 500 in accordance with a fourth embodiment of the present disclosure. The imaging setup 500 relates to performance characteristics such as depth accuracy of a depth camera for an application such as bin picking. The imaging setup 500 includes a bin 502, golf balls 504, conical object (e.g., traffic cones 506), a depth camera 520, and a mounting device 508. The depth camera 520 is mounted on the mounting device 508 such that the depth camera 520 faces downward.
[78] The golf balls 504 and the traffic cones 506 are placed inside the bin 502. The bin 502 may have any suitable shape and size. The golf balls 504 and the traffic cones 506 can be of any suitable sizes and colors. The bin 502 is placed on a non-shiny floor 510.
[79] Fig. 2E depicts an exemplary imaging setup 600 in accordance with a fifth embodiment of the present disclosure. The imaging setup 600 relates to performance characteristics such as efficiency of a depth camera for object separation capability for various applications. The imaging set-up 600 includes a depth camera 620, a mounting device 608, boxes 602 of different sizes and colors, a non-shiny black floor 610 and/or an aluminum shiny floor (not shown). The mounting device 608 is mounted on one of walls of a room such that the depth camera 620 faces vertically downward.
[80] The boxes 602 may be made up of materials, without limitation, like paper, plastic, wood, cardboard, etc. The boxes 602 may be used with and without an aluminum cover. In an embodiment, the boxes 602 are rectangular.
[81] Fig. 2F depicts an exemplary imaging setup 700 in accordance with a sixth embodiment of the present disclosure. The imaging setup 700 relates to performance characteristics such as depth accuracy of a depth camera for applications such as counting objects, measurement estimation of object surfaces and speed estimation of a moving object. The imaging set-up 700 includes a depth camera 720, a mounting device 708 for the depth camera 720, a floor 712 of types: non-shiny black floor, non-shiny white floor, shiny aluminum floor, golf balls 706, and a mirror 710 behind the golf balls 706. In an embodiment, the imaging set-up 700 may further include another depth camera 722 enabling a more direct comparison of the performance of the two depth cameras.
[82] The golf balls 706 may be of suitable dimension and may be made of materials including without any limitation rubber, plastic, surlyn, urethane etc. The mirror 710 has suitable dimensions and shape.
[83] Fig. 3 depicts an exemplary flowchart of a method 800 with respect to instructions executed by the computing unit 20a and/or the processor 10a for assessing spatial accuracy of a depth camera. At step 802, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 0.5m, 1m, 1.5m, 2m, 2.5m, 3m, 3.5m, 4m, etc. for placing the mounting device on which the depth camera is mounted to capture images of the objects (e.g., flat objects like a wall) placed in front of the depth camera.
[84] At step 804, the computing unit 20a places the depth camera at a first determined distance to obtain a set of images (say, 5 – 10 images). The computing unit 20a is configured to activate the depth camera to record the rosbag of images at the first determined distance of an identifier (e.g., a logo) present on the wall. The computing unit 20a sends the rosbag of the images to the processor 10a.
[85] At step 806, the processor 10a extracts region of interest (ROI) for an image. For example, the processor 10a calculates an image center using the height and the width of the image. Thereafter, the processor 10a extracts four corners of region of interest (ROI) using the image center and the size of ROI. The size of the ROI may be 10x10 pixels in an embodiment. The image center may be determined based upon dimensions of the image.
[86] At step 808, the processor 10a calculates mean depth and standard deviation of the extracted ROI at the first determined distance. The steps 802 to step 808 are repeated for each and every image. After performing the above-mentioned steps for each image of the set of images, at step 810, the processor 10a estimates the mean of the mean depth for the set of images and mean of the standard deviation for the set of images at the first determined distance.
[87] At step 812, the processor 10a compares the mean of the mean depth and the mean of the standard deviations for the first determined distance with the corresponding threshold. The processor 10a may also plot a graph of mean vs number of images and standard deviation vs number of images. At step 814, the steps from 802-812 are repeated the remaining determined distances. When the mean of the mean depth and the mean of the standard deviations are less than respective thresholds, the processor 10a determines that the depth camera has the spatial accuracy required for the target application.
[88] Fig. 4 depicts an exemplary flowchart of a method 1000 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing temporal accuracy of a depth camera. At step 1002, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 0.5m, 1m, 1.5m, 2m, 2.5m, 3m, 3.5m, 4m, etc. for placing the mounting device on which the depth camera is mounted to capture images of the objects (e.g., flat objects like a wall) placed in front.
[89] At step 1004, the computing unit 20a places the depth camera at a first determined distance to obtain a set of images (say, 5 – 10 images). The computing unit 20a is configured to activate the depth camera to record the rosbag of images at the determined distance of an identifier (e.g., a logo) present on the wall at a determined time span (e.g., every 20 minutes). The set of images taken at a given determined distance may be considered as a sample. The computing unit 20a sends the rosbag of the images to the processor 10a.
[90] At step 1006, the processor 10a calculates the image center using image height and width to extract four corners of ROI using the image center and size of the ROI.
[91] At step 1008, the processor 10a calculates mean depth and standard deviation of the extracted ROI at the first determined distance. The steps 1002 to step 1008 are repeated for each and every image present within the sample. After performing the above-mentioned steps for each and every sample, at step 1010, the processor 10a calculates the mean of mean and mean of standard deviation for each sample.
[92] At step 1012, the processor 10a compares the mean of mean and the mean of standard deviation with the respective thresholds. The steps 1004 – 1012 are then repeated for all samples. The processor 10a may also plot a graph of the mean of mean vs time and the mean of standard deviation vs time for all samples. When the mean of the mean depth and the mean of the standard deviations are less than respective thresholds, the processor 10a determines that the depth camera has the temporal accuracy required for the target application.
[93] Fig. 5 depicts an exemplary flowchart of a method 1200 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing motion impact of a depth camera. At step 1202, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 1m, 1.5m, 2m, 2.5m, 3m etc. for placing the depth camera on a mounting device to record bag files of a static scene.
[94] At step 1204, the computing unit 20a places the depth camera on a mounting device at the determined distance. The computing unit 20a thereafter activates the depth camera to record bag files of a static scene.
[95] At step 1206, the computing unit 20a instructs a user to move the depth camera at different speeds, e.g., high, medium, low. The user may move the depth camera in a random pattern or in a pre-defined pattern. The computing unit 20a records bag files at the different speeds. In an embodiment, the depth camera may be mounted on an AMR and the computing unit 20a may move the AMR as desired. The computing unit 20a sends the recorded set of data for the static scene and for scenes corresponding to the depth camera’s motion at the different speeds to processor 10a.
[96] At step 1208, the processor 10a calculates normalized sum of squared differences of consecutive frames in bag files recorded for the static scene and the scenes corresponding to the depth camera’s motion.
[97] At step 1210, the processor 10a compares the normalized sum of squared differences with a threshold to assess stability of the depth camera under motion. In this case, the smaller the normalized sum of squared differences the better is the performance of the depth camera.
[98] Fig. 6 depicts an exemplary flowchart of a method 1400 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing impact of lighting conditions on a depth camera. At step 1402, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 0.5m, 1m, 1.5m, 2m, 2.5m, 3m etc. for placing a mounting device on which the depth camera is mounted to capture images of the objects.
[99] At step 1404, the computing unit 20a places the desired set of objects for example, without any limitation, a traffic cone on a white non-shiny carpet, a chair on a white non-shiny carpet, and three golf balls on a white non-shiny carpet in front of the wall and a toy ball on a white non-shiny carpet.
[100] At step 1406, the computing unit 20a maneuvers the mounting device (e.g., via the motorized camera sliding mechanism) on which the depth camera is mounted and places the mounting device at the determined distance. The computing unit 20a thereafter activates the depth camera to capture a set of images (say 5 – 10 images) of the objects present in front of the wall and capture point cloud data of them.
[101] At step 1408, the computing unit 20a repeats the steps 1402-1406 under different lighting conditions i.e., at (0-30 lux) and (300-400 lux), and for (2000-4000 lux). The computing unit 20a controls the lighting by sending control signal(s) interfaced with the lighting devices as explained earlier. The computing unit 20a sends the captured data to the processor 10a.
[102] At step 1410, the processor 10a compares the point clouds of an image under the different lighting conditions with respective point clouds taken with a different depth camera and/or respective point simulated point clouds. In this case, respective point clouds taken with a different depth camera and/or respective point simulated point clouds are considered as the thresholds. Comparing the point clouds generated by one depth camera with the point clouds generated by the other depth camera enables the processor 10a to compare or benchmark performance of the depth cameras with each other and provide a recommendation regarding which depth camera may be better suited for the target application.
[103] Fig. 7 depicts an exemplary flowchart of a method 1600 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing depth quality of a depth camera for shiny and transparent surfaces. At step 1602, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 1m, 1.5m, 2m, 2.5m, 3m etc. for placing a mounting device on which the depth camera is mounted to capture images of the objects.
[104] At step 1604, the computing unit 20a places the desired set of objects, for example, without any limitation, a wine glass, a display screen, and a metal bowl on a white non-shiny carpet in front of the depth camera.
[105] At step 1606, the computing unit 20a maneuvers the mounting device on which the depth camera is mounted and places the mounting device at the determined distance. The computing unit 20a thereafter is configured to activate the depth camera to capture the set of images of the desired set of objects present in front it and record corresponding point clouds.
[106] At step 1608, the computing unit 20a repeats the steps 1602-1606 for different determined distances. The computing unit 20a sends the recoded data to the processor 10a.
[107] At step 1610, the processor 10a compares the point clouds with corresponding simulated point clouds for assessing depth quality of the depth camera for shiny and transparent surfaces. In this case, the simulated point clouds are considered as the thresholds. The simulated point clouds may be generated using the digital twin technology.
[108] Fig. 8 depicts an exemplary flowchart of a method 1800 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing object separation capability of a depth camera. At step 1802, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 1m, 1.5m, 2m, 2.5m, 3m etc. for placing a mounting device on which the depth camera is mounted to capture images of the objects.
[109] At step 1804, the computing unit 20a places the desired set of objects, for example, rectangular boxes (of three different sizes), or balls (of three different sizes), or balls (of three different sizes) with an aluminum foil placed under the depth camera. The set of objects may be in the FOV and distance range.
[110] At step 1806, the computing unit 20a maneuvers and places the mounting device on which the depth camera is mounted at the determined distance. The computing unit 20a thereafter activates the depth camera to capture the set of images of the desired set of objects. The computing unit 20a sends depth maps of the captured images to the processor 10a.
[111] At step 1808, the computing unit 20a repeats steps 1804-1806 for different distances from the depth camera.
[112] At step 1810, the processor 10a detects shapes of the desired set of objects for each determined distance based upon the depth map of the images. In an embodiment, the processor 10a generates a histogram of average depth based upon the depth map of the images and detects the shapes of the desired set of objects based upon the histogram.
[113] At step 1812, the processor 10a calculates a distance between adjacent objects based upon the detected shapes.
[114] At step 1814, the processor 10a compares the calculated distances with a threshold. In this case, the threshold may be equal to known distances between the set of objects.
[115] Fig. 9 depicts an exemplary flowchart of a method 2000 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing depth quality of a depth camera for various object shapes and structures. At step 2002, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 1m, 1.5m, 2m, 2.5m, 3m etc. for placing a mounting device on which the depth camera is mounted to capture images of the objects.
[116] At step 2004, the computing unit 20a places the desired set of objects, for example, without any limitation, cuboid boxes (of three different sizes), or balls (of three different sizes), or paper roll(s), or pyramid crate(s) on a white and a black non-shiny carpet.
[117] At step 2006, the processor 10a generates simulated CAD files for the above-mentioned objects which would be matched with real-time objects. The simulated CAD files are used to generate simulated point clouds corresponding to the desired set of objects.
[118] At step 2008, the computing unit 20a maneuvers and places the mounting device on which the depth camera is mounted at the determined distance. The computing unit 20a thereafter is configured to activate the depth camera to capture the set of images of the desired set of objects present in front of it. The desired set of objects may be placed one-by-one. For example, the cuboid boxes may be placed and the corresponding images taken. The balls may then be placed and corresponding images taken and so on.
[119] At step 2010, the computing unit 20a records the bag files containing the point clouds. The computing unit 20a sends the point clouds to the processor 10a.
[120] At step 2012, the processor 10a calculates a performance parameter indicating a similarity between the recorded points clouds and the corresponding simulated point clouds. Examples of the performance parameter include a Euclidean distance, a mean square error, a count of valid/invalid pixels, etc.
[121] At step 2014, the processor 10a compares the performance parameter with a threshold and assesses the performance characteristics based upon the comparison. For examples, a value of the Euclidean distance or the mean square error lower than a respective threshold indicates a higher similarity between the recorded point cloud and the simulated point cloud, which represents a better performance (and hence, suitability) of the depth camera for the target application. Similarly, the count of invalid pixels greater than a respective threshold indicates a poor quality of the depth map, which may mean that the depth camera may not be suitable for the target application. The steps 2002-2014 are then repeated for each and every desired set of objects.
[122] Fig. 10 depicts an exemplary flowchart of a method 2200 with respect to the instructions executed by processor 10a and/or the computing unit 20a for assessing bin picking scenario of a depth camera. At step 2202, the computing unit 20a determines the distance at which the depth camera has to be fixed. For example, the determined distances may be 1m, 1.5m, 2m, 2.5m, 3m etc. on the floor on which the depth camera is mounted on a mounting device to capture images.
[123] At step 2204, the computing unit 20a places a bin having the desired set of objects such as mixture of golf balls, small cones and pyramids.
[124] At step 2206, the processor 10a generates a simulated data (e.g., simulated point clouds) for the bin.
[125] At step 2208, the computing unit 20a maneuvers and places the mounting device on which the depth camera is mounted at the determined distance facing downwards to the floor. Upon receiving confirmation, the computing unit 20a is configured to activate the depth camera to capture the set of images of the objects.
[126] At step 2210, the computing unit 20a records a bag file from the depth camera and calculates the point cloud data. The computing unit 20a sends the point cloud data to the processor 10a.
[127] At step 2212, the processor 10a compares the point cloud data obtained at step 2210 with the simulated point cloud generated at step 2206, for assessing performance parameters of the depth camera for the application of bin picking scenario. In this case, the simulate point cloud acts as the threshold.
[128] Fig. 11 depicts an exemplary flowchart of a method 2400 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing performance of a depth camera for the SLAM and navigation application. At step 2402, the computing unit 20a instructs the user to place an AMR. The computing unit 20a sends a control signal(s) to a controller coupled with the AMR to move the AMR to a desired position.
[129] At step 2404, the computing unit 20a instructs a user to place flower pots at desired positions (e.g., the corners, one or more corners and center) in a room with textured curtains.
[130] At step 2406, the computing unit 20a instructs the user to mount the depth camera on the AMR. Upon receiving confirmation, the computing unit 20a is configured to activate the depth camera to record bag files. The computing unit 20a sends the recorded bag files to the processor 10a.
[131] At step 2408, the processor 10a runs a VSLAM algorithm on recorded bag file to determine optimized poses and a map of the environment.
[132] At step 2410, the processor 10a calculates a root mean square error (RMSE) based upon the optimized poses and ground truth poses. The RMSE value is compared with a threshold to assess the performance of the depth camera for the SLAM and navigation application. Further, the processor 10a may also analyze the generated map of the environment to evaluate its mapping quality. In an embodiment, the processor 10a displays the generated maps to the user on the user device 30.
[133] Fig. 12 depicts an exemplary flowchart of a method 2600 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing measurement estimation of object surfaces by a depth camera. At step 2602, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 1m, 1.5m, 2m, 2.5m, 3m etc. for placing a mounting device on which the depth camera is mounted to capture images of the objects.
[134] At step 2604, the computing unit 20a places the desired set of objects, for example, box(es), traffic cone(s) and paper roll(s).
[135] At step 2606, the computing unit 20a places the depth camera at the determined distance facing in a tilted manner. Upon receiving the confirmation, the computing unit 20a is configured to activate the depth camera to capture the set of images of the objects and record corresponding depth maps.
[136] At step 2608, the computing unit 20a repeats the steps 2602-2606 for two different types of floors. The computing unit 20a sends the recorded depth maps to the processor 10a.
[137] At step 2610, the processor 10a calculates an estimate of one or more dimensions of the desired set of objects (e.g., height, width, length of the boxes, or diameter and length of the paper rolls) using a suitable dimension measurement technique. The above steps are repeated for 10 trials per dimension.
[138] At step 2612, the processor 10a calculates an average of the estimated values of the dimensions across the 10 trials.
[139] At step 2614, the processor 10a calculates a difference between the average estimated value of a dimension and an actual value of the dimension (e.g., a difference between an average of estimated length of a box and an actual length of the box). The processor 10a compares the difference with a threshold to assess performance of the depth camera when assessing measurement estimation of object surfaces.
[140] Fig. 13 depicts an exemplary flowchart of a method 2800 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing the performance of a depth camera for speed estimation of a moving object. At step 2802, the computing unit 20a places a mobile robot to provide a reference ground truth speed.
[141] At step 2804, the computing unit 20a instructs the user to place depth camera at a pre-defined distance. Upon receiving the confirmation, the computing unit 20a is configured to activate the depth camera and record bag files.
[142] At step 2806, the computing unit 20a controls the mobile robot to move in a straight line in the FOV of the depth camera from one point to another.
[143] At step 2808, the computing unit 20a instructs the mobile robot to run in a diagonal line in the FOV of the depth camera from one point to another.
[144] At step 2810, the computing unit 20a repeats the step 2806 and 2808 for a black carpet, a white carpet and an aluminum sheet. The computing unit 20a sends the recorded data to the processor 10a.
[145] At step 2812, the processor 10a estimates a speed of the mobile robot for each floor type based upon the recorded data using a suitable speed estimation technique.
[146] At step 2814, the processor 10a calculates a difference between the estimated speed and the actual speed of the mobile robot. The processor 10a then compares the difference with a threshold to assess the performance of the depth camera for assessing speed estimation of moving objects.
[147] Fig. 14 depicts an exemplary flowchart of a method 3000 with respect to the instructions executed by the processor 10a and/or the computing unit 20a for assessing the performance of a depth camera for counting of objects. At step 3002, the computing unit 20a determines distances at which measurements are to be taken. For example, the determined distances may be 1m, 1.5m, 2m, 2.5m, 3m etc. for placing the mounting device on which the depth camera is mounted to capture images of the objects placed in front.
[148] At step 3004, the computing unit 20a instructs the user to place the depth camera at a first predefined distance to obtain a set of images. Upon receiving the confirmation, the computing unit 20a is configured to activate the depth camera to record bag file at the determined distances.
[149] At step 3006, the computing unit 20a places (or instructs the user to place) golf balls in front of the depth camera.
[150] At step 3008, the computing unit 20a places (or instructs the user to place) a transparent glass between the golf balls and the depth camera.
[151] At step 3010, the computing unit 20a places (or instructs the user to place) a mirror behind the golf balls.
[152] At step 3012, the computing unit 20a implements (or instructs the user to implement) occlusion of multiple golf balls.
[153] At step 3014, the computing unit 20a repeats the steps 3002-3010 for a white floor, and a shiny aluminum floor. The computing unit 20a sends the recorded data (e.g., the bag files having the depth maps and/or the point clouds) to the processor 10a.
[154] At step 3016, the processor 10a estimates a count of objects (e.g., the golf balls) based upon the recorded data using a suitable object counting technique. The steps 3002 – 3014 may be repeated for a pre-defined number (say, 10) of trials for each scenario (e.g., with/without the mirror, with/without the transparent glass, different types of floors).
[155] At step 3018, the processor 10a calculates a performance parameter based upon the estimated count of objects and the actual count of objects. In an embodiment, the performance parameter is a counted accuracy defined as a percentage of the number of times the object were detected correctly and the total number of trials. In an embodiment, the performance parameter is a mean absolute error defined as an average of the absolute difference between the estimated count and the actual count of objects for the number of trials. The processor 10a then compares the performance parameter with the corresponding threshold to assess the performance of the depth camera for this application.
[156] The scope of the invention is only limited by the appended patent claims. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and/or configurations will depend upon the specific application or applications for which the teachings of the present invention is/are used. ,CLAIMS:We claim
1. A system (100) for assessing performance of a depth camera, the system (100) comprising:
a. an imaging unit (20) comprising a computing unit (20a) communicatively coupled to a depth camera; and
b. an assessment unit (10) comprising a processor (10a) communicatively coupled to the computing unit (20a) and configured to:
i. obtain an identifier of the depth camera and a target application for the depth camera;
ii. determine an imaging set-up and a performance characteristic associated with the target application; wherein the imaging set-up comprises a desired position and a desired orientation of the depth camera, a desired set of objects, desired positions of the desired set of objects, and desired surrounding conditions;
iii. communicate a set of instructions to be executed corresponding to the imaging set-up to the computing unit (20a), wherein the set of instructions comprise instructions to capture one or more images of the desired set of objects using the depth camera;
iv. receive, from the computing unit (20a), point clouds or depth maps corresponding to the one or more images captured by the depth camera based upon the set of instructions;
v. calculate a performance parameter based upon the point clouds or the depth maps corresponding to the one or more images, the performance parameter being indicative of the performance characteristic;
vi. compare the performance parameter with a threshold; and
vii. provide a recommendation to the user indicating usability of the depth camera for the target application based upon the comparison;
c. wherein, based upon the set of instructions, the computing unit (20a) is configured to:
i. control the depth camera to capture the one or more image;
ii. record the point clouds or the depth maps corresponding to the one or more images; and
iii. communicate the point clouds or the depth maps corresponding to the one or more images to the processor (10a).
2. The system (100) as claimed in claim 1, wherein the imaging unit (20) comprises the desired set of objects comprising one or more of: a traffic cone, a chair, a ball, a paper roll, a vase, a wine glass, a metal bowl, a cuboid box, a pyramid crate, a flowerpot, a bin, an aluminum cover, a mirror, a glass, a golf ball, a display device.
3. The system (100) as claimed in claim 1, wherein the imaging unit (20) comprises one or more of: a lighting device, a curtain, a mounting device, a motorized camera slider, a carpet, a mobile robot, an autonomous mobile robot (AMR).
4. The system (100) as claimed in claim 1, wherein the processor (10a) is configured to compare the point clouds or the depth maps corresponding to the one or more image with a simulated point cloud or a simulated depth map, wherein the simulated point cloud or the simulated depth map is determined based upon a simulated imaging set-up corresponding to the imaging set-up.
5. The system (100) as claimed in claim 1, wherein the processor (10a) or the computing unit (20a) is configured to display the desired position and the desired orientation of the depth camera, the desired set of objects, the desired positions of the desired set of objects, and the desired surrounding conditions on a display of a user device (30).
6. The system (100) as claimed in claim 1, wherein the desired surrounding conditions comprises desired lighting conditions and the set of instructions comprises instructions associated with the desired lighting conditions, wherein the computing unit (20a) is configured to send, based upon the set of instructions, a control signal to a controller, interfaced with a lighting device, to change intensity of the light emitted by the lighting device according to the desired lighting conditions.
7. The system (100) as claimed in claim 1, wherein the set of instructions comprises instructions associated with desired positions of the desired set of objects, wherein the computing unit (20a) is configured to send, based upon the set of instructions, a control signal to a controller, interfaced with an object of the desired set of objects, to change a position of the object to a corresponding desired position.
8. The system (100) as claimed in claim 1, wherein the set of instructions comprises instructions associated with the desired orientation of the depth camera, wherein the computing unit (20a) is configured to send, based upon the set of instructions, a control signal to a controller, interfaced with a mounting device, to control an orientation of the depth camera to the desired orientation, wherein the mounting device is configured to mount the depth camera.
9. The system (100) as claimed in claim 1, wherein the set of instructions comprises instructions associated with the desired position of the depth camera, wherein the computing unit (20a) is configured to send, based upon the set of instructions, a control signal to a controller interfaced with a motorized camera slider to change a position of the depth camera to the desired position.
10. A method (150) for assessing performance of a depth camera, the method (150) comprising:
a. obtaining, by a processor (10a) of an assessment unit (10), an identifier of a depth camera and a target application for the depth camera;
b. determining, by the processor (10a), an imaging set-up and a performance characteristic associated with the target application; wherein the imaging set-up comprises a desired position and a desired orientation of the depth camera, a desired set of objects, desired positions of the desired set of objects, and desired surrounding conditions;
c. communicating, by the processor (10a), a set of instructions to be executed corresponding to the imaging set-up to a computing unit (20a) of an imaging unit (20), wherein the set of instructions comprise instructions to capture one or more images of the desired set of objects using the depth camera, wherein the computing unit (20a) is communicatively coupled to the depth camera;
d. receiving, by the processor (10a) from the computing unit (20a), point clouds or depth maps corresponding to the one or more images captured by the depth camera based upon the set of instructions;
e. calculating, by the processor (10a), a performance parameter based at least upon the point clouds or the depth maps corresponding to the one or more images, the performance parameter being indicative of the performance characteristic;
f. comparing, by the processor (10a), the performance parameter with a threshold; and
g. providing, by the processor (10a), a recommendation to the user indicating usability of the depth camera for the target application based upon the comparison.
11. The method (150) as claimed in claim 10, wherein the method (150) comprises comparing, by the processor (10a), the point clouds or depth maps corresponding to the one or more image with a simulated point cloud or a simulated depth map, wherein the simulated point cloud or the simulated depth map is determined based upon a simulated imaging set-up corresponding to the imaging set-up.
12. The method (150) as claimed in claim 10, wherein the method (150) comprises displaying, by the processor (10a) or the computing unit (20a), the desired position and the desired orientation of the depth camera, the desired set of objects, the desired positions of the desired set of objects, and the desired surrounding conditions on a display of a user device (30).
13. The method (150) as claimed in claim 10, the method (150) comprises:
a. controlling, by the computing unit (20a), the depth camera to capture the one or more image based upon the set of instructions;
b. recording, by the computing unit (20a), the point clouds or the depth maps corresponding to the one or more images based upon the set of instructions; and
c. communicating, by the computing unit (20a), the point clouds or the depth maps corresponding to the one or more images to the processor (10a).
14. The method (150) as claimed in claim 10, wherein the desired surrounding conditions comprises desired lighting conditions and the set of instructions comprises instructions associated with the desired lighting conditions, wherein the method (150) comprises sending, by the computing unit (20a) based upon the set of instructions, a control signal to a controller, interfaced with a lighting device, to change intensity of the light emitted by the lighting device according to the desired lighting conditions.
15. The method (150) as claimed in claim 10, wherein the set of instructions comprises instructions associated with desired positions of the desired set of objects, wherein the method (150) comprises sending, by the computing unit (20a) based upon the set of instructions, a control signal to a controller, interfaced with an object of the desired set of objects, to change a position of the object to a corresponding desired position.
16. The method (150) as claimed in claim 10, wherein the set of instructions comprises instructions associated with the desired orientation of the depth camera, wherein the method (150) comprises sending, by the computing unit (20a) based upon the set of instructions, a control signal to a controller, interfaced with a mounting device, to control an orientation of the depth camera to the desired orientation, wherein the mounting device is configured to mount the depth camera.
17. The method (150) as claimed in claim 10, wherein the set of instructions comprises instructions associated with the desired position of the depth camera, wherein the method (150) comprises sending, by the computing unit (20a) based upon the set of instructions, a control signal to a controller interfaced with a motorized camera slider to change a position of the depth camera to the desired position.