Claims:1. A processor-implemented method for encroachment detection in a geographical area, the method comprising:
identifying, via one or more hardware processors, one or more image features from one or more first images of the geographical area, the one or more first images captured using an in-flight unmanned aerial vehicle (UAV);
retrieving, via the one or more hardware processors, one or more second images of the geographical area pre-stored in an archive database based at least on the one or more image features of the one or more first images;
segmenting, via the one or more hardware processors, the one or more first images into a first plurality of image portions and the one or more second images into a second plurality of image portions based on the one or more image features using a Deep Neural Network (DNN) model trained using a training data for urban environment;
selecting, via the one or more hardware processors, a first set of image portions from amongst the first plurality of image portions, and a second set of image portions from amongst the second plurality of image portions, the first set of image portions and the second set of image portions comprising a region of interest (ROI) associated with the geographical area;
registering, via the one or more hardware processors, the first set of image portions with the second set of image portions to obtain a first set of registered image portions; and
comparing, via the one or more hardware processors, the first set of registered image portions with the first set of image portions to identify the encroachment in the geographical area.

2. The method as claimed in claim 1, wherein the one or more image features comprises
color, texture, shape, motion, location; and specialized features comprising edges, corner points, blobs, and ridges.

3. The method as claimed in claim 1, wherein the DNN network comprises a multi-level DNN network.

4. The method as claimed in claim 1, wherein segmenting the one or more first images into the first plurality of image portions and the one or more second images into the second plurality of image portions comprises:
identifying, based on the training data, areas associated with public properties and private properties of the geographical area in the first plurality of image portions and the second plurality of image portions

5. The method as claimed in claim 1, further comprising computing an amount of encroachment in the geographical area using a 3D point cloud model.

6. The method as claimed in claim 1, further comprising computing an amount of encroachment in the geographical area using a 2D change detection model.

7. The method as claimed in claim 1, further comprising detecting one or more house numbers in the at least first image of the geographical area, wherein the one or more house numbers facilitates in retrieving one or more second images of the geographical area pre-stored in an archive database.

8. The method as claimed in claim 1, wherein identifying the encroachment comprises:
estimating a position of one or more objects in the geographical area by performing a perspective correction based on a height at which the at least first image is captured by the UAV;
categorizing the one or more objects as one of a temporary object and a mobile object; and
determining encroachment by the one or more objects in the geographical area based on the type of object and the position of the one or more objects in a 3D space.

9. A system for encroachment detection, the system comprising:
one or more memories storing instructions; and
one or more hardware processors coupled to said one or more memories, wherein the one or more hardware processors configured by said instructions to:
identify one or more image features from one or more first images of the geographical area, the one or more first images captured using an in-flight unmanned aerial vehicle (UAV);
retrieve one or more second images of the geographical area pre-stored in an archive database based at least on the one or more image features of the one or more first images;
segment the one or more first images into a first plurality of image portions and the one or more second images into a second plurality of image portions based on the one or more image features using a Deep Neural Network (DNN) model trained using a training data for urban environment;
select a first set of image portions from amongst the first plurality of image portions, and a second set of image portions from amongst the second plurality of image portions, the first set of image portions and the second set of image portions comprising a region of interest (ROI) associated with the geographical area;
register the first set of image portions with the second set of image portions to obtain a first set of registered image portions; and
compare the first set of registered image portions with the first set of image portions to identify the encroachment in the geographical area.

10. The system as claimed in claim 9, wherein the one or more image features comprises color, texture, shape, motion, location; and specialized features comprising edges, corner points, blobs, and ridges.

11. The system as claimed in claim 9, wherein the DNN network comprises a multi-level DNN network.

12. The system as claimed in claim 9, wherein to segmenting the one or more first images into the first plurality of image portions and the one or more second images into the second plurality of image portions, the one or more hardware processors are configured by said instructions to:
identify, based on the training data, areas associated with public properties and private properties of the geographical area in the first plurality of image portions and the second plurality of image portions

13. The system as claimed in claim 9, wherein the one or more hardware processors are further configured by said instructions to compute an amount of encroachment in the geographical area using a 3D point cloud model

14. The system as claimed in claim 9, wherein the one or more hardware processors are further configured by said instructions to compute an amount of encroachment in the geographical area using a 2D change detection model.

15. The system as claimed in claim 9, wherein the one or more hardware processors are further configured by said instructions to detect one or more house numbers in the at least first image of the geographical area, wherein the one or more house numbers facilitates in retrieving the one or more second images of the geographical area pre-stored in an archive database.

16. The system as claimed in claim 9, wherein to identify the encroachment, the one or more hardware processors are further configured by said instructions to:
estimate a position of one or more objects in the geographical area by performing a perspective correction based on a height at which the at least first image is captured by the UAV;
categorize the one or more objects as one of a temporary object and a mobile object; and
determine encroachment by the one or more objects in the geographical area based on the type of object and the position of the one or more objects in a 3D space.
, Description:FORM 2

THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENT RULES, 2003

COMPLETE SPECIFICATION
(See Section 10 and Rule 13)

Title of invention:
SYSTEM AND METHOD FOR ENCROACHMENT DETECTION

Applicant:
Tata Consultancy Services Limited
A company Incorporated in India under the Companies Act, 1956
Having address:
Nirmal Building, 9th floor,
Nariman point, Mumbai 400021,
Maharashtra, India

The following specification particularly describes the embodiments and the manner in which it is to be performed.

TECHNICAL FIELD
[0001] The embodiments herein generally relate to encroachment detection, and, more particularly, to method and system for detecting encroachment in urban areas using Unmanned Aerial Vehicle (UAV).

BACKGROUND
[0002] With growing infrastructure in cities and towns, the possibilities of encroachment are also increasing manifold. Encroachment may refer to intrusion or advance into a property not belonging to self beyond an acceptable limit. Encroachment may be effected through various objects, such as poles, walls, temporary structures or objects, rails, and other public or private infrastructure. In order to maintain decorum, and law and order, encroachment is to be kept under check.
[0003] Currently, encroachment detection is performed through manual processes, in which field officers are responsible for physically visiting locations of encroachment to ascertain the same. Alternatively, there are certain sophisticated encroachment detection systems that can capture satellite images of geographical areas to detect encroachment. Such satellite images are compared with previous images of the same geographical area to ascertain encroachment in the areas. The inventors here have recognized several technical problems with such conventional systems, as explained below.
[0004] In certain instances, areas are encroached for long periods of time due to which it may be difficult for the field officers to establish encroachment, and accordingly manual detection and verification of encroachment becomes difficult. The conventional sophisticated systems are dependent on satellite images, however such satellite images may provide unsatisfactory results in case the encroachment is to be detected at micro levels, for example in public places. In addition, comparing the satellite images with previous images of the area involves huge processing and memory requirements, thereby loading the encroachment detection system.

SUMMARY
[0005] The following presents a simplified summary of some embodiments of the disclosure in order to provide a basic understanding of the embodiments. This summary is not an extensive overview of the embodiments. It is not intended to identify key/critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some embodiments in a simplified form as a prelude to the more detailed description that is presented below.
[0006] In view of the foregoing, an embodiment herein provides a processor-implemented method for encroachment detection in a geographical area. The method includes identifying, via one or more hardware processors, one or more image features from one or more first images of the geographical area, the first image captured using an in-flight UAV. Further the method includes retrieving, via the one or more hardware processors, one or more second images of the geographical area pre-stored in an archive database based at least on the one or more image features of the first image. Furthermore the method includes segmenting, via the one or more hardware processors, the one or more first images into a first plurality of image portions and the one or more second images into a second plurality of image portions based on the one or more image features and a Deep Neural Network (DNN) model trained using a training data for urban environment. Moreover, the method includes selecting, via the one or more hardware processors, a first set of image portions from amongst the first plurality of image portions, and a second set of image portions from amongst the second plurality of image portions. The first set of image portions and the second set of image portions includes regions of interest (ROI) associated with the geographical area. Also, the method includes registering, via the one or more hardware processors, the first set of image portions with the second set of image portions to obtain a first set of registered image portions. Also, the method includes comparing, via the one or more hardware processors, the first set of registered image portions with the first set of image portions to identify the encroachments in the geographical area.
[0007] In one aspect, a system for encroachment detection is provided. The system includes one or more memories storing instructions; and one or more hardware processors coupled to said one or more memories. The one or more hardware processors configured by said instructions to identify one or more image features from one or more first images of the geographical area. The first image is captured using an in-flight UAV. Further, the one or more hardware processors configured by said instructions to retrieve one or more second images of the geographical area pre-stored in an archive database based at least on the one or more image features of the first image. Furthermore, the one or more hardware processors configured by said instructions to segment the one or more first images into a first plurality of image portions and the one or more second images into a second plurality of image portions based on the one or more image features and a Deep Neural Network (DNN) model trained using a training data for urban environment. Moreover, the one or more hardware processors configured by said instructions to select a first set of image portions from amongst the first plurality of image portions, and a second set of image portions from amongst the second plurality of image portions. The first set of image portions and the second set of image portions includes regions of interest (ROI) associated with the geographical area. Also, the one or more hardware processors configured by said instructions to register the first set of image portions with the second set of image portions to obtain a first set of registered image portions. Also, the one or more hardware processors configured by said instructions to compare the first set of registered image portions with the first set of image portions to identify the encroachments in the geographical area.
[0008] In yet another implementation, a non-transitory computer-readable medium having embodied thereon a computer program for executing a method for encroachment detection is provided. The method includes identifying one or more image features from one or more first images of the geographical area, the one or more first images captured using an in-flight UAV. Further the method includes retrieving one or more second images of the geographical area pre-stored in an archive database based at least on the one or more image features of the one or more first images. Furthermore the method includes segmenting the one or more first images into a first plurality of image portions and the one or more second images into a second plurality of image portions based on the one or more image features and a Deep Neural Network (DNN) model trained using a training data for urban environment. Moreover, the method includes selecting a first set of image portions from amongst the first plurality of image portions, and a second set of image portions from amongst the second plurality of image portions. The first set of image portions and the second set of image portions includes regions of interest (ROI) associated with the geographical area. Also, the method includes registering the first set of image portions with the second set of image portions to obtain a first set of registered image portions. Also, the method includes comparing the first set of registered image portions with the first set of image portions to identify the encroachments in the geographical area.
[0009] It should be appreciated by those skilled in the art that any block diagram herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computing device or processor, whether or not such computing device or processor is explicitly shown.

BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The embodiments herein will be better understood from the following detailed description with reference to the drawings, in which:
[0011] FIG. 1 illustrates a network implementation for encroachment detection, in accordance with an example embodiment.
[0012] FIG. 2 illustrates a block diagram of a system for encroachment detection, in accordance with an example embodiment.
[0013] FIG. 3 illustrates a process flow representative of registration of images for encroachment detection, in accordance with an example embodiment.
[0014] FIG. 4 illustrates a process flow representative of segmentation of images for encroachment detection, in accordance with an example embodiment.
[0015] FIG. 5 illustrates a flow diagram of a method for encroachment detection, in accordance with an example embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS
[0016] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.
[0017] The encroachment of urban areas at micro levels can be determined effectively by utilizing images which can provide information of such areas at micro levels. For instance, some of the urban areas may be encroached by objects such as poles, walls, temporary structures or objects, rails, and so on. However, detection of encroachment of such small objects at micro level using satellite images is a challenging task.
[0018] Various embodiments of the present disclosure provide methods and system for detecting encroachment in urban areas at micro levels, effectively and efficiently. For example, in an embodiment, the method for encroachment detection in a geographical area includes comparing the captured media with prestored media of the same geographical area. However, a significant contribution of the embodiments disclosed herein is that instead of comparing entire images (or video) of the geographical area, in the disclosed embodiments, only regions of interest (ROI) in the geographical area are compared against prestored images of the ROI. By comparing only the ROI in the captured image and the prestored images, the disclosed method and system enables efficient computation, thereby saving in processing time, power and memory of the system.
[0019] Additionally, the embodiments disclose use of Unmanned Aerial Vehicles (UAVs) for capturing media, including images and/or videos, of the areas where encroachment is to be detected. In general, a UAV is an aircraft which flies without a human pilot on-board. A UAV recognizes a path of flight thereof based on programmed instructions provided to the UAV by a remote control station or by UAV embedded controllers. The UAVs, also known as “drones” can be utilized for various urban civilian applications, including encroachment detection. For example, the UAV is being utilized for urban civilian applications such as surveillance, fire brigades, disaster control, emergency response crews, while remote rural civilian applications include periodic monitoring of long linear infrastructures towards critical utilities, such as power line, oil/gas pipelines, and so on. An example network implementation for encroachment detection is described further with reference to FIG. 1.
[0020] Referring now to the drawings, and more particularly to FIGS. 1 through 5, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method.
[0021] FIG. 1 illustrates a network implementation 100 of a system 102 for encroachment detection, in accordance with an example embodiment. The system 102 is configured to detect encroachment in geographical areas at a micro-level. Herein, ‘micro-level’ may refer to a very small level, which may be difficult to detect using satellite images of the area. For example, there may be encroachment in geographical areas due to poles, walls, temporary structures, and other similar infrastructure or objects. Encroachment detection at such small level using the satellite images is complex and inaccurate.
[0022] Although the present subject matter is explained considering that the system 102 is implemented on a server, it may be understood that the system 102 may also be implemented in a variety of computing systems, such as a laptop computer, a desktop computer, a notebook, a workstation, a mainframe computer, a server, a network server, a cloud-based computing environment and the like. It will be understood that the system 102 may be accessed by multiple users through one or more user devices 104-1, 104-2…104-N, collectively referred to as user devices 104 hereinafter, or applications residing on the user devices 104. In one implementation, the system 102 may include a cloud-based computing environment in which a user may operate individual computing systems configured to execute remotely located applications. Examples of the user devices 104 may include, but are not limited to, a portable computer, a personal digital assistant, a handheld device, and a workstation. The user devices 104 are communicatively coupled to the system 102 through a network 108.
[0023] The servers, such as the server 106, include but are not limited to application servers, database servers, computation farms, data centers, virtual machines, cloud computing devices, mail or web servers and the like. The server 106 includes one or more computing devices or machines capable of operating one or more Web-based and/or non-Web-based applications that may be accessed by other computing devices (e.g. client devices, other servers) via the network 108. One or more servers 106 may be front end Web servers, application servers, and/or database servers. Such data includes, but is not limited to Web page(s), image(s) of physical objects, user account information, and any other objects and information. It should be noted that the server 106 may perform other tasks and provide other types of resources.
[0024] The server 106 may include a cluster of a plurality of servers which are managed by a network traffic device such as a firewall, load balancer, web accelerator, gateway device, router, hub and the like. In an aspect, the server 106 may implement a version of Microsoft® IIS servers, RADIUS servers and/or Apache® servers, although other types of servers may be used and other types of applications may be available on the servers 106.
[0025] In an embodiment, the network implementation 100 includes one or more databases such as an archive database 110, communicatively coupled to the servers 106. The archive database 110 may be configured to allow storage and access to data, files or otherwise information utilized or produced by the system 102. Herein, it is assumed that the archive database 110 is embodied in computing devices configured external to the servers 106. It will however be noted that in alternative embodiments, the archive databases 110 may be embodied in the servers 106.
[0026] In one implementation, the network 108 may be a wireless network, a wired network or a combination thereof. The network 108 can be implemented as one of the different types of networks, such as intranet, local area network (LAN), wide area network (WAN), the internet, and the like. The network 108 may either be a dedicated network or a shared network. The shared network represents an association of the different types of networks that use a variety of protocols, for example, Hypertext Transfer Protocol (HTTP), Transmission Control Protocol/Internet Protocol (TCP/IP), Wireless Application Protocol (WAP), and the like, to communicate with one another. Further the network 108 may include a variety of network devices, including routers, bridges, servers, computing devices, storage devices, and the like.
[0027] FIG. 2 illustrates a block diagram of a system 200 for encroachment detection, in accordance with an example embodiment. The system 200 may be an example of the system 102 (FIG. 1). In an example embodiment, the system 200 may be embodied in, or is in direct communication with the system, for example the system 102 (FIG. 1). The system 200 includes or is otherwise in communication with one or more hardware processors such as a processor 202, one or more memories such as a memory 204, and an I/O interface 206. The processor 202, memory 204, and the I/O interface 206 may be coupled by a system bus such as a system bus 208 or a similar mechanism.
[0028] The I/O interface 206 may include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like. The interfaces 206 may include a variety of software and hardware interfaces, for example, interfaces for peripheral device(s), such as a keyboard, a mouse, an external memory, a camera device, and a printer. Further, the interfaces 206 may enable the system 102 to communicate with other devices, such as web servers and external databases. The interfaces 206 can facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example, local area network (LAN), cable, etc., and wireless networks, such as Wireless LAN (WLAN), cellular, or satellite. For the purpose, the interfaces 206 may include one or more ports for connecting a number of computing systems with one another or to another server computer. The I/O interface 206 may include one or more ports for connecting a number of devices to one another or to another server.
[0029] The hardware processor 202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the hardware processor 202 is configured to fetch and execute computer-readable instructions stored in the memory 204.
[0030] The memory 204 may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, the memory 204 includes a plurality of modules 220 and a repository 240 for storing data processed, received, and generated by one or more of the modules 220. The modules 220 may include routines, programs, objects, components, data structures, and so on, which perform particular tasks or implement particular abstract data types. In one implementation, the modules 220 may include programs or coded instructions that supplement applications and functions of the system 200.
[0031] The repository 240, amongst other things, includes a system database 242 and other data 244. The other data 244 may include data generated as a result of the execution of one or more modules 220.
[0032] In an embodiment, the I/O interface 206 may include, in addition to other components, a media sensor. The media sensor may include a media capturing module, such as a camera, video and/or audio module, in communication with the processor 202. The media sensor may be any means for facilitating capturing images, video and/or audio for storage, display, or transmission. For example, in an exemplary embodiment in which the media sensor may be embodied in a camera, such that the camera may be configured to form and save a digital image file from an image of the marker captured by the camera. The media sensor may include hardware such as a complementary metal-oxide semiconductor/charged coupled device (CMOS/CCD) sensors configured for capturing the media. In an embodiment, the media sensor may be configured to capture media items in accordance with a number of capture settings such as focal length, zoom level, lens type, aperture, shutter timing, white balance, color, style (e.g., black and white, sepia, or the like), picture quality (e.g., pixel count), flash, date, time, or the like. In some embodiments, the values of the capture settings (e.g., degree of zoom) may be obtained at the time a media item comprising the object image is captured and stored in association with the captured media item in a memory device, such as, memory 204. The media sensor can include all hardware, such as circuitry, a lens or other optical component(s), and software for creating a digital image file from a captured image.
[0033] In some example embodiments, the image sensor may include only the hardware needed to view an image, while a memory device, such as the memory device of the system 200 stores instructions for execution by the processor 202 in the form of software to create a digital image file from a captured image. In an exemplary embodiment, the media sensor may further include a processor or co-processor which assists the processor 202 in processing image data and an encoder and/or decoder for compressing and/or decompressing image data. The encoder and/or decoder may encode and/or decode according to, for example, a joint photographic experts group (JPEG) standard or other format.
[0034] In an example embodiment, the system 200 may be embodied in the UAV. In another embodiment, the system 200 may be implemented at a remote server communicable coupled with the UAV such that the media is captured by the media sensors deployed on the UAV, and said media may be transmitted wirelessly to the system 200 configured/embodied in the remote server. The system 200 may then detect encroachment, if any, and transmit information regarding encroachment back to the UAV in real-time. In this embodiment, the system 200 may include a communication interface element embodied in the I/O interface 206 to facilitate communication between the UAV and the remote server (implementing the system 200).
[0035] In an embodiment, to detect the encroachment, at first, the system 200 is caused to capture media of the geographical area via the media sensors. The media captured by the media sensors may include images and/or video clips of the geographical area. For instance, the system may capture one or more first images of the geographical area using the media sensors. Additionally, the system 200 may retrieve one or more second images of the geographical area from an archive database. In an embodiment, the second images may be pre-stored in the archive database. Herein, it will be noted that the terminology “one or more first images” and “one or more second images” are representative of the captured media and pre-stored media, respectively of the geographical area. Also, the terms “first” and “second” are used merely to differentiate the media that is captured, from the media, which is pre-stored. The pre-stored media may include a historic data associated with the geographical area. For instance, the pre-stored media may include archive images of a region such as a road, a colony, a business park and so on. Such pre-stored media may be retrieved from government resources, online maps of the geographical area available through online resources, through crowdsourcing platform, and so on. In an embodiment, a time-series data of the geographical area can be stored in a repository, and such repository can be accessed by the system to retrieve pre-stored media. Herein, the time-series data refers to map data of the geographical area which may contain different versions according to the time of storing such date. In other words, such data may be timestamped with the time capturing and storing the data. The timestamped data may be useful in determining the time period/time instant during which actual encroachment has taken place. In an embodiment, the second image is associated with the first image, meaning thereby, that the second image is the image of same portions of geographical area as that of the first image. In order to retrieve the image of the same portion of the geographical area as is covered in the first image, the system 200 utilizes location coordinates of the geographical area. For instance, the system 200 may derive Global Positioning System (GPS) coordinates of the geographical area and utilize the same GPS co-ordinates to retrieve the pre-stored image of the geographical area.
[0036] The system 200 may segment the one or more first images into a first plurality of image portions based on one or more image features and a training data. Image features may refer to characteristic properties of images which may distinguish one image from another image. Examples of image features may include, but are not limited to, color, texture, shape, motion, location, visual saliency map and so on. Additionally, there may be certain specialized image features including edges, corner points, blobs, and ridges that may be local or global. The image features may be extracted using a variety of image processing techniques, including, but not limited to extraction techniques based on spatial features, textural features, appearance based features, histogram features, motion features, transform features, point features, region features, edge features, and so on. A detailed process flow for segmenting the first image into the first plurality of image portions is described further in detail with reference to FIG. 3.
[0037] In an embodiment, based on the image features, the system 200 may identify the segments from the first image that may be associated with public properties and private properties. For example, trees on green patch and in a house, grass on green strip or in the home lawn, concrete and asphalt in case of pavements and driveways may be examples of objects that may be automatically identified to be one of public objects and private objects. In an embodiment, the system 200 may be caused to utilize a spatial relationship approach to classify/categorize the identified objects as private objects or public objects. In an embodiment, to identify the objects as private or public objects, the system 200 may be caused to utilize labelled regions in a given scene. For example, the system 200 may create a graph with nodes as objects and relationships between the nodes may be represented using the edges. Using a training data, a relationship graph can be created and stored. The relationship graph may be representative of relationship between different labelled objects and unlabeled (or identified objects) in the scene and also the association between the labelled and unlabeled objects. The system 200 may construct a Logistic Regression classifier, a Tree Augmented Naive Bayes classifier or any classifier that does not assume full Naïve independence for declaring associations during a testing phase.
[0038] In an embodiment, the system 200 may additionally utilize house/street numbers that may be retrieved from the captured images to retrieve the second image from the images database of the geographical area. Based on information obtained from the image segments, said segments are sorted in the order of likelihood of the segments to include the characters and text information. In an embodiment, the segments may be sorted by performing a texture analysis of each segment based on information extracted from the training data. Due to the complex nature of the scene, and presence of outliers such as vehicle number plates, multiple segments can be detected which contain text information.
[0039] In an embodiment, the system 200 may be caused to take in the location and directional information from GPS to eliminate false detections. In an embodiment, in order to identify the house/street numbers from the captured images, the system may be trained using previous images of the geographical area. In an example embodiment, a multi-class classifier can be trained for identifying house numbers from the training images. In order to train the multi-class classifier, the system may be caused to first perform texture analysis of the entire scene associated with the geographical area to determine texts therein. The text can be normal or designed, but may have fixed shapes. The texts may appear in different wavelet packets. During the training phase, the wavelet packets may be identified. The result will contain false alarms, which will be made up of non-text regions. Using k-Means clustering on moments calculated on original color data, the text regions are identified. The identified text regions are normalized and input to an optical code reader (OCR) engine, and the multi-class classifier may be trained to detect house numbers therefrom. In an alternate embodiment, the text localization for identifying the house numbers may include training a deep neural network for text detection and localization.
[0040] The system 200 may further segment the one or more second images into a second plurality of image portions. The system may segment the second images into multiple image portions in a similar manner as the first image is segmented into a multiple image portions.
[0041] In order to detect the encroachment, the system compares the captured media, (for example, the captured first images) and the pre-stored media (for example, the second images of the geographical area). In an embodiment, to accurately detect encroachment, the system aligns the captured media with the pre-stored media. For example, the first image may be registered with respect to the second image. Herein, a significant contribution of the disclosed embodiments is that the system identifies one or more regions of interest (ROIs) in the first image and the second image, and aligns only those image portions of the first image and the second image that include the ROIs. For example, in a forest scenario, non-green patches in the captured image may be selected as the ROI. Additionally or alternatively, from urban scenarios, mobile objects such as vehicles and human beings/animals can be detected. Such regions may not be considered during registration as they can cause errors. Such parts can be omitted from ROI. Even trees and plants can be treated in similar manner so that only man-made changes/encroachments can be detected. In an embodiment, the system may select the ROI in the images based on anomaly detection, saliency and other higher level features of the application (or the scene), and accordingly instead of registering entire first image, the system selects only the ROIs based on domain specific features.
[0042] In an embodiment, in order to identify the ROI in the first image, the system may set suitable weights in saliency calculation algorithms or anomaly detection algorithms. For instance, the system 200 may set suitable weights in object detection algorithm based on the segmentation of the first image into public and private regions/objects. The private objects may be subject to change as they may be in control of the individuals (owners of the property). However, public objects may be subject to comparatively less amount of change over time. Hence, the public areas in the scene can be used for registration. Further, the shape details of the private object instead of appearance models can be used for registration as the shape of the house is unlikely to change without due permissions from the authorities. These unchanged objects or objects that are unlikely to change frequently are used as ROIs, and the weights of the unchanged/unlikely to change objects may be adjusted accordingly. For instance, in case of grass, due to constant growth the grass can grow tall, whether belonging to green areas in public or private places. However, a lower weight can be given to green patches in public region. The pavements and asphalt can be given higher weightage in region of interest identified for registration or other purposes.
[0043] Upon registration, the system 200 may detect the objects that may be categorized as encroaching objects in the geographical area. In an embodiment, the system 200 may utilize machine learning approach for detection of the encroaching objects. For example, the system 200 may utilize Deep learning networks as one of the ways, and may further generalize it to work with supervised machine learning techniques. In an embodiment, the system may include a fast embedded implementation of Deep Neural Network (DNN),that can be employed for real-time detection of objects in the images. The objects that can be detected in an urban environment may include common street assets like pavements, houses, green patches, poles, trees, vehicles, roads, fences, bridges, fly-overs and pavements. Once the objects in the geographical area in the first image and the second image are identified, the system compares said objects indicated in the portions of the pre-stored images for detecting the encroachment. In an embodiment, a multi-layer deep neural network (DNN) can be trained to determine the encroachment. An example of the multi-layer DNN is described further with reference to FIG.5. The parallel network will enable constant fine tuning of specific strand/application of the network independent of the other working models. In an alternative embodiment, pre-trained networks can be utilized, where the fully connected layers are disconnected and the system is re-trained for recognizing change in urban objects.
[0044] In an embodiment, the system 200 may detect the encroaching objects by first estimating a position of the one or more objects. In an embodiment, the position of one or more objects can be determined by performing a perspective correction in the first image based on a height at which the at least first image is captured by the UAV. As is understood data pertaining to the height from which the first image is captured, can be derived from the UAV. Then, the system 200 categorizes the one or more objects as one of a temporary object and a mobile object. In an embodiment, the mobile objects may include vehicles, for instance, two-wheeler and/or four wheeler vehicles, walking individuals, and so on. Upon detecting the encroachment, the information associated with the encroaching objects may be derived using machine learning method, where wavelet feature extraction and support vector machine (SVM) classifier are performed to ascertain whether or not the encroaching object is a mobile object.
[0045] Based on the type of the objects and the position of the one or more objects in a three-dimensional (3D) space, the system determines encroachment by the one or more objects in the geographical area. It will be noted that the system is trained using the DNN network to categorize the objects as one of the encroaching objects and non-encroaching objects based on the height of image capture and the position of said objects.
[0046] In an embodiment, the system 200 is further capable of detecting an amount of encroachment. In an embodiment, the system may compute a metric indicative of the amount of encroachment in the geographical area. In an embodiment, the system may compute said metric by using a 3D point cloud model. In order to compute the amount of encroachment, the system may generate a 3D point cloud from the media (for example, video data captured using the UAV) that is captured using any existing methods. The 3D point cloud represents a 3D scene along with a depth information thereof. The depth information can be computed using point correspondence from different views of the scene. In an embodiment, the UAV may include a stereoscopic camera for capturing the depth information of the scene. The 3D point cloud contains a set of points with respect to some reference co-ordinate system. In an implementation, standard state of the art structure-from-motion techniques can be used to obtain the 3D point cloud. In an embodiment, based on the 3D point cloud, the system estimates values for area of coverage and calculates approximate volume of geographical area. Herein, it will be noted that accurate computation of the 3D volume may be challenging, and hence using certain references, an approximate value of amount of encroachment can be determined. For example, it may be determined that the encroaching object is covering half of the width of a road, and the width of the road is around 2 meters, then dimension of the encroaching object may be assumed to be approximately one meter. Said approximate value can be converted to amount of encroachment. A map of the geographical area may provide details about the public spaces including the area that may be common to public. The calculated encroachment and the area on city map may give details about the amount of encroachment. Alternatively, the system 200 may detect amount of encroachment using 2D change detection model/methodology. For instance, the system 200 may calculate the geographical area covered in the reference image and register the reference image with respect to the captured image. The system 200 may then determine a difference image indicative of the difference between the registered image and captured image. The changed area may then be compared with segmented public spaces in the reference image. Said comparison results in two areas, including (a) public spaces area in reference image; and (b) public spaces area in the changed image. The system combines these values, thereby resulting in a unique number that may reflect the area of encroachment. For example, the system 200 may compute a sum of all the pixels from the absolute difference between the segmented binary images as:
sum(Iref – Ichanged)
Herein, it will be noted that a higher value of sum is indicative of a higher level of encroachment.
[0047] In an alternative embodiment, wherein it may be complex to determine the extent of encroachment using the 2D images of the scene, the system may create a 3D point cloud from the video captured by the UAV. The information contained in the 3D point cloud may be utilized to calculate the approximate volume that may provide an extent of encroachment. In a more generalized case, in case a type of object is known, an approximate size of the object can be determined. Hence, accurate detection of the type of encroachment may facilitate in determination of a level of encroachment. For example, if the object is a vehicle such as a car, the size of the car can be used for determining the extent of encroachment. If the identified encroachment is a rubbish bin, the bin volume can be quantified to declare the level of encroachment. If the object is not detected due to lack of knowledge, the 3D point cloud can be reconstructed and the encroachment level can be estimated. In case the 3D point cloud calculation fails due to smooth surfaces of the encroachment, approximate level of encroachment can be determined by counting the pixel in change detected image.
[0048] FIG. 3 illustrates a process flow 300 representative of registration of images for encroachment detection, in accordance with an example embodiment. The images that are captured using the UAV are subject to noise, for example, in the form of pedestrians, vehicles, and changes due to natural processes such as tree growth. Registration of the captured images using conventional techniques may lead to erroneous results due the presence of the noise. The disclosed embodiments include registering the reference images obtained from pre stored repository with images captured by the UAV. Herein, only relevant segments from the images are considered for segmentation, leaving out all the noisy segments which could lead to error.
[0049] Herein, the images may include a first image 302 and a second image 304, where the first image may be a captured image of the geographical area and the second image represents a pre-stored image of the geographical area. In an embodiment, the first image is captured by the image sensors embodied in the UAV. Additionally, the system embodied in the UAV may determine the location information of the geographical area, using for example, GPS position and orientation of the UAV. Using the location information of the geographical area, pre-stored images 304 of the geographical area are retrieved from a repository. Using the captured image and the pre-stored image, the system may then register the images at 308, and also segment the captured image and the pre-stored image at 306. The captured image may be registered with respect to the reference image using conventional registration techniques. It will however be noted that in the present disclosure the registration of the captured image with reference to the pre-stored image refers to registration of only selected portions of the both the images, and such portions are referred to ROI, as is already discussed with reference to FIG. 2. In an embodiment, the registered images may then be utilized for detecting the encroachment at 312.
[0050] In an embodiment, the captured image and the pre-stored image may be segmented at 306 to obtain a first plurality of image portions and a second plurality of image portions, respectively. In an embodiment, the segmented images may be utilized for detecting house number at 310. For example, optical character recognition (OCR) techniques may be used to detect the house number from the images captured from UAV. The detected house number may facilitate in localization of the UAVs in while navigating through a map of the geographical area. The information, such as house numbers, that is received from segmentation step 310 may be utilized to speed up the process of encroachment detection and improve robustness of the process. A detailed process flow for segmenting the images is described further with reference to FIG. 4.
[0051] FIG. 4 illustrates a process flow 400 representative of segmentation of images for encroachment detection, in accordance with an example embodiment. As illustrated in FIG. 4, the UAV may capture media, for instance images of a geographical area at 402. A plurality of image features may be extracted from said images. As is understood, the images captured by the UAV are referred to as ‘first images’ throughout the description, however, for the brevity of description and ease of understanding, the images captured by the UAV are referred to as images only. The goal of the segmentation is to divide the image into smaller segments which helps in further image analysis.
[0052] The images are segmented to obtain relevant segments therefrom, such relevant segments facilitates in identifying image regions of the geographical area showing public and private properties. The public properties may further be sub-divided into several classes such as nature strip, road, pavement, entry path and known assets (such as telephone box or an electricity junction box).
[0053] The system, for example, the system 200 (of FIG. 2) may extract image features from the images at 404. Examples of image features may include, but are not limited to, color, texture, shape, motion, location, visual saliency map and so on. Additionally there may be certain specialized image features including edges, corner points, blobs, and ridges that may be local or global. The image features may be extracted using a variety of image processing techniques.
[0054] In an embodiment, based on the image features, the system 200 may segment regions of the first image at 406. In an embodiment, the system 200 may segment the first image to identify ROIs in the first image. For example, the system may identify regions that may be associated with public properties and private properties located in the geographical area. In an embodiment, the system may be trained using DNN to identify the ROIs based on the geographical area at 408. For instance, in case the geographical area is a forest region, the system may be trained to identify non-green patches in the images of the geographical area. Using the training data and the image features, the system may detect objects associated with ROI in the first image at 410. An example of a multi-layer DNN is described further wit reference to FIG. 5.
[0055] FIG. 5 illustrates an example representation of a multi-layer DNN network 500, in accordance with an example embodiment. The multi-layer DNN network 500 (hereinafter referred to as network 500), may include six layers. In an embodiment, the six Layer DNN network may include two or more parallel networks, for example, networks 510, 530, and 550, consisting of convolution layers, activation layers (like Rectified Linear Unit - ReLU), pooling layers, and a fully connected layer. For example, the network 510 may include convolution layers such as a convolution layer 512, activation layers (like Rectified Linear Unit - ReLU) such as ReLU 514, pooling layers such as a pooling layer 516, and a fully connected layer such a fully connected layer 518. The output of the parallel layers may be provided to a fully connected network (with one or more fully connected layers) that may facilitate detection of change in scene between registered images (i.e., prestored image and captured images of the geographical area). Herein, the fully connected layers can be Gaussian. Alternatively, said layers can be replaced by Support Vector Machine (SVM) for binary classification. It will be noted herein that the layers and the number of layers may be variable during implementation of various embodiments disclosed herein. In an embodiment, the disclosed layers may be activation layer, convolutional layer, and sub-sampling layer, and these layers may be intelligently combined to form different architectures. The combination of the layers to generate a multi-layered DNN allows in optimizing memory required and speed of the processing of identifying the encroachment in the geographical areas. A flow diagram illustrating a method for encroachment detection is described further in FIG. 6.
[0056] FIG. 6 illustrate a flow chart of a method 600 for encroachment detection in accordance with an example embodiment. The method 600 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types. The method 600 may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communication network. The order in which the method 600 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method 600, or an alternative method. Furthermore, the method 600 can be implemented in any suitable hardware, software, firmware, or combination thereof. In an embodiment, the method 600 depicted in the flow chart may be executed by a system, for example, the system 200 of FIG. 2. In an example embodiment, the system 200 may be embodied in a computing device, for example, the computing device 104 (FIG. 1).
[0057] At 602, the method 600 includes, the identifying, via one or more hardware processors, one or more image features from one or more first image of the geographical area. The first image is captured using an in-flight UAV. At 606, the method 600 includes retrieving, via the one or more hardware processors, one or more second images of the geographical area pre-stored in an archive database based at least on the one or more image features of the first image. At 608, the method 600 includes segmenting, via the one or more hardware processors, the at least first image into a first plurality of image portions and the at least first image into a second plurality of image portions. In an embodiment, the segmenting of the first and second image is performed based on one or more image features and a DNN model trained using a training data for urban environment. An example of segmenting an image using the training data is described in FIG. 4
[0058] At 610, the method 600 includes selecting, via the one or more hardware processors, a first set of image portions from amongst the first plurality of image portions, and a second set of image portions from amongst the second plurality of image portions. The first set of image portions and the second set of image portions include ROIs associated with the geographical area. At 612, the method 600 includes registering, via the one or more hardware processors, the first set of image portions with the second set of image portions to obtain a first set of registered image portions. At 614, the method 600 includes comparing, via the one or more hardware processors, the first set of registered image portions with the first set of image portions to identify the encroachments in the geographical area.
[0059] The written description describes the subject matter herein to enable any person skilled in the art to make and use the embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope of the claims if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language of the claims.
[0060] Various embodiments disclosed herein provide method and system for robust detection of encroachment, in particular, in urban geographical areas. In an embodiment, the system for encroachment detection retrieved images of the geographical area from a databased, based upon captured images of the area as well as house/street addresses that may be identified in the captured images. An important contribution of the disclosed embodiments is that the disclosed embodiments utilize house/street addresses for localizing the search to retrieve relevant images from the database, thereby enhancing the robustness of the process. In addition, another significant contribution of the disclosed embodiments is that while registering the captured image with respect to the pre-stored images, the system registers only those image portions which include ROI, instead of registering the whole of the images. This approach of registering the images enhances system efficiency and saves system memory.
[0061] It is, however to be understood that the scope of the protection is extended to such a program and in addition to a computer-readable means having a message therein; such computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means can include both hardware means and software means. The method embodiments described herein could be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.
[0062] The embodiments herein can comprise hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. The functions performed by various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
[0063] The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Examples of a computer-readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read/write (CD-R/W) and DVD.
[0064] A data processing system suitable for storing and/or executing program code will include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements can include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution.
[0065] Input/output (I/O) devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intervening I/O controllers. Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modem and Ethernet cards are just a few of the currently available types of network adapters.
[0066] A representative hardware environment for practicing the embodiments may include a hardware configuration of an information handling/computer system in accordance with the embodiments herein. The system herein comprises at least one processor or central processing unit (CPU). The CPUs are interconnected via system bus to various devices such as a random access memory (RAM), read-only memory (ROM), and an input/output (I/O) adapter. The I/O adapter can connect to peripheral devices, such as disk units and tape drives, or other program storage devices that are readable by the system. The system can read the inventive instructions on the program storage devices and follow these instructions to execute the methodology of the embodiments herein.
[0067] The system further includes a user interface adapter that connects a keyboard, mouse, speaker, microphone, and/or other user interface devices such as a touch screen device (not shown) to the bus to gather user input. Additionally, a communication adapter connects the bus to a data processing network, and a display adapter connects the bus to a display device which may be embodied as an output device such as a monitor, printer, or transmitter, for example.
[0068] The preceding description has been presented with reference to various embodiments. Persons having ordinary skill in the art and technology to which this application pertains will appreciate that alterations and changes in the described structures and methods of operation can be practiced without meaningfully departing from the principle, spirit and scope.