FORM 2
THE PATENTS ACT 1970
[39 OF 1970]
&
THE PATENTS (AMENDMENT) RULES, 2006 COMPLETE SPECIFICATION
[See Section 10; rale 13]
"A SYSTEM FOR TRAJECTORY BASED POTHOLE DETECTION AND A METHOD
THEREOF"
HELLA INDIA AUTOMOTIVE PVT. LTD., an Indian company, of Nanospace, Baner-Pashan Link Road, Pune-411045, India,
The following specification particularly describes the invention and the manner in which it is to be performed:

The present Invention relates to a system for trajectory based pothole detection and a method thereof. More particularly, the present invention relates to a method and a system for detecting the pothole lying in the trajectory of the vehicle.
Aging roads and poor road-maintenance have resulted in a large number of potholes on roads. A pothole deteriorates lifespan of a vehicle and eventually, increases the costs of repair and maintenance of the vehicle. Despite advancement in technology when routes and maps have been digitised for assisting the driver, little has been done to provide a prior warning system to avoid potholes on the road. A timely warning or an alert signal to the driver will reduce chances of occurrence of serious incidents on roads like vehicle break down. Perhaps, visibility during day time may allow the drivers to remain cautious about the potholes on the road. However, in night time the visibility limitations make it further difficult for the driver to avoid the potholes.
Generally, driving on evenly and smooth roads improves efficiency of the vehicle. However, frequent sloping and starting of the vehicle delivers poor efficiency. Not only this, some roads are shaped irregularly which further makes it difficult for the driver to react instantly so as to avoid the potholes.
The existing system provides alert warning to the driver of the upcoming potholes on the roads after capturing the image of the road. Such systems have camera mounted on the

vehicle to capture the Images of road and assist the driver in driving by alert warnings. The cameras mounted on the vehicles provide a broader view of roads which may not be visible from naked eyes. The system continuously provides images and other information of potholes on roads irrespective of the position of the potholes, i.e., whether the pothole cutting the trajectory of the vehicle or not Thus, the driving is regularly Interrupted even when the potholes do not lie in the trajectory of the vehicle.
However, such systems are not efficient in giving effective results in detecting the pothole and providing prior warning to the driver only when the pothole is present in the trajectory of the vehicle. The major drawbacks of the prior art are continuous capturing and sharing the Information which distracts the driver while driving. Therefore, no device currently exits that assists the driver to avoid potholes cutting the trajectory of the vehicle. The prior art does not take into account the size, shape and position of the pothole while detecting the presence of the pothole in the path of the vehicle.
Thus, there is a requirement of a method for trajectory based pothole detection and a system
to mitigate the above mentioned drawbacks.
It is the principal object of the present invention to provide a trajectory based pothole detection.
It Is an object of the present Invention to provide a system for trajectory based pothole detection and a method thereof.
It is another object of the present Invention to provide prior warnings to driver of a vehicle.

It is another object of the present Invention to avoid unnecessary warnings to the driver of a vehicle.
According to an embodiment of the present invention, a trajectory based pothole detection method comprising capturing an image of road by a camera device mounted on a vehicle, transferring said captured image to a processor using a camera serial interface, converting said captured image to a grey scale image, applying differentiation based edge algorithm on said grey scale image to obtain a black-and-white image, filtering and refining said black-and-white image to remove unwanted noise using filters, detecting blobs on said image to determine position of potholes in the extracted road section of the captured image, plotting centroid and minor axis length on said potholes appearing in the image, plotting a circle around the potholes covering maximum pothole area using the centroid as centre and considering minor axis length as radius r; plotting a centre line on the road captured in said image to plot a safe trajectory, calculating distance between a centroid of said pothole and a centre line, comparing distance with minor axis length of said pothole, confirming presence of said pothole if distance is less than minor-axis length and indicating driver when said pothole lying in trajectory of said vehicle by an LED,
According to an embodiment of the present invention, the trajectory based pothole detection method further comprising morphological filtering of said image.
According to an embodiment of the present invention, the trajectory based pothole detection method further comprising restarting the process when distance is greater than minor axis length of said pothole.

According to an embodiment of the present Invention, the trajectory based pothole detection method further comprising confirming presence of said potholes lying in trajectory of said vehicle for each pothole separately.
According to an embodiment of the present invention, the trajectory based pothole detection method further comprising sending signals to an LED driver circuit by said processor to indicate said pothole lying in the trajectory of said vehicle.
According to an embodiment of the present Invention, the trajectory based pothole detection method further comprising sending signals to an LED by said LED driver circuit to indicate driver of said pothole lying in trajectory of the vehicle.
According to an embodiment of the present invention, the trajectory based pothole detection method further comprising said LED circuit and said LED receiving power supply from a power supply unit.
According to an embodiment of the present invention, the trajectory based pothole detection method as claimed in claim 1 further comprising said camera device and said processor
receiving power supply from a power supply unit.
According to an embodiment of the present invention, the trajectory based pothole detection method further comprising stepping down of voltage using a voltage regulator In said power supply unit.
According to an embodiment of the present invention, the trajectory based pothole detection method further comprising a canny edge detector to produce a black-and-white image.
According to an embodiment of the present Invention, a trajectory based pothole detection system comprising a camera device mounted on a vehicle to capture images of a road, a

processor for detecting a pothole lying in a trajectory of a vehicle in connection with a camera device, said processor further comprising a central processing unit, a storage means, a graphic processing unit, a video output means, a camera serial interface, an input means and an output means, an indication means to indicate the driver of a pothole lying in the trajectory of said vehicle, said indicating means includes an LED driver circuit and an LED, and a power supply unit supplying power to a camera device, a processor and an indicating means including at least one battery, a voltage regulator and a power supplying means.
According to an embodiment of the present invention, said captured image is transferred to said processor using said camera serial interface, said captured image is converted to a grey scale image and to a black-and-white image by an edge detector, said black-and-white image is filtered to remove unwanted noise and to detect a blob in said image, the blobs in the image are detected as potholes in the extracted road section of the captured image, a centroid and a minor axis length are plotted on said potholes, a circle is plotted on the detected potholes covering maximum pothole area using the centroid as centre and minor axis length as radius r, a safe trajectory is plotted by a centre line on said road captured in said image, distance d is determined from centroid of said pothole to said centre line and is compared with minor axis length of said pothole, and the driver is indicated of said pothole lying in the trajectory of said vehicle if the compared distance is less than the minor axis length of said pothole.
FIG. 1 illustrates a block diagram of a trajectory based pothole detecting system according to an embodiment of the present invention.
FIG. 2 illustrates steps of identifying pothole according to an embodiment of the present invention.

FIG. 3 illustrates steps of detecting a pothole according to an embodiment of the present invention
FIG. 4a illustrates steps of detecting a trajectory of a vehicle path according to an embodiment of the present invention.
FIG. 4b illustrates steps for detecting a pothole lying in the trajectory of the vehicle according to an embodiment of the present invention.
FIG. 5 illustrates a flow diagram of trajectory based pothole detection system according to an embodiment of the present invention.
FIG. 6 illustrates an application of the trajectory based pothole detection system according to an embodiment of the present invention.
The detailed description set forth below in connection with the appended drawings is intended as a description of various embodiments of the present invention and is not intended to represent the only embodiments in which the present invention may be practiced. Each
embodiment described in this disclosure is provided merely as an example or illustration of the present invention, and should not necessarily be construed as preferred or advantageous over other embodiments. The detailed description includes specific details for the purpose of providing a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. The embodiment provided herein is for the purpose of filing the complete specification.
The terms "minor axis length" and "radius r" have been used interchangeably.

The terms "blob" and "object'' have been used Interchangeably.
The terms "captured image" and "captured frame" have been used interchangeably.
FIG. 1 illustrates a block diagram of a trajectory based pothole (103) detecting system (90) according to an embodiment of the present invention. The present invention discloses a system (90) and a method for detecting a pothole (103) on the road (131) and particularly, detecting the pothole (103) lying in the trajectory of a vehicle (100) driven on that road (131). The trajectory based pothole detection system (90) comprises of an imaging apparatus (102) mounted on the vehicle (100) for capturing an image (120) of the road (131), a computer-enabled-hardware-medium (104) to receive the captured image (120) from the imaging apparatus (102) and carry the processing of the said captured image (120) for detecting the pothole (103). The process uses an algorithm and morphological operations. Further, the system (90) has an indicating means (110) that indicates a pothole (103) lying in the trajectory of the vehicle (100) and a power supply unit (108) to provide a power source to the imaging apparatus (102), the computer-enabled-hardware-medium (104) and the indicating means (110). Thus, after detecting the pothole (103) lying in the trajectory of the vehicle (100), the computer-enabled-hardware-medium (104) gives the indication of the pothole (103) to the driver by the indicating means (110).
According to an embodiment of the present invention, the imaging apparatus (102) is a camera device (102) that captures the image (120). The said camera device (102) is mounted by a mounting means on a body of the vehicle (100) such that it provides a broader field of view and uninterrupted images to the trajectory based pothole detection system (90) for accurate processing of the captured image (120) through the computer-enabled-hardware-medium (104). The camera device (102) of the present invention Is further provided with a camera serial interface for enabling data communication with the computer-enabled-

hardware-medium (104) of the present invention. Thus, the images (120) captured by the camera device (102) are transferred to the computer-enabled-hardware-medium (104) to achieve the aforesaid objectives of the present invention.
According to an embodiment of the present invention, the computer enabled hardware medium (104) is a processor (104) or a microchip (104) that is enabled with an edge detector to detect the pothole lying in the trajectory of the vehicle (100). Preferably, the edge detector is a canny edge detector. The processor (104) of the present invention comprises a Central Processing Unit (CPU) (1045), a Graphic Processing Unit (GPU) (1041), a video output means (1046) like an HDMI means, storage means (1042) to store data and also to store the captured Images (120), for example an in-built memory allocation in the processor (104) or a detachable memory card, Camera Serial Interface (1043) and a General-Purpose Input/Output (GPIO) pins (1044).
The indicating means (110) of the present invention is an LED (Light Emitting Diode) indicator (110) enabled with a LED driver circuit (106). Once the processor (104) detects the pothole (103) lying in the trajectory of the vehicle (100), the processor (104) sends signals in form of a 5 Volt input to the LED driver circuit (106). Thereafter, on receiving the Input from
the processor (104), the LED driver circuit (106) lights up an LED bulb (110) mounted on the body of the vehicle (100). Thus, an effective indication is provided to the driver.
The power supply unit (108) of the present Invention has a battery backup (1081), a voltage regulator (1082) and a power output means (1083). According to an embodiment of the present invention, the battery backup of 12 volts provides direct power supply to the LED driver circuit (106) and the LED (110). The said battery backup (1081) is further connected to the voltage regulator (1682) to step down the voltage to 5 volts which is supplied to the

camera device (102) and the processor (104) of the trajectory based pothole detection system (90) of the present Invention.
FIG, 2 illustrates steps to identify the potholes (103) according to an embodiment of the present invention. The camera (102) mounted on the vehicle (100) captures the image (120) of the road (131) on receiving input signals from the processor (104) and provides Image output to the processor (104) via the camera serial interface (1043). The processor (104) converts the captured image (120) of the road (131) to a greyscale image (122). The grey scale image (122) is processed using a simple differentiation-based edge algorithm on the extracted road section (131), i.e., the road (131) appearing in the captured image (120). The canny edge detector is used to carry out edge detection on the captured frame (120) to produce a black and white image (124). As a next step, a filter is applied on the black and white Image (124) to remove noise in the image existing due to the movement of the vehicle (100) or due to intraoperative working of the camera (102) during the running of the vehicle (100). The filter performs a morphological operation like dilation, erosion and opening to remove the unwanted noise from the black and white image (124) and to further refine the image. After refinement, the image Is obtained with filled and clear borders (126). Further, a blob detector Is used to detect the potholes (103) in the captured frame (120). The blob detector detects the regions in a digital image that has differential properties, such as brightness or color, as compared to surrounding regions. The refinement and filtration (503) Is performed to bring the image parameters within the predefined limits of the trajectory based pothole (103) detection system (90) of the present invention and thus, showing only objects (127) or the blobs (127) lying on the road (131). The objects (127) may be the potholes (103) on the road. The blobs (127) are used to plot circles (129) around the detected potholes (103) covering the maximum pothole area as shown in the image (130). The circles

(129) are plotted by finding the centroid and minor axis length of the objects (127) In the binary image.
FIG. 3 illustrates steps of detecting a pothole (103) according to an embodiment of the present invention. Now to detect the pothole (103) lying in the trajectory of the vehicle (100), the centroid and minor axis are plotted on the objects (127) appearing in the image. The centroid is plotted for the objects (127), preferably a pothole (103), found in the image (130) and the minor axis length is determined. Taking centroid as the centre and the minor axis length as the radius, the circle (129) is plotted around the potholes (103) as shown in the image (130). The said circles (129) covering the maximum area of the objects (127) are determined as potholes (103) on the extracted road section (131) of the image (120).
FIG. 4a illustrates a trajectory of vehicle path according to an embodiment of the present invention. The methodology described above gives a clear and defined image (130) having circles (129) which cover the maximum area of the potholes (103) on the road section (131) of the image. Another step in detecting the pothole (103) lying in the trajectory of the vehicle (100) is to identify the trajectory of the vehicle path where the vehicle (100) is driven. The trajectory of the vehicle path varies with different size and shape of the road (131). Generally, the roads (131) are either curved or straight. Accordingly, a centre line (1055) is drawn on the road section (131) of the captured Image (120) to indicate the trajectory (1055) of the vehicle path. The present invention projects the trajectory for all kinds of vehicle path, i.e. vehicle path for various road contours. For example, in the instant figure, a centre line (1055) Is drawn on the road (131) which is of straight contour and similarly, a centre line (1055) is drawn on the road (131) which is of curved contour. Thus, the trajectory of vehicle path is plotted.

FIG. 4b illustrates detecting a pothole (103) lying in the trajectory of the vehicle (100) according to an embodiment of the present invention. After plotting the circles (130) and identifying the trajectory of the vehicle path, the next step is to find whether the pothole (103) is in the path of the vehicle (100), i.e., if the pothole (103) is lying in the trajectory of the vehicle (100). For example, the instant figure shows small sized potholes (103) present on the road (131) that is not cutting the trajectory of the vehicle (100). Therefore, the driver will not be indicated of the potholes (103) thereof.
Hence, to avoid unnecessary indications to the driver, the present invention provides steps for a confirmation if the pothole (103) is lying in the trajectory of the vehicle (100). The instant figure further shows the presence of a pothole (103) lying in the trajectory of the vehicle (100). Thus, as per the embodiment of the present invention, confirmation and further indication to the driver of the vehicle (100) is provided only on detecting the pothole (103) lying in the trajectory of the vehicle (100). The steps for checking the presence of a pothole (103) on the trajectory of the vehicle (100) are as follows:
a. As aforesaid, the centre line (1055) is drawn on the road section (131) of the captured
image (120) to indicate the trajectory of the vehicle path,
b. determining the distance d from the centroid of each pothole (103) to the centre line
(1055) as drawn in step (a),
c. comparing the distance d calculated from step (b) with the minor axis length of the
pothole (103) as detected in preceding section,
d. confirming the presence of the pothole (103) if the distance d (as calculated from step
(b)) is less than the minor-axis length.
e. indicating to the driver about a pothole (103) lying in the trajectory of the vehicle
(100) by lighting the LED light (110).

FIG. 5 Illustrates a flow diagram according to an embodiment of the present invention. The steps to Indicate the pothole (103) lying in the trajectory of the vehicle path starts with an image acquisition step (501) wherein the camera device (102) captures the image (120) of the road (131) ahead and transfers the image (120) to the processor (104) using the Camera Serial Interface (1043). In step (502), the captured image (120) undergoes various operations by the processor (104) wherein the captured image (120) is first converted to a grey scale image (122) and then to a black-and-white image (124) by an edge detector. In step (503), the black-and-white image (124) undergoes refinement and filtration using filters and morphological operations to obtain the image (126) with filled and clear border. The black-and-white image (124) with filled and clear borders as shown in image (126) is further processed to obtain blobs (127) or objects (127) of the detected potholes (103). That is, the image (128) is generated wherein the white colour objects (127) or the blobs (127) are detected potholes (103) on the black background of the road section (131). This helps in clearly detecting the potholes (103).
In step (504), a centroid is plotted on the object (127) as a centre and minor axis is plotted on the object (127) having radius r as the length. In step (505), the circle (129) is plotted around
the object (127) using the centroid as centre and the minor axis length as radius r.
Step (506) describes plotting of the safe trajectory wherein the trajectory of the vehicle path is plotted as per the shape of road, i.e. according to the various road contours (straight, curved, etc), as shown in Fig. 4a. In plotting the safe trajectory, a centre line (1055) Is drawn on the road section (131) of the captured image (120) which indicates the trajectory of the vehicle path.
In step (507), a check for presence of the pothole (103) which intersects with the trajectory of the vehicle path is conducted. In this step, distance d is calculated from the centroid of the

pothole (103) to the trajectory of the vehicle path or the centre line (1055). Thereafter, the said distance d is then compared with the radius r of the pothole (103).
In step (508), if the distance d is found less than the radius (minor axis length) r, the system (90) confirms that the pothole (103) lies in the trajectory (1055) of the vehicle path and the LED (110) is lighted up as an indication to the driver, as Illustrated in step (509).
However, if the distance d is greater than the radius r of the pothole (103), the system (90) is directed to step (600) wherein it is confirmed that the pothole (103) lies outside the trajectory (1055) of the vehicle path and therefore, no indication is provided to the driver.
Accordingly, after completing of either the step (509) or the step (600), the same process restarts from step (601) to detect and confirm the presence of the pothole (103) in the trajectory of the vehicle path for subsequent captured images (120) and/or for other potholes (102) appearing in the road section (131) of the captured image (120).
FIG. 6 illustrates an application of the trajectory based pothole detection system (90) according to an embodiment of the present invention. The pothole detection system (90) of the present invention has an application in various kinds of vehicles (100), i.e., preferably, a
bicycle, two wheelers and three wheeler vehicles. An example for implementation of an embodiment of the present Invention is illustrated herein only to describe the application of the present invention on two wheeler vehicles and in no way the scope of the present invention shall be limited to this example. The camera device (102) of the trajectory based pothole detection system (90) is mounted on a two-wheeler vehicle (100), preferably, a bike (100). The camera device (102) is mounted on a front surface of the bike (100). The camera (102) is preferably mounted above the windscreen facing towards the road. The camera device (102) faces outwards he. towards the path where the vehicle (100) Is being driven such that the said camera (102) is able to capture images (120) of the road ahead. The images

captured (120) are transferred to a processor (104) which is mounted on the vehicle (100) In connection with the camera (102) and the indicating means (110). The LED (110) is fitted on the front pane! of the bike. Preferably, the LED (110) is fitted near to the meter section of the bike where all other indicators are arranged. The LED circuit (106) on receiving signals from the processor (104) Indicates the driver about the pothole (103) lying In the trajectory of the vehicle (100). Thus, the driver can take timely steps to evade the said pothole (103).
Various modifications to these embodiments are apparent to those skilled in the art from the description and drawings herein. The principles associated with the various embodiment defined herein may be applied to other embodiments. Therefore, the description is not intended to be limited to the embodiments shown along with the accompanying drawings but is to be provided the broadest scope consistent with the principles and novel and inventive features describe/disclosed or suggested herein. Any modifications, equivalent substitutions, improvements etc. within the spirit and principle of the present invention shall all be included in the scope of protection of the present invention.

We claim:
1. A trajectory based pothole detection method comprising:
capturing an image (120) of road by a camera device (102) mounted on a
vehicle (100);
transferring said captured image (120) to a processor (104) using a camera
serial interface (1043);
converting said captured image (120) to a grey scale image (122);
applying differentiation based edge algorithm on said grey scale image
(122) to obtain a black-and-white image (124);
- filtering and refining said black-and-white image (124) to remove unwanted noise using filters;
- detecting blobs (127) on said image (124) to determine position of potholes (103) in the extracted road section (131) of the captured image (120);
- plotting centroid and minor axis length on said potholes (103) appearing in the image (120);
plotting a circle around the potholes (103) covering maximum pothole area using the centroid as centre and considering minor axis length as radius r; plotting a centre line (1055) on the road captured in said image (120) to plot a safe trajectory;
calculating distance between a centroid (1055) of said pothole (103) and a centre line (1055);
- comparing distance with minor axis length of said pothole (103),
confirming presence of said pothole (103) if distance is less than minor-
axis length; and

indicating driver when said pothole (103) lying in trajectory of said vehicle (100) by an LED (110).
2. The trajectory based pothole detection method as claimed in claim 1 further comprising morphological filtering of said image (120).
3. The trajectory based pothole detection method as claimed in claim 1 further comprising restarting (601) the process when distance is greater than minor axis length of said pothole (103).
4. The trajectory based pothole detection method as claimed in claim 1 further comprising confirming presence of said potholes (103) lying in trajectory of said vehicle (100) for each pothole (103) separately.
5. The trajectory based pothole detection method as claimed in claim 1 further comprising sending signals to an LED driver circuit (106) by said processor (104) to indicate said pothole (103) lying in the trajectory of said vehicle (100).
6. The trajectory based pothole detection method as claimed in claim 5 further comprising sending signals to an LED (110) by said LED driver circuit (106) to indicate driver of said pothole (103) lying in trajectory of the vehicle (100).
7. The trajectory based pothole detection method as claimed in claim 6 further comprising said LED circuit (106) and said LED (110) receiving power supply from a power supply unit (108).
8. The trajectory based pothole detection method as claimed in claim 1 further comprising said camera device (102) and said processor (104) receiving power supply from a power supply unit (108).
9. The trajectory based pothole detection method as claimed In claim 8 further comprising stepping down of voltage using a voltage regulator (1082) in said power

10. The trajectory based pothole detection method as claimed In claim 1 further comprising a canny edge detector to produce a black-and-white image (124).
11. A trajectory based pothole detection system comprising:
a camera device (102) mounted on a vehicle (100) to capture images of a
road,
a processor (104) for detecting a pothole (103) lying in a trajectory of a
vehicle (100) in connection with a camera device (102), said processor
further comprising a central processing unit (1045), a storage means
(1042), a graphic processing unit (1041) , a video output means (1046), a
camera serial interface (1043), an input means and an output means
(1044),
an indication means to indicate the driver of a pothole (103) lying in the
trajectory of said vehicle (100), said indicating means includes an LED
driver circuit (106) and an LED (110), and
a power supply unit (108) supplying power to a camera device (102), a
processor (104) and an indicating means (110) including at least one
battery (1081), a voltage regulator (1082) and a power supplying means,
wherein
said captured image (120) is transferred to said processor (104) using said
camera serial interface (1043),
said captured image (120) is converted to a grey scale image (122) and to a
black-and-white image (124) by an edge detector,
said black-and-white image (124) is filtered to remove unwanted noise and
to detect a blob (127) in said image (126),

the blobs (127) in the image (128) are detected as potholes (103) in the extracted road section (131) of the captured image (120), a centroid and a minor axis length are plotted on said potholes (103), a circle is plotted on the detected potholes (103) covering maximum pothole area using the centroid as centre and minor axis length as radius r,
- a safe trajectory is plotted by a centre line (1055) on said road (131)
captured in said image (120),
distance d is determined from centroid of said pothole (103) to said centre line (1055) and is compared with minor axis length of said pothole (103), and
the driver is indicated of said pothole (103) lying in the trajectory of said
vehicle (100) if the compared distance is less than the minor axis length of
said pothole (103).