DESC:FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
The Patent Rules, 2003
COMPLETE SPECIFICATION
(See sections 10 & rule 13)
1. TITLE OF THE INVENTION
A DEPTH-ENHANCED NEURAL STATE SYSTEM AND METHOD FOR ACCURATE TRAFFIC LIGHT DETECTION AND CONTEXTUAL DECISION-MAKING IN AUTONOMOUS VEHICLES
2. APPLICANT (S)
S. No. NAME NATIONALITY ADDRESS
1 NMICPS Technology Innovation Hub On Autonomous Navigation Foundation IN C/o Indian Institute of Technology Hyderabad, Kandi, Sangareddy, Telangana– 502284, India.
2 Indian Institute Of Technology Hyderabad IN Kandi, Sangareddy, Telangana– 502284, India.
3. PREAMBLE TO THE DESCRIPTION
COMPLETE SPECIFICATION

The complete specification particularly describes the invention and the manner in which it is to be performed.

FIELD OF INVENTION:
[001] The present invention relates to the field of autonomous vehicle systems. The present invention in particular relates to a depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles under variable traffic conditions.
DESCRIPTION OF THE RELATED ART:
[002] Real-time Traffic Light Detection and Recognition (TLDR) is a critical challenge for autonomous vehicles (AVs), particularly in complex traffic scenarios like those found in India. Existing neural network-based object detection models struggle with the various aspects of Indian traffic:
[003] Traffic light positions, shapes, and even colors can vary significantly compared to standardized systems in other countries. Additionally, the chaotic nature of Indian traffic, with frequent occlusions and background clutter, poses difficulties for detection algorithms.
[004] High false positive rates and reduced accuracy lead to the models incorrectly identifying objects as traffic lights (false positives) or missing actual traffic lights altogether, compromising the safety of AV operation. Hence Improved TLDR Systems are needed.
[005] Reference may be made to the following:
[006] Patent No. US11837090 relates to a method that may include obtaining input information relating to an environment in which an autonomous vehicle (AV) operates. The input information may describe a state of the AV, an operation of the AV within the environment, a property of the environment, or an object included in the environment. The method may include setting up a traffic rule profile for the AV that specifies societal traffic practices corresponding to a location of the environment. The method may include identifying a first traffic rule relevant to the AV based on the obtained input information and determining a first decision corresponding to the traffic rule profile and the first traffic rule. The method may include sending an instruction to a control system of the AV, the instruction describing a given operation of the AV responsive to the traffic rule profile and the first traffic rule according to the first decision.
[007] Publication No. CN114707359 relates to an automatic driving automobile decision planning method based on value distribution reinforcement learning, and belongs to the field of automatic driving automobiles. The method comprises the following steps: S1, constructing a traffic light-free crossroad scene considering uncertainty; s2, constructing a full-parameterized quantile function model as an automatic driving automobile control model; and S3, based on the state-action return distribution information learned in the full-parametric quantile function model, introducing a conditional value-at-risk, and generating a driving behavior with risk awareness. According to the method, value distribution reinforcement learning is utilized to improve the safety and stability of the decision planning strategy of the autonomous vehicle in an uncertain environment.
[008] Publication No. CN114613177 relates to a method for planning the speed and acceleration of an automatic driving vehicle passing through a traffic light intersection. The method comprises the following information judgment steps: judging whether the vehicle drives into a traffic light road section or not; judging the state of the vehicle before driving into the parking line; judging the longitudinal distance between the target vehicle and the vehicle; the relative speed of the target vehicle and the vehicle is judged; judging whether the zebra crossing pedestrians are in the left-right front area of the vehicle or not; judging the current traffic light state; judging the conditions of the intersection passing through the traffic light; and the emergency braking condition is judged. And performing planning and state decision-making on the speed and the acceleration passing through the traffic light intersection at the current moment by combining the judgment results, and if decision-making conditions are met, performing planning of the corresponding speed and the acceleration, and outputting corresponding parameters to the vehicle.
[009] Publication No. CN106920403 relates to a single-point adaptive control method based on an array radar, wherein the method mainly relates to the field of intersection traffic signal adaptive control. The single-point adaptive control method realizes dynamic prediction and signal optimization for traffic state of a single intersection through a characteristic that a novel array radar detector can detect positions and speeds of vehicles in an inlet lane. The method comprises the steps of determining a signal base solution; detecting an initial queue length at the intersection according to the array radar, and calculating initial green light time; acquiring vehicle information of each inlet lane, and predicating time in arriving at a stopping line, and determining state of the vehicle in passing the stopping line; calculating the delay and number of parking times of each vehicle; and determining a phase switching decision. Through arrayed radar technology, the single-point adaptive control method has the advantages of performing accurate detection for real-time traffic state of each inlet lane of the single intersection, performing adaptive control and improving operation efficiency and service level of the intersection traffic.
[010] Publication No. US2017243073 relates to a method and system to determine whether a traffic light applies to a vehicle. Traffic light count, visibility duration and spatial position are analyzed to determine the applicability of a traffic light to a vehicle.
[011] IN Publication No. 202441030125 relates to a sophisticated deep reinforcement learning framework specifically designed to enhance the performance and safety of autonomous vehicles. The framework integrates a novel combination of data collection, advanced learning algorithms, real-time decision-making processes, and a dynamic feedback system. It employs a unique architecture that allows the autonomous vehicle to continuously learn and adapt to diverse and changing environments. By leveraging real-time environmental data and historical driving data, the vehicle autonomously refines its driving strategies, improving its decision-making capabilities and adaptability to unforeseen situations. This results in a robust autonomous driving system that excels in various driving conditions, thereby significantly advancing the field of autonomous transportation.
[012] IN Publication No. 202411009126 relates to traffic sign recognition, and plays a critical role in enhancing road safety and supporting the development of autonomous vehicles. This research presents a novel traffic sign recognition system leveraging Convolutional Neural Networks (CNNs) and real-time sensor data fusion. The primary objective is to improve the accuracy, robustness, and real-time performance of traffic sign detection and interpretation under varying environmental conditions. The present invention involves the collection of a diverse dataset comprising thousands of traffic sign images, encompassing various sign types, lighting conditions, and sign orientations. We employ a state-of-the-art CNN architecture for feature extraction and classification, optimizing model hyper parameters to minimize over fitting. The details are shown in figures of the present enclosure.
[013] Publication No. US2024005779 relates to a technology for an autonomous vehicle cloud system (AVCS). The AVCS provides sensing, prediction, decision-making, and/or control instructions for specific vehicles at a microscopic level using data from one or more of other vehicles, a roadside unit, cloud-based platform, or traffic control center/traffic control unit. Specifically, the autonomous vehicles are effectively and efficiently operated and controlled by the AVCS. The AVCS provides individual vehicles with detailed time-sensitive control instructions for vehicles to fulfill driving tasks.
[014] Patent No. US10147320 relates to self-driving vehicles safety system, comprising synthesized and coordinated components and entities, including vehicles, pedestrians, and traffic control light mechanisms, exchanging information, employing lidar (light imaging detection and ranging), radar and intelligent computer-based decision support algorithm systems that analyze images and extract information, to provide safety and vehicle control, regulated and prioritized traffic, and reduced vehicle emissions.
[015] Patent No. US10235877 relates to self-driving vehicles safety system, comprising synthesized and coordinated components and entities, including vehicles, pedestrians, and traffic control light mechanisms, exchanging information, employing lidar (light imaging detection and ranging), radar and intelligent computer-based decision support algorithm systems that analyze images and extract information, to provide safety and vehicle control, regulated and prioritized traffic, and reduced vehicle emissions.
[016] Patent No. US10005460 relates to determining whether a vehicle should continue through an intersection. For example, one or more of the vehicle's computers may identify a time when the traffic signal light will turn from yellow to red. The one or more computers may also estimate a location of a vehicle at the time when the traffic signal light will turn from yellow to red. A starting point of the intersection may be identified. Based on whether the estimated location of the vehicle is at least a threshold distance past the starting point at the time when the traffic signal light will turn from yellow to red, the computers can determine whether the vehicle should continue through the intersection.
[017] The article entitled “An improved traffic lights recognition algorithm for autonomous driving in complex scenarios” by Ziyue Li, Qinghua Zeng, Yuchao Liu, Jianye Liu, and Lin Li; International Journal of Distributed Sensor Networks; 20 January 2021 talks about the image recognition is susceptible to interference from the external environment. It is challenging to accurately and reliably recognize traffic lights in all-time and all-weather conditions. This article proposed an improved vision-based traffic lights recognition algorithm for autonomous driving, integrating deep learning and multi-sensor data fusion assist (MSDA). We introduce a method to obtain the best size of the region of interest (ROI) dynamically, including four aspects. First, based on multi-sensor data (RTK BDS/GPS, IMU, camera, and LiDAR) acquired in a normal environment, we generated a prior map that contained sufficient traffic lights information. And then, by analyzing the relationship between the error of the sensors and the optimal size of ROI, the adaptively dynamic adjustment (ADA) model was built. Furthermore, according to the multi-sensor data fusion positioning and ADA model, the optimal ROI can be obtained to predict the location of traffic lights. Finally, YOLOv4 is employed to extract and identify the image features. We evaluated our algorithm using a public data set and actual city road test at night. The experimental results demonstrate that the proposed algorithm has a relatively high accuracy rate in complex scenarios and can promote the engineering application of autonomous driving technology.
[018] Current traffic light detection systems for autonomous vehicles struggle in complex environments like India's chaotic streets. These systems often rely solely on 2D image data, making it difficult to distinguish traffic lights from other objects, especially when they're partially hidden by other vehicles or signs. Additionally, traditional models lack the ability to understand the context of the scene or how traffic lights change over time (like color transitions). This leads to a high number of false positives, where the system mistakes regular objects for traffic lights, and reduced accuracy in detecting real ones.
[019] This invention is a new system for autonomous vehicles to improve how they detect traffic lights, especially in complex traffic situations like those found in India. Existing systems often struggle because they only rely on cameras and pictures (2D data) to see the world. This makes it hard to tell traffic lights apart from other objects, particularly when they're hidden by other vehicles or signs. Additionally, these systems can't understand the bigger picture of what's happening around them.
[020] Hence to ensure safe AV navigation in India's diverse traffic environments, new models are needed with advancements in:
• Extracting more informative features from traffic lights despite their small size and complex surroundings.
• Understanding the surrounding traffic scene to better identify traffic lights.
• Analyzing how traffic lights change over time (e.g., color transitions) to improve detection accuracy.
[021] The conventional system does not capture the unique characteristics of Indian traffic lights. Traffic lights are relatively small compared to the overall scene, making them challenging to detect accurately.
[022] In order to overcome the above listed prior art, the present invention aims to provide a depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles under variable traffic conditions. The system provides significant improvement in traffic light detection for autonomous vehicles operating in challenging environments. This not only improves the overall accuracy of the system but also reduces the risk of accidents caused by misinterpreting traffic signals.
OBJECTS OF THE INVENTION:
[023] The principal object of the present invention is to provide a depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles under variable traffic conditions.
[024] Another object of the present invention is to provide a depth-enhanced neural state system and method which provides safer navigation and reduces the risk of accidents caused by misinterpreting traffic lights.
[025] Yet another object of the present invention is to provide a depth-enhanced neural state system and method ensuring smoother and more efficient driving, especially when considering traffic light timings.
SUMMARY OF THE INVENTION:
[026] The present invention relates to the depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles under variable traffic conditions. The invention provides a modular and scalable solution for traffic light detection in autonomous vehicles. This flexibility can be beneficial for future advancements and integration with other systems within the car.
[027] The invention includes depth perception and a neural state machine. Depth perception, potentially achieved through stereo vision cameras, provides a 3D understanding of the scene. This allows the system to more accurately identify traffic lights, even when they're partially obscured. The neural state machine analyzes the surrounding environment and temporal information like color changes. This contextual reasoning helps filter out false positives and ensures the system focuses on genuine traffic lights.
BRIEF DESCRIPTION OF THE INVENTION
[028] It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered for limiting its scope, for the invention may admit to other equally effective embodiments.
[029] Figure 1 object detection architecture according to the present invention;
[030] Figure 2 shows block diagram of depth-enhanced neural state system according to the present invention;
[031] Figure 3 shows flowchart according to the present invention.
DETAILED DESCRIPTION OF THE INVENTION:
[032] The present invention provides a depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles under variable traffic conditions.
[033] The system comprises cameras (1) to capture images of the road ahead, but unlike regular cameras, they can also see depth information, creating a 3D picture. Processing unit (2) is a powerful computer that analyzes the images from the cameras. The system senses the 3D data and identifies traffic lights (fig 1).
[034] Neural state machine (3) analyzes the information from the processing unit, along with the traffic light's color and how it changes over time and distinguishes real traffic lights from anything else. Once the system identifies a traffic light and its state (red, yellow, green) (4), this information is sent to the vehicle's control unit. Vehicle control unit (2). Using the information about the traffic light from the system, the control unit can make decisions about stopping, going, or slowing down. The cameras (1) capture the visual data, including depth information, and transmit it digitally to the processing unit. The processing unit (2) performs the initial analysis of the visual data, extracting relevant features and preparing it for the neural state machine (3).
[035] The processed data is then transferred electronically to the neural state machine. The neural state machine analyzes the data from the processing unit, considering additional factors like traffic light color changes over time. Once the neural state machine identifies a traffic light and its state (red, yellow, green), it sends a digital signal to the vehicle's control unit. The entire system operates electronically within the vehicle's network. This allows for efficient communication and data exchange between components. The physical placement of the cameras is crucial for optimal performance, requiring a clear view of the road. The processing unit and neural state machine can be located more flexibly within the car's computer system as long as they have reliable connections.
[036] The invention provides a modular and scalable solution for traffic light detection in autonomous vehicles. This flexibility can be beneficial for future advancements and integration with other systems within the car.
[037] The invention includes depth perception and a neural state machine. Depth perception, potentially achieved through stereo vision cameras, provides a 3D understanding of the scene. This allows the system to more accurately identify traffic lights, even when they're partially obscured. The neural state machine analyzes the surrounding environment and temporal information like color changes. This contextual reasoning helps filter out false positives and ensures the system focuses on genuine traffic lights.
[038] The system provides a 3D understanding of the scene, allowing for more accurate detection of occluded traffic lights (hidden by other objects) which are common in Indian traffic scenarios.
[039] The neural state machine (e.g., mealy machine controller) to analyze the surrounding traffic scene and temporal information (like traffic light color changes). This contextual analysis helps reduce false positives and improve overall accuracy.
[040] This is not a typical feature of existing traffic light detection systems, but it allows the system's output (traffic light state and position) to directly influence the vehicle's navigation decisions, leading to a more optimized and safe driving experience.
[041] By incorporating depth perception and a neural state machine, the invention can distinguish real traffic lights from background clutter and other objects more effectively. This leads to a significant reduction in false positives and a higher detection rate of actual traffic lights, even when they're partially hidden. With more accurate and reliable traffic light detection, autonomous vehicles can make better decisions based on real-time traffic signals. This provides safer navigation and reduced risk of accidents caused by misinterpreting traffic lights.
[042] By integrating with the vehicle's on-board control unit, the invention's output (traffic light state and position) directly influences the vehicle navigation decisions which leads smoother and more efficient driving, especially when considering traffic light timings. Thus it significantly improves the performance of autonomous vehicles in complex traffic environments, paving the way for safer and more reliable self-driving.
[043] The system allows better understanding of the position of objects in the scene, making it easier to identify traffic lights even if they're partially hidden. The neural state machine analyzes the surroundings and considers how things change over time, like traffic lights switching colors. With this extra information, the system can focus on real traffic lights and ignore anything else.
[044] A neural state machine (NSM) controls vehicle movement, adapts to changing environments and tasks by learning to recognize and respond to new situations. NSMs processes sensory information in real-time, allowing the robot to respond quickly to changing situations. By learning to recognize and respond to errors and uncertainties, NSMs can improve the robustness of the robot's behavior. NSMs can learn to perform complex tasks without requiring extensive programming or manual tuning (Fig 1).
[045] The invention uses stereo vision cameras to create a three-dimensional picture of the environment. This depth of information is crucial for distinguishing traffic lights from other similar-looking objects in congested urban scenes. The cameras capture not just visual images but also spatial depth, which enables the processing unit to accurately locate and identify traffic lights, even when they are obscured by other vehicles or street furniture. This capability reduces errors commonly seen in 2D systems where depth cues are absent.
[046] The system incorporates a neural state machine (NSM) that processes the depth-enriched data from the cameras. This machine not only recognizes traffic lights based on color and shape but also understands their temporal changes (e.g., from green to red) and contextual surroundings. The NSM evaluates how traffic lights change over time, integrating these changes with real-time traffic conditions to make more informed decisions. This dynamic analysis helps minimize the false positives typical of systems that cannot differentiate between traffic lights and other high-luminance objects.
[047] The output from the neural state machine is directly fed into the vehicle's control unit. This integration allows for immediate actions based on the traffic light states detected, such as stopping, slowing down, or continuing through an intersection. By reducing the latency typically involved in decision-making processes in autonomous vehicles, this system ensures faster and safer responses to traffic light changes, crucial for effective navigation in unpredictable traffic scenarios.
[048] The system is modular and scalable, accommodating enhancements such as integration with other vehicle systems or updates in sensor technology. This flexibility allows for the system to be updated with future technologies without needing significant overhauls, ensuring longevity and adaptability of the autonomous vehicle’s navigation capabilities.
[049] By employing depth perception and a sophisticated neural state machine, the system provides a comprehensive understanding of the 3D scene, significantly improving the detection of occluded or partially visible traffic lights. This is particularly beneficial in environments like India, where traffic conditions can vary dramatically, and traffic lights are often obscured by other elements. It ensures a robust detection mechanism, reducing the risk of accidents caused by misinterpreted or unseen traffic lights.
[050] The system provides significant improvement in traffic light detection for autonomous vehicles operating in challenging environments. This not only improves the overall accuracy of the system but also reduces the risk of accidents caused by misinterpreting traffic signals.
[051] Numerous modifications and adaptations of the system of the present invention will be apparent to those skilled in the art, and thus it is intended by the appended claims to cover all such modifications and adaptations which fall within the true spirit and scope of this invention.
,CLAIMS:WE CLAIM:
1. A depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles under variable traffic conditions comprises-
a) Cameras (1) to capture images of the road ahead, but unlike regular cameras, they can also see depth information, creating a 3D picture to identify traffic lights
b) Control and processing unit (2) is a powerful computer that analyzes the images from the cameras and output from the neural state machine is directly fed into the vehicle's control unit.
c) Neural state machine (3) analyzes the information from the processing unit, along with the traffic light's color, changes over time and distinguishes real traffic lights from anything else and once the system identifies a traffic light and its state (red, yellow, green) (4) and vehicle's control unit, using the information make decisions about stopping, going, or slowing down.
d) The cameras (1) capture the visual data, including depth information, and transmit it digitally to the processing unit.
2. The depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles, as claimed in claim 1, wherein neural state machine (NSM) that processes the depth-enriched data from the cameras, recognizes traffic lights based on color and shape but also understands their temporal changes, evaluates traffic lights change over time, integrating these changes with real-time traffic conditions to make more informed decisions.
3. The depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles, as claimed in claim 1, wherein depth perception is achieved and a neural state machine, through stereo vision cameras, provides a 3D understanding of the scene which accurately identifies traffic lights, analyzes the surrounding environment and temporal information like color changes and filters out false positives and ensures the system focuses on genuine traffic lights.
4. The depth-enhanced neural state system and method for accurate traffic light detection and contextual decision-making in autonomous vehicles, as claimed in claim 1, wherein dynamic analysis helps minimize the false positives typical of systems that cannot differentiate between traffic lights and other high-luminance objects.