Field of Invention:
The present invention related to. robotic vision system ^ artificial intelligence, Big Data Analytics machine learning and predictive analytics in Robotics , Unmanned System and for blind or visually impaired persons .More specifically, it describes a novel artificial intelligence system that is readily
deployable, efficient and easy to install and use involving intelligent analysis of imaging records of a person(s) based on predefined algorithms and self-evolving artificial intelligence and predict his future state and generate results in a desirable form. Data set of present conditions of environment and their previous state is used as raw data in the system.
Background:
The present invention applies artificial intelligence and predictive analysis on the data available in the system to generate required results for the navigation or data collection of Unmanned Aerial System during day and or Night time using N-SLAM.
Summary:
The present invention related to artificial intelligence, Data Mining, machine learning and predictive
analytics in robotic system for the navigation or data collection of Unmanned Aerial System during
day and or Night time using N-SLAM.
Brief description of drawings:
The detailed description is described with reference to the accompanying figures. In the figures, the
left most digit in the reference number identifies the figure in which the reference number first
appears. The same numbers are used throughout the drawings to reference like features and
components.
FIG1. Illustrates a block diagram representation of the localization , estimation and control for
Aerial Vehicle during navigation at day or night time using N-SLAM.( Night System Localization and
Mapping)
Detailed description of drawings:
Exemplary embodiments will now be described with reference to the accompanying drawing. The
invention may, however, be embodied in many different forms and should not be construed as
limited to the embodiments set forth herein ; rather, these embodiments are provided so that this
invention will be thorough and complete , and will fully convey its scope to those skilled in the art.
The terminology used in the detailed description of the particular exemplary embodiments
illustrated in the accompanying drawings is not intended to be limiting. In the drawings, like
numbers refer to like elements.
Reference in this specification to "one embodiment"or "an embodiment" means that a particular
feature , structure ,or characteristic described in connection with the embodiment is included in at
least one embodiment of the disclosure. The appearances of the phrase "in one embodiment" in
various places in the specification are not necessarily all referring to the same embodiment, nor are
separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various
features are described which may be exhibited by some embodiments and not by others. Similarly,
various requirements are described which may be requirements for some embodiments but not
other embodiments.
The specification may refer to "an", "one" or "some" embodiment(s) in several locations. This does
not necessarily imply that each such reference is to the same embodiment(s), or that the feature
only implies to a single embodiment. Single features of different embodiments may also be
combined to provide other embodiments.
As used herein , the singular forms "a", "an" and "the" are intended to include the plural forms as
well , unless expressly stated otherwise. It will be further understood that the terms "includes" ,
"comprises" , "including", and/or "comprising" when used in this specification, specify the presence
of stated features, integers, steps, operations, elements, and/or components, but do not preclude
the presence or addition of one or more other features, integers, steps, operations, elements,
components, and/or groups thereof. It will be understood that when an element is referred to as
being "connected" or "coupled" to another element, it can be directly connected or coupled to the
other element or intervening elements may be present. Furthermore, "connected" or "coupled" as
used herein may include wirelessly connected or coupled. As used herein, the term "and/or"
includes any and all combinations and arrangements of one or more of the associated listed items.
Unless otherwise defined, all.terms (including technical and scientific terms) used herein have same
meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
It will be further understood that term, such as those defined in commonly used dictionaries, should
be interpreted as having a meaning that is consistent with their meaning in the context of the
relevant art and will not be interpreted in an idealized or overlay formal sense unless expressly so
defined herein.
The terms used in this specification generally have their ordinary meanings in the art, within the
context of the disclosure, and in the specific context where each term is used. Certain terms that are
used to describe the disclosure are discussed below, or elsewhere in the specification, to provide
additional guidance to the practitioner regarding the description of the disclosure. For convenience,
certain terms may be highlighted, for example using italics and/or quotation marks. The use of
highlighting has no influence on the scope and meaning of a term; the scope and meaning of a term
is the same, in the same context, whether or not it is highlighted. It will be appreciated that same
thing can be said in more than one way.
The figures depict a simplified structure only showing some elements and functional entities, all
being logical units whose implementation may differ from what is shown. The connections shown
are logical connections; the actual physical connections may be different.
Consequently, alternative language and synonyms may be used for any one or more of the terms
discussed herein, nor is any special significance to be placed upon whether or not a term is
elaborated or discussed herein. Synonyms for certain terms are provided. A recital of one or more
synonyms does not exclude the use of other synonyms. Th£*ssag§f examples anywhere in this
specification including examples of any terms discussed herein is illustrative only, and is not
intended to further limit the scope and meaning of the disclosure or of any exemplified term.
Likewise, the disclosure is not limited to various embodiments given in this specification.
Without intent to further limit the scope of the disclosure, examples of instruments, apparatus,
methods and their related results according to the embodiments of the present disclosure are given
below. Note that titles or subtitles may be used in the examples for convenience of a reader, which
in no way should limit the scope of the disclosure. Unless otherwise defined, all technical and
scientific terms used herein have the same meaning as commonly understood by one of ordinary
skill in the art to which this disclosure pertains. In the case of conflict, the present document,
including definitions will control.
According to the preferred embodiment of the present invention In order to initialize a visual
localization and mapping algorithm, a good estimate of the position based on the internal sensors of
the MAV is essential. The design includes a powerful onboard computer which makes it possible to
run high-level task, in particular visual localization and mapping, onboard the MAV. The position
controller takes as input the poses from visual localization and the setpoints ( image markers )
generated from the planner. For MAV controlling the attitude measurements and the vision pose
estimates need to be synchronized. In our system the synchronization is solved.by hardware triggering
the cameras by the 1MU and timestamping the measurements. Vision-IMU Synchronization Data
from different sensors will be synchronized by an electronic shutter signal and put together into a
timestamped sensor message.
The MAV is equipped with a number of sensors, which give frequent updates about the motion.
The inertial sensor measures the body accelerations and angular velocities. A proprietary filter on
board of the Hawk Eye converts the angular velocities to an estimated attitude (orientation). An
ultrasound sensor is used to measure the altitude of the vehicle. In addition to the body accelerations,
the MAV sends an estimate of its estimated body velocity . This estimate is based on the inertia
measurements, aerodynamic model and visual odometry obtained from the relative motion between
camera frames.
The information from the sensors are used in an Extended Kalman Filter to estimate the current pose.
An EKF has been considered the facto standard in the theory of nonlinear state estimation. The EKF
state vector comprises a position vector pW, velocity vector vW, acceleration vector aW and attitude
(orientation) vector qW. All vectors are 3-dimensional Cartesian coordinates. The position -vector pW
includes the estimated altitude, which . An additional altitude vector hW is added to the EKF state
vector, which contains the unmodified ultrasound measurement without the estimated
elevation.Furthermore, it contains first-order and second-order derivatives of the ultrasound
measurement. These derivatives are used by the elevation mapping method .
The sensor data from the MAV is used to fill a measurement matrix. This measurement matrix
is used by the EKF to update the state . The attitude (orientation) information from the MAV's
onboard filter is written to the attitude vector of the measurement matrix. This is possible because the
attitude measurements are filtered and debiased onboard of the MAV The position of the MAV cannot
be estimated directly and is derived from the estimated velocities.
The velocity estimate sent by the MAV is written to the velocity vector of the measurement
matrix.Based on the previous position , the state's velocity vt and time between measurements _t, the
new position pt can be calculated as follows: The sensor data from the MAV is used to fill a
measurement matrix. This measurement matrix is used by the EKF to update the state . The attitude
(orientation) information from the AR.Drone's proprietary onboard filter is written to the attitude
vector of the measurement matrix. . The position of the AR.Drone cannot be estimated directly and is
derived from the estimated velocities where_t is the variable time (seconds) between the last two
measurements. Instead of using the onboard velocity estimate, the visual odometry method can be
used to estimate velocity.
In theory, the velocity can be estimated by integrating the body acceleration measurements.
However,the acceleration sensor from the provides unreliable data, which would result in large
velocity errors. Therefore, the body acceleration measurements are not written to the EKF's
measurement matrix.
Similar to the estimated position, the vertical velocity estimate is used to estimate the vehicle's
altitude. However, the ultrasound altitude measurement can be integrated into the EKF's measurement
matrix to improve the altitude estimate. The ultrasound sensor is not sensitive to drift because it
provides an absolute altitude measurement. Relying only on the ultrasound sensor is not optimal since
the measurements are affected by the material and structure of the floor .
Mapping :
When an estimate of the MAV pose is available, a map can be constructed to make the MAV
localize itself and reduce the error of the estimated pose. The map consists of a texture map and a
feature map. The texture map is used for human navigation and the feature map is used by the MAV
to localize itself.
Texture map
Now that we have an estimate of the MAV position and attitude in the world, we can use this
information to build a texture map of the environment. The MAV is equipped with a down-looking
camera that has a fixed resolution . The frames captured by this camera can be warped on a flat
canvas to create a texture map. Directly merging the frames on the canvas is not possible, because
individual frames can be taken from a broad range of angles and altitudes. Instead, perspective
correction is applied and all frames are normalized in size and orientatiomMAV camera is modeled
using a pinhole camera model . In this model, a scene view is formed by projecting 3D points into the
image plane using a perspective transformation.
where xf and yf represent a 2D point in pixel coordinates and xw, yw and zw represent a 3D point in
world coordinates. The 3 X 3 camera intrinsic matrix A includes the camera's focal length and
principal point.
The 3 X 4 joint rotation-translation matrix [Rjt] includes the camera extrinsic parameters, which
denote the coordinate system transformations from 3D world coordinates to 3D camera coordinates.
Equivalently, the extrinsic parameters define the position of the camera center and the camera's
heading (attitude) in world coordinates.In order to warp a camera frame on the correct position of the
canvas, the algorithm needs to know which area of the world (floor) is captured by the camera. This is
the inverse operation of the image . formation described above. Instead of mapping 3D world
coordinates to 2D pixel coordinates, 2D pixel coordinates are mapped to 3D world coordinates (mm).
It is impossible to recover the exact 3D world coordinates, because a pixel coordinate maps to a line
instead of a point (i.e., multiple points in 3D world -coordinates map to the same 2D pixel
coordinate). This ambiguity can be resolved by assuming that all 3D world points lie on a plane (zw =
0), which makes it possible to recover xw and yw. The 2D world coordinates of the frame's corners
are obtained by casting rays from the four frame's corners The 2D world coordinate that corresponds
to a frame corner is defined as the point (xw; yw; 0) where the ray intersects the world plane (zw = 0).
Both xw and yw are computed.
Both the camera's extrinsic and intrinsic parameters are required for this operation. The extrinsic
parameters (position and attitude of the camera) are provided by the EKF state vector. The camera
intrinsic parameters are estimated .
"Now, a relation between the pixel coordinates and world coordinates is known. However, the 2D
world coordinates (mm) need to be mapped to the corresponding 2D pixel coordinates (xm; ym) of
the map's canvas. A canvas with a fixed resolution of 4:883mm=px is used.
where smap = 1=4:884.
Now, the map's pixel coordinates of the frame corners are known . In order to transform a frame to
the map's canvas, a relation between the frame's pixels and map's pixels is required.
A transformation from the frame's corner pixel coordinates and the corresponding pixel coordinates
of the map's canvas can be expressed. The local perspective transformation H is calculated by
minimizing the back-projection using a leastsquares algorithm. Find Homography is used to solve the
perspective transformation H. This
transformation describes how each frame's pixel needs to be transformed in order to map to the
corresponding (sub) pixel of the map's canvas. The transformation is used to warp the frame on the
map's canvas. By warping the frame on a flat canvas, implicit perspective correction is applied to the
frames.warpPerspective is used to warp the frame on the map's canvas.
In this way a texture map can be built. This map consists of a set overlapping textures. The placement
and discontinuities of the overlapping textures can be further optimized by map stitching,
Feature map :
The inertia measurements and visual odometry provide frequent position estimates.
If the MAV is able to relate a video frame to a position inside the feature map, the vehicle is able to
correct this drift long enough to build a map.
A 2D grid with a fixed resolution of 100 X 100mm per cell is used. From each camera frame the
method extracts Speeded-Up Robust Features (SURF) that are invariant with respect to rotation and
scale. Each feature is an abstract description of an interesting part of an image . A SURF feature is
described by a center point in image sub-pixel coordinates and a descriptor vector that consists of 64
floating point numbers. Each feature that is detected in a camera frame is mapped to the
corresponding cell of the feature map. A feature's center point (xf ; yf ) is transformed to its
corresponding position in 2D world coordinates (xw; yw), . This is done by casting a ray from the
features pixel coordinates in the frame.
The method is similar to the method used for casting ray's from the frame's corners . Finally,
the 2D world position (xw; yw) of each feature is transformed to the corresponding cell indices (xbin;
ybin),
Localization :
The feature map created in the can be used for absolute position estimates, at the moment of loopclosure
. This allows to correct the drift that originates from the internal sensors of the MAV. The
sensor measurements provide frequent velocity estimates. However, the estimated position will drift
over time, because the errors of the velocities estimates are being accumulated. The feature map that
is created can be exploited to reduce this drift,because localization against this map provides absolute
positions of the vehicle. These absolute positions are integrated into the EKF and improve the
estimated position and reduce the covariance of the state.When a camera frame is received, SURF
features are extracted. Each feature consists of a center positionin pixel coordinates (xf; yf) and a
feature descriptor. A feature's center point (xf; yf) is transformed to its corresponding position in 2D
world coordinates (xw; yw), . This is done by casting a ray from the features pixel coordinates in the
frame. The method is similar to the method used for casting ray's from the frame's corners The next
step is matching the feature descriptors from the camera frame against the feature descriptors from the
feature map. When the feature map is quite large, this process becomes slow. However, the estimated
position of the vehicle can be used to select a subset of the feature map. This can be done by placing a
window that is centered at the vehicle's estimated position. The covariance of the estimated position
can be used to determine the size of the window. The set of frame descriptors (query descriptors -Dq)
is matched against the map descriptors (training descriptors Dt). Matching is done using a brute force
matcher that uses the L2 norm as similarity measure. For each query descriptor dq from the frame,
function C(dq) selects the training descriptor dt from the map that minimizes the L2 norm:
where DT is the set of map descriptors within a window around the estimated position. The L2
distance between two descriptors a and b is defined where N = 64 is the length of the SURF descriptor
vector.
Each query descriptor (frame) is matched against the descriptor from the training descriptors (map)
that is most similar. Please note it is possible that multiple descriptors from the frame are matched
against a single descriptor from the map.
For each match C(dq; dt) the 2D world coordinates (xw;dq ; yw;dq ) and (xw;dt ; yw;dt ) of both
descriptors are already computed. These point pairs can be used to1 calculate a transformation between
the query points (frame) and the training points (map). This transformation describes the relation
between the EKF estimated vehicle position and the position according to the feature map.
Payload ( Optical Sensors ) - IR Based weighing less than 300 gram.Required Optical Sensors ( IR
Based ) will be used in the development of N- SLAM and delivered on the project delivery.
Suitable RF downlink for the Payload to be connected to the Ground control station - A 2.4 GHz
and 5.8 GHz frequency based radio sets will be used for uplinking and downlinking the commands
and visual data between the Micro Air Vehicle and Ground Control Station .
GCS will display the map of the flight area of MAV , Flight parameters , live video and flight path overlay on the Flight Map .

CLAIMS
We claim:
1. A system in artificial intelligence, data mining , Big Data , machine learning and predictive
analytics in day and / or Night System localization and mapping for Unmanned systems.
2. A system as claimed in claim 1 where unmanned system can be partially controlled via a
wireless or tethered link.
3. A system as claimed in Clairri 1 and 2 where the system can be a Unmanned Aerial Vehicle (
UAV), Unmanned Ground Vehicle ( UGV) or an Unmanned Underwater Vehicle (UUV).