Description:
Technical Field
[0001] The invention relates to event-based vision sensors generally and, more particularly, to the processing of an event queue in situations where events occur in bursts.
Background
[0002] Figure 1 shows a block-diagram of a typical event-based vision sensor. It comprises an array of pixels 10 wherein each pixel is configured to produce events asynchronously as contrast changes are detected in the pixel. Each event represents a relative increase or decrease in light intensity that exceeds a threshold. An arbiter 12 assigned to the array detects the events and, with the aid of a control circuit 14, organizes their readout synchronously through a readout circuit 16, at the pace of a system clock CK. Upon readout, each event is assigned a timestamp TS, its coordinates (X, Y) in the array, its signed threshold level ∆, and is placed in an event stream ESTRM for transmission and further processing in an application program.
[0003] With such a configuration, situations occur that may cause a disparity be-tween the assigned timestamps and the actual occurrence of the events.
Summary
[0004] An event-based vision sensor generally comprises an array of event-based pixels, each pixel configured to record events based on variations of a sensed pho-tocurrent and enable, at the occurrence of an event, a request line common to pix-els aligned along an axis of the array; a request arbiter configured to respond asynchronously to the enabled request lines by enqueueing corresponding line readout requests in a request queue, and serve the request queue sequentially; an event readout circuit connected to the request arbiter and configured to readout in parallel the events of a line of pixels corresponding to a current line readout re-quest served by the arbiter, and transmit the read-out events with a timestamp indicative of the time of readout in an event stream; a queue length counter connected to the request lines to produce the request queue length as the number of enabled request lines; a queue monitor connected to the queue length counter, configured to generate an event burstiness indicator based on the current request queue length; and means configured to carry out a burstiness compensation action based on the event burstiness indicator.
[0005] The readout circuit may be configured to insert successive event burstiness indicator values in the event stream, and the sensor may further comprise a pro-cessor receiving the event stream and programmed to apply processing measures on the event stream in response to the event burstiness indicator values.
[0006] The burstiness indicator may be the queue length and the arbiter may be controlled, when the queue length exceeds a threshold, to subsample the request queue such that only requests corresponding to regions of interest of the pixel ar-ray are served, and to reset resulting unserved requests in the request queue.
[0007] The sensor may further comprise a row request arbiter configured to serve row readout requests enqueued in a row request queue; a row readout circuit asso-ciated with the row request arbiter; a column request arbiter configured to serve column readout requests enqueued in a column request queue; a column readout circuit associated with the column request arbiter; a row queue length counter and a column queue length counter respectively assigned to the row and column re-quest queues; the request queue monitor associated with the row and column re-quest arbiters, to: activate the row request arbiter for serving the row request queue when the row request queue length exceeds the column request queue length by a threshold, and activate the column request arbiter for serving the col-umn request queue when the column request queue length exceeds the row request queue length by a threshold.
[0008] The row and column request arbiters may be configured to operate simul-taneously to enqueue the respective row and column readout requests and the re-quest queue monitor may be configured to reset the request queue of an arbiter switched from an active state to an inactive state.
Brief Description of the Drawings
[0009] Embodiments will be exposed in the following description provided for exemplary purposes only, in relation to the appended drawings, in which:
[0010] Figure 1 is a block diagram of an event-based vision sensor organized to process events by rows in a pixel array;
[0011] Figure 2 is a block diagram of an event-based vision sensor implementing an embodiment of event burst mitigation based on a burstiness indicator; and
[0012] Figure 3 is a block diagram of an event-based vision sensor illustrating another embodiment of event burst mitigation based on a burstiness indicator.
Detailed Description
[0013] In an event-based vision sensor, when a burst of events occurs, for instance upon a sudden scene brightness change, significant latency may arise between the occurrence of an event and its actual readout, introducing an error in the timestamp that is generally assigned upon readout.
[0014] Moreover, the transmitted event stream sees a burst of event data, that may cause a large traffic load on a chip-to-chip interface and a high event processing load on a host processor.
[0015] It is provided in the present disclosure to monitor event generation and transmit event burstiness information for use by downstream processing to apply compensation measures where possible.
[0016] Referring again to Figure 1, an exemplary situation is illustrated than can cause an event burst. The sensor array 10 observes a sudden brightness variation along a line spanning eight pixels, generating eight events EV1-EV8. In this ex-ample, the pixels happen to belong to three columns and eight rows.
[0017] In a typical event-based sensor, each row of the pixel array is assigned a request line REQ monitored by the arbiter 12. A pixel that detects an event imme-diately enables the request line of its row, by pulling the line to an active level. Any subsequent events in the same row have no effect on the request line if the request line is already active. The arbiter eventually serves the request by enabling an acknowledge line ACK common to the row of pixels, which causes the transfer of the event data of the entire row to latches of the readout circuit 16, and the clearing of the request line. Only the data of the active events of the row are parsed by the readout circuit, and concatenated into the event stream ESTRM, together with the current timestamp TS and the X, Y coordinates provided by the positions of the active events in the read-out row and the rank of the current re-quest line served by the arbiter.
[0018] If the arbiter does not have time to serve a request of a first row before events occur in other rows, generating subsequent requests, these subsequent re-quests are put into a request queue 18 and served later. The enqueuing of the re-quests is performed asynchronously by combinatorial logic in the arbiter. Such may be the situation for the depicted events EV1-EV8, which generate eight dis-tinct requests asynchronously. The arbiter and control circuitry can only serve one such request at a time, synchronously, at the pace of the system clock CK. If the events occur faster than the system clock, the arbiter serves the first request and puts the others in the queue. The remaining requests are served according to a given priority rule, preferably a first-in-first-out rule. Other priority rules may be implemented, especially if it is difficult to trace the asynchronous order of occur-rence of the requests, such as a fair arbitration rule based on the position of the first row served, such that a given row is served a second time only if the other rows have been served at least once.
[0019] When the arbiter eventually serves a request in the queue, it is apparent that the timestamp assigned to the corresponding event, which is the time of readout, lags behind the time of the event. The lag, translating to a timestamp error in the event stream, is variable and depends on the position of the request in the queue, which position can vary between 1 and a variable length reached by the queue. The length of the queue depends on the number of rows affected by a burst of events that occurs within one period of the system clock.
[0020] The position of a request in the queue does not, in general, provide a direct indication of the lag. Indeed, if the events occur at the same time, such as when a flash is observed in the scene, the requests will be enqueued at different positions that all represent the same time of occurrence. Hence, a position N in the queue could, upon service by the arbiter, represent any time lag between one system clock period and N clock periods.
[0021] Moreover, as a pending request awaits in the queue, a more recent event may occur in the corresponding row. Since the pending request line is still active, the new event does not change anything. Thus, a same pending request may repre-sent different event occurrence times within the time it takes to serve the request.
[0022] It is provided herein to use an event burstiness indicator based on the queue length to apply burst mitigation measures. The burstiness indicator may simply be the current queue length, or a statistical indicator representative of the queue length evolution.
[0023] In a simple embodiment, the burstiness indicator may be inserted with the event data in the event stream, where the indicator transmitted with a current event represents the burstiness to be considered for a group of previously trans-mitted events. The indicator may also be communicated on an available hardware interface. A downstream processor of the sensor or application program may use the burstiness indicator to adapt the event data processing to account for the pos-sible timestamp errors or apply processing that does not require accurate timestamps. In any case, the burstiness indicator may be used by the downstream processing to detect a return-to-normal condition after an event burst condition, where the timestamps can be trusted again. Indeed, in some situations, bursts of events may be generally ignored, whereby the timestamp information is irrelevant, but it is then useful to know when a burst condition starts and finishes.
[0024] Figure 2 is a block diagram of an event-based vision sensor implementing another embodiment of event burst mitigation based on the request queue length.
[0025] The sensor is a supplemented version of that of figure 1, and comprises same elements, such as the pixel array 10, the row request arbiter 12, here labeled Y-ARBITER, the row readout circuit 16, here labeled X-READOUT, and the row request queue 18, here labeled Y-QUEUE. These elements preserve their previous functionality, i.e., organizing the readout of rows of pixels of the array based on pixels that activate row requests Y-REQ. In other words, these elements are configured to process horizontal lines of pixels (rows) that are aligned along the Y-axis.
[0026] In addition, the sensor comprises elements to process the array perpendicu-larly, along the X-axis, i.e., organizing the readout of vertical lines of pixels (col-umns) based on pixels that activate column requests X-REQ. These elements comprise a column request arbiter X-ARBITER 20, associated with a column re-quest queue X-QUEUE and a column readout circuit Y-READOUT 24. The array 10 is therefore completed by a vertical column request line X-REQ connected to the x-arbiter 20 for each column of pixels, and a number of horizontal lines (not shown) connected to the y-readout circuit 24 for reading out in parallel the events of a column served by the y-arbiter 20. An event stream output by the sensor is produced by merging the outputs of the x- and y-readout circuits, which, as de-scribed later, may operate one at a time.
[0027] Basically, these additional elements operate with columns of pixels like the traditional elements operate with rows of pixels.
[0028] In addition, the control circuitry of the sensor includes a queue monitor QMON 26 that monitors both the row and column request queues Y-QUEUE, X-QUEUE, and each arbiter is assigned a queue length counter YQLEN, XQLEN. Each queue length counter may have a parallel counter structure having one input for each request line and an output encoding the number of inputs that are at an active state. For a sensor having a 1280x728 pixel array, for example, the counter YQLEN has 728 inputs and a 10-bit output for encoding any number between 0 and 727, and the counter XQLEN has 1280 inputs and an 11-bit output.
[0029] The exemplary occurrence of events EV1-EV8 is shown again in the array 10. It is assumed that these events occur simultaneously or at least within one readout cycle (or system clock period CK). According to the traditional row-approach, the events occur in eight distinct rows and thus enable eight distinct row request lines Y-REQ that cannot be served immediately. The y-arbiter puts the requests in the y-queue, which attains a size of 8 (assuming the first of the re-quests is not served immediately).
[0030] In parallel, the events EV1-EV8, occurring in three distinct columns, ena-ble three distinct column request lines X-REQ. The x-arbiter connected to these request lines puts these requests in the x-queue.
[0031] Enqueueing the requests in the x- and y-queues occurs asynchronously and quasi simultaneously.
[0032] When a time comes to serve a queue, the queue monitor 26 compares the queue lengths of the x- and y-queues and enables the arbiter having the shortest queue, here the x-arbiter, for servicing its queue while the other arbiter, here the y-arbiter, is disabled and its request queue is reset. The x-arbiter then serves its three requests, representing the eight pending events in the array, saving five cycles compared to the case where the y-arbiter would have been alone to handle the events.
[0033] The following table further illustrates the above operation in successive cycles of the system clock CK.
Time CK Events Y-Queue X-Queue Readout
< t0
t0 EV1-EV8 8
0 3
2 X:
X: EV1, EV2
t1 0 1 X: EV3-EV5
t2 0 0 X: EV6-EV8
[0034] Just before a cycle time t0, the events EV1-EV8 have triggered the corre-sponding x- and y-requests, the requests have been put in the x- and y-queues, and the queue lengths have been compared by the queue monitor QMON, here ena-bling the x-arbiter. These operations happen asynchronously through combinatorial logic.
[0035] At time t0, the first column of the x-queue, with events EV1 and EV2, is served, the x-queue length is decremented to 2, and the y-queue is reset.
[0036] At time t1, the next column in the x-queue, with events EV3-EV5, is served and the x-queue length is decremented to 1.
[0037] At time t2, the last column in the x-queue, with events EV6-EV8, is served and the x-queue length is decremented to 0. At this time, all requests involved in the previous comparison have been served, whereby a new comparison is initiated. In this example, no further requests have been enqueued, so the system is idle, and the queue monitor performs a comparison at each new cycle to detect a new queue length imbalance.
[0038] While the x-arbiter serves its requests, new events may occur, for instance events E9-E11 occurring in two rows and three columns, as shown in Figure 2. For sake of clarity, it is assumed that these rows and columns differ from those of events E1-E8. As previously mentioned, both x- and y-arbiters are configured to enqueue new requests although only one is active at a time for serving its requests. In other words, the disabled arbiter continues to detect and enqueue requests to prepare for the next comparison.
[0039] The following table illustrates the additional impact of events EV9-EV11, assumed to all occur slightly before time t1.
Time CK Events Y-Queue X-Queue Readout
< t0
t0 EV1-EV8 8
0 3
2 X:
X: EV1, EV2