DESC:FIELD OF THE INVENTION:
The present invention relates to facilitate video rendering operation. More specifically, the present invention is directed to develop a method for selecting candidate frames of video frame for rendering based on computing resource availability of recipient of the video frames and improves the viewing experience keeping the processing lower.

BACKGROUND OF THE INVENTION:
Monitoring of environment in video surveillance domain, live telecast of incidents – everywhere it is required to display the video frames on monitor without any significant latency. The challenge lies in the fact that the source of the video (e.g. Cameras, Media streamers) usually streams the video at a particular bit rate and fps simultaneously to multiple recipients. But, the recipients may have varying amount of computing resources to receive, decode, process and render the video frames; it varies from one recipient machine to another.
Adaptive frame rate control prior to encoding at the video source in the domain of videoconferencing, announcements and live streaming etc is considered in Rate control algorithm for scalable video encoding with disposable p-frames, US20140376617A1.
There are few works for buffer management at the source [ref: Content server media stream management, US20090259765A1], however receiver specific self-adjusted mechanism to enable smooth rendering of video by the said buffer management scheme is absent.
In a scenario where multicast video is streamed, it is impractical to change FPS or resolution of the video specific to any particular recipient at the source. The recipients themselves should adjust the rendering mechanism to suit availability of computing resources at its own end. So there is a need to control the video rendering mechanism at the recipients’ end, rather than at the source end.

OBJECT OF THE INVENTION:
It is thus basic object of the present invention is to develop a method for selecting candidate frames of video stream for rendering based on computing resource availability of recipient of the video frames.

Another object of the present invention is to develop a method which would be adapted to adjust the video rendering mechanism to suit availability of computing resources at the recipient end to improve the viewing experience and keep the processing lower.

Yet another object of the present invention is to develop a method which would be adapted to implement a control of the video rendering mechanism at the recipients’ end, rather than at the video source end.

SUMMARY OF THE INVENTION:
Thus according to the basic aspect of the present invention there is provided a method for selecting candidate frames of decoded video frames for rendering based on computing resource availability of recipient of the video stream comprising
involving indirect measurement of computing resource availability in the recipient end including
decoding the video frames arriving at the recipient end,
storing the decoded video frames in a queue,
continuously monitoring length of the queue and thereby recognizing computing resource availability based on the queue length;
dynamically reducing video streaming rate to the recipient by selecting candidate frames of the video stream for rendering depending on the computing resource availability ensuring better viewing experience and keeping the processing lower; and
gradually increasing the video streaming rate to the recipient on availability of the computing resource availability.

In the present method, if length of the queue goes beyond a predefined limit determined by admissible latency, it is inferred that the frame rendering and accordingly the frame processing rate is reduced.

In the present method, the selection of the candidate frames of the video stream for rendering depending on the computing resource availability includes dropping few of the frames of the video stream to reduce frame processing rate at the recipient without compromising the viewing experience.

In the present method, the frames arriving at the recipient end are decoded to get raw frames (R,G,B or Y,U,V).

In the present method, the frames are dropped as a multiple of modulo factor (y), whereby the effective FPS (FPSd) after dropping the frames is
FPSd = FPS*(1-(1/y));
where FPS is original frames per second in the video stream and all the incoming frames are numbered in sequence, 1, 2, 3, …..

In the present method, for given values of the modulo factor (y>1), the frames whose frame number in one of the y-values are dropped.

In the present method, the modulo factor (y) is calculated as inverse of an incrementing variable (x) as y = (FPS)/x.

In the present method, increment of the x value reduces the FPS (FPSd) while decrement of the x value increases the FPS (FPSd).

In the present method, the effective FPS is lowered if the length of the queue grows beyond a upper threshold level by increasing the x value and the effective FPS is increased if the length of the queue goes below a lower threshold level by decreasing the x value.

In the present method, if the FPSd becomes 0 or negative then the x with –ve value is started for rendering the frames indicated by modulo factor (y) which is y = -x resulting the calculated or effective FPS (FPSd) = FPS * /y.

BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS:
Figure 1 shows block diagram showing data flow from source of video stream to rendering system in accordance with the present invention.
Figure 2 shows controlling fps based on Q length in accordance with the present invention.
Figure 3 shows basic scheme of controlling processing and rendering FPS in accordance with the present invention.
Figure 4 shows Comparison of simple modulo operation and proposed modulo operation at 25 FPS in accordance with the present invention.
Figure 5 shows comparison of simple modulo operation and proposed modulo operation at 12 FPS in accordance with the present invention.

DESCRIPTION OF THE INVENTION WITH REFERENCE TO THE ACCOMPANYING DRAWINGS:
As described in the accompanying figure 1, the source of the video emits video frames as a constant rate and the recipient receive them at that rate only. However, the processing step between receiving a video frame and rendering it takes a variable amount of time. This time can be more than the inter-frame gap. As for example if the video frames are emitted at 25 fps, the frames arrive at the recipient in real time at an interval of 40ms (=1000ms/25). The processing and rendering stage can take a variable amount of time. The time it takes depend on what processing resource is available at that moment of time and also what type of processing is required on the frame before it is sent for rendering. So the processing time is a time dependent entity, denoted by f1(t) in the diagram. The time it takes may be more than 40ms for a continuous stretch of time. Use of a memory buffer in between Stage 2 and Stage 3 to temporarily hold the frames does not help, as it would create latency in viewing the video.
It is also challenging for the Video renderer application to assess the computing load available to it, as lot of other applications or multiple instances of the same renderer application may be running in the machine. So, an indirect way of assessing the available CPU is developed. As and when the video frames arrive at the recipient end, it is decoded and stored in a queue. The application continuously monitors the length of the queue, and if the length goes beyond a certain limit (determined by admissible latency), it is inferred that the frame rendering and possibly the frame processing rate should be reduced (Fig 2). A technique of gradual reduction of fps and subsequent increase when computation bandwidth is available is also proposed.
In essence, based on current utilization of computation bandwidth by the renderer and processor desired fps at which the frames are processed or rendered is generated at the recipient end.
For example, let the frames be streamed from the source at 12 fps, and are received at the recipient’s end at that fps. For compression formats like H.264 or H.265 or similar intra-frame compression schemes, all the frames are to be decoded to get raw frames (R,G,B or Y,U,V).

Modulo operation to achieve smooth frame drop:
The objective of the method is to select candidate frames for rendering in an attempt to have better viewing experience and keep the processing lower.
Let us consider a modulo factor (y). The frames are dropped as a multiple of the modulo factor (y).
The effective FPS (FPSd) after dropping the frames can be calculated to as below:
FPSd = FPS*(1-1/y) (1a)
Comparison of simple modulo operation and proposed modulo operation is shown in Figure 1 and Figure 2.
The proposed method changes the FPS very rapidly still affects the visual experience in minimum possible manner. The reduction in FPS gives more time for processor to process.

Description of the proposed method:
The modulo factor (y) is calculated as inverse of another incrementing variable (x).
The relationship with y and x as follows:
y = (FPS)/x; (2a)
All the incoming frames are numbered in sequence, 1, 2, 3, …..
If (y > 1) then drop the frames whose frame number in one of these y-values.
The calculated or effective FPS (FPSd), FPSd = FPS * (1 – 1/y) (2a)
If FPSd becomes 0 or negative then start the x with –ve value, and start displaying the frames indicated by modulo factor (y)
y = -x (1b)
The calculated or effective FPS (FPSd),
FPSd = FPS * /y (2b)

In equation number (1a) and (2b) gives the effective FPS (FPSd) calculated from original FPS (FPS) and modulo factor (y)
FPSd = FPS*(1-1/y) (1a)
FPSd = FPS * /y (2b)
The calculated FPSd is shown in the table for an input FPS of 25.
The calculated FPSd varies from 11 to 2.4 gradually. Depending upon chosen x the FPSd can go further down.
Equations (2a) and (1b) shows relation between (y) and (x). Here (x) is the only variable in the system.
We keep on incrementing (x) untill the calculated FPS (FPSd) reaches 0. Once the calculated FPS (FPSd) reaches 0 we decrement the (x) as shown in the following table:
For simplicity we took x=1, but it can be other values also based on how fast the rendering fps is required to come to a steady state and what should be the extent of granularity of fps reduction.
x Y FPSd Comment
1 12 11 Drop every 12th frame
2 6 10 Drop every 6th and 12th frames
3 4 9 Drop every 4th, 8th and 12th frames
4 3 8 Drop every 3rd, 6th , 9th and 12th frames
5 2 6 Drop every 2nd, 4th ,6th, 8th , 10thand 12th frames
6 2 6 If the display FPS (FPSd) repeats itself then increment x so that FPSd can reach the next lower value or 0.
7 1 0 If the display FPS (FPSd) reaches 0 then change the x to start with a negative value
-3 3 4 Display every 3rd , 6th , 9th and 12th frames
-4 4 3 Display every 4th,8th and 12th frames
-5 5 2.4

The effective FPS is lowered if the encoded frame buffer length grows beyond a upper thershold level. (Refer Figure 3)
To achieve that algorithm explained in equations (1a), (2a), (1b) and (2b) is applied.
Similarly effective FPS is increased if the encoded frame buffer length goes below a lower threshold level. (Refer Figure 3)
The same algorithm as in equations (1a), (2a), (1b) and (2b) is applied to increase the FPS.
The reduction in rendering and processing FPS, reduces the load in the system. The reduction in the load of the system brings the system to equilibrium.
In equilibrium the processing and rendering FPS remains constant.
,CLAIMS:WE CLAIM:
1. A method for selecting candidate frames of video stream for rendering based on computing resource availability of recipient of the video stream comprising
involving indirect measurement of computing resource availability in the recipient end including
decoding the video frames arriving at the recipient end,
storing the decoded video frames in a queue,
continuously monitoring length of the queue and thereby recognizing computing resource availability based on the queue length;
dynamically reducing video streaming rate to the recipient by selecting candidate frames of the video stream for rendering depending on the computing resource availability ensuring better viewing experience and keeping the processing lower; and
gradually increasing the video streaming rate to the recipient on availability of the computing resource availability.

2. The method as claimed in claim 1, wherein if length of the queue goes beyond a predefined limit determined by admissible latency, it is inferred that the frame rendering and accordingly the frame processing rate is reduced.

3. The method as claimed in claim 1 or 2, wherein the selection of the candidate frames of the video stream for rendering depending on the computing resource availability includes dropping few of the frames of the video stream to reduce frame processing rate at the recipient without compromising the viewing experience.

4. The method as claimed in anyone one of claims 1 to 3, wherein the frames arriving at the recipient end are decoded to get raw frames (R,G,B or Y,U,V).

5. The method as claimed in anyone of the claims 1 to 4, wherein the frames are dropped as a multiple of modulo factor (y), whereby the effective FPS (FPSd) after dropping the frames is
FPSd = FPS*(1-1/y);
where FPS is original frames per second in the video stream and all the incoming frames are numbered in sequence, 1, 2, 3, …..

6. The method as claimed in anyone of the claims 1 to 5, wherein for given values of the modulo factor (y>1), the frames whose frame number in one of the y-values are dropped.

7. The method as claimed in anyone of the claims 1 to 6, wherein the modulo factor (y) is calculated as inverse of an incrementing variable (x) as y = (FPS)/x.

8. The method as claimed in anyone of claims 1 to 7, wherein increment of the x value reduces the FPS (FPSd) while decrement of the x value increases the FPS (FPSd).

9. The method as claimed in anyone of claims 1 to 8, wherein the effective FPS is lowered if the length of the queue grows beyond a upper thershold level by increasing the x value and the effective FPS is increased if the length of the queue goes below a lower threshold level by decreasing the x value.

10. The method as claimed in anyone of the claims 1 to 9, wherein if the FPSd becomes 0 or negative then the x with –ve value is started for rendering the frames indicated by modulo factor (y) which is y = -x resulting the calculated or effective FPS (FPSd) = FPS * /y.

Dated this 25th day of March, 2019 Anjan Sen
Of Anjan Sen & Associates
(Applicants Agent)

IN/PA-199