FIELD OF INVENTION
The present invention relates to a method for bandwidth matched media streaming
across a network by dynamic encoded media stream adaptation. Media streaming
means delivering of media as it is played out at the receiver. The present invention
relates specifically, to dynamic adjusting of the rate of encoding media stream for
streaming over a variable bandwidth network. It attempts to find a no overhead (time
for media stream adaptation, communication information involved) requiring method
for achieving fine grain scalability.
BACKGROUND OF THE INVENTION
For streaming content over the network, media content is encoded at the transmitting
station before sending and received content is decoded appropriately before rendering
at the receiver. The encoding reduces media content to a level suitable for transmission
according to the bandwidth available. This leads to loss of quality of the content.

However, as the bandwidth varies a lot while the streaming is going on, for best media
content delivery and bandwidth utilization, the transmitting system needs to
dynamically and finely adjust its output bit-rate in accordance with the current
bandwidth requirements.
Essentially, for achieving rate control the known systems function by varying the
encoding parameters like quantization to achieve the desired bit-rate. Alternatively,
layered encoding (with detail represented in optional layers above the base layer) is
used, dropping / adding layers according to the desired output bit-rate based on delay
/ buffer constraints and network bandwidth. Simultaneously multiple streams are
encoded for various bit-rates and switched between the streams in case of long term
and significant bandwidth changes. So the output rate of sent bit-stream has to be
scaled as per the network bandwidth requirements.
For this various standards have been developed; like AVC standard (advanced video
coding standard) and SVC (scalable video coding standard). These are not
interoperable. These standards allow for scalability of an encoded bit-rate to various
bit-rate levels. Based on network bandwidth feedback received from the receiver, the
appropriate bit-rate level media stream can be transmitted.

One technical paper by Suman Banerjee, Seungjoon Lee, Ryan Braud, Bobby
Bhattacharjee, Aravind Srinivasan describes scalable resilient media streaming.
A more relevant paper on A VBR Rate Control Algorithm for MPEG-2 Video Coders with
Perceptually Adaptive Quantization and Traffic Shaping by Mark R. Pickering, John F.
Arnold and Michael C. Cavernor describes a variable bit-rate (VBR) rate control
algorithm for the MPEG-2 video coder. This document uses perceptually adaptive
quantization by varying the quant step-size of the coder according to the spatial
correlation, temporal correlation and a prediction of the quantization distortion of each
macro-block. The algorithm further shapes the traffic characteristics of the output
bit-stream of the coder by varying the quantiser step-size to ensure that the traffic
statistics of the output bit-rate of the coder comply with a predetermined traffic
descriptor. The results presented show that there exists a perceptually optimum set of
traffic statistics which provide a consistent perceived decoded image quality for a given
probability of cell loss.
US Patent 6,895,054 provides a process for dynamic bit-rate control with asymmetrical
reactions to changes in the signal density of the digital video signals. For meeting a
target of long-term bit-rate and achieving a short-term quick reaction two averaging

periods are used in this process. In order to allow high peaks of bit-rate the bit-rate
control is relaxed during high motion scenes at those points while encoding a motion
picture. To preserve the best overall video quality, the coding process reacts slowly to
transitions from low to high motion sequences enhancing the image quality. The coding
process reacts quickly to transitions from high to low motion sequences to enhance the
image quality.
References can also be made to other related documents like US 7,099,954; US 5,731,
835; Application No. 11/260, 650; US 7,143,432; Application No. 814061 ?; US 6,445,
828; 6, 996, 173; 7, 209, 519; US 6, 792, 149; EP 1090491.
The prior art systems do not provide a fine grain control of bit-rate control. They do not
provide fast adaptation of output bit-rate for a variety of bandwidth changes that may
be required. They are not compliant with other codec standard for scalability and rate
control (e.g. AVC can't be used on top of or along with SVC). The rate control
mechanism is fixed upon the codec standard used and so if the codec, fixed in hardware
or third party etc., does not have the scalability feature then the above described
methods will not be able to achieve rate control since the rate control mechanism is built
into the codec. Known systems require overhead / latencies to transmit control

parameters for achieving rate control (e.g. the quant parameter or the differential
quant parameter etc.), i.e extra information (the encoding parameters which have been
adjusted) are required to be sent for the decoder to decode it properly and loss of
information is still there. This is necessary because the receiving end of the streaming
system requires to be informed about and needs to adjust to the new encoding
parameters / style by transmitting system. The levels of scalability are fixed, i.e. the
levels of supported bit-rates are at fixed levels corresponding to the levels (or layers)
defined and built into the system. Once such a system (H/w S/w) is prepared, there is
no way that the levels of scalability etc., can be changed when used in different
environments.
SUMMARY OF THE INVENTION
The main object of the present invention is to develop a method for adjusting the
out-put bit-rate of the encoder (whose output stream is to be streamed onto a variable
bandwidth network) to respond to network bandwidth changes without requiring
making any changes or adjustments to the encoder parameters on the fly, thus allowing
for a flexible rate control strategy with different supported bit-rates to closely track
available network bandwidth and adjust according to its variation that can work with a

variety of different compression standards (WMV, MPEG-4, MPEG-1 etc.) whether they
support media stream scaling or not, hence we need not change the codec of the
streaming system to one which supports scalability standard like SVC, AVC.
The purpose of the present invention is to provide a flexible, low overhead method for
achieving dynamic rate scaling in a media streaming system that can be used with a
variety of different compression standards (for compressing the media stream before
transmission).
The media will be streamed over the network with variable bandwidth. To achieve
maximum quality in delivered media stream along with maximum channel (network
bandwidth) utilization, we control the output bit-rate of the compressed media stream,
which is transmitted over the network.
A unique method of modifying the content of the media stream fed into the encoder is
used to achieve reduction in the size of encoded stream for any block-based codec
(which makes use of frame prediction from future or past frames and compresses blank
regions of video much better than ones carrying actual moving object).

For matching the size of encoded media stream with available bandwidth (a bit-rate
estimate), the raw media stream is temporally sub-sampled (frame-rate reduction as
required) up to a preset minimum threshold and for reducing data rate further (if
necessitated by further bandwidth fall) the frame is resized to a new size and pasted on
the top-left corner of frame (a similar case would be the concept of picture-in-picture).
The rest of the frame being black. The video and blank region is separated by marking
a boundary and these regions are aligned along the block (according to the maximum
block size the codec supports). The boundary is marked according to certain rules by a
pre-processing module at the transmitter end.
The frame rate is adjusted to match bandwidth change first and once it reaches the
pre-defined thresholds the resizing mechanism described above is also applied to give
best quality video. Thus, the transmission-side buffer filling rate is reduced and
accordingly the packet sending (buffer emptying) is adjusted to match the bandwidth
estimate. The same applies to receive-side buffer filing rate.

At the receiver's end, an intelligent post-processing unit analyzes the decoded frame to
find, if it is a case of video content being embedded in the frame or the whole frame is
the video content and if the video content is in a sub-portion of the frame then it
extracts that portion and resizes it back to original size before sending it to the renderer.
Hence, a method is proposed for adapting the size of encoded multimedia stream
(which has to be sent across any network) to match the available bandwidth by
adapting raw video sequence used for generating the stream.
In accordance with the present invention, the raw video sequence is transformed into a
video sequence with better compression characteristic to achieve the desired output
data-rate to match available bandwidth. The decision is based on the current
transmission rate and bandwidth.
The receiving end post-processes the decoded frames intelligently (without requiring
any extra information than otherwise transmitted normally) to recover the original
video frames.

In one preferred embodiment the present invention provides a method for media
streaming across a variable bandwidth network by dynamic encoded media stream
adaptation, comprising the steps of: obtaining bandwidth estimate from a bandwidth
estimator module; adjusting / modifying input video sequence from the bandwidth
estimate, utilizing feedback on the current output bit rate of an encoder; encoding video
sequence in the encoder with adjusted bit rate of the encoder; and transmitting the
encoded media stream with adjusted bit rate over the variable bandwidth network.
The method measures and utilizes feedback regarding the current output bit-rate of the
encoder rate (of the transmitting media streaming station) for making dynamic rate
scaling decisions. Different encoders with different efficiencies will yield different level
of adjustments for same bandwidth change.
The dynamic rate scaling can be deployed with a wide variety of codec standards,
irrespective of whether they support rate scalability features or not.
The control of bit rate of the encoded media stream with no pre-specified constraints to
adhere to, but a variable bit rate can be adjusted to continuously to match the network
bandwidth over which the media is being streamed across.

In the mechanism described the rate scaling can be introduced without any delay such
as may be associated with bit stream switching or any additional overhead as in scalable
streaming. It is to be noted that neither frame rate change nor any resize change
requires any other information to be conveyed to the receiver end.
The rate scaling can be achieved by pre-processing the media frame data input to the
encoder and hence, not requiring adjustment of encoder parameters on-the-go.
According to the method the video frame containing the content is resized and refitted
into the frame of same size while keeping the rest of the content of the frame not
containing any useful video information. Hence, a method for rate control which does
not require reconfiguring of encoder parameters and sending the same to the receiver.
The original frame information can be intelligently reconstructed using ROI detection as
mentioned earlier, which scans the edges for the ROI or boundary marker (non-zero)
content.
The frame resize-refitting involves marking boundaries, partial or complete as required,
at right and bottom edges of the actual video content region of the frame, if necessary
(in case extreme edges of frame are completely black), to detect the region of interest

in the frame (containing the actual video frame data). Thus, not necessitating
transmission of new size. The width of the boundary marker is configured as per the
width of the blocks for the block based encoder, according to the block size, the codec
supports.
Frame post-processing for re-construction of the originally transmitted frame from the
decoded frame which is actually a smaller-resized frame, refitted into a bigger
empty-black frame (of frame size specified at the start of streaming) which involves
detection of bounding rectangle for region of interest containing the video data and
detecting the size of the actual ROI and resizing it back to original frame size.
The method is capable of detecting whether resizing to bigger size is necessary or not.
Rate scaling as mentioned above uses the described resize-refit strategy in combination
with frame rate adaptation of the video stream to achieve rate scaling to dynamically
adapt output bit rate of the encoded media stream.

The method can provide the benefit of lower spatial resolution streams of a number of
resolutions and which are not restricted to a few as specified by profiles and levels of
some standards. Example, the benefit of a video stream having bit rate comparable to
QCIF resolution can be obtained as well as those with CIF resolution, 4 CIF resolutions
etc. Example, existing compression standards may specify CIF resolution like
QCIF - Quarter CIF (resolution 176 x 144)
SQCIF - Sub quarter CIF (resolution 128 x 96)
4CIF - 4 x CIF (resolution 704 x 576)
16CIF -16 x CIF (resolution 1408 x 1152)
So scalable streams for these would only support four of these bit rates, which may not
give the best quality video for available bandwidth (assuming that video will be shown
in maximum size to receiver). But according to the method of the invention, a variety
of sizes are possible, which can support more bit rates than the above. Thus, spatial
scalability will have more levels irrespective of whether the compression standard
supports the same or not.
For achieving spatial scalability with block based (including VOP based) compression
schemes require transmission of no additional overhead, is compatible and can be used
with a variety of compression schemes, supports greater levels (finer granularity).

The spatial scalability in which the switching between levels can be instantaneous and
does not require shifting an alternate bit stream (which has / is been / being
pre-encoded) before / simultaneously) to different bit rate or encoding for separate
levels at the same time.
The method makes use of efficient encoding of still regions for rate scaling.
For embedding video frame into another same frame and for detection and recovery of
original picture from the decoded frame without separately sending any flags or data
(uncompressed or loss-lessly compressed data or appending additional headers as in
VOL layers of MPEG-4 etc.) to help the same.
Rate scaling involves adaptation of the raw video stream to achieve desired change in
data rate which includes use of scene detection for estimating scene change duration,
max key-frame interval, encoder efficiency for compressing still regions in key and delta
frames (configured before streaming begins) and which takes different decisions for
different compression schemes depending on the encoder's overall efficiency at
run-time as well as data characteristics, using the current output data rate as feedback.

BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS
The invention can now be described in detail with the help of the figures of the
accompanying drawings in which
Figure 1 shows the ROI marked on the video screen content.
Figure 2 is a block diagram of the present invention.
Figures 3 is a block diagram for video pre-processing
module of the present invention.
Figure 4 is a flow chart of the operation of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
As shown in the block diagram of the invention, (Figure 2) raw frame sequence from
media source 1 is fed to a video frame sequence pre-processing module 2 to control
bit-rate. The invention relies upon this rate adaptor module 2, which takes in the

feedback 3 of the current output bit-rate and the bandwidth estimate from current
bandwidth estimator 4 to arrive at the bit-rate reduction strategy to be used, which will
be a combination of image frame modification and frame rate manipulation, as
described below. This again will be updated dynamically. The rate adaptor block 2
decides how much change is to be effected upon the frame rate and the amount of
resizing (image content modification as described below) required.
According to the current network bandwidth, the frame rate is adjusted (increased or
decreased) dynamically to match the output bit-rate to the network bandwidth
requirements. If the target bit-rate cannot be achieved by this alone then the image is
modified as before by refitting a down-sized image of the original frame into the output
image frames (of the pre-processing module). The rest of the frame is kept with a fixed
pre-defined / known color and is the empty part of the frame containing no necessary
video information bounded by a boundary-marker (line of pixels marking right and
bottom edge of the region of interest (ROI) which is the actual video content in the
actual processed frame with the remaining portion as the empty area of known /
pre-defined color). This particular change (resizing video content) is done at always
done on an l-frame (key-frame) and if necessary, the encoder is forced to insert such a
frame.

The frame rate can be reduced up to a pre-set threshold. If the desired target bit-rates
cannot be achieved then the image frame modification (resizing frame) is applied as
required. These thresholds can be specified according to required video quality
acceptance criteria and / or according to the nature of video (the amount of activity and
length and frequency of such periods of high activity). The amount of resizing is again
decided based on the bandwidth estimates and current output rates.
The amount of frame rate and image frame adaptation that is applied based on
estimation of bandwidth as received from the bandwidth estimator module 4 in Figure
1 and the current output data rate feedback received.
The feedback from the receiver 8 is guaranteed and time-gap between sending two
such successive feedbacks is pre-configured.
The threshold minimum frame-rate and resize amounts along with other
encoding-decoding parameters like height, width of frames etc. are known to both
receivers 8, transmitter stations 7.

The receiver station 8 based on inter-packet delay, packets dropped, buffer size at
receiver end etc. maintains available bandwidth data estimates and updates the
transmitter whenever bandwidth changes drastically within a minimum set
pre-configured gap time (M, say pre-configured as 1 sec) from last feedback or
periodically (T, say once in 5 sec) or upon receiving buffer hitting threshold marks.
Any or all such triggers, constraints may be used for sending bandwidth feedbacks to
transmitter. The receiver monitors each l-frame received and to detect when the sizes
close its estimates have arrived and resizes the frame accordingly. Since resizing will
only be applied when frame-rate reductions is not sufficient, the receiver can know
whether resizing is necessary or not.
The resize decisions are made using knowledge of encoder's efficiency in encoding a
sequence of black frames i.e. the efficiency with which it encodes black key-frames and
delta frames (P-/B-/S-frames).
Rate adaptor module finds the amount of frame rate change and resizing required
matching the bandwidth as described above.

In addition if resizing is being used and bandwidth falls then it may apply a lower size of
region of interest (ROI) to video frame. In such a case, an l-frame is inserted in case the
resizing is more than a few blocks' dimensions (i.e. drastic change of size compared to
present size, the threshold for which can be set) and then such a decision is done at the
next key frame interval.
Also if bandwidth is falling continuously and difference between bandwidth and data
rate goes beyond threshold then smaller and smaller sizes shall be applied frame by
frame without inserting l-frame.
This is a sample implementation. In general any order of decreasing size operations as
deemed fit can be applied as necessary. The amount of resizing on successive frames
shall depend on the block size of the codec as well as the search distance (i.e. the
number of neighboring blocks it uses for prediction) of encoder.
However, if bandwidth increases, then from the next key-frame interval a new
increased size ROI is used. This key frame interval is the maximum interval between
key-frames.

-Image content / layout adjustment (pre-processing)
The frame is resized (spatial smoothing, resizing) to generate a smaller - sized frame
ROI from the original frame and then the smaller image is copied onto the top-left
corner of the frame the rest of the frame (blank / Null region) is made of a chosen pixel
color value (say black). The intensity value may, optionally be given to be a
representative of the image size to aid in making faster post-processing (resizing)
decisions. Additionally / alternatively, the color component values for all this region
may also be set to help in identifying the actual video content region within the frame.
The resize configuration found in such a manner is applied to a frame and each after
that frame and a different resize configuration will be applied only at the next l-frame,
the interval between two l-frames which can bear a new resize configuration, may be
pre-configured at both receiver and transmitter end so that resize decisions may only
respond to overall average bandwidth change observed during the period. Such a
method might be used for high (frequent) bandwidth variability networks like WiFi
wherein short-term changes will be frequent.

The color can be defined to be black or can be chosen using previous frame; in fact the
sections outside of this region can be altered in any suitable manner for post-processing
for example, market boundary lines can be inserted before the empty parts of the frame
starts on a ROI-blank region boundary. The frame pre-processed in this manner is fed
into the encoder and this sort of adjustment is applied at key (l-frame) boundaries. In
any case, the pre-processed frame is bounded by only one boundary marker for
identifying the ROI in the frame, which is the outer-most one. The only exception to
this rule being when a raw frame contains a boundary marker and black region arrives,
which is similar to a resized frame. For such a case a boundary line is drawn at the
bottom of the frame along its width.
The image resizing may be done so as to deviate the least from the aspect ratio of the
frame and but new size of image must be aligned along the block boundaries of the
block size the compression scheme supports.
All the frame resizing and modifications are done in the spatial domain only and the
resulting frames have perceptual meaning to any ordinary human observer. This
output can be fed to any encoder but will generate less output bits after encoding as
only a small area of the frame is having the video content, the rest is blank / empty

which gets efficiently encoded in key-frames and for which no other information is
required to be sent except flags (skip MB) in the following predicted frames.
It is to be noted that the boundary markers will only be present once resize-refitting
strategy is adopted or if original frame has completely black edges (right and bottom)
but is not to be resized to smaller size ROI boundary-marking algorithm:
Case A: If resizing is not being applied:
1. Check raw frame's last rows (of the last vertical blocks / slice) and similarly the
last columns (last / right-most block column of frame) for non-zero pixels except
the top row of last slice and the leftmost column of the last column-block.
2. In case all pixels along all these blocks are zero (except the top row), then mark
the last row of frame with a boundary marker and / or for the last column. Check
the decode version of this frame, if this boundary is not preserved then mark all
the pixels of this block as white.

Alternatively, only the last row or column need to be checked for non-zero pixels in the
decoded version of the frame, if last row is zero then change the last pixel value to some
other big value so that it is non-zero in the raw frame and then encode the frame. If the
implementation of the invention is configured to bring about resize changes after a fixed
pre-configured run of frames then this decode operation is only necessary for an
l-frame and may be applied only on those specific instants on which resizing can occur.
Case B: If resizing is necessary:
1. Find the amount of resizing necessary taking into account the encoder's
efficiency in encoding black key - frames as well as delta frames, Max interval
between key frames, average duration of scene (A scene change detector is
used which notifies upon detection of scene change).
2. Round the resize dimensions to multiple of corresponding block dimensions.
3. Resize the frame and paste it on a black picture of original frame size.
4. Make a boundary (say of white color) just below the resized video area and just
to the right side of the embedded video content of the frame.

Basically, according to the invention the last row of blocks can have:
1. Their top / leftmost pixels non-zero (say white), in case resizing has been done
for the frame and rest of the block as zero (ROI is of dimension one block less
then the frame dimension).
2. Or the complete block as zero in case the boundary of ROI is smaller than that
described above.
3. When resizing has not been applied and the decoded frame is the original frame.
A. The last row of pixels non-zero along with one. This frame designated to
be the no-resize condition detection frame can also be defined as follows:
The entire rows and columns corresponding to the last blocks of the
frame as decided above shall be of a particular color-intensity value (say
white). This depends on whether the codec guarantees error free
transmission of the frame described above or not and this may be
pre-configured.

B. One or more pixels anywhere else in these blocks as non-zero (the top
row of pixels in these blocks are not checked for this or any other
condition)-
The size of the ROI is one less than the horizontal and vertical dimension of the
boundary.
After image content has been resized, a smoothing filter may be applied for smoothing
the image if the complexity of the scene goes up due to introduction of noise (if
sub-sampled values had been interpolated). For this a comparison is made of the
complexity (or noise content) of the scene in original and the resized version of the
image.
-Frame post processing:
At the receiver's end, the frame is decoded. After this the useful portion of the frame is
detected and the original frame can be reconstructed by resizing it to a larger size. The
detection of actual region-of-interest (area rectangle bounded by boundary-marker and
left-top edges of frame image) in the decoded frame is done without knowing its

dimensions (as these were not transmitted). The region of interest is scaled to occupy
the full-frame. The ROI area and empty are non-overlapping areas with empty areas
being of the same color or being distinguishable from video contents for that frame.
ROI-detection mechanism: The max key frame interval is fixed and known at both the
receiver and the transmitter. Also the dimensions of the blocks of the block-based
codec are known. The different possible ROI sizes that can be generated are all aligned
along the block boundaries, i.e. the dimensions of the ROI are multiples of
corresponding block dimensions. E.g. ROI's height is a multiple of block-height. So all
such boundaries are examined.
Algorithm for detecting whether resizing decoded frames back to original frame size:
1. Check for the last horizontal blocks of the frame and check if a horizontal
boundary line as described above is drawn at the last row of the frame or
if any of its pixels are non-zero (or any color-intensity different from that
of possible empty region of frame which may be chosen to have only
black and white values or some fixed color-intensity). If yes, then the
frame need not be resized. To speed up this step, first just the last row

/ column are checked for non-zero pixels. If such pixels do not exist, then
go to step 2 else resizing is not necessary. Or if all pixels of this frame are
white. Basically in this step, it is determined whether the frame is a
no-resize detection frame or a normal frame in which case resizing is not
necessary.
2. Check the rows (aligned on block size boundaries) of frame for boundary
marker (i.e. existence of any non-zero pixels) and similarly for columns. In
case empty regions intensity value is taken of specific value, then hint from
the same may also be used to quicken this search.
Additionally, based on frame rate, bandwidth calculations, last sent feedbacks, it may
be guessed whether resizing is necessary or not.
The resizing changes and frame rate control changes are applied discretely (not
continuously), i.e. a new frame rate and a new resized refitted image are generated and
kept the same for a sequence of frames after which bandwidth feedback from receiving
station and sending rate is taken into account and a new appropriate frame rate and
new resizing (stream adaptation measures) is chosen for the next sequence of frames.

The length of this sequence of frames can be configured and the algorithm can make
stream adaptations as frequent to as long (say 5 minutes) as desired. In the former
case bandwidth will be closely monitored and in the latter it will be considering more of
an average behaviour and will respond less to bandwidth changes.
The minimum length of such frames is known before-hand (assuming no scene change)
and resizing changes are applied assuming the befit over all such frames.
The pixels are scanned from bottom - left corner until a row is arrived where a non-zero
pixel is reached, similarly for y directions. The rows are scanned at the top of each block
i.e. rows that are multiple of block boundary height are scanned similarly for height the
boundary is drawn. The chief advantage in this approach is that the encoder (at the
transmitting end) and the decoder (at the receiving end) do not need to be configured
dynamically to work for different frame sizes according to the desired bandwidth.
A subtle difference with other rate scaling mechanisms is that according to the invention
it is necessary to find the current output data rate which gives an idea of the encoder's
efficiency and rate adjustment will be subject to the current output data rate and hence,
the encoder, thus catering to different compression standards. This feedback is
provided by the "current bit-rate feedback" block in Figure 1.

Based on the percentage gap between this rate and the current network bandwidth and
along with estimate of the order of the short term fluctuation of bandwidth, the frame
rate and image resize-refitting is done. The bit-rate is reduced with the aim of equaling
the pessimistically estimated bandwidth in the following manner.
Additionally, for a video object based codec like MPEG-4; the embodiment of the
invention may chose a different strategy for sending ROI and the empty region to
receiver (which may eliminate the need for intelligent resizing through boundary
detection), the actual ROI may be sent as a video object; with a black frame as sprite
object with updates of boundary.
So the receiving end of the station sends the packet statistics at its end and sends it
over to the transmitter. The transmitter estimates the bandwidth, based on the change
in bandwidth the target bit-rate is revised and the rate adjustment mechanism - frame
rate and frame refitting, tuned to closely track the desired output bit-rate.
One of the advantages of the present invention over known system is that this invention
aims to bring rate scalable delivery of media content even on non-scalable encoding /
rate adaptable streaming systems and can be applied with any compression standards.

It detects the efficiency of the compression scheme and generates the rate adaptation
measures accordingly.
The method of the present invention allows for a fine grained rate adaptation approach
which is not possible with current art primarily support fixed discrete levels of bit-rate
support (number of detailed layers / number of alternate bit-rate streams are fixed and
corresponding to a particular level of bandwidth). The scalability of the resulting
bit-stream is flexible and not limited to a few discrete bit-rate levels / profiles and such
changes in scalability can be adjusted dynamically according to available bandwidth.
It is to be noted that the frame resizing does not require any extra overhead in the form
of information / headers to be sent to the receiving station for recovering the original
raw video sequence. Hence, the method for rate scaling of video stream does not
require transmitting any extra overhead information. Instead it reconstructs this
information from intelligently as discussed above from the processed frames.
The system of the present invention can work with a variety of available codec
standards / compression schemes.

In the light of the above the commercial importance of the invention is primarily
allowing for better quality viewing experience of the delivered media stream. Any
stream system can be adapted to follow this rate control technique by simply adding the
pre-post-processing modules along with feedback for bandwidth and current
transmission rate (data sending rate or the data generation rate) at the transmitting
end regardless of whether it supports the scalability features while encoding or supports
rate scalability of the encoder or not.
It can be applied to any streaming system and works well for any network with
fluctuating bandwidth requirements. Other streaming systems either give lower quality
or cause congestion and / or suffer more packet losses (being unable to cope with
widely fluctuating bandwidth of a network like WiFi or even the internet).
A video pre-processing module, which adapts the input media, stream so that the
encoder may encode it more efficiently. An intelligent post-processing module at the
receiving end to recover the original pre-processed video frame. An input received
regarding currently estimated bandwidth.

The bandwidth estimator module calculates current bandwidth from feedback of the
receiver and the pre-processing module then decides upon the adjustment of rate
scaling factors (frame rate and image-resize factor) to achieve the best matching
bit-rate for streaming media. The motion estimates and the configured delay allowable
are used to arrive at a decision for frame dropping for each frame to achieve the desired
bit-budget.
A long-term as well as a short-term motion estimates (depending on configurable
delay) are used to arrive at rate scaling decisions, namely the re-adjustment of size
following by refitting the smaller sized frame in the original frame and frame-drop
decisions (for each frame) will be taken dynamically. This is done to achieve a change
in the output bit-rate to match the available bandwidth according to the estimates.
At the receiver-end the inter-packet delays and packet loss-fraction are analyzed to spot
a trend and estimate the bandwidth bottleneck. Whenever a fall is detected, feedback
is sent to transmitter. The feedbacks are generated at some fixed pre-defined intervals
and do not occupy much bandwidth. The transmitter prepares estimates depending on
the configured delay and feeds it to rate control module which makes the decisions, as
described above, along with estimates of short-term fluctuation as observed and sent
by receiving station.

Some of the effects / advantages of the invention include a rate control method which
can be applied to a variety of codec standards (whether or not it supports scalability)
and can be (but not limited to) dct-based, wavelet based or any other Eg. mpeg-1, -2,
-4, H.264, Dirac etc. The amount to which a codec responds to stream adaptations and
produces stream-size reductions (based on adaptation of raw frame sequence)
depends on the efficiency with which it can encode constant color portions of video.
Moreover, the codecs responsiveness to stream adaptation can be increased by
adjusting (reducing) bits for motion vector encodings dynamically as the invention
adapts raw video frame.
Invention provides a unit / module for measuring the rate scalability achieved by media
stream adaptation measures used by the invention, estimates the effect of such
adaptations on raw frame sequence to final transmitted encoded media stream, and
hence is able to match / adapt the media streams generated by a variety of codecs to
dynamic bandwidth constraints of the network used for streaming.
It also includes a rate adaptation strategy capable of supporting various bit-rates and
not limited to a fixed few ones as in certain standards like SVC, AVC, the supported
output bit-rates can be dynamically adjusted as per current bandwidth trends. In a

combination of high and low motion sequences the high motion sequence will suffer
lesser frame loss while the lower motion sequence will bear more losses. It provides a
rate control technique with no overhead information required to be transmitted, and a
pro-active bandwidth estimation method, which detects bandwidth fall before losses
occur. The inventive method has very little overhead for rate scaling even considering
the feedbacks from receiver. The method requires no change to the encoder / decoder
and re-configuring of the same in hardware or software. The chief advantage of the
invention is that the parameters (i.e. size) of the encoders are not required to be
adjusted while streaming is going on and the codec can function with the same frame
size as configured at the start.
Also the interval between intra-coded frames need to be known and then ability of
invention to have configurable responsiveness to bandwidth change (as described in
the document) shall depend on the ability of the codec to be configured to insert intra
coded frame in the coded frame sequence.

WE CLAIM
1. A method for media streaming across a variable bandwidth network by dynamic
encoded media stream adaptation, comprising the steps of:
- obtaining bandwidth estimate from a bandwidth estimator module;
- adjusting / modifying input video sequence from the bandwidth estimate,
utilizing feedback on the current output bit rate of an encoder;
- encoding video sequence in the encoder with adjusted bit rate of the
encoder; and
- transmitting the encoded media stream with adjusted bit rate over the
variable bandwidth network.
2. The method as claimed in claim 1, wherein said bandwidth estimation utilizes
feedback on current output bit rate of encoder rate for arriving at dynamic rate
scaling decision.

3. The method as claimed in claim 2, wherein for achieving the dynamic rate
scaling a wide variety of codec standards can be utilized independent of whether
they support rate scalability feature or not.
4. The method as claimed in claim 1, wherein control of said bit rate of an encoded
media stream comprises adjusting the variable bit rate to continuously match
the network bandwidth over which media is being streamed.
5. The method as claimed in the preceding claims, wherein said rate scaling can be
introduced without any delay that may be associated with scalable streaming,
like bit stream switching.
6. The method as claimed in preceding claims, wherein rate scaling can be
achieved by pre-processing media frame data input to the encoder, without
needing adjustment of encoder parameters, and resizing for generating a
smaller sized frame for regions of interest (ROI) and refitting into the frame of
same size while keeping the rest of the content of the frame not containing any
useful video information.

7. The method as claimed in claim 6, wherein edges of the ROI or boundary marker
content are scanned for re-constructing the original frame information.
8. The method as claimed in claims 6 or 7, wherein said frame resize-refitting
comprises marking boundaries, partial or complete, at right and bottom edges of
the actual video content region of the frame, for detecting the ROI in the frame.
9. The method as claimed in preceding claims, wherein reconstruction of the
originally transmitted frame from the smaller-resized decoded frame refitted
into a bigger empty-black frame comprises detection of bounding rectangle for
region of interest containing the video data, and detecting of the size of the
actual ROI and resizing it back to original frame size.
10. The method as claimed in claim 9, wherein said method is capable of detecting
whether resizing to bigger size is required or not.
11. The method as claimed in preceding claims, wherein said rate scaling utilizes
said resize-refit in combination with frame rate adaptation of the video stream to
achieve rate scaling to dynamically adapt output bit rate of the encoded media
stream.

12. The method as claimed in claim 1, wherein said method provides the benefit of
lower spatial resolution streams of a number of resolutions so as to provide a
variety of sizes which can support more bit rates allowing more levels of spatial
scalability irrespective of whether the compression standard support the same
or not.
13. The method as claimed in claim 12, wherein for achieving spatial scalability,
block based compression schemes, requiring transmission of no additional
overhead, is compatible which support greater levels.
14. The method as claimed in preceding claims, wherein switching between levels
can be instantaneous without shifting an alternate bit stream to different bit rate
or encoding for separate levels at the same time.
15. The method as claimed in preceding claims, wherein still regions are efficiently
encoded for rate scaling.
16. The method as claimed in preceding claims, wherein the video frame is
embedded into another frame for detection and recovery of original picture from
the decoded frame without separately sending any flags or data to help the
same.

17. The method as claimed in preceding claims, wherein the rate scaling comprises
adaptation of the raw video stream to achieve desired change in data rate with
use of scene detection which takes different decisions for different compression
schemes depending on the encoder's overall efficiency at run-time as well as
data characteristics.
18. A method for media streaming across a variable bandwidth network by dynamic
encoded media stream adaptation, substantially as herein described and
illustrated in the figures of the accompanying drawings.

The main object of the present invention is to develop a method for adjusting the
out-put bit-rate of the encoder (whose output stream is to be streamed onto a variable
bandwidth network) to respond to network bandwidth changes without requiring
making any changes or adjustments to the encoder parameters on the fly, thus allowing
for a flexible rate control strategy with different supported bit-rates to closely track
available network bandwidth and adjust according to its variation that can work with avariety of different compression standards (WMV, MPEG-4, MPEG-1 etc.) whether they
support media stream scaling or not, hence we need not change the codec of the
streaming system to one which supports scalability standard like SVC, AVC.
The purpose of the present invention is to provide a flexible, low overhead method for
achieving dynamic rate scaling in a media streaming system that can be used with a
variety of different compression standards (for compressing the media stream before
transmission).