THE PATENTS ACT 1970
[39 OF 1970]
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
[See section 10; rule 13]
Title: “AN ADAPTIVE IMAGE ENCODING METHOD IN A VIRTUAL
NETWORK COMPUTING (VNC) ENVIRONMENT”
Name and address of the Applicant:
LG SOFT INDIA PRIVATE LIMITED, Cherry Hills, Embassy Golf Links
Business Park, Bangalore – 560 071, Karnataka, India
Nationality: India
The following specification particularly describes the invention and the manner in
which it is to be performed.
2
FIELD
The instant disclosure relates to the field of virtual network computing (VNC).
More particularly, it relates to providing an optimized VNC environment by
implementing adaptive images encoding in VNC server.
BACKGROUND
In today’s world of high speed communication, a network of computing
device is essential for sharing text and multimedia data, between multi-users. VNC
system is widely used for graphical desktop sharing on the network and to remotely
control another computer or other devices like mobile phone, personal computer etc.
VNC uses remote frame buffer (RFB) protocol to transmit the keyboard and mouse
events from one computer or other device to another, relaying the graphical screen
updates over the network.
A VNC system consists of a server that passively allows one or more clients to
take control of it by using a communication protocol. The client watches, controls and
interacts with the server using communication protocol like RFB protocol. The VNC
server sends small rectangles of the frame buffer to the client. In its simplest form, the
VNC protocol can use a lot of bandwidth, so various methods have been devised to
reduce the communication overhead. For example, there are various encodings
methods to determine the most efficient way to transfer these rectangles. The VNC
system allows the client and server to negotiate which encoding will be used. After
original full screen has been transmitted, it only transfers rectangles that change. This
encoding works very well if only a small portion of the screen changes from one
frame to the next, for example like a mouse pointer moving across a desktop, or text
being written at the cursor. But bandwidth demands get very high if a lot of pixels
change at the same time, such as when scrolling a window or viewing full-screen
video
The disadvantage of existing VNC system is that VNC server will not take into
account of how fast the Desktop Graphical image Change (i.e. Image Change Rate -
Rate at which the image changes per second) which in-turn may produces frame loss
and may loss crucial information. Also, quality of the image is always fixed and it is
3
not based on image change rate. In light of the foregoing discussion, there is a need
for a method and system to solve the above mentioned problems.
SUMMARY
Exemplary embodiments of the present disclosure relate to a method, a server
and a system for providing adaptive image encoding in a virtual network computing
(VNC) environment.
The shortcomings of the prior art are overcome and additional advantages are
provided through the provision of a method, server and system as described in
description.
The present disclosure solves the limitations of existing techniques by
providing adaptive encoding of the images in the VNC environment.
In one embodiment of the disclosure the adaptive encoding of the images
provides for optimized utilization of channel bandwidth.
In one embodiment of the disclosure quality of the image encoded varies
based on image change rate, thus emphasis is given not to loose any crucial
information.
Additional features and advantages are realized through the techniques of the
present disclosure. Other embodiments and aspects of the disclosure are described in
detail herein and are considered a part of the claimed disclosure.
In one embodiment is an adaptive image encoding method in a virtual network
computing (VNC) environment, wherein the VNC environment comprises a server
and one or more clients communicating with the server, said method comprising acts
of: calculating percentage change between previously stored images with current
image being displayed on the server, selecting lossy encoding if the percentage
change is greater than a predetermined encoder threshold, else selecting lossless
encoding; wherein the lossy encoding selection leads to applying non-predictive lossy
4
encoding if the percentage change is less than a predetermined lossy threshold else
predictive lossy encoding is applied; wherein the lossless encoding selection leads to
applying non-predictive lossless encoding if percentage of image change is less than
predetermined lossless threshold else predictive lossless encoding is applied; and
transmitting the encoded image to the one or more clients.
In another embodiment is a VNC server comprising frame grabber to store an
image being displayed on the server, frame comparator to calculate percentage change
between the previous stored images with current image being displayed on the server,
a processor responsive to instructions for, selecting lossy encoding if the percentage
change is greater than predetermined encoder threshold, else selecting lossless
encoding; wherein the lossy encoding selection leads to non-predictive lossy encoding
of the image if the percentage change is less than predetermined lossy threshold else
predictive lossy encoding; wherein the lossless encoding selection leads to nonpredictive
lossless encoding of the image if percentage of image change is less than
predetermined lossless threshold else predictive lossless encoding; and a means to
transmit the encoded image to one or more clients.
BRIEF DESCRIPTION OF THE DRAWINGS
The novel features and characteristic of the disclosure are set forth in the
appended claims. The disclosure itself, however, as well as a preferred mode of use,
further objectives and advantages thereof, will best be understood by reference to the
following detailed description of an illustrative embodiment when read in conjunction
with the accompanying drawings. One or more embodiments are now described, by
way of example only, with reference to the accompanying drawings wherein like
reference numerals represent like elements and in which:
FIG. 1 illustrates VNC system in accordance with one embodiment.
FIG. 2 is block diagram of a VNC system in accordance with one embodiment.
FIG. 3 shows a flow chart of adaptive encoding method in accordance with an
exemplary embodiment.
5
The figures depict embodiments of the disclosure for purposes of illustration
only. One skilled in the art will readily recognize from the following description that
alternative embodiments of the structures and methods illustrated herein may be
employed without departing from the principles of the disclosure described herein.
DETAIL DESCRIPTION
The foregoing has outlined rather broadly the features and technical
advantages of the present disclosure in order that the detailed description of the
disclosure that follows may be better understood. Additional features and advantages
of the disclosure will be described hereinafter which form the subject of the claims of
the disclosure. It should be appreciated by those skilled in the art that the conception
and specific embodiment disclosed may be readily utilized as a basis for modifying or
designing other structures for carrying out the same purposes of the present
disclosure. It should also be realized by those skilled in the art that such equivalent
constructions do not depart from the spirit and scope of the disclosure as set forth in
the appended claims. The novel features which are believed to be characteristic of the
disclosure, both as to its organization and method of operation, together with further
objects and advantages will be better understood from the following description when
considered in connection with the accompanying figures. It is to be expressly
understood, however, that each of the figures is provided for the purpose of
illustration and description only and is not intended as a definition of the limits of the
present disclosure.
FIG.1 shows a VNC system capable of performing adaptive image encoding in
accordance with one embodiment of the disclosure. It shows a VNC server 101
interacting with a plurality of VNC clients (105, 106 and 107). The Adaptive image
encoding contains a frame buffer grabber 102 which captures and stores the raw
images in uncompressed format such as RGB, YUV format etc. Frame buffer
comparator 103 calculates the percentage of scene change between previous stored
images with the current displayed image on the VNC server 101. The percentage of
image change is then used by a processor 104 to perform selection of encoding
scheme to be adopted and encodes the image to be transmitted to one or more VNC
clients (105, 106 and 107). The VNC system uses a standard protocol like remote
6
frame butter (RFB) to transmit the encoded image. The encoding scheme to be
adopted by the processor 104 to encode the image depends on predetermined
threshold and the client request. The processor 104 performs predictive lossless / non
predictive lossless / predictive lossy / non predictive lossy encoding depending on the
percentage change between previous stored images with the current displayed image.
The percentage change most of the time have an impact depending on frame rate
i.e.rate at which the image changes per second. .
FIG. 2 shows block diagram of VNC system having VNC server 101 capable
of performing adaptive image encoding and interacting with plurality of VNC clients
using RFB protocol. RFB is truly a “thin client” protocol. The emphasis in the design
of the RFB protocol is to make very few requirements of the client so that it can run
on widest range of hardware. The RFB client is stateless. If a RFB client (202, 203
and 204) disconnects from a given RFB server 201 and subsequently reconnects to
that same server, the state of user interface is preserved. Furthermore, a different
client endpoint can be used to connect to the same RFB server. At the new endpoint,
the user will see exactly the same graphical user interface as at the original endpoint.
The RFB server 201 is capable of producing a stream of rectangles of screen
data. It repeatedly dumps the whole of a frame buffer as fast as possible. A better
implementation of RFB server 201 sends updates only when changes to the screen are
required, and only sends the areas of the screen which have changed.
Building artificial intelligence i.e. providing adaptive encoding before sending
update frame buffer makes efficient use of network bandwidth without compromising
on quality of frame buffer.
The adaptive image encoding takes two important things into consideration.
• Number of Images per second (Rate at which the image changes per
second).
• No loss of information.
7
After exchanging handshaking message with RFB server 201, the RFB client
202 does its initialization and request graphical screen updates based on its inputs like
transmitting keyboard and mouse events.
The Adaptive image encoding initially captures and stores the raw image in
uncompressed format such as RGB, YUV format etc in a frame buffer grabber 105.
Frame comparator 106 then calculates the percentage of scene change between
previous stored images with current displayed image. The percentage of image change
is then input to an encoder decision maker 108 and if the percentage of image change
is greater than a predetermined encoder threshold then lossy encoding method would
be employed else if the percentage of image change is less than the predetermined
encoder threshold then lossless encoding method would be employed. However, if the
entire image needs to be transmitted based on the request by the VNC client, then the
percentage of image change will be set to 100%, thus resulting in selection of lossy
encoding method.
After selection of lossless or lossy encoding, the next step involves making
decision of whether to do predictive lossless / non predictive lossless / predictive
lossy / non predictive lossy encoding. This decision is made by lossless encoding
decision maker 109 and lossy encoding decision maker 110.
The non-predictive lossless encoding method is selected by the lossless
encoding decision maker 109 if the percentage of image change is less than
predetermined lossless threshold else predictive lossless encoding method is selected.
Lossless non predictive coding 111 encodes the image to be transmitted using lossless
non predictive encoding scheme. Lossless predictive coding 112 encodes the image
to be transmitted using lossless predictive encoding scheme.
The non-predictive lossy encoding method is selected by the lossy encoding
decision maker 110 if the percentage of image change is less than predetermined lossy
threshold else predictive lossy encoding method will be selected. Lossy non predictive
coding 113 encodes the image to be transmitted using lossy non predictive encoding
scheme. Lossy predictive coding 114 encodes the image to be transmitted using lossy
predictive encoding scheme.
8
In accordance with an embodiment of the invention, RFB client 202 hints
about encoding preference to RFB server 201 while receiving graphical screen
updates. RFB server 201 may or may not choose to make use of this hint. Based on
the type of encoding scheme adapted by the VNC server 101, the decoder 205 will
decode the encoded data and output the decoded data in required format such as RGB,
YUV etc. which can be easily displayed on different display devices. The VNC server
101 informs the VNC client about the encoding scheme adopted to encode the image.
Similarly, the adaptive encoded image is shared with the plurality of RFB clients
(202, 203 and 204) wherein the respective decoders (205, 206 and 207) decode the
image to display it on the VNC client machine.
In this embodiment the VNC system uses RFB protocol for remote desktop
sharing. However, for a person skilled in the art it is apparent that an equivalent
protocol can be easily substituted. Even though the illustration shows VNC server
interacting with more than one client it is apparent that it can perform VNC with a
single client.
FIG. 3 shows a flow chart of adaptive encoding method in accordance with an
exemplary embodiment. At step 301 frame buffer comparator compares previous
stored images with current displayed image and provides percentage of scene changes
between the images. If the entire image needs to be transmitted as requested by the
VNC client, then percentage of image change can be set to 100%. This percentage is
then used to take a decision whether to select lossy encoding or lossless encoding in
the next step 302. At step 302 if the percentage of image change is greater than the
predetermined encoder threshold then lossy encoding is selected, else lossless
encoding is selected.
The encoding threshold depends on application for with VNC is used. For
example for Video Conference System the encoder threshold can be set in the range
of 0 to 20% and if it is content sharing the encoder threshold can be set in the range of
0 to 40%.
9
At step 304 a decision to select lossless predictive encoding or lossless nonpredictive
encoding is carried out. Here percentage of image change is compared with
predetermined lossless threshold. Lossless predictive encoding scheme is employed at
step 308 if the percentage of image change is greater than the predetermined lossless
threshold, for example video lossless encoding; else lossless non-predictive encoding
scheme is employed at step 307, for example image lossless encoding.
As an example, the lossless threshold selected should be less than 20% for
lossless non-predictive encoding and can vary based on the application need.
For video conference application the lossless threshold can be set in the
range of 0 to 10%. Where as, in content sharing application the lossless threshold can
be set in the range of 0 to 20%.
It step 303 a decision to select lossy predictive encoding or lossy nonpredictive
encoding is carried out. Here percentage of image change is compared with
the predetermined lossy threshold. Lossy predictive encoding scheme is employed at
step 306 if the percentage of image change is greater than the predetermined lossy
threshold, for example video lossy encoding; else lossy non-predictive encoding
scheme is employed at step 305 for example image lossy encoding. At step 309 the
encoded image is transmitted to one or more clients after exchanging handshake
between the VNC server and the VNC clients about encoding mechanism.
In one exemplary embodiment, the lossy threshold selected is in the range of
40% to 70% for lossy non-predictive encoding and can vary based on the application
need. For video conference application the lossy threshold can be set in the range of
40 to 50%. Where as, in content sharing application the lossless threshold can be set
in the range of 40 to 60%.
The non-predictive coding does not employ any kind of prediction. Lossless
non-predictive encoding is used when it is important that the original and the
decompressed data need to be identical. In case of lossy non-predictive encoding
redundancy part in the image to be encoded is not noticeable visually is removed and
reproduced in the VNC client, without deteriorating the quality.
10
Predictive coding employs prediction. Only the difference (residue) between
the two images i.e. current image and its previous images is encoded. Lossless
Predictive encoding is used when it is important that the original and the
decompressed data need to be identical. In case of lossy predictive encoding the
redundancy part in the current image frame which is not noticeable visually is
removed and reproduced in the receiver side, without deteriorating the quality, here
only residue part will be coded.
Few example of lossless encoding are Huffman Coding, Adaptive Huffman
Coding, LZW algorithm, Arithmetic coding etc. Few example of lossy encoding are
JPEG, EXIF, JPEG2000, MPEG4, H263, H264, SVC etc.
Following paragraphs shows examples of different possible scenarios when
the predetermined encoder threshold, predetermined lossless threshold, predetermined
lossy threshold are set to particular values.
In one example let the predetermined encoder threshold be set to 40%,
predetermined lossless threshold be set to 20% and predetermined lossy threshold be
set to 60%.
Scenario 1: Let the percentage image change between previous stored images
with current displayed image be 80%. In this case a lossy encoding scheme is
selected. In the next step since the percentage image change is greater than
predetermined lossy threshold, predictive lossy encoding is employed.
Scenario 2: Let the percentage image change between previous stored images
with current displayed image be 50%. In this case a lossy encoding scheme is
selected. In the next step since the percentage of image change is less than
predetermined lossy threshold, thus non-predictive lossy encoding scheme is
employed.
Scenario 3: Let the percentage image change between previous stored images
with current displayed image be 30%. In this case a lossless encoding scheme is
selected. In the next step since the percentage of image change is greater than
11
predetermined lossless threshold, hence predictive lossless encoding scheme is
employed.
Scenario 4: Let the percentage image change between previous stored images
with current displayed image be 5%. In this case a lossless encoding scheme is
selected. In the next step since the percentage of image change is less than
predetermined lossless threshold, non-predictive lossless encoding scheme is
employed.
Scenario 5: When the VNC client request entire frame, the percentage of
image change between previous stored images with current displayed image is
considered 100% and hence lossy predictive coding is employed.
In one embodiment, the instant disclosure provides plurality of benefits. For
example the benefits includes, but not limiting to access, operate, and configure a
computer, mobile or other device remotely from another computer, mobile or other
device. This can be accomplished via the internet and/or a local area connection.
In an exemplary embodiment the instant disclosure provides for remote
desktop sharing for example, shared working in the workplace or classroom,
distributing educational context to a group of students simultaneously to view a
computer/mobile or other device screen being manipulated by an instructor or to
allow the instructor to take control of the students' computer/mobile or other device to
provide assistance.
In an exemplary embodiment the instant disclosure provides support on how
to use computer/mobile or other device within a geographically spread family. For
example, like control of your relative's/friend mobile to show them how to install and
use some new software package (games, applications) or to install software or an
application.
In one embodiment the adaptive image encodings selects encoding scheme
based on percentage of image change.
12
In one embodiment the adaptive image encoding provides for effective usage
of network bandwidth, which in turn allows more number of VNC clients to connect
VNC server.
The embodiments of the instant invention provide significant improvement in
the performance of VNC.
The present disclosure is not to be considered limited in terms of the particular
embodiments described in this application, which are intended as illustrations of
various aspects. Many modifications and variations can be made without departing
from its spirit and scope, as will be apparent to those skilled in the art. Functionally
equivalent methods and apparatuses within the scope of the disclosure, in addition to
those enumerated herein, will be apparent to those skilled in the art from the
foregoing descriptions. Such modifications and variations are intended to fall within
the scope of the appended claims. The present disclosure is to be limited only by the
terms of the appended claims, along with the full scope of equivalents to which such
claims are entitled. It is also to be understood that the terminology used herein is for
the purpose of describing particular embodiments only, and is not intended to be
limiting.
With respect to the use of substantially any plural and/or singular terms herein,
those having skill in the art can translate from the plural to the singular and/or from
the singular to the plural as is appropriate to the context and/or application. The
various singular/plural permutations may be expressly set forth herein for sake of
clarity.
While various aspects and embodiments have been disclosed herein, other
aspects and embodiments will be apparent to those skilled in the art. The various
aspects and embodiments disclosed herein are for purposes of illustration and are not
intended to be limiting, with the true scope and spirit being indicated by the following
claims.
13
We Claim:
1) An adaptive image encoding method in a virtual network computing (VNC)
environment, wherein the VNC environment comprises a server and one or
more clients communicating with the server, said method comprising acts of:
a. calculating percentage change of image between previous stored image
with current image displayed on the server
b. selecting lossy encoding if the percentage change is greater than a
predetermined encoder threshold, else selecting lossless encoding;
wherein the lossy encoding selection leads to applying nonpredictive
lossy encoding if the percentage change is less than a
predetermined lossy threshold else predictive lossy encoding is
applied;
wherein the lossless encoding selection leads to applying nonpredictive
lossless encoding if percentage of image change is less than
predetermined lossless threshold else predictive lossless encoding is
applied; and
c. transmitting the encoded image to the one or more clients after
exchanging handshake messages between the server and the one or
more clients about type of encoding employed.
2) The method as claimed in claim 1, wherein the percentage change is set to
100, if entire image has to be transmitted.
3) The method as claimed in claim 1, wherein the percentage change is set to a
predetermined value on the basis of client’s request.
4) The method as claimed in claim 1, wherein the encoder threshold is in range
of 0 to 40%.
5) The method as claimed in claim 1, wherein the lossless threshold is in range of
0 to 20%.
6) The method as claimed in claim 1, wherein the lossy threshold is in range of
40 to 70%.
7) A VNC server comprising:
a frame grabber to store an image being displayed,
14
a frame comparator to calculate percentage change of image between
previous stored image with current image displayed on the VNC
server,
a processor responsive to instructions for,
selecting lossy encoding if the percentage change is greater than
predetermined encoder threshold, else selecting lossless encoding;
wherein the lossy encoding selection leads to non-predictive
lossy encoding of the image if the percentage change is less than
predetermined lossy threshold else predictive lossy encoding;
wherein the lossless encoding selection leads to non-predictive
lossless encoding of the image if the percentage change is less than
predetermined lossless threshold else predictive lossless encoding; and
a means to transmit the encoded image to one or more clients.
8) A system having at least one VNC server as claimed in claim 7 and at least
one client interacting with the server using remote frame buffer protocol.
Dated this 16th day of July, 2010
Naveen.C.
IN/PA-1419
Of K & S Partners
Agent for the Applicant
15
AN ADAPTIVE IMAGE ENCODING METHOD IN A VIRTUAL NETWORK
COMPUTING (VNC) ENVIRONMENT