DESCRIPTION
IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD
TECHNICAL FIELD
[0001]
The present technology relates to an image
processing device and an image processing method, and
more particularly, to a technology for reducing
deterioration in quality of a predicted image and
buppressing deterioration in compression efficiency.
PACKGROUND ART
In recent years, devices that treat image
information as digital information and transmit and store
high-efficient information at that time, for example,
devices that conlform with a scheme such as MPEG for
performing compression by orthogonal transform such as
discrete cosine' transform and motion compensation have
.been proliferated in general households and broadcast
stations.
In particular, MPEG2 (ISO/IEC 13818-2) is defined
as a general image encoding scheme, and thus has
currently been in wide use in a wide range of
applications for professional use and consumer use. Wheri
the MPEG2 compression scheme is used, a high compression
rate and good image quality can be realized by assigning
a code rate (a bit rate) of 4 to 8 Mbps, for ex$mple, in
a case of an interface scanned image with a standard
resolution of 720 x 480 pixels. Further, in a case - of an
interface scanned image with the high resolution of 1920
x 1088 pixels, a high compression rate and good image
quality can be realized by assigning a code rate of 18 to
22 Mbps.
[0004]
A considerable amount of calculation is required in
encoding and decoding than in a conventional encoding
scheme such as MPEG2 or MPEG4. However, standardi2ation
for realizing higher encoding efficiency has been carried
out as Joint Model of Enhanced-Compression Video Coding,
and thus an international standard has been created in
the name of H.264 and MPEG-4 Part 10 (hereinafter,
referred to as H.264/AVC (Advanced Video Coding)).
[00051
In H.264/AVCI as illustrated in Fig. 1, one macro
block constituted by 16 x 16 pixels can be divided as one
of the pixel areas of 16 x 16, 16 x 8, 8 x 16, and 8 x 8,
3
and each pixel area has each independent motion vector.
Further, as illustrated in Fig. 1, the 8 x 8 pixel area
can be divided as one of the sub-areas of 8 x 8 pixels, 8
x 4 pixels, 4 x 8 pixels, and 4 x 4 pixels, and each subarea
can have each independent motion vector. According
to MPEG-2, a motion prediction and compensation pTocess
is performed in a motion prediction and compensation
process unit of 16 x 16 pixels in a case of a frame
motion compensation mode and in a motion prediction and
compensation process unit of 16 x 8 pixels for each of
the first and second fields in a case of a field motion
compensation mode.
[0006]
In H.264/AVCI as disclosed in Patent Document 1, a
s
motion prediction and compensation process is performed
with decimal pixel accuracy such as 1/4 pixel accuracy.
Fig. 2 is a diagram illustrating the motion prediction
ahd compensation process with the 1/4 pixel a&curacy. In
Fig. 2, a position "A" indicates the position of an
I
integer accuracy pixel stored in a frame memory,
positions "b," "c," and "d" are positions of 1/2 pixel
accuracy, positions "el," "e2," and "e3" are positions of
1/4 pixel accuracy.
[0007]
Clip10 is defined below, as in Expression (1).
[Expression 11
0; if (a < 0)
Clipl(a) = 1 a; otherwise
max pix; if (a > max - pix)
In Expression (I), when an input image has 8-bit
accuracy, the value of max pix -p ix is 255.
[0008]
Pixel values at the positions "b" and "d" are
generated using a 6-tap FIR filter, as in Expression (2)
and Expression (3).
F = A - 2 - 5 . A - 1 + 2 0 - A o + 2 0 . A l - 5 . A 2 +A3 ...
(2)
b, d = Clip1 ((F +-16) >> 5) ... (3)
[0009]
A pixel value at the position "c" is generated
using a 6-tap FIR filter, as in Expression (6) and one of
Expression (4) and Expression (5) .
[OOlO]
F=b-2 - 5 .b-1 + 20 .bo + 20 - b l - 5 - b 2 + b3 ...
(4
F=d-2 - 5 + 2 0 - d o + 20 .dl - 5 .dz + d3 ...
(5
c = Clip1 ( (F + 512) >> 10) ... (6)
Further, a Clipl process is performed only once,
after both product sum processes in horizontal and
vertical directions are performed.
[OOll]
Pixel values at the positions "el" to "e3" are
generated by linear interpolation, as in Expression (7)
to Expression (9) .
el = (A + b + 1) >> 1 ... (7)
e2 = (b + d + 1) >> 1 ... ( 8 )
e3 = (b + c + 1) >> 1 ... (9)
[0012]
In an image compression technology, standardization
of HEVC (High Efficiency Video Coding) for realizing
higher encoding efficiency than the H.264/AVC scheme has
been examined. HEVC is an encoding scheme of which
standardization is in progress by a joint standardization
organization of ITU-T and ISO/IEC, JCTVC (Joint
Collaboration Team-Video Coding). In HEVC, a basic .unit
called a coding unit (CU~ extended from the concept of a
macro block is defined. Non-Patent Document 1 suggests a
technology for enabling image compression with a block
size extended more than the macro block of 16 x 16 pixels.
Further, in HEVC, a prediction unit (PU) which is a basic
unit for division of a coding unit and prediction-is also
defined.
[0013]
In HEVC, as disclosed in Non-Patent Document 2, a
PQAO (Picture Quality Adaptive Offset) is considered to
be provided between a deblock filter and adaptive loop
filter. The kinds of offset, there are two kinds of
offset called band offset and there are six kinds of
offset called edge offset. Further, the offset may not
be adapted. Encoding efficiency is improved by
partitioning an image in a quad-tree and selectively
perfbrming encoding in each area according to one of the
above-described kinds of offset.
CITATION LIST
PATENT DOCUMENT
[0014]
Patent Document 1: Japanese Patent Application Laid-Open
No. 2010-016453
NON-PATENT DOCUMENT
[0015]
Non-patent Document 1: "Video Coding Using Extended Block
Sizes" (Study Group 16, Contribution 123, ITU, COM16-
C123-E in January, 2009) -
Non-patent Document 2: "CE8 Subtest 3: Picture Quality
Adaptive Offset," JCTVC-D122, in January 2011)
SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0016]
When predicted image data is generated by
performing motion compensation using reference image data
based on a motion vector detected through motion
prediction and noise is contained in reference image data,
the predicted image data also contains no,ise, thereby
resulting in deterioration in compression efficiency.
[0017]
Accordingly, an object of the present technology is
to provide an image processing device and an image
processing method capable of suppressing deterioration in
compression efficiency by reducing deterioration in
quality of a predicted image.
'SOLUTIONS TO PROBLEMS
[0018]
A first aspect of the present technology lies in an
image processing device including: a loop filter unit
that executes a filter process on reference image data
referred to by a current block; an interpolation filter
unit that generates predicted image data using the
reference image data and a motion vector corresponding to
the current block;,and a filter control unit that
switches a filter coefficient of an interpolation filter
according to the filter process performed by the loop
filter unit on the reference image data used by the
interpolation filter.
[0019]
According to the technology, for example, in an '
image processing device, such as an image encoding device
that encodes a difference between input image data and
predicted image data by dividing the input image data
into a plurality of pidel blocks and performing a
prediction process on each pixel block using reference
image data or an image decoding device that performs a
decoding process on compression image information
generated by the image encoding device, a loop filter
unit performs a filter process on the reference image
data referred to by a current block, for example, a
deblocking filter process or an adaptive loop filter
process. An interpolation filter unit generates
predicted image data using the reference image data and a
motion vector corresponding to the current block.
According to the filter process performed by the loop
filter unit on the reference image data used by the
interpolation filter, for example, noise removal strength
is adjusted by switching the filter coefficient of the
interpolation filter unit depending on whether the
reference image data used by the interpolation filter
unit is subjected to the filter process by the loop
filter unit. The filter coefficient not subjected to the
filter process by the loop filter unit is set to a
coefficient in which the'noise removal strength is
enhanced more than that of the filter coefficient
subjected to the filter process. When an offset process
is adaptively performed and when the offset process is
not performed, a coefficient in which the noise removal
strength is enhanced more than a case wheA the offset
process is performed is set.
[0020]
A second aspect of the present technology lies in
an image processing method including the steps of:
executing a filter process on reference image data
referred to by a current block; generating predicted
image data using the reference image data and a motion
vector corresponding to the current block; and switching
a filter coefficient of the generating of the predicted
image data according to the filter process in the
executing of the filter process on the reference image
data used in the generating of the predicted image data.
EFFECTS OF THE INVENTION
[0021]
According to the technology, the loop filter unit
performs the filter process on the reference image data
referred to by the current block. The interpolation
filter unit generates the predicted image data using the
reference image data and the motion vector corresponding
to the current block. The filter coefficient of the
interpolation filter is switched according to the filter
process performed on the reference image data used in the
interpolation filter by ;he loop filter unit. Therefore,
when the loop filter unit does not perform the filter
process, the predicted image data with small noise can be
generated, for example, by causing the interpolation
filter unit to perform the filter process on the
reference image data with the filter characteristics in
which the noise removal strength is enhanced. Therefore,
deterioration in the quality of a predicted image is
reduced, and thus deterioration in compression efficiency
can be suppressed.
BRIEF DESCRIPTION OF DRAWINGS
[0022]
Fig. 1 is a diagram illustrating block sizes in
H.264/AVC.
Fig. 2 is a diagram illustrating a motion
prediction and compensation process of 1/4 pixel accuracy.
Fig. 3 is a diagram illustrating the configuration
of an image encoding device.
Fig. 4 is a diagram illustrating the configuration
a motion prediction/compensation unit.
Fig. 5 is a diagram illustrating an example of
filter characteristics.
Fig. 6 is a diagram illustrating a hierarchical
structure when the size of a macro block is extended.
Fig. 7 is a flowchart illustrating a process of an
image encoding device.
Fig. 8 is a flowchart illustrating prediction
processes. -
Fig. 9 is a flowchart illustrating an intraprediction
process.
Fig. 10 is a flowchart illustrating an interprediction
process.
Fig. 11 is a flowchart illustrating a motion
I
compensation process.
Fig. 12 is a diagram iilustrating a case in which
ON/OFF of an adaptive loop filter process is' switched
between in units of frames.
Fig. 13 is a diagram illustrating the configuration
of an image decoding device.
1
Fig. 14 is a diagram illustrating the- cdnfiguration
of a motion compensation unit.
Fig. 15 is a flowchart illustrating a process of
the image decoding device.
Fig. 16 is a flowchart illustrating prediction
' processes.
Fig. 17 is a flowchart illustrating an interprediction
process.
Fig. 18 is a diagram illustrating another
configuration of the image encoding device.
Fig. 19 is a diagram illustrating a quad-tree
structure.
Fig. 20 is a diagram illustrating band offset.
Fig. 21 is a diagram illustrating edge offset.
Fig. 22 is a diagram illustrating rule lists of the
edge offset.
Fig. 23 is a flowchart illustrating a motion
compensation process. -
Fig. 24 is a diagram illustrating another
configuration of the image decoding device.
Fig. 25 is a diagram illustrating an example of the
overall configuration of a computer apparatus.
Fig. 26 is a diagram illustrating an example of the
overall configuration of a television apparatus.
Fig. 27 is a diagram illustrating an example of the
overall configuration of a portable telephone.
Fig. 28 is a diagram illustrating an example of the
overall configuration of a recording/reproduction
apparatus.
Fig. 29 is a diagram illustrating an example of the
overall configuration of an imaging apparatus.
MODE FOR CARRYING OUT THE INVENTION
[0023]
:Hereinafter, a mode for carrying out the present
technology will be described. The description will be
made in the following order.
1. Configuration of Image Encoding Device
2. Process of Image Encoding Device
3. Configuration of Image Decoding Device
4. Process of Image Decoding Device
5. Another Configuration and Process of Image
Encoding Device I
6. Another Configuration and Process of Image
Decdding Device
7. Case of Software Process
8. Case 'of Application to Electronic Apparatus
[0024]
<1. Configuration of Image Encoding Device>
Fig. 3 is a diagram illustrating t h ~co nfiguration
t
of an image encoding device to which an image processing
device is applied. An image encoding device 10 inTcludes
an analog/digital conversion unit (A/D conversion unit)
11, a screen rearrangement buffer 12, a subtraction unit
13, an orthogonal transform unit 14, a quantization unit
15, a lossless encoding unit 16, an accumulation buffer
17, and a rate control unit 18. The image encoding
device 10 further includes an inverse quantization unit
21, an inverse orthogonal transform unit 22, an addition
unit 23, a loop filter unit 24, a frame memory 25, an
intra-prediction unit 31, a motion
prediction/compensation unit 32, and a predicted
image/optimum qode selection unit 33. Further, the loop
filter unit 24 includes, for example, a deblocking filter
unit 241 and an ALF (Adaptive Loop Filter) unit 242.
[0025]'
The A/D conversion unit 11 converts an analog image
signal into digital image data and outputs the'digital
image data to the screen rearrangement buffe~ 12.
The screen rearrangement buffer 12 sorts frames of
the image data output from the A/D conversion unit 11.
The screen rearrangement buffer 12 sorts the frames
5 according to a GOP (Group of Pictures) structure relevant
to an encoding process and outputs the - image data
obtained after the sorting to the subtraction unit 13,
the ALF unit 242, the intra-prediction unit 31, and the
motion prediction/compensation unit 32.
The subtraction unit 13 is supplied with the. image
data output from the screen rearrangement buffer 12 and
predicted image data selected by the predicted
image/optimum mode selection unit 33 to be described
15 below. The subtraction unit 13 calculates prediction
error data which is a difference between the image data
output from the screen rearrangemenLbuffer 12 and the
predicted image data supplied from th,e predicted
image/optimum &ode selection unit 33, and then outputs
20 the prediction,!error data to the orthogonal transform
I
unit 14.
The orthogonal transform unit 14 performs an
orthogonal transform process such as discrete cosine
25 transform (DCT)' or Karhunen-Loeve transform,on the
prediction error data output from the subtraction unit 13.
The orthogonal transform unit 14 outputs transform
coefficient data obtained through the orthogonal
transform process to the quantization unit 15.
30 [0029]
The quantization unit 15 is supplied with the
transform coefficient data output from the orthogonal
transform unit 14 and a rate control signal output from
the rate control unit 18 to be described below. The
quantization unit 15 per-f orms quantization on the
transform coefficient data and outputs quantization data
to the lossless encoding unit \16 and the inverse
i
quantization unit 21. The quahtization unit 15 switches
a quantization parameter (quankization scale) based on
the rate control signal from the rate control uni& 18 and
changes the bit rate of the quantization data.
[0030]
The lossless encoding unit 16 is supplied with the
quantization data output from the quantization unit 15,
prediction mode information from the intra-prediction
unit 31 to be described below or prediction mode
information, a differential motion vector, and loop
filter information, and the like from the motion
prediction/compensation unit 32. Further, information
indicating whether an optimum mode is used for the intraprediction
or the inter-prediction is supplied from the
predicted image/optimum mode selection unit 33. The
prediction mode information includes a prediction mode,
block size information on a motion prediction unit, or
the like according to the intra-prediction or interprediction.
The lossless encoding unit 16 performs a
lossless encoding process on the quantization data
through, for example, variable-length coding or
arithmetic coding to generate image compression
information and outputs the image compression information
to the accumulation buffer 17. The lossless encoding
&
unit 16 performs the lossless encoding process on the
prediction mode information supplied from the intraprediction
unit 31, when-the optimum mode is the intraprediction.
The lossless encoding unit 16 performs
lossless encoding on the prediction mode information, the
differential motion vector, or the like supplied from the
motion prediction/compensation unit 32, when the optimum
moge is the inter-prediction. The lossless encoding unit
16 causes information obtained through the lossless
encoding to be included in the image compression
information. For example, the lossless encoding unit 16
adds the information to header information of an encoding
stream which is the image compression information. The
lossless encoding unit 16 causes the loop filter
information or loop filter information subjected to the
lossless encoding to be included in the image compression
information.
[0031]
The accumulation buffer 17 accumulates the image
compression information from the lossless encoding unit
16. The accumulation buffer 17 outputs the accumulated
image compression information at a transmission rate
suitable for a transmission path.
[0032]
The rate control unit 18 monitors a free space of
the accumulation buffer 17, generates a rate control
signal according to the free space, and outputs the rate
control signal to the quantization unit 15. For example,
the rate control unit 18 - acquires information indicating
the free space from the accumulation buffer 17. When the
free space is small, the rate control unit 18 reduces the
I
bit rate df the quantization data according to the rate
control signal. Further, when the free lspace of the
accumulation buffer 17 is sufficiently large, the rate
control unit 18 increases the bit rate of the
quantization data by the rate control signal.
[00331
*
The inverse quantization unit 21 performs an
inverse quantization process on the quantization data
supplied from the quantization unit 15. The inverse
quantization unit 21 outputs the transform coefficient
data obtained through the inverse quantization process to
the inverse orthogonal transform unit 22.
[0034 I
The inverse orthogonal transform unit 22 performs
an inverse orthogonal transform process on the transform
coefficient data supplied from the inverse quantization
unit 21 and outputs obtained data to the addition unit 23.
[00351
The addition unit 23 generates decoded image data
by adding the data supplied from the inverse orthogonal
transform unit 22 to the predicted image data supplied
from the predicted image/optimum mode selection unit 33,
and then outputs the decoded image data to the deblocking
filter unit 241 and the intra-prediction unit 31. The
.decoded image data is used as image data of a reference
image.
[00361 -
The loop filter unit 24 is a filter that is
embedded in an encoding loop and performs subsequent
prediction using the decoded image data as the reference
image data. The loop filter unit 24 excludes the
{ influence of noise froin a prediction error in theaotion
compensation prediction by generating the reference image
data from which the noise is removed, and thus improves
e
encoding efficiency.
The deblocking filter unit 241 of the loop filter
unit 24 performs a filter process of reducing block
distortion occurring when an image is encoded. The
deblocking filter unit 241 performs a deblocking filter
process of removing block distortion from the decoded
image data supplied from the addition unit 23 and outputs
the processed decoded image data to the ALE unit 242.
The deblocking filter unit 241 adaptively performs the
deblocking filter process so that the deblocking filter
process may not be pe-rformed in a partial area in a frame.
[0038]
The ALE unit 242 performs a filter process of
removing distortion which has not be removed in the
filter process of the deblocking filter unit 241 or noise
of an area not subjected to the filter process by the
deblocking filter unit 241. The ALE unit 242 performs an
adaptive loop filter process only on an area in which the
encoding efficiency is improved when the filter process
is perforrhed to reduce an error between the input image
data supplied from the sheen rearrangement buffer 12 and
the decoded image data supplied from the deblocking
filter unit 241. The ALE unit 242 outputs the processed
decoded image data to the frame memory 25. The ALF unit
242 outputs filter information on the adaptive loop
filter process to the lossless encoding unit 16 s6 as to
be inclbded in the ima,ye cor$pression information, so that
the Sam$ adaptive lood filt d r process as the image
encoding process can be performed in an image decoding
process. As the filter information, for example, ALF
flag information and a filter coefficient or the like
used in the adaptive loop filter process are used. The
5 ALF unit 242 outputs the ALF flag information to the
motion prediction/compensation unit 32.
[0039]
As the ALF flag information, for example,
adaptive - loop - filter - flag, alf - cu - control - flag, alf - flag,
10 or the like is used. Here, adaptive - loop - filter - flag is
a flag indicating whether the adaptive loop filter
process is performed 'in the current slice. Further,
alf - cu - control - flag is a flag indicating whether the
adaptive loop filter process is performed on luminance
15 data. Furthermore, alf flag is a flag identifying a
I -
I coding unit performing the adaptive loop filter process. ~ Accordingly, an area in which the adaptive loop filter
process is performed on the reference image data can be
I 1 determined based on the ALF flag information.
The frame memory 25 retains the decoded image data
*
subjected to the filter process and supplied from the ALF
1 unit 242. The decoded image data retained in the frame
I .memory 25 is supplied as the reference image data to the
1 ; 25 motion prediction/compensation unit 32.
[0041] -
The intra-prediction unit 31 determines an optimum
intra-hrediction mode by performing prediction in all of
the intra-prediction mode candidates using the input
30 image &ta of an encoding target image supplied from the
screen rearraniement bu5fer 12 and the reference image
data supplied from the addition unit 23. For example,
the intra-prediction unit 31 calculates a cost function
value in each intra-prediction mode and sets, as the
optimum intra-prediction mode, the intra-prediction mode
in which the encoding efficiency is the best based on the
calculated cost function value. The intra-prediction
unit 31 outputs the predicted image data generated in the
optimum intra-prediction mode and the cost function value
in the optimum intra-prediction mode to the predicted -
image and optimum mode selection unit 33. The intrappediction
unit. 31 outputs prediction mode information
indicating the optimum intra-prediction mode to the
lossless encoding unit 16.
[0042]
The motion prediction/compensation unit 32
determines an optimum inter-prediction mode by performing
prediction in all of the inter-prediction mode candidates
using the input' image data of an encoding target image
supp1ie.d from the screen-rearrangement buffer 12 and the
reference image data supplied from the frame memory 25.
For example, the motion prediction/compensation unit 32
calculates a cost function value in each inter-prediction
mode and sets, as the optimum inter-prediction'mode, the
inter-prediction mode in which the encoding efficiency is
the best based on the calculated cost function value.
The motion prediction/compensation unit 32 outputs the
predicted image data generated in the opti;mutn inter-
,I
prediction mode and the cost function value in the
optimum inter-prediction mode to the predicted image and
optimum mode selection unit 33. The motion
prediction/compensation unit 32 outputs prediction mode
information on the optimum inter-prediction mode to the
lossless encoding unit 16. The motion
prediction/compensation unit 32 switches a filter
coefficient according to the filter process performed on
5 the reference image data by the loop filter unit 24 in
order to suppress deterioration in the encoding
efficiency caused due-to noise of the reference image
data.
[0043]
10 Hereinafter, a case will be described in which the
filter coefficient when the adaptive loop filter process
,
is not performed is switched to a coefficient for which
noise removal strength is enhanced, compared to the
filter coefficient when the adaptive loop filter process
15 is performed.
[0044]
Fig. 4 is a diagram illustrating the configuration
of the motion prediction/compensation unit 32. The
motion prediction/compensation unit 32 includes a motion
20 detection unit 321, a mode determination unit 322, a
motion compensation processing unit 323, and a motion
vector buffer 324. C
[0045]
The motion detection unit 321 is supplied with the
25 sorted input image data supplied from the screen
rearrangement buffer 12 and the reference image data read
from the frame memory 25. The motion detection unit 321
detects a motion vector by performing motion search in
all of the inter-prediction mode candidates. The motion
30 detection unit 321 outputs, to the mode determination
unit 322, a motion vector indicating the detected motion
vector together with the input image data and the
reference image data at the time of detecting the motion
vector.
, [0046]
5 The mode determination unit 322 is supplied with
I
r the motion vector and the input image data from the
t motion detection unit 321, the predicted image data from
I,
the motion compensation processing unit 323, and a motion
I
I vector of an adjacent prediction unit from the motion
10 vector buffer 324. The mode determination unit 322 sets
a prediction motion vector by performing median
prediction or the like using the motion vector of the
i adjacent prediction unit and calculates a differential
motion vector indicating a difference between the
15 prediction motion vector-and the motion vector detected
by the motion detection unit 321. The mode determination
unit 322 calculates the cost function values in all of
the inter-prediction mode candidates using the input
image data, the predicted image data, and the
20 differential motion vector. The mode determinatign unit
322 determines a mode in which the calculated cost
function value id the minimum as the optimum interprediction
mode. The mode determination unit 322 outputs,
to the motion compensation processing unit 323,
1 25 prediction mode information indicating the determined
I optimum inter-prediction mode and the cost function value
together with the motion vector relevant to the optimum
i
I inter-prediction moded, the differential motion vector,
and the like. The mode determination unit 322 outputs
30 the prediction mode information relevant to the interprediction
mode and t-h e motion vector to the motion
compensation processing unit 323 in order to calculate
the cost function values in all of the inter-prediction
-
mode candidates.
[00471
For example, the cost function values are
calculated based on one method of a High Complexity mode
or a Low Complexity mode so as to be determined in JM
(Joint Model) which is reference software in the
H.264/AVC scheme.
[0048]
That is, in the High Complexity mode, a tentatively
lossless encoding process is performed in all - of the
prediction mode candidates to calculate the cost function
value expressed by Expression (10) below for each
prediction mode.
Cost (Mode E a) = D + h e R ... (10)
[0049] -
a indicates the whole set of the prediction mode
candidfes in which the image of the prediction unit is
encode D indicates a differential energy (distortion)
between the predicted image and the input image when the
encoding is performed in the prediction mode. R
indicates an amount of occurring code including an
orthogonal transform coefficient, the prediction mode
information, or the like, h indicates the Lagrange
multiplier given as a function of a quantization
parameter QP.
[0050]
That is, in order to perform the encoding in the
High Cbmplexity mode, a tentative encoding process is
required to be performed once in all of the prediction
-
mode candidates to calculate the parameters D and R, and
thus a larger amount of calculation is necessary.
[0051]
In the Low Complexity mode, on the other hand, the
5 predicted image is generated and header bits including
the differential motion vector, the prediction mode
information, or the like are generated in all of the
prediction mode candidates to calculate the cost function
values expressed by Expression (11) below.
10 Cost (Mode E 0) = LD + QP2Quant (QP) . Header - Bit ...
R indicates the whole set of the prediction mode
candidates in which the image of the prediction unit is
15 encoded. D indicates a differential energy (distortion)
between the predicted image and the input image when the
encoding is performed in the prediction mode. Header - Bit
is the header bits for the prediction mode and QP2Quant
is'a function given as a function of the quantization
20 parameter QP. I
[0053]
That is, in the Low Complexity mode, the prediction
process is required to be performed on each prediction
mode, but even a decoded image is not necessary.
i
25 Therefore, the encoding can be realized with the lower
1
I amount of calculation than the amount of calculation in
the High Complexity mbde.
[0054]
The motion compensation processing unit 323
30 includes a compensation control unit 3231, a coefficient
table 3232, and a filter unit 3233. The motion
compensation processing unit 323 determines whether the
reference image data is subjected to the adaptive loop
filter process based on the ALF flag information
generated by the ALF unit 242. The motion compensation
5 processing unit 323 switches filter characteristics in
the filter process on the reference image data used to
generate the predicted image data according to the
determination result (hereinafter, referred to as an "ALF
process determination result"). For example, the motion
10 compensation processing unit 323 switches the filter
characteristics in the filter process by switching a
filter coefficient supplied from the coefficient table
3232 to the filter unit 3233. In Fig. 4, the filt%r
coefficient is configured to be supplied from the
15 coefficient table 3232 to the filter unit 3233. However,
'the filter coefficient may be configured to be supplied
' from the compensation control unit 3231 to the filter
unit 3233.
The compensation control unit 3231 includes an ALF
information memory 3231a that stores the ALF flag
information supplied from the ALF unit 242. The
compensation control unit 3231 controls reading the
reference image data from the frame memory 26 - based on
the block size (including a shape) of the prediction unit
supplied from the mode ,determination unit 322, the motion
vector, and a reference index. The compensation control
unit 3231 determines whether the read reference image
data is subjected to the adaptive loop filter process
based on the ALFjflag information stored in the,ALF
information memoiy 3231a and outputs an ALF process
determination result to the coefficient table 3232. For
example, when the compensation control unit 3231
determines that the reference image data is subjected to
the adaptive loop filter process based on the ALF flag
5 information, the compensation control unit 3231 sets -
alf - applied - flag as the ALF process determination result
to be true. Conversely, when the compensation control
unit 3231 determines that the reference image data is not
subjected to the adaptive loop filter process, the
10 compensation control unit 3231 sets alf - applied - flag to
A
be false. The compensation control unit 3231 outputs
alf - applied - flag indicating the ALF procTss determination
result to the coefficient table 3232.
[0056]
15 The coefficient table 3232 is supplied with a
decimal part of the motion vector and the ALF process
determination result generated by the compensation
control unit 3231. The coefficient table 3232 has a
filter coefficient (filter- c oef- w ithout- A LF) used for the
20 reference image data not subjected to the adaptive loop
filter process and a filter coefficient
(filter - coef - with - ALFJ used for the reference image data
subjected to the adaptive loop filter process.
[0057]
25 When the reference image data is not subjected to
the adaptive loop filter process, noise of the reference
image data is not removed. Accordingly,
filter - coef - without - ALF is set as a coefficient with
filter characteristics in which a noise removal strength
30' is greater than that of the filter coefficient used for
the reference image data subjected to the adaptive loop
*
filter~process. Conversely, when the reference image
data is subjected to the adaptive loop filter process,
the noise is expected to be removed. Accordingly,
*
filter - coef with ALF is set as a coefficient - - with filter
characteristics preserving components from a lowfrequency
component to a high-frequency component. Fig.
5 is a diagram illustrating an example of the filter
characteristics. The characteristics obtained usi-n g
filter - coef - without - ALF are configured as low-frequency
pass filter characteristics in which attenuation of the
high-frequency is large, compared to the characteristics
obtained using filter - coef - with - ALF. Further, when the
reference image data is not subjected to the adaptive
loop filter process, the filter characteristics may be
filter characteristics in which the noise removal
strength is increased, compared to the case in which the
reference image data is subjected to the adaptive loop
filter process, and are not limited to the
characteristics illustrated in Fig. 5. For example, the
filter characteristics may be attenuation characteristics
different from the characteristics illustrated in Fig. 5.
[0058]
Even when the motion vector has decimal pixel
accuracy, the coefficient table 3232 st0res.a plurality
of filter coefficients corresponding to decimal pixel
positions so that the predicted image data with the
decima'l image accuracy can be generated through an
interpolation filter process, as described above. That
is, the coefficient table 3232 stores a filter
coefficient set (filter - coef - set - wo - ALE) used for the
reference image data not subjected to the adaptive loop
filter process and a filter coefficient set
(filter- c oef- s et- w - A LF) used for the reference image data
subjected to the adaptive loop filter process.
[0059]
The coefficient table 3232 outputs the decimal part
of the motion vector and the filter coefficient selected
according to the ALF process determination result to the
filter unit 3233. For example, when alf - applied - flag is
true, the coefficient table 3232 selects the filter
coefficient according to the decimal part of the motion
vector from filter - coef - set - w - ALF and outputs the filter
coefficient to the filter unit 3233. Further, when
alf - applied - flag is false, the coefficient table 3232
selects the filter coefficient according to the decimal
part of the motion vector from the filter - coef - set - wo - ALF
and outputs the filter coefficient to the filter unit
3233.
[0060]
The filter unit 3233 performs an interpolation
filter process of obtaining given predicted image data
with the decimal pixel accuracy in the reference image
data of a current block by using the filter coefficient
supplied from the coefficient table 3232. When the
reference image data is not subjected to the adaptive
loop filter process, the filter unit 3233 performs the
filter process with the filter aharacteristics in which
the noise removal streng-t h is edhanced, compared to the
filter coefficient used for the reference image data
subjected to the adaptive loop filter process. When the
mode determination unit 322 calculates the cost function
value, the filter unit 3233 outputs the generated
I predicted image data to the mode determination unit 322
so that the optimum inter-prediction mode can be
determined. The filter unit 3233 outputs the predicted
image data generated in the optimum inter-predictron mode
5 to the predicted image and optimum mode selection unit 33:
[0061]
Although not illustrated, the motion compensation
processing unit 323 outputs the motion vector used for
motion compensation to the motion vector buffer 324 and
10 outputs the prediction mode information of the optimum
inter-prediction and the differential motion vector or
the like in the corresponding mode to the lossless
I encoding unit 16. T.he motion compensation processing
I
unit 323 outputs the cost function value in the optimum
15 inter-prediction to the predicted image and - optimum mode
selection unit 33 illustrated in Fig. 3.
I
1 The predicted image and optimum mode selection unit
i 33 compares the cost function value supplied from the
20 intra-prediction unit 31 to the cost function value
supplied from the motion prediction/compensation unit 32
and selects the mode with the smaller cost function value
as the optimum mode in which the encoding efficiency is
the best. The predicted image and optimum mode selection
25 unit 33 outputs the predicted image data generated in the
optimum mode to the subtraction unit 13 and the addition
unit 23. The predicted image and optimum mode selection
unit 33 outputs information indicating whether the
optimum mode is the intra-prediction mode or the inter-
30 prediction mode to the lossless encoding unit 16.&The
predicted image and optimum mode selection unit 33
switches between the intra-prediction and the inte..r -
prediction in units of slices.
<2. Process of Image Encoding device>
5 The image encoding device performs the encoding
process by extending the size of the macro block more
than, for example, the H.264/AVC scheme. Fig. 6 is a
diagram illustrating an example of a hierarchical
structure when the size of the macro block is extended.
10 In Fig. 6, (C) and (D) of Fig. 6 illustrate cases in
which coding units have the size of 16 x 16 pixels of the
macro block and the size of 8 x 8 pixels of the sub-macro
block defined in the H.264/AVC scheme. Further, (A) of
Fig. 6 illustrates a case in which the block size of the
15 coding unit is 64 x 64 pixels and (B) of Fig. 6
illustrates a case in which the block size of the coding
unit is 32 x 32 pixels. In Fig. 6, llSkip/direct"
indicates a block size when a skip micro-block and a
direct mode are selected.
20 LO0641
In one hierarchy, a plurality of prediction units
including sizes with which the coding unit is divided are
set. dor example, in the hierarchy of the macro block of i
the 64'x 64 pixels illustrated in (A) of Fig. 6, sizes of
25 64 x 64 pixels, 64 x 32 pixels, 32 x 64 pixels, and 32 x
32 pixels are set with the block sizes of the prediction
units in the same hierarchy. Although not illustrated,
prediction units in which the coding unit is divided into
two pieces with asymmetric block sizes may be provided.
30 Further, vME'' indicates the block size of a prediction
unit. Furthermore, "P8 x 8" indicates that a block size
can be divided further to be smaller in a lower-level
hierarchy.
i Next, a process of the image encoding device will
i 5 be described with reference to the flowchart illustrated
in Fig. 7. In step ST11, the A/D conversion unit 11
performs A/D conversion on an input image signal.
1 In step ST12, the screen rearrangement buffer 12 1 10 performs image sorting. The screen rearrangement buffer
12 stores the image &ata supplied from the A/D conversion
I unit 11 and performs sorting from a displaying order of
I
I respective pictures to an encoding order.
I
15 In step ST13, the subtraction unit 13 generates the
prediction error data. The subtraction unit 13 generates
the prediction error data by calculating a difference
I
, I
between the image data of the images sorted in step ST12
I
I and the predicted image data selected by the predicted
I
20 image and optimum mode selection unit 33. The prediction
x
error data has a smaller amount of data than that of the
I original image data. Accordingly, the amount of data can
I
I be compressed, compared to a case in which an image is
directly encoded.
25 [0068]
..
In step ST14, the orthogonal transform unit 14
performs an orthogonal transform process. The orthogonal
transform unit 14 performs the orthogonal transform on
the prediction error data supplied from the subtraction
I
I 30 unit 13. Specifically, the orthogonal transform unit 14
I
i performs orthogonal transform such as discrete cosine
transform or Karhunen-Loeve transform on the prediction
error data and outputs the transform coefficient data.
[0069]
In step ST15, the quantization unit 15 performs the
5 quantization process. The quantization unit 15 quantizes
the transform coefficient data. When the quantization
process is performed, rate control is performed, as
described in the process of step ST25 to be described
below.
In step ST16, the inverse quantization unit 21
performs the inverse ,quantization process. The inverse
quantization unit 21 performs the inverse quantization on
the transform coefficient data quantized by the
15 qu,antization unit 15 according to the characteristics
corresponding to the characteristics of the quantization
unit 15.
In step ST17, the inverse orthogonal transform unit
20 22 performs the inverse orthogonal transform process.
The inverse orthogonal transform unit 22 performs the
inverse orthogonal transform on the transform coefficient
data subjected to the inverse quantization by the inverse
C
i
quantization unit 21 according to the characteristics
25 corresponding to the characteristics of the orthogonal
transform unit 14.
[0072]
In step ST18, the addition unit 23 generates the
reference image>data. The addition unit 23 generates
30 decoding data (referknce image data) by adding the
predicted image data'supplied from the predicted image
>
and optimum mode selection unit 33 to the data obtained
after the inverse orthogonal transform at a position
corresponding to the predicted image.
[0073]
In step ST19, the deblocking filter unit 241
performs the filter process. The deblocking filter unit
241 removes block distortion by filtering the decoded
image data output from the addition unit 23.
[0074]
In step ST20, the frame memory 25 stores the
reference image data. The frame memory 25 stores the
decoding data (reference image data) obtained after the
filter process.
[0075]
In step ST21, the intra-prediction unit 31 and the
motion prediction/compen.sation unit 32 each perform the
prediction process. That is, the intra-prediction unit
31 performs the intra-prediction process in the intraprediction
mode and the motion prediction/compensation
unit 32 performs the motion prediction and compensation
process in the inter-prediction mode. The predict-ion
process will be described in detail below with reference
to Fig. 8. Through these processes, the predictioq
process is each performed in all of the prediction mode
candidates and the cost function values are calculated in
all of the prediction mode candidates. The optimum
intra-prediction mode and the optimum inter-prediction
mode are selected based on the calculated cost function
values, and then the predicted image.generated in the
selected prediction mode, its cost function, and the
prediction mode information are supplied to the predicted
image and optimum mode selection unit 33.
In step ST22, the predicted image and optimum mode
selection unit 33 selects the predicted image data. The
5 predicted image and optimum mode selection unit 33
determines the optimum mode in which the encoding
efficiency is the best based on the respective cost
function values output from the intra-prediction unit 31
and the motion prediction/compensation unit 32. That is,
10 the predicted image and optimum mode selection unit 33
determines the coding unit in which the encoding
efficiency is the best from, for example, the respective
hierarchies illustrated in Fig. 6 and the shape of the
prediction unit in the &ding unit and determines whether
15 to perform the intra-prediction and the inter-prediction.
The predicted image and optimum mode selection unit 33
outputs the predicted image data of the determined
optimum mode to the subtraction unit 13 and the addition
unit 23.. The predicted image data is used for the*
20 calculation of step ST13 and step ST18, as described
above.
[0077 I
In step ST23, the lossless encoding unit 16
performs the lossless encoding process. The lossless
25 encoding unit 16 performs the lossless encoding on the
quantization data output from the quantization unit 15.
That is, the lossless encoding such as variable-length
coding or arithmetic coding is performed on the
quantization data .to compress the data. The lossless
30 encoding unit 16 performs the lossless encoding - on the
prediction mode information corresponding to the
predicted image data selected in step ST22, the
differential motion vector generated through the interprediction,
or the like, so that the lossless encoded
data such as the prediction mode information can be
5 included in the image compression information generated
by performing the lossless encoqing on the quantization
data. The lossless -encoding unit 16 causes the loop
filter information or the loop filter information
subjected to the lossless encoding to be included in the
image compression information.
[0078] -
In step ST24, the accumulation buffer 17 performs
an accumulation process. The accumulation buffer 17
accumulates the image compression information output from
15 the lossless encoding unit 16. The image compression
information accumulated in the accumulation buffer - 17 is
appropriately read and is transmitted to the decoding
side via the transmission path.
[0079]
20 In step ST25, the rate control unit 18 performs the
rate control. When the accumulation buffer 17
accumulates the image compression information, the rate
control unit 18 controls a rate of the quantization
process performed by the quantization unit 15 so that
25 overflow or underflow does not occur in the accumulation
buffer 17.
[0080]
Next, the,prediction process in step ST21 of Fig. 7
will be described with reference to the flowchart of Fig.
30 8.
[0081]
In step ST31, the intra-prediction unit 31 performs
1
the intra-prediction process. The intra-prediction unit
31 performs the intra-prediction on an image of the
prediction unit to be encoded in all of the intraprediction
mode candidates. The decoded image data
before the blocking filter process is performed by the
deblocking filter unit 241 is used as the image data of
the decoded image referred to in the intra-prediction.
Through the intra-predic_tion process, the intraprediction
is performed in all of the intra-prediction
mode candidates to calculate the cost function values in
all of the intra-prediction mode candidates. Then, based
on the calculated cost function values, one intraprediction
mode in which the encoding efficiency js the
best is selected from all of the intra-prediction modes.
In step ST32, the motion prediction/compensation
unit 32 performs the inter-prediction process. The
motion prediction/compensation unit 32 performs the
d
inter-prediction process in the inter-prediction mode
candidates using the decoded image data obtained after
the deblocking process and stored in the frame memory 25.
Through the inter-prediction process, the prediction
process is performed in all of the inter-prediction mode
candidates to calculate the cost function values in all
of the inter-predicti'bn mode candidates. Then, based on
the calculated cost function values, one inter-prediction
mode in which the encoding efficiency is the best is
selected from all of the inter-prediction modes.
[0083]
Next, the inter-prediction process of step ST31 in
Fig. 8 will be described with reference to the flowchart
of Fig. 9.
[00841
In step ST41, the intra-prediction unit 31 performs
the intra-prediction in each prediction mode. The intraprediction
unit 31 generates the predicted image data in
each intra-prediction mode using the decoded image data
before the blocking filter process.
[0085]
In step ST42, the intra-prediction unit 31
calculates the cost function value in each prediction
mode. As described above, for example, the cost function
values are calculated based on one method of a High
Complexity mode or a Low Complexity mode so as to be
determined in JM (Joint Model) which is reference
software in the H.264/A.VC scheme. That is, in the High
Complexity mode, as the process of step ST42, the cost
function value expressed by Expression (10) described
above is calculated in each prediction mode by performing
even a tentative lossless 'encoding process on all of the
prediction mode candidates. Further, in the Low
Complexity mode, as the process of step ST42, the cost
function value expressed by Expression (11) described -
above is calculated in each prediction mode by generating
the predictkd image and calculating even the header bits
of the motion vector, the prediction mode information, or
the like for all of the prediction mode candidates.
[0086]
In step ~ ~ 4 t3he, i ntra-prediction unit 31
determines the optimum intra-prediction mode. The intraprediction
unit 31 determines the optimum intraprediction
mode by selecting one intra-prediction mode in
I
which the cost function value is the minimum from the
cost function values based on the cost function values
calculated in step ST42.
s
5 [0087]
Next, the inter-prediction process of step ST32 in
Fig. 8 will be described with reference to the flowchart
of Fig. 10.
[0088]
10 In step ST51, the motion prediction/compens~tion
unit 32 performs the motion prediction process. The
motion grediction/compensation unit 32 detects the motion
vector by performing the motion prediction in each
& prediction mode, and then the process proceeds to step
15 ST52.
[0089]
In step ST52, the motion prediction/compensation
unit 32 performs the motion compensation process. The
motion prediction/compensation unit 32 generates the
I
20 predicted image data'by performing the motion
compensation on the r-e ference image data based on the
motion vector detected in step ST51.
[0090]
Fig. 11 is a flowchart illustrating the motion
25 compensation process. In step ST61, the motion
prediction/compensation unit 32 reads the reference image
data. The compensation control unit 3231 of the motion
prediction/compensation unit 32 determines a reading area
based on the size of the prediction unit performing the
30 motion compensation, the motion vector detected for the
prediction unit performing the motion compensation, and
the reference index indicating the reference image used
to detect the motion vector. Further, - the compensation
control unit 3231 reads the image data of the determined
reading area from the frame memory 25, and then the
process proceeds to step ST62.
[00911
In step ST62, the motion prediction/compensation
C
unit 32 generates the ALF process determination result.
The compensation control unit 3231 of the motion
prediction/compensation unit 32 generates the ALF process
determination result by determining whether the reference
image data is subjected to the adaptive loop filter
process based on the flag information generated by the
ALF unit 242, and then the process proceeds to step ST63.
[0092]
In step ST63, the motion prediction/compensation
unit 32 determines whether the adaptive loop filter
process is performed. When the reference image data re&d
in step ST61 is subjected to the adaptive loop filter
process, the motion prediction/compensation unit 32
causes the process to proceed to step ST64. When the
reference image data is not subjected to the adaptive
loop filter process, the motion prediction/compensation
unit 32 causes the process to proceed to step ST65.
[0093]
In step ST64, the motion prediction/compensation
unit 32 outputs filter - coef - with - ALF according to the
decimal part of the motion vector. When the ALF process
determination result indicates the application of the
filter, the coefficient table 3232 of the motion
prediction/compensation unit 32 selects
I I '
*
filter - coef - with - ALF according to the decimal part of the
motion vector from filter - coef - set - w - ALF and outputs
filter - coef - with - ALF to the filter unit 3233. Then, the
process proceeds to step ST.
5 [0094] -
In step ST65, the motion prediction/compensation
unit 32 outputs filter - coef - with ALF according to the
decimal part of the motion vector. When the ALE process
determination result does not indicate the application of
10 the filter, the coefficient table 3232 of the motion
prediction/compensation unit 32 selects
filter - coef - without - ALF a'ccording to the decimal part of
the motion vector from filter - coef - set - wo - ALE and outputs
filter - coef - without ALF to the filter unit 3233. Then,
15 the process proceeds to step ST66.
[00951
Fig. 12 is a diagram illustrating an example in
which the execution or non-execution of the adaptive loop
. .
filter process is switched in units of frames. In Fig.
20 12, frames LO (O), LO (I), and LO (3) are frames (ALF ON)
' for which the adaptive loop filter process is performed
' in all of the areas in the frame. The frame LO (2) is a
frame (ALE OFF) for which the adaptive loop filter
process is not performed in all of the areas in the frame.
For example, when the predicted image data is
"
generated using LO (0) as the reference frame,
filter - coef - without - ALF according to the decimal part of
the motion vector is selected from filter coef set - -s et- w o- A LF.
30 When the predicted image data is generated using LO (2)
as the reference frame, filter- c oef- w ithout- A LF ac- cording
to the decimal part of the motion vector is selected from
filter - coef - set - wo - ALF.
[00971
In step ST66, the motion prediction/compensation
5 unit 32 generates the predicted image data. The filter
unit 3233 of the motion prediction/compensation unit 32
generates the predicted image data by performing the
filter process on the reference image data using the
filter coefficient supplied from the coefficient tab19
10 3232 in step ST64 or step ST65.
[0098]
Thus, in the motion compensation process of step
ST52 in Fig:lO, the predicted image data is generated,
as described above, and then the process proceeds to step
*
15 ST53.
[0099]
In step ST53, the motion prediction/compensation
unit 32 calculates the cost function values. The motion
prediction/compensation unit 32 calculates the cost
20 function values using the input image data of the
I
prediction unit to be encoded, the predicted image data
generated in step ST52, and the like, as described above,
and then the process proceeds to step ST54.
[01001 . h
25 In step ST54, the motion prediction/compensation
unit 32 determines the optimum inter-prediction mode.
The motion prediction/compensation unit 32 determines the
reference index in which the cost function value
calculated in step ST53 is the minimum value, theblock
30 size of the coding unit, and the shape and size of the
prediction unit in the coding unit, and sets the optimum
L
inter-prediction mode. When the mode in which the cost
function is the minimum is determined, the cost function
value when the inter-prediction is performed in a skip
mode is also used.
5 [OlOl]
When the predicted image and optimum mode selection
unit 33 selects the optimum inter-prediction mode as the
optimum prediction mode, the motion
prediction/compensation unit 32 generates the predicted
10 image data so that the predicted image data of the
optimum inter-prediction mode can be supplied to the
subtraction unit 13 apd the addition unit 23.
Thus, in the image encodjng device 10, when the
15 reference image data used to g'enerate the predicted image
data is not subjected to the adaptive loop filter process,
the coefficient of the filter characteristics in which
the noise removal strength is increased is selected as
the filter coefficient used for the reference image data.
20 Therefore, since the predicted image data with small
noise can be generated, the deterioration in the quality
of the predicted image can be reduced, and thus
deterioration in the compression efficiency can be
suppressed. Further, when the reference image data used
25 to generate the predicted image data is subjected to the
. adaptive loop filter process, for example, the
coefficient with filter characteristics preserving
components from a low-frequency component to & highfrequency
component is selected as the filter coefficient
30 used for the reference image data. heref fore,: the
deterioration in the encoding efficiency can d e prevented
by causing the motion prediction/compensation unit 32 to
perform the filter process on the reference image data to
which the filter is applied in the adaptive loop filter
process.
8
[0103]
Even when the loop filter unit 24 adaptively
performs the deblocking filter process, the deterioration
in the encoding efficiency can be suppressed by switching
the filter characteristics depending on whether the
reference image data is likewise subjected to the
deblocking filter process.
[01041
<3. Configuration of Image Decoding device>
Next, a case in which the image processing device
is applied to an image decoding device will be described.
The image compression information generated by encoding
the input image is supplied to the image decoding device
via a predetermined transmission path, a recording medium,
or the like and is decoded.
Fig. 13 is a diagram illustrating the configuration
pf the image decoding device. An image decoding device
50 includes an accumulation buffer 51, a lossless
decoding unit 52, an inverse quantization unit 53, an
.c
inverse orthogonal transform unit 54, an addition unit 55,
a loop filter unit 56, a screen rearrangement buffer 57,
a digital/analog conversion unit (D/A conversion unit) 58.
The image decoding device 50 further includes a frame
memory 61, selectors 62 and 73, an intra-prediction unit
71, a motion compensation unit 72. The loop filter unit
56 includes, for example, a deblockihg filter unit 561
and an ALF (Adaptive Loop Filter) unit 562.
[0106]
The accumulation buffer 51 accumulates the
transmitted image compression information. The lossless
5 decoding unit 52 decodes the image compression
information supplied from the accumulation buffer 51
according to a scheme corresponding to the encoding
scheme of the losslegs encoding unit 16 in Fig. 3.
[0107]
The lossless decoding unit 52 outputs the
prediction mode information obtained by decoding the
image compression information to the intra-prediction
unit 71 or the motion compensation unit 72. Further, the
lossless decoding unit 52 outputs the differential motion
vector obtained by decoding the image compression
information or the loop filter information obtained from
the image compression information to - the motion
compensation unit 72.
[0108]
20 The inverse quantization unit 53 performs inverse
quantization on the quantization data decoded by the
lossless decoding unit 52 according to a scheme ..
corresponding to the quantization scheme of the
quantization unit 15 in Fig. 3. The inverse orthogonal
25 transform unit 54 performs inverse orthogonal transform
on the output of the inverse quantization unit 53
according to a scheme corresponding to the orthogonal
transform scheme of the orthogonal transform unit 14 in
Fig. 3 and outputs data to the addition unit 55.
30 [0109]
The addition unit 55 adds the data obtained after
the inverse orthogonal transform to the predicted image
data supplied from the selector 73 to generate decoded
image data and outputs the decoded image"data to the
deblocking filter unit 561 and the intra-prediction unit
The loop filter unit 56 performs the same filter
process as that of the loop filter unit 24 of the image
encoding device 10 and stores the reference image data
from which noise is removed in the frame memory 61.
[Olll]
The deblocking filter unit 561 of the loop filter
unit 56 performs a deblocking filter process on the
decoded image data supplied from the addition unit 55 to
remove block distortion and outputs the decoded image
data to the ALF unit 562.
[0112]
The ALF unit 562 performs an adaptive loop filter
process based on the filter information supplied from the
lossless decoding unit 52, supplies the processed-decoded
image data to the frame memory 61 to accumulate the
decoded image data, and outputs the decoded image data to
the screen rearrangement buffer 57.
[0113]
The screen rearrangement buffer 57 sorts images.
That is, the order of the frames sorted in the encoding
order by the screen rearrangement buffer 12 in Fig. 3 is
sorted to the original display order, and the original
display order is output to the D/A conversion unit 58.
[0114]
TJle D/A conversion unit 58 performs the D/A
-
conversion on the image data supplied from the screen
rearrangement buffer 57 and outputs the image data to a
display (not illustra-ted) to display the images.
[0115]
The frame memory 61 stores the decoded image data
obtained after the filter process by the loop filter unit
24 as reference image data.
[0116]
The intra-prediction unit 71 generates the
predicted image data based on the prediction mode
information sypplied from the lossless decoding unit 52
and the decoded image data supplied from the addition
unit 55 and outputs the tenerated predicted image data to
the selector 73.
[0117]
The motion compensation unit 72 reads the reference
image data from the frame memory 61 based on the
prediction mode information or the differential m6tion
vector supplied from the lossless decoding unit 52,
generates the predicted image data, and outputs the
generated predicted image data to the selector 73.
[0118]
The motion compensation unit 72 reads the reference
image data from the frame memory 61 based on the
prediction mode information and the differential motion
vector supplied from the lossless decoding unit 52 and
performs motion compensation to generate the predicted
image data. The motion compensation unit 72 outputs the
ge,nerated predicted image data to the selector - 73. The
motion$compensation unit 72 switches the filter
characteristics based on the loop filter information and
generates the predict2d image data.
[0119]
Based on the prediction mode information supplied
from the lossless decoding unit 52, the selector 73
5 selects the intra-prediction unit 71 in a case of the
intra-prediction and selects the motion compensation unit
72 in a case of the inter-prediction. The selector 73
outputs the predicted image data generated by the
selected intra-prediction unit 71 or the selected motion
10 compensation unit 72 to the addition unit 55. -
[0120]
Fig. 14 is a diagram illustrating the configuration
of the motion compensation unit 72. The motion
compensation unit 72 includes a motion vector
15 synthesizing unit 721, a motion compensation processing
*
unit 722, and a motion vector buffer 723. Fig. 14
illustrates a case in which the filter coefficient when
the adaptive loop filter process is not performed in the
reference image data is switched to a coefficient for
20 which noise removal strength is enhanced, compared to the
filter coefficient when the adaptive loop filter prodess
is performed.
[0121]
The motion vector synthesizing unit 721 adds the
25 differential motion vector and the prediction motion
vector of the decoding target prediction unit supplied
from the lossless dec-oding unit 52 to calculate a motion
vector of the prediction unit and outputs the motion
vector to the motion compensation processing unit 722.
30 The motion vector synthesizing unit 721 generates a
prediction motion vector by pkrforming median predicti'on i
or the like, for example, using the motion vector of
adjacent prediction units stored in the motion vector
buffer 723.
[ 0122 I
5 The motion compensation processing unit 722
includes a compensation control unit 7221, a coefficient
table 7222, and a filter unit 7223. The compensation
control unit 7221 reads - the reference image data from the
frame memory 61 based on the prediction mode information
10 supplied from the lossless decoding unit 52 and the
motion vector supplied from the motion vector
synthesizing unit 721. The compensation control unit
7221 determines whether the read reference image Gata is
subjected to the adaptive loop filter process based on
15 the ALE flag information supplied from the lossless
decoding unit 52 and outputs an ALE process determination
result to the coefficient table 7222. For example, when
the compensation control unit 7221 determines that the
reference image data is subjected to the adaptive loop
20 filter process based on the ALE flag information, the
compensation control unit 7221 sets alf - applied - flag as,
the ALE process determination result to be true.
Conversely, when the compensation control unit 7221
determines that the reference image data is not subjected
25 to the adaptive loop filter process, the compensation
control unit 7221 sets alf - applied - flag to be false. The
compensation control unit 7221 outputs alf - applied - flag
indicating the ALF process determination result to the
coefficient table 7222.
30 [0123]
The coefficient table 7222 is supplied with a
decimal part of the motion vector and the ALE process
determination resuLt generated by the compensation
control unit 7221. As in the coefficient table 3232 of
the image encoding device 10, the coefficient table 7222
stores a filter coefficient set
("filter- c oef- s et- w o- A LE:') used foq the reference image
data not subjected to the adaptive loop filter process
and a fllter coefficient set ("filter- c oef- s et- w - A LE")
usdd fdr the reference image data subjected to the
adaptive loop filter process.
[0124] i -
The coefficient table '7222 outputs the filter
coefficient selected according to the decimal part of the
motion vector and the ALE process determination result to
the filter unit 7223. For example, when alf - applied - flag
is true, the coefficient table 7222 selects the filter
coefficient according to the decimal part of the motion
vector from filter- c oef- s et- w - A LE and outputs the filter
coefficient to the filter unit 7223. Further, when
alf - applied - flag is false, the coefficient table 7222
selects the filter coefficient according to the decimal
part of the motion vector from the filter- c oef- s et- w o- A LE
and outputs the filter coefficient to the filter unit
The filter unit 7223 performs an interpolation
filter process of obtaining given predicted image data
with the decimal pixel accuracy in the reference image
data of a current block by using the filter coefficient
supplied from the coefficient table 7222. When the
reference image data is not subjected to the adaptive
loop filter process, the filter unit 7223 performs the
filter process with the filter characteristics in which
the noise removal strength is enhanced, compared to the
filter coefficient used Tor the reference image data
5 subjected to the adaptive loop filter process. The
filter unit 7223 outputs the predicted image data
obtained by performing the filter process on the
reference image data to the selector 73 illustrated in
Fig. 13. When the selected filter coefficient is *
10 included in the image compression information, the filter I
coefficient included in the image compression information I
may be stored in the coefficient table 7222, and the
filter unit 7223 may perform the interpolation filter
process using the filter coefficient stored in the
15 coefficient table 7222 and output the predicted image
data to the selector 73.
[0126]
Based on the prediction mode information supplied
from the lossless decoding unit 52, the selector 73
,20 selects the intra-prediction unit 71 in the case .. of the
intra-prediction and selects the motion compensation unit
72 in the case of the inter-prediction. The selector 73
outputs the predicted image data generated by the
selected intra-prediction unit 71 or the selected motion
25 . compensation unit 72.to the addition unit 55.
[0127]
<4. Process of Image Decoding device>
Next, an image decoding process performed by the
image decoding device 50 will be described with reference
30 to the flowchart of Fig.-15.
[0128]
In step ST81, the accumulation buffer 51
*
accumulates the supplied image compression information.
In step ST82, the lossless decoding unit 52 performs a
lossless decoding process. The lossless decoding unit 52
decodes the image compression information supplied from
the accumulation buffer 51. That is, quantization- data
of each picture encoded by the lossless encoding unit 16
in Fig. 3 can be obtained. The lossless decoding unit 52
performs the lossless decoding on the prediction mode
information included in the image compression informatior?.
When the obtained prediction mode information is
information on the intra-prediction mode, the prediction
mode information is output to the intra-prediction unit
71. When the prediction mode information is information
on the inter-prediction mode, the lossless decoding unit
52 outputs the prediction mode information to the motion
compensation unit 72. The lossless decoding unit 52
outputs the differential motion vector obtained by
decoding the image compression information or the loop
filter information obtained from the image compression
information to the motion compensation unit 72.
[0129]
In step ST83, the inverse quantization unit 53
performs an inverse quantization process. The inverse
quantization unit 53 performs the inverse quantization on
the quantization data decoded by the lossless decoding
unit 52 with characteristics corresponding to the
cha-racteristics of the qiiantization unit 15 in Fig. 3.
[0130]
In step ST84, the inverse orthogonal transform unit
54 performs an inverse orthogonal transform process. The
inverse orthogonal transform unit 54 performs the inverse
orthogonal transform on the transform coefficient data
subjected to the inverse quantization by the inverse
quantization unit 53 with characteristics corresponding
to the characteristics of the orthogonal transform unit
14 in Fig. 3.
[0131]
In step ST85, the addition unit 55 generates the
decoded image data. The addition unit 55 generates the
decoded image data by adding the data obtained by
performing the inverse orthogonal transform process to
the predicted image data selected in step ST89 to be
described below. Thus, the original image is decoded.
[0132]
In step ST86, the deblocking filter unit 561
performs a filter pro'cess. The deblocking filter unit
561 removes block distortion contained in the decoded
image by performing a deblocking filter process on the
decoded image data output from the addition unit 55.
[0133]
In step ST87, the frame memory 61 performs a
process of storing the decoded image data. The decoded
image data stored in the frame memory 61 or the decoded
image data output from the addition unit 55 is used to
generate the predicted image data as the reference image -
data.
[0134]
In step ST88, the intra-prediction unit 71 and the
motion compensation unit 72 perform prediction processes.
The intra-prediction unit 71 and the motion compenSation
*
unit 72 each perform the prediction process according to
the prediction mode information supplied from the
lossless decoding unit 52. -
[0135]
That is, when the prediction mode information of
5 the intra-prediction is supplied from the lossless
decoding unit 52, the intra-prediction unit 71 generates
the predicted image data based on the prediction mode
information. Further, when the prediction mode
information of the inter-prediction is supplied from the
I 10 lossless decoding unit 52, the motion compensation unit
72 generates the predicted image data by performing the
motion compensation based on the prediction mode
information.
w
[0136]
15 In step ST89, the selector 73 selects the predicted
image data. The selector 73 selects the predicted image
supplied from the,intra-prediction unit 71 and the ,
I
predicted image data supplied from the motioh
compensation unit 72, supplie$ the selected predicted
20 image data to the addition unit 55, and adds the
I predicted image data to the output of the inverse ~ I
orthogonal transform unit 54 in step ST85, as described
above. -
[ b137 1
25 In step ST90, the screen rearrangement buffer 57
performs image sorting. That is, the screen
rearrangement buffer 57 sorts the order of the frames
sorted'for the encoding by the screen rearrangement
buffer 12 of the image encoding device 10 in Fig. 3 to
30 the original display ofder.
[0138]
In step ST91, the D/A conversion unit 58 performs
the D/A conversion on the image data from the screen
rearrangement buffer 57. The image is output to a
display (not illustrated) and the image is displayed.
[0139]
Next, the prediction process of step ST88 in Fig.
15 will be described with reference to the flowchart of
Fig. 16.
[0140]
In step ST101, the lossless decoding unit 52
determines whether th_e prediction unit to be decoded is
subjected to the intra-encoding. When the prediction
mode information obtained by performing the lossless
decoding is the prediction mode information of the intraprediction,
the lossless decoding unit 52 supplies the
prediction mode information to the intra-prediction unit
71, and then the process proceeds to step ST102. Further,
the prediction mode information is the prediction mode
information of the inter-prediction, the lossless
decoding unit 52 supplies the prediction mode information
to the motion compensation unit 72, and then the process
proceeds to step ST103.
[0141]
In step ST102, the intra-prediction unit 71
performs the intra-prediction process. The intraprediction
unit 71 generates the predicted image data by
performing the intra-prediction using the decoded image
!
data supplied from the addition unit 55 and not subjected
to the deblocking filter process and the prediction mode
30 information,
[0142]
In step ST103, the motion compensation unit 72
performs the inter-prediction process. The motion
compensation unit 72 reads the reference image data from
the frame memory 61 and generates the predicted image
data based on information such as the prediction mode
information supplied from the lossless decoding unit 52.
[0143]
Fig. 17 is a flowchart illustrating the interprediction
process of-step ST103. In step ST111, the
motion compensati~n unit 72 acquires the prediction mode
information and the ALF flag information. The motion
4
compensation unit 72 acquires the prediction mode
information and the ALE flag information from the
lossless decoding unit 52, and then the process proceeds
to step ST112. The ALF unit 562 performs the filter
process using the ALE flag information acquired from the
lossless decoding unit 52 and the filter coefficient.
[0144]
In step ST112, the-motion compensation unit 72
restructures the motion vector. The motion compensation
unit 72 restructures the motion vector of the prediction
unit, for example, by adding a prediction motion vector
generated through median prediction or the like using the
motion vectors of the adjacent prediction units t: the
differential motion vector supplied from the lossless
decoding unit 52, and then the process proceeds to step
In step ST113, the motion compensation unit 72
performs the motion compensation process. The motion
compensation unit 72 reads the reference image data fiom
the frame memory 61 based on the prediction mode
information acquired in step ST111 or the motion vector
restructured in step ST112. The motion compensation unit
72 generates the predicted image data by performing the
5 filter process on the read reference image data based on
the motion vector or the ALF flag information, as in the
motion compensation p>ocess illustrated in Fig. 11.
Thus, as in the image encoding device 10, in the
10 image decoding device 50, the coefficient with the filter
characteristics in which,the noise removal strength is
enhanced is selected as the filter coefficient used for
the reference image data, when the reference image data
used to generate the predicted image data is not
15 subjected to the adaptive loop filter process in the
inter-prediction. Further, when the reference - image data
used to generate the predicted image data is subjected to
the adaptive loop filter process, for example, the
coefficient with the filter characteristics preserving
20 components from a low-frequency component to a highfrequency
component is selected as.the filter coefficient
*
used for the reference image data. Therefore, even when
the image compression information is generated using the
filter coefficient according to the adaptive loop filter
25 process by the image encoding device 10, the image
decoding device 50 can perform the decoding process
corresponding to the encoding process of the image
encoding device 10. When the image compression
information is generated using the filter coefficient
30 according to the deblocking filter process by the image
encoding device 10, the image decoding device 50 can
perform the decoding process corresponding to the
encoding process of the image encoding device'l0 by
transforming the filter characteristics depending on
whether the reference image data is subjected - to the
5 deblocking filter process by the motion compensation unit
<5. Another Configuration and Process of Image Encoding
10 Fig. 18 is a diagram illustrating another
configuration when the image processing device is applied
to an image encoding device. In another configuration of
the image encoding device, a loop filter unit 24 includes
an SAO (Sample Adaptive pffset) 245 between a deblocking
15 filter unit 241 and an ALE (Adaptive Loop Filter) unit
242. The SAO corresponds to the above-described PQAO
(Picture Quality Adaptive Offset).
[0148]
As the kinds of offset, there are two kinds-of
20 offset called band~offset and six kinds of offset called
edge offset. Further, the offset may not be adapted. An
image can be partitioned in a quad-tree and execution of
encoding can be selkcted in each area according to one of
the above-described kinds of offset.
25 [0149]
This selectioh information can be encoded by the
lossless encoding unit 16 and can be included in a bit
stream. By using this method, the encoding efficiency
can be improved.
30 [0150]
Hereinafter, a ?pad-,tree structure will be
described with reference to Fig. 19. For example, in the
image encoding device,10, a cost function value JO of
Level-0 (partition depth 0) indicating that an area 0 is
not partitioned is calculated, as illustrated in (A) of
Fig. 19. Further, cost function values J1, J2, J3, and
54 of Level-1 (partition depth 0) indicating that the
area 0 is partitioned into four areas, areas 1 to 4, are
calculated.
[0151]
As illustrated .in (B) of Fig. 19, the cost function
values are compared to each other and the partitioned
areas (Partitions) of Level-1 in which the cost function
value is small are selected by "JO > (J1 + J2 + J3 +
J4) ."
[0152]
Likewise, as illustrated in (C) of Fig. 19, cost
function values 55 to J20 of Level-2 (partition depth 2)
indicating that the area 0 is partitioned sixteen areas,
areas 5 to 20, are calculated.
[0153]
As illustrated in (D) of Fig. 19, the cost function
values are compared to each other and the partitioned
areas (Partitions) of Level-1 are selected in the area 1
by "J1 < (J5 + J6 + J9 + J10) . I 1 In the area 2, the
partitioned areas (Partitions) of Level-2 are selected by
"J2 > (J7 + J8 + Jll + J12) ." In the area 3, the
partitioned areas (Partitions) of Level-2 are selected by
"J3 > (J13 + J14 + J17 + J18) ." In the area - 4, the
partitioned areas (Partitions) of Level-1 are selected by
As a result, the final Quad-tree areas (Partitions)
are determined in the Quad-tree structure, as illustrated
in (D) of Fig. 19. In each of the determined areas of
the Quad-tree structure, the cost function values are
calculated for all of the two kinds of band offset, the
six kinds of edge offset, and non-offset and the offset
by which the encoding is performed is determined.
[0155]
For example, in the area 1, EO (4), that is, the
fourth kind of edge offset is determined among the edge
offsets, as illustrated in (E) of Fig. 19. In the area 7,
OFF, that is, the non-offset is determined. In the area
8, EO (2), that is, the second kind of edge offset is
determined among the edge offsets. In the areas 11 and
12, OFF, that is, the non-offset is determined.
-
[0156]
In the area 13, BO (I), that is, the first kind of
band offset is determined among the band offsets. In the
area 14, EO (2), that is, the second kinds of edge offset
is determined among the edge offsets. In the area 17, BO
(2) ,, that is, the second kind of band offset is
determined among the band offsets. In the area 18, BO
(I), that is, the first kind of band. offset is determined
among the band offsets. In the area 4, EO (I), that is,
the first kind of edge offset is determined amopg the
edge offsets. -
[0157]
Next, the baud offsets will be described in detail
with reference to Fig. 20. In regard to the band offsets,
in the example of Fig. 20, 1 scale indicates "1 band = 8
pixels," the luminance pixel values can be divided into
32 bands, and each band independently has an offset value.
That is, in the example of Fig. 20, among 0 to 255 pixels
(32 bands), 16 middle bands can be classified to a first
group and each 8 bands on both sides can be classified to
5 a second group.
[0158]
The offsets of only one of the first and second
a
groups are encoded and transmitted to the decoding side.
In general, black and white is distinct or subtle shade
10 is realized in one area in many cases, and thus a case in
which pixels are present in all of the first and second
groups is rare. Therefore, by sending only one side of\ *
the offset, in each Quad-tree area, an amount of encoding
is prevented from increasing due to the transmission of
15 the pixel value of a value which is not included.
[0159]
When input signals are broadcast, luminance signals
are restricted within bhe range of 16,235 and colordifference
signals are restricted within the range of
20 16,240. At this time, broadcast legal illustrated in the
lower part of Fig. 20 is applied, and thus offset values
of each two bands indicated by x marks on both sides are
not transmitted.
[0160] .
25 Next, the edge offsets will be described in detail
with reference to Fig. 21. In the edge offsets, a
corresponding pixel value is compared to neighboring
pixel values neighboring on this pixel value and an
offset value is transmitted for a category corresponding
30 thereto.
I patterns illustrated in (A) to (D) of Fig. 21 and 2 twodimensional
patterns illustrated in (E) and (F) of Fig.
21, each offset is transmitted with categories
5 illustrated in. Fig. 22. -
[0162]
In (A) of Fig. 21, neighboring pixels are disposed
one-dimensionally on the right and left sides of a pixel
C. That is, a 1-D 0-degree pattern forming 0 degrees
with respect to the pattern of (A) of Fig. 21 is -
illustrated. In (B) of Fig. 21, neighboring pixels are
disposed one-dimensionally on the upper and lower sides
of the pixel C. That is, a 1-D 90-degree pattern forming
90 degrees with respect to the pattern of (A) of Fig. 21
is illustrated.
[0163]
In (C) of Fig. 21, neighboring pixels are disposed
one-dimensionally on the upper left and lower right sides
of the pixel C. That is, a 1-D 135-degree pattern
20 forming 135 degrees with respect to the pattern of (A) of
Fig. 21 is illustrated. In (D) of Fig. 21, neighboring
pixels are disposed oie-dimensionally on the upper right
and lower left sides of the pixel C. That is, a I-D 135-
degree pattern forming 45 degrees with respect to the
25 pattern of (A) of Fig. 21 is illustrated.
[0164]
In (E) of Fig. 21, neighboring pixels are disposed
two-dimendionally on the upper, lower, right, and left
sides of the pixel C. That is, a 2-D cross pattern in
30 which the ,pixel C is crossed is illustrated. In (F) of
Fig. 21, neighboring pixels are - disposed twodimensionally
on the upper right, lower left, upper left,
and lower right sides of the pixel C. That is, a 2-D
diagonal pattern in which the pixel C is diagonally
crossed is illustrated.
[0165]
(A) of Fig. 22 illustrates a rule list
(Classification rule for 1-D patterns) of the onedimensional
patterns. The patterns in (A) to (D) 'bf Fig.
21 are classified to five kinds of category illustrated
in (A) of Fig. 22, the offsets are calculated by the
category, and the offsets are transmitted to the decoding
unit.
[0166]
When the pixel value of the pixel C is less than
the pixel values of two neighboring pixels, the pattern
is classified to category 1. When the pixel value of the
pixel C is less than the pixel value of one neighboring
pixel and is equal to the pixel value of the other
neighboring pixel, the pattern is classified . to category
2. When the pixel value of the pixel C is greater than
the pixel value of one neighboring pixel and is equal to
the pixel value of the other neighboring pixel, the
pattern is classified to category 3. When the pixel
value of the pixel C is greater than the pixel values of
two neighboring pixels, the pattern is classified to
category 4. In none of the above, the pattern is
classified to'category 0.
(B) of Fig. 22 illustrates a rule list
(Classification rule for 2-D patterns) of the twodimensional
patterns. The patterns in (E) and (F) of Fig..
-
21 are classified to seven kinds of category illustrated
in (B) of Fig. 22 and the offsets are transmitted to the
decoding unit by the category.
5 When the pixel value of the pixel C is less than
*
the pixel values of the four neighboring pixels, the
pattern is classified to category 1. When the pixel
value of the pixel C is less than the pixel values of
three neighboring pixels and is equal to the pixel value
10 of the fourth neighboring pixel, the pattern is i
classified.to category 2. When the pixel value of t-he
pixel C is less than the pixel values of three
neighboring pixels and is greater than the pixel value of
the fourth neighboring pixel, the pattern is cla,ssified
15 to category 3.
[0169]
When the pixel yalue of the pixel C is greater than
the pixel values of three neighboring pixels and is less
than the pixel value of the fourth neighboring pixel, the
20 pattern is classified to category 4. When the pixel
value of the pixel C is greater than the pixel values of
three neighboring pixels and is equal to the pixel value
of the fourth neighboring pixel, the pattern is
classified to category 5. When the pixel value of the
25 pixel C is greater than the values of the four
neighboring pixels, the pgttern is classified to category
6. In none of the above; the pattern is classified to
category 0.
30 As described above, in the edge offset, the amount
uf calculation is decreased for the one-dimensional t
i
+ SP312351WOOO
+
pattern, since comparison only with two neighboring
pixels may be performed. In a high efficiency encoding
condition, the value of 1-bit offset is transmitted to
the decoding side with higher accuracy, compared to the
low delay encoding condition.
[01711
As described above, the offset process is
adaptively performed. The offset process may be
performed only on the luminance signal or may be
performed on the luminance signal and the colordifference
signal.
The motion prediction/compensation unit 32
generates the predicted image data by switching the
filter coefficient when the SAO unit 245 does not perform
the offset process to the coefficient in which the noise
removal strength is enhanced, compared to the filter
coefficient when the offset process is performed.
[0173]
In this case, the coefficient table 3232 is
supplied with the decimal part of the motion vector, the
ALE process determination result generated by the
compensation control unit 2231, and the SAO information
on the offset process from the SAO unit 245. When the
offset process is not pe;formed by the SAO. unit 245, the
coefficient table 3232 has the filter coefficient
(filter- c oef- w ithout- A LF) use? for the reference image
data not subjected to the adagtive loop filter process
l
and the filter coefficient (filter- c oef- w ith- A LF) used
for the reference image data subjected to the adaptive
loop filter process. When the offset process is
performed by the SAO unit 245, the coefficient table 3232
has a filter coefficient (filter- c oef-without -A LE-wAO)
used for the reference image data not subjected to the
adaptive loop filter process and a filter coefficient
(filter- c oef- w ith- A LF- w AO) used for the reference image
data subjected to the adaptive loop filter process. The
coefficient table 3232 stores a filter coefficient set,
when the offset process is not performed by the SAO unit
245, and a filter coefficient set, when the offset
process is performed.
[0174]
The filter coefficient when the offset process is
not performed by the SAO unit 245 is set to the filter
coefficient in which the noise removal strength is
enhanced more than the filter coefficient when the offset
process is performed by the SAO unit 245.
[0175]
The filter unit 3233 performs an interpolation
filter process of obtaining given predicted image data
with the decimal pixel accuracy in the reference image
data of a current block using the filter coefficient
selected based on the de-c imal part of the motion vector,
the ALF process determination result, and the SAO
information in the coefficient table 3232. When the mode
determination unit 322 calculates the cost function value
to determine the optimum inter-prediction mode, the
filter unit 3233 outputs the generated predicted image *
data to the mode determination unit 322. The filter unit
3233 outputs the predicted image data generated in the
30 qptimum inter-prediction mode to the predicted image and
optimum mode selection unit 33.
I Fig. 23 is a flowchart illustrating the motion
compensation process. In step ST121, the motion
1 prediction/compensation unit 32 reads the reference image
data. The compensation control unit 3231 of the motion
prediction/compensation unit 32 determines a reading area
based on the size of the prediction unit performing the
motion compensation, the motion vector detected for the
prediction unit performing the motion compensation, and
the reference index indicating the reference image used
to detect the motion vector. Further, the compensation
control unit 3231 reads the image data of the determined
reading area from the frame memory 25, and then the
process proceeds to step ST122.
[0177]
In step ST122, the motion prediction/compensation
unit 33 generates the ALF process determination result
i
and the SAO information. The compensation control unit
3231 03 the motion prediction/compensation unit 32
generaees the ALF process determination result by
I
determining whether the reference image data is subjected
to the adaptive loop filter process based on the flag
information generated by the ALF unit 242. The
compensation control uhit 3231 acquires the SAO
information from the SAO unit 245, and then the process
proceeds to step ST123.
[0178]
In step ST123, the motion prediction/compensation
unit 32 determines whether the offset process is
performed. When the motion prediction/compensation unit
32 determines that the offset process is perpormed based
on the SAO information, the process proceeds to step
ST124. When it is determined that the offset process is
performed. the procdss proceeds to step ST127.
[0179]
In step ST124, the motion prediction/compensation
unit 32 determines whether the adaptive loop filter
process is petformed. When the adaptive loop filter
process is performed Dn the reference image data read in
step ST121, the motion prediction/compensation unit -32
cause the process to proceed to step ST125. When the I
adaptive loop filter process is not performed, the
process proceeds to step ST126.
[01801
In step ST125, the motion prediction/compensation
unit 32 outputs filter - coef - with - ALF according to the
decimal part of the motion vector. When the ALE process
determination result indicates application of the filter,
the coefficient table 3232 of the motion
prediction/compensation unit 32 selects
filter - coef - with - ALE according to the decimal part of the
motion vector from filter coef set w ALE - - - -A LE and outputs
filter - coef - with - ALE to the filter unit 3233. Then the
process proceeds to step ST130. -
[0181]
In step ST126, the motion prediction/compensation
unit 32 outputs filter- c oef- w ith- A bE according to the
decimal part of the motion vector. When the ALE process
determination result does not indicate the application of
the filter, the coefficient table 3232 of the motion
prediction/compensation unit 32 selects
filter coef - without - ALE according to the decimal part of
65
the motion vector from filter - coef - set - wo - ALE and outputs
filter - coef - without - ALE to the filter unit 3233. Then,
the process proceeds to step ST130.
[0182]
In step ST127, the motion prediction/compensation
unit 32 determines wh'ether the adaptive loop filter
process is performed. When the reference image data read
in step ST121 is subjected to the adaptive loop filter
process, the motion prediction/compensation unit 32
causes the process to proceed to step ST128. When the
adaptive loop filter process is not performed, the
process proceeds to step ST129.
I
In step ST128, the motion prediction/compensation
unit 32 outputs filter - coef - with - ALE - wAO according - to the
decimal part of the motion vector. When the ALE process
determination result indicates the application of the
filter, the coefficient table 3232 of the motion
prediction/compensation unit 32 selects
filter - coef - with - ALE - wAO according to the decimal part of
*
the motion vector from the filter coefficient set qnd
outputs filter- c oef- w ith- A LE- w AO to the filter unit 3233.
Then, the process proceeds to step ST130.
[0184]
In step ST129, the motion prediction/compensation
unit 32 outputs filter- c oef- w ith- A LE- w AO according to the
decimal part of the motion vector. When the ALE process
determination result does not indicate the application of
the filter, the coefficient table 3232 of the motion
prediction/compensation unit 32 selects
filter- c oef- w ithout- A LE- w AO according to the decimal part
of the motion vector from the filter coefficient set and
outputs filter - coef - without - ALF - wAO to the filter unit
3233. Then, the process proceeds to step-ST130.
-
[0185]
5 Further, filter - coef - with - ALF is a coefficient in
which the noise removal strength is enhanced more than
that of filter - coef - with - ALF - wAO, and
filter - coef - without - ALE is a coefficient in which the
noise removal strength is enhanced more than that of
10 filter - coef - without - ALF - wAO.
[01861
In step ST130, the motion prediction/compensation
unit 32 generates the prsdicted image data. The filter
unit 3233 of the motion prediction/compensation unit 32
15 generates the predicted image data by performing the
filter process on the reference image data using the
filter coefficients supplied from the coefficient table
3232 through the processes from step ST123 to step_ ST129.
[0187]
20 Even when the SAO unit 245 is provided, the filter
coefficient is switched depending on whether the offset
process is performed by performing the above-described
processes. When the offset process is not performed, the
filter coefficient is considered as the filter
25 coefficient in which the noise removal effect is high,
and thus the deterioration in the encoding efficiency can
be suppressed.
[0188]
<6. Another Configuration and Process of Image Decoding
30 device> -
i Next, another configuration and process in a case
in which the image pr_ocessing device is applied to the
image decoding device will be described. Fig. 24 is a
diagram illustrating the other configuration of the image
decoding device. In the other configuration of the image
5 decoding device, the loop filter unit 56 includes an SAO
unit 565 between the deblocking filter unit 561 and the
ALE (Adaptive Loop Filter) unit 562. The SAO unit 565
performs the same process as that of the SAO unit 245 of
the image encoding device 10.
10 [0189]
The motion compensation unit 72 reads the reference
image data from the frame memory 61 based on the
prediction mode information and the differential motion
vector supplied from the lossless decoding unit 52 and
15 performs motion compensation to generate the predicted
image data. The motion compensation unit 72 outputs the
generated predicted image data to the selector 73. The
motion compensation unit 72 switches the filter
characteristics based on information on the filter
20 process of the loop filter unit and generates the
predicted image data.
[0190]
The motion compensation unit 72 generates the
predicted image data by switching the filter coefficient
<
25 when the SAO unit 565 does not perform the offset process
to the coefficient in which the noise reinoval strength is
enhanced, compared to the filter coefficient when the
*
offset process is performed.
[0191]
30 In this case, the coefficient table 7222 is
supplied with the decimal part of the motion vector, the
ALE process determination result generated by the
compensation control unit 7221, and the SAO information
indicating whether the filter process is, performed from
the SAO unit 565. The coefficient table 7222 has the
5 filter coefficient (filter- c oef- w ithout- A LE) used for the
reference image data not subjected to the adaptive loop
filter process, the filter coefficient
(filter- c oef- w ith-ALE) used for the reference image data
subjected to the adaptiv; loop filter process. The
10 coefficient table 7222 further has the filter coefficient
(filter- c oef- w ithout- A LE- w AO) used for the reference
image data not subjected to the adaptive loop filter
process and the filter coefficient
(filter- c oef- w ith- A LE- w AO) used for the referencewirnage
15 data subjected to the adaptive loop filter process, when
the SAO unit 245 performs the offset process.
[0192]
The filter unit 7223 performs an interpolation
filter process of obtaining given predicted image data
with the decimal pixel accuracy in the reference image
data of a current block using the filter coefficient
selected based on the decimal part of the motion vector,
the ALE process determination result, and the SAO
information in the coefficient table 7222. When the
25 reference image data -i s not subjected to the offset
'process, the filter unit 7223 performs the filter process
with the filter characteristics in which the noise
removal strength is enhanced, compared to the filter
coefficient used for the reference image data subjected
30 to the offset process. The filter unit 7223 outputs the
predicted image data obtained by performing the filter
process on the reference image data to the selector 73.
The motion compensation unit 72 can generate the
predicted image data by performing the same processes as
those of the flowchart illustrated in Fig. 23. When the
selected filter coefficient is included in the image
c'ompression information, the filter - coefficient included
in the imabe compression information may be stored in the
coefficient table 7222, and the filter unit 7223 may
perform the interpolation filter process using the filter
coefficient stored in the coefficient table 7222 and
output the predicted image data to the selector 73...
[0193]
Based on the prediction mode information supplied
from the lossless decoding unit 52, the selector 73
selects the intra-prediction unit 71 in the case of the
intra-prediction and selects the motion compensation unit
72 in the case of the inter-prediction. The selector 73
outputs the predicted image data generated by the
selected intra-prediction unit 71 or the selected motion
compensation unit 72 to the addition unit 55.
[0194]
Thus, as ,in the image encoding device 10, in the
image decoding device- 50, the coefficient with the filter
characteristics in which the noise removal strength is
enhanced more than that of the filter coefficient used
when the offset process is performed is selected as the
filter coefficient used for the reference image data,
when the offset process is not performed. Therefore,
even when the image compression information is generated
using the filter coefficient according to the offset
process by the image encoding device 10, the image
decoding device 50 can perform the decoding process
corresponding to the encoding process of the image
encoding device 10.
[0195]
5 In the above-described embodiment, the ALF unit is
I , configured to be provided. However, when the ALF unit is
not provided, the filter coefficient may be set based on
the SAO information or the like without using the ALFi
process determination result. -
10 [0196]
7. Case of Software Processing
The series of processes described in this
I specification can be performed by hardware, software, or
I I a combination configuration of the hardware and software. I
15 When the processes are performed by software, a program
I
irecording the processing sequence is installed and
executed in a memory of a computer embedded in dedicated
k
,fi 1 I
hardware. Further, the program may be installed and
executed in a general-purpose computer capable of
20 performing various processes.
[01971 -
Fig. 25 is a diagram illustrating an example of the
configuration of a computer apparatus that performs the
series of processes described above by a program. A CPU
I 25 801 of a computer apparatus 80 executes various processes I
according to the program recorded on a ROM 802 or a
recording unit 808.
A RAM 803 appropriately stores the program executed
30 by the CPU 801, data, or the like. The CPU 801, - the ROM
802, and the RAM 803 are connected to each othe~ via a,
bus 804.
[0199]
An input/output interface 805 is also connected to
the CPU 801 via the bus 804. An input unit 806 such as a
5 touch panel, a keyboard, a mouse, or a microphone and an
output unit 807 configured by a display or the like are
connected to the input/output interface 805. The CPU 801
performs various processes in response to instructions
input from the input unit 806. Then, the CPU 801 outputs
10 the processed results to the output unit 807.
[0200]
The recording unit 808 connected to the
input/output interface 805 is configured by, for example,
a hard disk and records the program executed by the CPU
15 801 or various data. A communication unit 809
communicates with an external apparatus via a network
such as the Internet or a local area network or a wired
or wireless communication medium such - as digital
broadcast. The computer apparatus 80 may acquire a
20 program via the communication unit 809 and record the
program on the ROM,802 or the recording unit 808.
[0201]
When a removable medium 85 such as a magnetic disk,
an optical disc, a magneto-optical disc, or a
25 semiconductor memory is mounted, a drive 810 drives the
removable medium 85 and acquires the recorded program,
data, or the like. The acquired program or data is
transmitted to the ROM 8J2, the RAM 803, or the recording
unit 808, as necessary.
30 [0202]
The CPU 801 reads and executes the program
configured to perform the series of processes described
above, and then performs the encoding process on an image
signal recorded on the recording unit 808 or the
removable medium 85 or an image signal supplied via the
*
5 communication unit 809 or a decoding process on image
compression information.
[0203]
<8. Case of Application to Electronic Apparatus>
As described above, the H.264/AVC scheme has been
10 used as the encoding scheme/decoding scheme. However,
the present technology can be also applied to an image
encoding device/image decoding device that uses an
encoding scheme/decoding scheme to perform other motion
prediction and compensation processes.
Further, the prssent technology can be applied to
an image encoding device and an image decoding device
that is used when image information (bit stream)
compressed by orthogonal transform such as discrete
20 cosine transform and motion compensation, for example, as
in MPEG, H.26x1 or the like is received via a network
medium such as satellite broadcast, a cable TV
(television), the Internet, and a portable telephone or
is used when processing is performed on a storage medium
25 such as light, a magnetic disk, and a flash memory.
Next, electronic apparatuses to which the image
encoding device 10 or the image decoding device 50
described above is applied will be described.
30 [0206]
Fig. 26 is a diagram illustrati.ng an example of the
*
r SP312351WOOO
overall configuration of a television apparatus to which
the present technology is applied. A television ..
apparatus 90 includes an antenna 901, a tuner 902, a
demultiplexer 903, a decoder 904, a video signal
5 processing unit 905, a display unit 906, a sound signal
processing unit 907, a speaker 908, and an external
interface unit 909. The television apparatus 90 further
I includes a control unit 910, a user interface unit 911,
and the like.
10 [0207]
The tuner 902 selects a desired channel from a,
broadcast wave signal received by the antenna 901,
performs demodulation, and outputs an obtained stream to
the demultiplexer 903-.
15 [0208]
The demultiplexer 903 extracts packets of a video
I or an audio of a program to be viewed from the stream and I
I
outputs data of the extracted packets to the decoder 904.
The demultiplexer 903 outputs the packets of the data
20 such as EPG (Electronic Program Guide) to the control
unit 910. Further, when scrambling is performed, the
scrambling is cancelled by the demultiplexer or the like.
The decoder 904 pe;forms a process of decoding the
25 packets and outputs video data and audio data generated
through the decoding process to the video signal
processing unit 905 and the sound signal processing unit
907, respectively.
[0210]
*
30 The video signal processing unit 905 removes noise
.of the video data and performs video processing or the
like on the video data according user's setting. The
video signal processing unit 905 generates video data of
a program to be displayed on the display unit 906, or
image data or the like through a process based on
5 application supplied via a network. The video signal
processing unit 905 generates video data used to display
a menu screen or the like for selection or the like of an
item and superimposes the generated video data on the
video data of the program. The video signal processing
10 unit 905 generates a,driving signal based on the video
data generated in this way and drives the display unit
906.
[0211]
The display unit 906 drives a display device (for
15 example, a liquid crystal display element or the like)
based on the driving signal from the video signal
procesging unit 905 and displays the video or the like of
the program.
I
20 The sound signal processing unit 907 outputs an
audio by performing a predetermined process such - as noise
removal on the audio data, performing a D/A conversion
process or an amplification process on the pro'cessed
audio data, and supplying the audio data to the speaker
[ 02 13 ]
*
The external interface unit 909 is an interface
that is connected to an external apparatus or a network,
and transmits and receives data such as video data br
30 audio data.
[0214]
The user interface unit 911 is connected to the
control unit 910. The user interface unit 911 includes
an operation switch and a remote control signal receiving
unit and supplies an operation signal according to a
5 user's operation to the control unit 910.
[0215]
The control unit 910 is configured to include a CPU
(Central Processing Unit) or a memory. The memory stores
a program to be executed by the CPU, - various data
10 necessary for the CPU to perform a process, EPG data,
data acquired via the network, and the like. The program
stored in the memory is read and executed at a
predetermined timing such as an activation time of the
television apparatus 90 by the CPU. The CPU controls
15 each unit so that the television apparatus 90 can operate
according to a user's operation by executing the program.
[0216]
The television apparatus 90 further includes a bus
912 that connects the tuner 902, the demultiplexer 903,
20 the video signal processing unit 905, the sound signal
processing unit 907, the external interface unit 909, and
the like to the control unit 910.
[0217]
Ih the television apparatus having the above-
25 described configuration, the decoder 904 is provided with
a function of the image decoding device (image decoding
method) according to this specification. Therefore, when
the filter characteristics are switched according to the
loop filter process on the reference image data and the
30 predicted image data is generated in the image encoding
process on a broadcast station side, the predicted image
data can be generated by switching the filter
characteristics in the image decoding process of the
television apparatus, as in the broadcast station side.
Accordipgly, even when the image compression information
5 is generated on the broadcast station side so that the
deterioration in the quality of a predicted image can be
reduced and the deter-ioration in the compression
efficiency can be suppressed, the decoding process can be
correctly performed by the television apparatus.
Fig. 27 is a diagram illustrating an example of the
overall configuration of a portable telephone to which
the present technology is applied. A portable telephone
92 includes a communication unit 922, an audio codec 923,
15 a camera unit 926, an image processing unit 927, a
multiplexing/separating unit 928, a
recording/reproduction unit 929, a display unit 930, and
a control unit 931. These units are connected to each
other via a bus 933.
20 [0219]
An antenna 921 is connected to the communication
unit 922 and a speaker 924 and a microphone 925 are
connected to the audio codec 923. Further, an operation
unit 932 is connected to the control unit 931.
25 [0220]
The portable telephone 92 performs various
processes such as transmission and reception of a sound
signal, transmission and reception of an electronic mail
or image data, image photographing, and data recording in
30 various modes such as an audio calling mode or a data
communication mode.
[0221]
In the audio calling mode, a sound signal generated
by the microphone 925 is subjected to conversion to the
audio data or data compression by the audio codec 923, .
5 and is supplied to the communication unit 922. The
communication unit 922 generates a transmission signal by
performing a modulation process, a frequency conversion
process, or the like on the audio data. The
communication unit 922 supplies the transmission signal
10 to the antenna 921 to transmit the transmission signal to
a base station (not ill~stra~ed).T he communication unit
922 supplies audio data obtained by performing an
amplification process, a frequency conversion process, a
demodulation process, and the like on a received signal
-
15 received by the antenna 921 to the audio codec 923. The
audio codec 923 performs data decompression on the audio
data or conversion to an analog sound signal and outputs
the analog sound signal to the speaker 924.
[0222]
a'
20 When mail transmission is performed in the data
Communication mode, the control unit 931 receives
character data input through an operation of the
operation unit 932 and displays the input characters on
the display unit 930. The control unit 931 generates
25 mail data based on a user's instruction or the like in
the operation unit 932 and supplies the mail data to the
communication unit 922. The communication unit 922
transmits a transmission signal obtained by performing a
modulation process, a frequency conversion prpcess, or
30 the like on the mail data from the antenna 921. The
communication unit 922 restores the mail data by
performing an amplification process or a frequency
conversion process, a demodulation process, and the like
on a received signal received by the antenna 921. The
mail data is supplied to the display unit 930 to display
5 the mail contents.
[0223]
The portable telephone 92 can cause the ,
recording/reproduction unit 929 to store the received
mail data in a storage medium. The storage medium is any
10 rewritable storage medium. Examples of the storage
medium include a semiconductor memory such as a RAM or a
built-in flash memory and a removable medium such - as a
hard disk, a magnetic disk, a magneto-opti a1 disc, an
, F
optical disc, a USB memory, or a memory card.
15 [0224]
When the image data is transmitted in the data
communication mode, the image data generatled by the
C
camera unit 926 is supplied to the image processing unit
927. The image processing unit 927 generates image
20 compression information by performing an encoding process
' on the image data.
LO2251
The multiplexing/separating unit 928 multiplexes
the image compression information generated by the image
25 processing unit 927 and the audio data supplied from the
audio codec 923 according to a predetermined scheme and
supplies the multiplexed data to the cominunication unit
922. The communication unit 922 transmits a transmission
signal obtained by performing a modulation process, a
30 frequency conversion process, or the like on the
multiplexed data from the antenna 921. Thle communication
-
unit 922 restores the multiplexed data by performing an
amplification process, a frequency conversion process, a
demodulation process, and the like on a received signal
received by the antenna 921. The multiplexed data is
supplied to the multiplexing/separating unit 928. The
multiplexing/separating unit 928 separates the
multiplexed data and supplies the image compression
information and the audio data to the image processing
unit 927 and the audio codec 923, respectively.
The image processing unit 927 generates image data
by performing a decoding process on the image compression
information. The image data is supplied to the display
unit 930 to display the received image. The audio codec
923 converts the audio data into an analog sound signal
and supplies the analog sound signal to the speaker 924
to output the received audio.
[0227 I
In the portable telephone apparatus with the abovedescribed
configuration, the image processing unit 927 is
provided with the function of the image processing device
(image processing method) according to this specification;
Accordingly, for example, by switching the filter
characteristics according to the loop filter process on
the reference image data in the encoding process on an
image to be transmitted, the deterioration in the quality
of a predicted image %an be reduced and the deterioration
in the compression efficiency can be suppressed. Further,
since the predicted image data can be generated by
switching the filter characteristics in the decoding
process on a received image, as in the encoding process,
the decoding process can be correctly performed.
[0228]
Fig. 28 is a diagram illustrating an example of the
overall configuration of a recording/reproduction
apparatus to which the present technology is applied.
For example, a recording/reproduction apparatus 94
records audio data and video data of a received broadcast
program on a recording medium and supplies the recorded
data to a user at a timing according to a user's
I
instruction. For ebtample, the recording/reproduction
apparatus 94 can acquire audio data or video data from
another apparatus and record the audio data and the video
data on a recording medium. The recording/reprodu?tion
appardtus 94 is configured to display an image on a
15 monitor apparatus or the like or output an audio by
decoding and outputting audio data and video data
recorded on a recording medium.
[0229]
The recording/reproduction apparatus 94 includes a
20 tuner 941, an external interface unit 942, an encoder 943,
an HDD (Hard Disk Drive) unit 944, a disk drive 945, a
selector 946, a decoder 947, an OSD (On-Screen Display)
unit 948, a control unit 949, and a user interface unit
The tuner 941 selects a desired channel from a
broadcask signal received by an antenna (not illGstrated) .
The tuner 941 outputs image compression information
obtained by demodulating the received signal of the
30 desired channel to the selector 946.
, LO2311
The external interface unit 942 includes at least
one of an IEEE 1394 interface, a network'interface unit,
a USB interface, a flash memory interface, and the like.
The external interface unit 942 is an interface that is
connected to an external apparatus, a network, a memory
<
card, or the like and receives data such as video data or
audio data to be recorded.
[0232]
When the video data or the audio data supplied from
the external interface unit 942 is not encoded, the
s
encoder 943 performs an encoding process according to a
predetermined scheme and outputs image compression
information to the selector 946.
[0233]
The HDD unit 944 records contents data such as a
video or an audio, various programs, other data, or the
like on a built-in hard disk and reads the data from the
hard disk, for example, when the data is reproduced.
[0234]
The disk drive 945 records and reproduces a signal
on a mounted optical disc. Examples of the optical disc
include a DVD disk (DYD-Video, DVD-RAM, DVD-R, DVD-RW,
DVD+R, DVD+RW, or the like) and a Blu-ray disk.
[0235]
When a video or .an audio is recorded, the selector
946 selects one of the4 streams from the tuner 941 or the
encoder 943 and suppli+s the selected stream to one of
the HDD unit 944 and the disk drive 945. When a video or
an audio is reproduced, the selector 946 supplies a
stream output from the HDD unit 944 or the disk drive 945
to the decoder 947.
-
[0236]
The decoder 947 performs a decoding process on the
stream. The decoder 947-supplies video data generated by
performing the decoding process to the OSD unit 948. The
decoder 947 outputs audio data generated by performing
the decoding process.
[0237]
The OSD unit 948 generates the video data +
configured to display a menu screen or the like for
selection or the like of an item, superimposes the video
data on video data output from the decoder 947, and
outputs the superimposed video data.
[0238]
The user interface unit 950 is connected to the
1
control unit 949. The user interface unit 950 includes
an operation switch and a remote control signal receiving
unit and supplies an operation signal according to a
user's operation to the control unit 949.
[0239]
The control unit- 949 is configured to include a CPU,
a memory, or the like. The memory stores a program to be
executed by the CPU and various data necessary for the
CPU to perform a process. The program stored in the
memory is read and executed at a predetermined timing
such as an activation time of the recording/reproduction
apparatus 94 by the CPU. The CPU controls each unit so
that the recording/reproduction apparatus 94 can operate
according to a user's operation by executing the program.
[0240]
In the recording/reproduction apparatus having the
above-described configuration, the encoder 943 is
provided with the function of the image processing device
(image processing method) according to this specification,.
Accordingly, for example, by switching the filter
characteristics according to the loop filter process on
the reference image data in the encoding process when an
image is recorded, the deterioration in the qualipy of a
predicted image can be reduced, and thus the
deterioration in the compression efficiency can be
suppressed. Further, since the predicted image data can
be generated by switching the filter characteristics in
the decoding process on the recorded image, as in the
encoding process, the decoding process can be correctly
performed.
[0241]
Fig. 29 is a diagram illustrating an example of the
overall configuration of an imaging apparatus to which
the present technology is applied. An - imaging apparatus
96 images a subject, displays an image of the subject on
a display unit, and records the image as image data on a
recording medium.
[0242 I
The imaging apparatus 96 includes an optical block
961, an imaging unit 962, a camera signal processing unit
963, an image data processing unit 964, a display unit
965, an external interface unit 966, a memory unit 967, a
media drive 968, an OSD unit 969, and a control unit 970.
A user interface unit 97-1 is connected to the control
unit 970. The image data processing unit 964, the
external interface unit 966, the memory unit 967, the
media drive 968, the OSD unit 969, the control unit 970,
and the like are connected to each other via a bus 972.
-
The optical block 961 is configured using a focus
lens, a diaphragm mechanism or the like. The optical
block 961 forms an optical image of a subject on &I
imaging surface of the imaging unit 962. The imaging
unit 962 is configured using a CCD or CMOS image sensor,
generates an electric signal according to the optical
image through photoelectric conversion, and supplies the
electric signal to the camera signal processing unit 963.
[0244]
The camera signal processing unit 963 performs
various kinds of camera signal processing such as knee
correction, gamma correction, or color correction on the
electric signal supplied from the imaging unit 962. The
camera signal processing unit 963 supplies the image data
obtained after the camera signal processing to the image
data processing unit 964.
[0245]
The image data processing unit 964 performs an
encoding process on the image data ypplied from the
1
camera signal processing unit 963. the image data
processing unit 964 supplies image compression
information generated by performing the encoding process
i
to the external interface unit 966 P the media drive 968.
The image data processing unit 964 erforqs a decoding
process on the image compression infbrmation supplied
from the external interface unit 966 or the media drive
968. The image data processing unit 964 supplies the
image data generated by performing the decoding process
to the display unit 965. The image data processirrq unit
964 supplies the image data supplied from the camera
signal processing unit 963 to the display unit 965, or
4
superimpose display data acquired from the OSD unit 969
on the image data and supplies the superimposed data to
the display unit 965.
[0246]
The OSD unit 969 generates display data such as a
menu screen or an icon including signs, characters, or
graphics and outputs the display data to the image data
processing unit 964.
[0247]
For example, the external interface unit 966
includes an USB input/output terminal or the like and is
connected to a printex when an image is printed. A drive
is connected to the external interface unit 966, as
necessary, a removable medium such as a magnetic disk or
an optical disc is appropriately mounted, a program read
from the removable medium is installed, as necessary.
Further, the external interface unit 966 includes a
network interface connected to a predetermined network
such as a LAN or the Internet. For example, the control
unit 970 reads the image compression information from the
memory unit 967 in response to an instruction from the
user interface unit 971 and supplies the image
compression information from the external interface unit
966 to another apparatus connected via a network. The
control unit 970 can acquire the image compression
in'formation or the image data supplied from another .
apparatus via the network via the external interfgce unit
966 and can supply the image compression information or
the image data to the image data processing unit 964:
'.
[ 02 4 8 ]
As a recording medium'driven by ,the media drive 968,
for example, any removable medium, such as a magnetic
disk, a magneto-optical disc, an optical disc, or a
semiconductor memory, capable of performing reading and
writing is used. As the recording medium, any kind of
removable medium may be used, a tape device may be used,
a disk may be used, or a memory card may be used. Of
course, a contactless IC card or the like may be used.
[0249]
The media drive 968 and the recording medium may be
integrated so as to b% configured by, for example, a nonportable
storage medium such as a built-in hard disk
drive or an SSD (Solid State Drive).
[0250]
The control unit 970 may be configured using a CPU,
a memory, or the like. The memory stores a program to be
executed by the CPU and various data necessary for the
CPU to perform a process. The program stored in the !
memory is read and executed at a predetermined timing
such as an activation time of the imaging apparatus 96 by
=
the CPU. The CPU controls each unit so that the imaging
apparatus 96 can operate according to a user's operation
by executing the program.
[0251]
In the imaging apparatus having the above-described
configuration, the image data processing unit 964 is
provided with the function of the image processing device
(image processing method) according to this specification.
Accordingly, by switching the filter characteristics
according to the loop filter process on the reference
image data in the encoding process when a photographed
image is recorded on the memory unit 967, a recording
medium, or the like, the deterioration in the quality of
a predicted image can be reduced, and thus the
deterioration in the compression efficiency can be
5 suppressed. Further, since the predicted image data can
be generated by switching the filter characteristics in
the decoding process on the recorded image, as in the
encoding process, the decoding process can be correctly
. -
performed.
' The present technology should not be construed to
be limited to the above-described embodiment. The
embodiment discloses the present technology as an
exemplified form, and thus it is apparent to those
15 skilled in the art that the embodiment may be corrected
or substituted within the scope of the present technology
without departing from the gist of the present technology.
That is, to determine th_e gist of the present technology)
the claims should be considered.
The image processing device according to the
present technology may be configured as follows.
(1) An image processing device includes: a Loop
filter unit that executes a filter process on reference
25 image data referred to by a current block; an
interpolation filter unit that generates predicted image
data using the reference image data and a motion vector
corresponding to the current block; and a filter control
I
I unit that switches a filter coefficient of an
30 interpolation filter according to the filter process
performed by the loop filter unit on the reference image
data used by the interpolation filter.
(2) In the image processing device described in (I),
the filter control unit may switch the filter coefficient
when the filter process is not performed by the loop
filter unit to a coefficient in which noise removal
strength is enhanced more than that of the filter
coefficient when the filter process is performed.
(3) In the image processing device described in (1)
or (2), the interpolation filter unit may perform a lowfrequency
pass filter process as the noise removal.
(4) In the image processing device described in any
one of (1) to ( 3 ) , the filter process of the loop filter
unit may include at least one of a deblocking filter
process and an adaptive loop filter process.
(5) In the image processing device described in any
one of (1) to ( 4 ) , the loop filter unit may perform
adaptively performs an offset process. The filter
control unit may set the filter coefficient not subjected
to the offset process as a coefficient in which noise
removal strength is enhanced more than that of the filter
coefficient subjected to the offset pr6cess. -
I
n
INDUSTRIAL APPLICABILITY
[0254]
In an image processing device and an image
processing method according to the present technology, a
loop filter unit executes a filter process on reference
image data referred to by a current block. An
interpolation filter unit generates predicted image data
using the reference image data and a motion vector
corresponding to the current block. A filter control
unit switches a filter coefficient of an interpolation
filter according to the filter process performed by the
loop filter unit on the reference image data used by the
interpolation filter.- Therefore, when the filter process
is not performed by the loop filter unit, a predicted
image data in which noise is small can be generited, for
example, by causing the interpolation filter unit to
perform the filter process on the reference image data by
the filter characteristics in which noise removal
strength is erihanced. Thus, the deterioration in the
quality of a predicted image can be reduced, and thus the
deterioration in the compression efficiency can be
suppressed. Accordingly, the present technology is
suitable for the image encoding device, the image
I
decodinb device, or the like used when the image
compression information (bit stream) obtained by
performing encoding in units of blocks is transmitted and
received via a network medium such as satellite broadcast,
a cable TV, the Internet, and a portable telephon6 oh is
used when processing is performed on a storage medium
such hs light, a magnetic disk, and a flash memory.
REFERENCE SIGNS LIST
10 Image encoding device
11 A/D conversion unit
12, 57 Screen rearrangement buffer
13 Subtraction unit
14 Orthogonal transform unit
15 Quantization un-i t
16 Lossless quantization unit
51 Accumulation buffer
Rate control ucit
53 Inverse quantization unit
54 Inverse orthogonal transform unit
55 Addition unit
56 Loop filter unit
61 Frame memory
71 Intra-prediction unit
Motion prediction/compensation unit
Predicted image/optimum mode selection unit
Image decoding device
52 Lossless decoding unit
58 D/A conversion unit
62, 73 Selector
15 72 Motion compensation unit
80 Computer apparatus
9 0 Television apparatus
9 2 Portable telephone
9 4 Recording/reproduction apparatus
20 96 Imaging apparatus
241, 561 Deblocking filter unit
242, 562 ALF unit
245, 565 SAO unit
321 Motion detection unit
25 322 Mode determination unit
323, 722 Motion coppensation processing unit
324, 723 Motion vector~buffer
721 Motion vector synthesizing unit
3231, 7221 Compensation control unit
30 3231a ALF information memory
3232, 7222 Coefficient table
3233, 7223 F i l t e r u n i t

CLAIMS
1. An image processing device comprising:
a loop filter unit that executes a filter process
on reference image data referred to by a current block;
an interpolation filter unit that generates
predicted image data using the reference image data and a
motion vector corresponding to the current block; and
a filter control unit that switches a filter
coefficient of an interpolation filter according to.the
filter process performed by the loop filter unit on the^
reference image data used by the interpolation firter.
1
1
2. The image processing device according to claim 1,
wherein the filter control unit switches the filter
coefficient when the filter process is not performed by
the loop filter unit to a coefficient in which noise
removal strength is enhanced more than that of the filter
coefficient when the filter process is performed.
3. The image processing device accordirfg to claim 2,
wherein the interpolation filter unit performs a lowfrequency
pass filter process as the - noise removal.
4. The image processing device according to claim 1,
wherein the filter process of the loop filter unit
includes at least one of a deblocking filter process and
an adaptive loop filter process.
5. The image processing device according to clai'm 1,
wherein the loop filter unit adaptibely performs an
o f f s e t process, and
I I 9
. wherei,n the f i l t q r -control u n i t s e t s the f i l t e r
I /
c o e f f i c i e n t n o t subjected t o the o f f s e t process as a
c o e f f i c i e n t i).Ii n which noise removal s t r e n g t h is enhanced
1 ' 1
more than t h a t of the f i i t e r c o e f f i c i e n t subjected t o the
I '
o f f s e t process.
1 1 1
1 1
I
6. An imAbe processing-method comprising the steps! o f :
I I
executing a f i l t e r process on reference image data
I i 1 r e f e r r e d t o ,1 '1 b' y a c u r r e n t block;
I I
generating predicted image data using .the reference
image data hnd a motion vector corresponding t o the
I 1 current block; and
I
switching a f i i t e r c o e f f i c i e n t of the generating of
the predicteh image :data according t d the f i l t e r process
I
i n the executing of t h e f i l t e r process on the reference
' I
image data us,ed i n the generating of t h e predicted Tmage
: data;