Description
Title of Invention
IMAGE PROCESSING DEVICE AND IMAGE PROCESSING METHOD
Technical Field
[OOOl]
The present disclosure relates to an image processing device and an image
processing method.
t
Background Art
[0002] 1
H.264/AVCY one of standard specifications for image encoding scheme,
applies a deblockiilg filter to a block boundary in units of blocks each containing 4x4
pixels, for example, in order to prevent image quality degradation due to block
distortion while an image is encoded. The deblocking filter requires a large amount
of processing and may account for 50% of the entire processing amount in image
decoding, for example.
[0003]
The standards work for High Efficiency Video Coding (HEVC), a nextgeneration
image 'encoding system, proposes application of the deblocking filter in
units of blocks each containing-8x8 pixels or more according to JCTVC-A119 (see
Non-Patent Literature 1). The technique proposed in JCTVC-A119 increases the
block size as a minimum unit for applying the deblockiig filter to perform filtering
processes in parallel on block boundaries in the same direction within one macro
block.
Citation List
Non-Patent Literature
[0004]
Non-Patent Literature 1 : K.Ugur (Nokia), K.R.Andersson (LM Ericsson),
A.Fuldseth (Tandberg Telecom), "JCTVC-A 1 19:Video coding technology proposal
by Tandberg, Nokia, and Ericsson", Documents of the first meeting of the Joint
Collaborative Team on Video Coding (JCT-VC), Dresden, Germany, 15-23 April,
20 10.
Summary of Invention
Technical Problem
[0005]
However, the technique proposed in JCTVC-A119 does not solve
dependency between the process on vertical block boundaries and the process on
horizontal block boundaries. It is still difficult to enable parallel '. processing on
block boundaries in different directions within one macro block (or one encoding
unit) and enable parallel processing between macro blocks. Accordingly, the abovedescribed
technique may not successfully solve problems of a delay and a decrease in
data rates due to a large processing amount while the deblocking filter is applied.
[0006]
The technology according to the disclosure aims at providing an image
processing device and an image processing method capable of providing parallel
processing when a deblocking filter is applied.
Solution to Problem
[0007]
According to an embodiment of the present disclosure, there is provided an
image processing device including a decoding section configured to decode an image
from an encoded stream, a determination section configured to perform
determination processes of determining whether to apply a deblocking filter to
neighboring blocks neighboring across a block boundary within an image to be
decoded by the decoding section, a filtering section configured to apply a deblocking
filter to neighboring blocks to which the determination section has determined to
apply a deblocking filter, and a control section configured to allow the determination
section to perform the determination processes for a vertical block boundary and a
horizontal block boundary using pixels of the neighboring blocks of a reconstruct
image as reference pixels.
[OOOS]
The image processing device can be realized typically as an image decoding
device for decoding an image.
[0009]
Further, according to an embodiment of the present disclosure, there is
5 provided an image processing method including decoding an image from an encoded
stream, performing determination processes of determining whether to apply a
deblocking filter to neighboring blocks neighboring across a block boundary within
an image to be decoded, applying a deblocking filter to neighboring blocks to which
it has been determined in the determination processes to apply a deblocking filter,
C
10 and controlling the determination processes in a manner that the determination
processes for a vertical block boundary and a horizontal block boundary are
performed using pixels of the neighboring blocks of a reconstru~tim age as reference
pixels.
[OO lo]
15 Further, according to an embodiment of the present disclosure, there is
provided an image processing device including a determination section configured to
. perform determination processes of determining whether to apply a deblocking filter
to neighboring blocks neighboring across a block boundary within an image to be
locally decoded when an image to be encoded is encoded, a filtering section
20 configured to apply a deblocking filter to neighboring blocks to which the
determination section has determined to apply a deblocking filter, a control section
configured to allow the determination section to perform the determination processes
for a vertical block boundary and a horizontal block boundary using pixels of the
neighboring blocks of a reconstruct image as reference pixels, and an encoding
25 section configured to encode the image to be encoded using an image filtered by the
filtering section.
[OOll]
The image processing device can be realized typically as an image encoding
device for encoding an image.
30 [OO12]
According to an embodiment of the present disclosure, there is provided an
image processing method including performing determination processes of
determining whether to apply a deblocking filter to neighboring blocks neighboring
across a block boundary within an image to be locally decoded when an image to be -.
encoded is encoded, applying a deblocking filter to neighboring blocks to which it
5 has been determined in the determination processes to apply a deblocking filter,
controlling the determination processes in a manner that the determination processes
for a vertical block boundary and a horizontal block boundary are performed using
pixels of the neighboring blocks of a reconstruct image as referenck pixels, and
encoding the image to be encoded using an image filtered by the deblocking filter.
10
Advantageous Effects of Invention
[00 131
As described above, the image processing device and the image processing
method according to the present disclosure further improves parallel processing
15 when a deblocking filter is applied.
Brief Description of Drawings
[00 141
[Fig. 11 Fig. 1 is a block diagram showing an example of a configuration of an image
20 encoding device according to an embodiment.
[Fig. 21 Fig. 2 is ablock diagram showing an example of a configuration of an image
decoding device according to an. embodiment.
[Fig. 31 Fig. 3 is an explanatory diagram showing an example of neighboring pixels
around a boundary.
25 [Fig. 41 Fig. 4 is an explanatory diagram illustrating reference pixels during filtering
need determination processes according to an existing technique.
[Fig. 51 Fig. 5 is an explanatory diagram illustrating pixels updated by filtering
processes.
[Fig. 61 Fig. 6 is an explanatory diagram illustrating identification of edges for
30 description of the embodiment.
[Fig. 71 Fig. 7 is an explanatory diagram illustrating a parallel process according to
5 / 7 7
an existing technique.
[Fig. 81 Fig. 8 is a first explanatory diagram illustrating dependency between
processes according to an existing technique.
[Fig. 91 Fig. 9 is a second explanatory diagram illustrating dependency between
5 processes according to an existing technique.
[Fig. 101 Fig. 10 is an explanatory diagram illustrating a sequence of processes
according to an existing technique.
[Fig. 1 I] Fig. I1 is a first explanatory diagram illustrating reference pixels during
filtering need determination processes according to a first working example.
4
10 [Fig. 121 Fig. 12 is a second explanatory diagram illustrating reference pixels during
the filtering need determination processes according to the first working example.
[Fig. 131 Fig. 13 is an explanatory diagram illustrating a first,example of process
sequence.
[Fig. 141 Fig. 14 is an explanatory diagram illustrating a second example of process
15 sequence.
[Fig. 151 Fig. 15 is a block diagram illustrating a detailed configuration of a
deblocking filter according to the first working example.
[Fig. 161 Fig. 16 is a block diagram illustrating a detailed configuration of a
determination section.
20 [Fig. 171 Fig. 17 is an explanatory diagram illustrating neighboring blocks around a
slice boundary.
[Fig. 181 Fig. 18 is an explanatory diagram illustrating a first example of a sequence
of processes for each slice.
[Fig. 191 Fig. 19 is an explanatory diagram illustrating a second example of a
25 sequence of processes for each slice.
[Fig. 201 Fig. 20 is a flowchart illustrating a first example of a process flow for the
deblocking filter according to an embodiment.
[Fig. 211 Fig. 21 is a flowchart illustrating a second example of a process flow for the
deblocking filter according to an embodiment.
30 [Fig. 221 Fig. 22 is a flowchart illustrating a flow of filtering need determination
processes according to an embodiment.
[Fig. 231 Fig. 23 is a block diagram illustrating a detailed configuration of the
deblocking filter according to a second working example.
[Fig. 241 Fig. 24 is an explanatory diagram illustrating first and second examples of a
determination technique provided by the second working example.
5 [Fig. 251 Fig. 25 is an explanatory diagram illustrating third and fourth examples of
the determination technique provided by the second working example.
[Fig. 261 Fig. 26 is an explanatory diagram illustrating fifth and sixth examples of the
determination technique provided by the second working example.
[Fig. 271 Fig. 27 is an explanatory diagram illustrating a process sequence for each
i
10 LCU.
[Fig. 281 Fig. 28 is a flowchart illustrating a process flow for each LCU.
[Fig. 291 Fig. 29 is an explanatory diagram illustrating an overview of a third
working example.
[Fig. 301 Fig. 30 is a block diagram illustrating a detailed configuration of a
15 deblocking filter according to the third working example.
[Fig. 3 11 Fig. 3 1 is an explanatory diagram illustrating determination of a weight for
weighted average.
[Fig. 321 Fig. 32 is an explanatory diagram illustrating an example of a weight for
weighted average.
20 [Fig. 331 Fig. 33 is an explanatory diagram illustrating an output pixel value from a
calculation section according to the third working example.
[Fig. 341 Fig. 34 is an explanatory diagram illustrating a first example of process
sequence for comparison.
[Fig. 351 Fig. 35 is an explanatory diagram illustrating a first example of process
25 sequence provided by the third working example.
[Fig. 361 Fig. 36 is an explanatory diagram illustrating a second example of process
sequence for comparison.
[Fig. 371 Fig. 37 is an explanatory diagram illustrating a second example of process
sequence provided by the third working example.
30 [Fig. 381 Fig. 38 is a flowchart illustrating a first example of a process flow for the
deblocking filter according to the third working example.
[Fig. 391 Fig. 39 is a flowchart illustrating a flow of a pixel value calculation process
shown in Fig. 38.
[Fig. 401 Fig. 40 is an explanatory diagram illustrating multiview codec.
[Fig. 411 Fig. 41 is an explanatory diagram illustrating an image encoding process
5 according to an embodiment applied to multiview codec.
[Fig. 421 Fig. 42 is an explanatory diagram illustrating an image decoding process
according to an embodiment applied to multiview codec.
[Fig. 431 Fig. 43 is an explanatory diagram illustrating scalable codec.
[Fig. 441 Fig. 44 is an 'explanatory diagram illustrating an image encoding process
10 according to an embodiment applied to scalable codec.
[Fig. 451 Fig. 45 is an explanatory diagram illustrating an image decoding process
according to an embodiment applied to scalable codec.
[Fig. 461 Fig. 46 is a block diagram illustrating a schematic configuration of a
television apparatus.
15 [Fig. 471 Fig. 47 is a block diagram illustrating a schematic configuration of a mobile
phone.
[Fig. 481 Fig. 48 is a block diagram illustrating a schematic configuration of a
recording/reproduction device.
[Fig. 491 Fig. 49 is a block diagram illustrating a schematic configuration of an
20 image capturing device.
Description of Embodiment
[00 151
Hereinafter, preferred embodiments of the present invention will be
25 described in detail with reference to the appended drawings. Note that, in this
specification and the drawings, elements that have substantially the same function
and structure are denoted with the same reference signs, and repeated explanation is
omitted.
[00 161
30 Description of Embodiment will be described in the following sequence.
1. Apparatus Overview
1 - 1. Image Encoding Device
1-2. Image Decoding Device
2. Existing Technique
2-1. Basic Configuration of Deblocking Filter
5 2-2. Dependency Between Processes According to an Existing
Technique
3. First Working Example
3-1. Deblocking Filter Configuration Example
3-2. Process Flow
4. Second Working Example
4- 1. Deblocking Filter Configuration Example
4-2. Process Flow
4-3. Process Example for Each LCU
5. Third Working Example
5- 1. Overview
5-2. Deblocking Filter Configuration Example
5-3. Process Sequence Example
5-4. Process Flow
6. Application to Various Codecs
6-1. Multiview Codec
6-2. Scalable Codec
7. Example Applications
8. Summing-up
[00 171
25 1. Apparatus Overview
With reference to Figs. 1 and 2, the following describes an overview of an
apparatus to which the technology disclosed in this specification is applicable. The
technology disclosed in this specification is applicable to an image encoding device
and an image decoding device, for example.
30 [0018]
[ 1 - 1. Image Encoding Device]
Fig. 1 is a block diagram showing an example of a configuration of an
image encoding device 10 according to an embodiment. Referring to Fig. 1, the
image encoding device 10 includes an A/D (Analogue to Digital) conversion section
11, a reordering buffer 12, a subtraction section 13, an orthogonal transform section
14, a quantization section 15, a lossless encoding section 16, an accumulation buffer
17, a rate control section 18, an inverse quantization section 21, an inverse
orthogonal transform section 22, an addition section 23, a deblocking filter 24a, a
frame memory 25, a selector 26, an intra prediction section 30, a motion estimation
section 40, and a mode selection section 50.
C
[00 1 91
The A/D conversion section 11 converts an image signal input in an
analogue format into image data in a digital format, and outputs a series of digital
image data to the reordering buffer 12.
The reordering buffer 12 reorders the images included in the series of image
data input from the A/D conversion section 11. AEter reordering the images
according to the a GOP (Group of Pictures) structure according to the encoding
process, the reordering buffer 12 outputs the image data which has been reordered to
the subtraction section 13; the intra prediction section 30, and the motion estimation
section 40.
[002 1 ]
The image data input from the reordering buffer 12 and predicted image
data selected by the mode selection section 50 described later are supplied to the
subtraction section 13. The subtraction section 13 calculates predicted error data
which is a difference between the image data input from the reordering buffer 12 and
the predicted image data input from the mode selection section 50, and outputs the
calculated predicted error data to the orthogonal transform section 14.
[0022]
The orthogonal transform section 14 performs orthogonal transform on the
predicted error data input from the subtraction section 13. The orthogonal
transform to be performed by the orthogonal transform section 14 may be discrete
cosine transform @CT) or Karhunen-Loeve transform, for example. The
orthogonal transform section 14 outputs transform coefficient data acquired by the
orthogonal transform process to the quantization section 15.
[0023]
5 The transform coefficient data input from the orthogonal transform section
14 and a rate control signal from the rate control section 18 described later are
supplied to the quantization section 15. The quantization section 15 quantizes the
transform coefficient data,, and outputs the transform coefficient data which has been
quantized (hereinafter, referred to as quantized data) to the lossless encoding section
%
16 and the inverse quantization section 21. Also, the quantization section 15
switches a quantization parameter (a quantization scale) based on the rate control
signal from the rate control section 18 to thereby change the bit sate of the quantized
data to be input to the lossless encoding section 16.
[0024]
The quantized data input from the quantization section 15 and information
described later about intra prediction or inter prediction generated by the intra
prediction section 30 or the motion estimation section 40 and selected by the mode
selection section 50 are supplied to the lossless encoding section 16. The
information about intra prediction may include prediction mode information
indicating an optimal intra prediction mode for each block, for example. Also, the
information about inter prediction may include prediction mode information for
prediction of a motion vector for each block, difference motion vector information,
reference image information, and the like, for example.
[0025]
2 5 The lossless encoding section 16 generates an encoded stream by
performing a lossless encoding process on the quantized data. The lossless
encoding by the lossless encoding section 16 may be variable-length coding or
arithmetic coding, for example. Furthermore, the lossless encoding section 16
multiplexes the information about intra prediction or the information about inter
30 prediction mentioned above to the header of the encoded stream (for example, a
block header, a slice header or the like). Then, the lossless encoding section 16
outputs the generated encoded stream to the accumulation buffer 17.
The accumulation buffer 17 temporarily stores the encoded stream input
from the lossless encoding section 16 using a storage medium, such as a
semiconductor memory. Then, the accumulation buffer 17 outputs the accumulated
encoded stream at a rate according to the band of a transmission line (or an output
line from the image encoding device 10).
The rate control section 18 monitors the free space of the accumulation
4
buffer 17. Then, the rate control section 18 generates a rate control signal according
to the free space on the accumulation buffer 17, and outputs the generated rate
control signal to the quantization section 15. For example, when there is not much
free space on the accumulation buffer 17, the rate control section 18 generates a rate
control signal for lowering the bit rate of the quantized data. Also, for example,
when the free space on the accumulation buffer 17 is sufficiently large, the rate
control section 18 generates a rate control signal for increasing the bit rate of the
quantized data.
[0028]
The inverse quantization section 21 performs an inverse quantization
process on the quantized data input from the quantization section 15. Then, the
inverse quantization section 21 outputs transform coefficient data acquired by the
inverse quantization process to the inverse orthogonal transform section 22.
[0029]
The inverse orthogonal transform section 22 performs an inverse orthogonal
transform process on the transform coefficient data input from the inverse
quantization section 21 to thereby restore the predicted error data. Then, the inverse
orthogonal transform section 22 outputs the restored predicted error data to the
addition section 23.
[0030]
The addition section 23 adds the restored predicted error data input from the
inverse orthogonal transform section 22 and the predicted image data input from the
mode selection section 50 to thereby generate decoded image data. Then, the
addition section 23 outputs the generated decoded image data to the deblocking filter
24a and the frame memory 25.
[003 11
5 A deblocking filter 24a performs filtering processes to decrease block
distortion that occurs during image encoding. For example, the deblocking filter
24a determines necessity of filtering for each block boundary of decoded image data
supplied from an addition section 23 and applies the deblocking filter to a boundary
that is determined to require the filter. The deblocking filter 24a is also supplied
4
10 with information used for the determination of filtering necessity (e.g., mode
information, transform coefficient information, and motion vector information) as
well as decoded image data from the addition section 23. After the filtering, the
I block distortion is eliminated from the decoded image data and the deblocking filter
24a outputs the decoded image data to frame memory 25. The process for the
deblocking filter 24a will be described in detail later.
[0032]
The frame memory 25 stores, using a storage medium, the decoded image
data input from the addition section 23 and the decoded image data after filtering
input from the deblocking filter 24a.
[0033]
The selector 26 reads, from the frame memory 25, the decoded image data
before filtering that is to be used for the intra prediction, and supplies the decoded
image data which has been read to the intra prediction section 30 as reference image
data. Also, the selector 26 reads, from the frame memory 25, the decoded image
data after filtering to be used for the inter prediction, and supplies the decoded image
data which has been read to the motion estimation section 40 as reference image data.
[0034]
The intra prediction section 30 performs an intra prediction process in each
intra prediction mode, based on the image data to be encoded that is input from the
reordering buffer 12 and the decoded image data supplied via the selector - 26. For
example, the intra prediction section 30 evaluates the prediction result of each intra
prediction mode using a predetermined cost function. Then, the intra prediction
section 30 selects an intra prediction mode by which the cost function value is the
smallest, that is, an intra prediction mode by which the compression ratio is the
highest, as the optimal intra prediction mode. Furthermore, the intra prediction
section 30 outputs, to the mode selection section 50, prediction mode information
indicating the optimal intra prediction mode, the predicted image data, and the
information about intra prediction such as the cost function value.
A motion estimation section 40 performs an inter prediction process
C
(prediction process between frames) based on image data for encoding supplied from
a reordering buffer 12 and decoded image data supplied via a selector 26. For
example, the motion estimation section 40 evaluates the prediction result of each
prediction mode usfng, a predetermined cost function. Then, the motion estimation
section 40 selects an optimal prediction mode, namely, a prediction mode that
minimizes the cost function value or maximizes the compression ratio. The motion
estimation section 40 generates predicted image data according to the optimal
prediction mode. The motion estimation section 40 outputs information about the
inter prediction such as prediction mode information indicating the optimal intra
prediction mode, the predicted image data, and the cost function value to a mode
selection section 50.
[0036]
The mode selection section 50 compares the cost function value related to
the intra prediction input from the intra prediction section 30 and the cost function
value related to the inter prediction input from the motion estimation section 40.
Then, the mode selection section 50 selects a prediction method with a smaller cost
function value, from the intra prediction and the inter prediction. In the case of
selecting the intra prediction, the mode selection section 50 outputs the information
about intra prediction to the lossless encoding section 16, and also, outputs the
predicted image data to the subtraction section 13 and the addition section 23. Also,
in the case of selecting the inter prediction, the mode selection section 50 outputs the
information about inter prediction described above to the lossless encoding section
16, and also, outputs the predicted image data to the subtraction section 13 and the
addition section 23.
[0037]
[I-2. Image Decoding Device]
5 Fig. 2 is a block diagram showing an example of a configuration of an
image decoding device 60 according to an embodiment. With reference to Fig. 2,
the image decoding device 60 includes an accumulation buffer 61, a lossless
decoding section 62, an inverse quantization section 63, an inverse orthogonal
transform section 64, an addition section 65, a deblocking filter 24b, a reordering
4.
10 buffer 67, a DIA (Digital to Analogue) conversion section 68, a fiarne memory 69,
selectors 70 and 71, an intra prediction section 80, and a motion compensation
section 90.
' [003 81
The accumulation buffer 61 temporarily stores an encoded stream input via
15 a transmission line using a storage medium.
[0039]
The lossless decoding section 62 decodes an encoded stream input from the
accumulation buffer 61 according to the encoding method used at the time of
encoding. Also, the lossless decoding section 62 decodes information multiplexed
20 to the header region of the encoded stream. Information that is multiplexed to the
header region of the encoded stream may include information about intra prediction
and information about inter prediction in the block header, for example. The
lossless decoding section 62 outputs the information about intra prediction to the
intra prediction section 80. Also, the lossless decoding section 62 outputs the
25 information about inter prediction to the motion compensation section 90.
[0040]
The inverse quantization section 63 inversely quantizes quantized data
which has been decoded by the lossless decoding section 62. The inverse
orthogonal transform section 64 generates predicted error data by performing inverse
30 orthogonal transformation on transform coefficient data input from the inverse
quantization section 63 according to the orthogonal transformation method used at
the time of encoding. Then, the inverse orthogonal transform section 64 outputs the
generated predicted error data to the addition section 65.
[004 11
The addition section 65 adds the predicted error data input from the inverse
orthogonal transform section 64 and predicted image data input from the selector 71
to thereby generate decoded image data. Then, the addition section 65 outputs the
generated decoded image data to the deblocking filter 24b and the frame memory 69.
The deblocking filter 24b performs filtering processes to decrease block
i
distal-tion appearing on a decoded image. The deblocking filter 24b determines the
necessity of filtering at each block boundary for decoded image data input from the
addition section 65, for example, and applies the deblocking filter to a boundary that
is determined to require the filter. The deblocking filter 24b is also supplied with
information used for the determination of filtering necessity as well as decoded
image data from the addition section 65. After the filtering, the block distortion is
eliminated from the decoded image data and the deblocking filter 24b outputs the
decoded image data to the reordering buffer 67 and the frame memory 69. The
process for the deblocking filter 24b will be described in detail later.
[0043]
The reordering buffer 67 generates a series of image data in a time sequence
by reordering images input from the deblocking filter 24b. Then, the reordering
buffer 67 outputs the generated image data to the D/A conversion section 68.
100441
The D/A conversion section 68 converts the image data in a digital format
input from the reordering buffer 67 into an image signal in an analogue format.
Then, the D/A conversion section 68 causes an image to be displayed by outputting
the analogue image signal to a display (not shown) connected to the image decoding
device 60, for example.
[0045]
The frame memory 69 uses a storage medium to store the decoded image
data input from the addition section 65 before filtering and the decoded image data
input from the deblocking filter 24b after filtering.
COO461
The selector 70 switches the output destination of the image data from the
frame memory 69 between the intra prediction section 80 and the motion
5 compensation section 90 for each block in the image according to mode information
acquired by the lossless decoding section 62. For example, in the case the intra
prediction mode is specified, the selector 70 outputs the decoded image data before
filtering that is supplied from the frame memory 69 to the intra predictibn section 80
as reference image data. Also, in the case the inter prediction mode is specified, the
10 selector 70 outputs the decoded image data after filteXng that is supplied from the
frame memory 69 to the motion compensation section 90 as the reference image data.
[0047]
The selector 71 switches the output source of predicted image data to be
supplied to the addition section 65 between the intra prediction section 80 and the
~ 15 motion compensation section 90 for each block in the image according to the mode
I information acquired by the lossless decoding section 62. For example, in ~ the case I the intra prediction mode is specified, the selector 71 supplies to the addition section
I 65 the predicted image data output from the intra prediction section 80. In the case
I the inter prediction mode is specified, the selector 71 supplies to the addition section
20 65 the predicted image data output from the motion compensation section 90.
[0048]
The intra prediction section 80 performs in-screen prediction of a pixel
value based on the information about intra prediction input from the lossless
decoding section 62 and the reference image data from the frame memory 69, and
25 generates predicted image data. Then, the intra prediction section 80 outputs the
generated predicted image data to the selector 71.
I The motion compensation section 90 performs a motion compensation
I process based on the information about inter prediction input from the lossless
I 30 decoding section 62 and the reference image data from the frame memory 69, and
the generated predicted image data to the selector 71.
[0050]
<2. Existing Technique>
[2-1. Basic Configuration of Deblocking Filter]
Generally, processes using the deblocking filter in an existing image
encoding system such as H.2641AVC or HEVC include two types of processes,
namely, filtering need determination processes and filtering processes. The
following describes these two processes in HEVC, for example.
[005 11
i
(1) Filtering Need Determination Processes
The filtering need determination processes determine whether the
deblocking filter needs to be applied to each boundary of bl~cksw ithin an input
image. Block boundaries include a vertical boundary between blocks horizontally
adjacent to each other and a horizontal boundary between blocks vertically adjacent
to each other. JCTVC-A119 uses a block size of 8x8 pixels as a minimum
processing unit. For example, a macro block of 16x 16 pixels includes four blocks
of 8x8 pixels. The process is applied to one (left) vertical boundary and one (top)
horizontal boundary for each block, namely, four boundaries plus four boundaries
equal to eight boundaries in total. The specification assumes that the macro block
as a technical term includes an coding unit (CU) in the context of HEVC.
[0052]
Fig. 3 is an explanatory diagram showing an example of pixels in two
blocks (neighboring blocks) Ba and Bb adjacent to each other around a boundary.
The following describes the vertical boundary as an example and the description is
obviously applicable to the horizontal boundary. The example in Fig. 3 uses
symbol p,j to represent a pixel in block Ba. In this symbol, i denotes a column
index and j denotes a row index. The column index i is numbered as 0, 1, 2, and 3
in order (from right to left) from the column nearest to the vertical boundary. . The
row index j is numbered as 0, 1, 2, . . . , 7 from the top to the bottom. The left half
of block Ba is omitted from the drawing. Symbol qkj is used to represent a pixel in
block Bb. In this symbol, k denotes a column index and j denotes -a row index. - .
The column index k is numbered as 0, 1, 2, and 3 in order (from left to right) from
the column nearest to the vertical boundary. The right half of block Bb is omitted
from the drawing.
[0053]
The following conditions can be used to determine the necessity of applying
the deblocking filter to the vertical boundary between blocks Ba and Bb shown in
Fig. 3.
[0054]
Determination condition of luma component (Lurna) ... The deblocking
i
filter is applied if conditions A and B are both true.
Condition A:
(Al) Block Ba or Bb enters the intra prediction mode; ,
(A2) Block Ba or Bb has a nonzero orthogonal transform coefficient; or
(A3) IMVAx-MVBxlM or IMVAy-MVBylM
Condition B:
Ip22-2p12+po21+lq22-2ql2+qo21+I~25-2p15+~o51+lq25-2ql5+qo5I

>2 j
(c2)(IP3j-~ojI+Iqoj-q3jI)<(P>>3)
(C~)IPO~-SO~I<1() (>~>~IC) +
where j denotes a row index for the vertical boundary or a column index for
the horizontal boundary. d=lp22-2p12+po2l+lq22-2qi2+qo2I+I~2s-2~1~+~05I+lq2~-
2q15+qo5l
[0060]
Weak filtering
A=Clip(-k,k,(l3(q0~-p0~)+4(q1~-pl~)-5(q2j-p2j)+l6)>>5))
PO~=C~~PO-~S~(~O~+A)
q0j=Clip0-255(q0j-A)
P I ~ = C ~ ~ P O - ~ ~ S ( ~ I ~ + A / ~ )
. .. q1j=Clip0-255(qlj-~/2)
[006 11
Strong filtering
poj'C1ip~-255(@2j+2p~j+2~j+2q~j+qlj+4)>>3)
qoj=~lipo-255((1jp + 2p~j+2q~j+l2j+qq 2j+4)>>3)
5 p 1j =Clip0-255(@2j+p1j +poj+q0j+2)>>2)
q1~=C1ip0-255((P0~+q0j+q1j+92j+2)'>2)
p2j=~lip~-255((2p31jj ++pj0 j+qoj+4)>>3)
q2j=C1ip~-255((p~j+q~j+qij+3q2j+2q3j+4)>>3)
[0062]
*,
10 where Clip(a,b,c) denotes a process to clip value c within the range of alclb
and C l i ~ ~ - ~de~n~ote(sc a) p rocess to clip value c within the range of 05~5255.
[0063]
Filtering chroma components
A=Clip(-tc,tc,((((q0j-p0j)<<2)+~1j-~1j+4)>>3))
15 poj'Clipo-255(poj~A)
qoj=Clip0-255(qoj-A)
[0064]
As indicated by broken-line frames C6 through C8 and C1 through C3 in
Fig. 5, the filtering processes (particularly strong filtering on luma components) on
20 general vertical boundaries update pixel values on the first through third and sixth
through eighth columns in each block. Similarly, the filtering processes on
horizontal boundaries update pixel values on the first through third and sixth through
eighth rows in each block.
100651
25 [2-2. Dependency Between Processes According to an Existing Technique]
For the purpose of description, as shown in Fig. 6, macro block MBx (MBO,
MB 1 ...) each having the size of 16x 16 pixels includes the top left vertical boundary
represented as Vx,O, the top center vertical boundary represented as Vx,l, the bottom
left vertical boundary represented as Vx,2, the bottom center vertical boundary
30 represented as Vx,3, the top left horizontal boundary represented as Hx,O, the top
right horizontal boundary represented as Hx, 1, the left center horizontal boundary
represented as Hx,2, and the right center horizontal boundary represented as Hx,3.
Concerning boundary Z, for example, the. filtering need determination process is
represented as Jz and the filtering process is represented as Fz.
[0066]
5 The above-described existing technique causes no dependency between
processes on boundaries in the same direction within one macro block. Therefore,
the technique can perform parallel filtering on vertical boundaries and horizontal
boundaries within one macro block, for example. As an example, ~ i7 m~ake.s i t
clear that there is no dependency among four filtering processes FVO,OF,v o,~F, vo~,
4
10 and Fvo,~(n o pixel updated redundantly) within macro block MBO and the filtering
processes can be performed in parallel.
[0067]
However, the above-described existing technique leaves the dependency
between the filtering .processes on vertical boundaries and the filtering need
15 determination processes on horizontal boundaries. The existing technique also
leaves the dependency between the filtering processes on horizontal boundaries and
the filtering need determination processes on vertical boundaries. If a vertical
boundary is processed prior to a horizontal boundary, for example, the filtering need
determination processes need to be performed on horizontal boundaries within a
20 given macro block after termination of the filtering processes on vertical boundaries.
As an example,. Fig. 8 shows that, within macro block MBO, filtering need
determination process JHO,Ode pends on results of filtering processes Fvo,~an d Fvo,l
and filtering need determination process JHO,d~e pends on a result of filtering
processes Fvo,~. Similarly, the filtering need determination processes need to be
25 performed on vertical boundaries within a given macro block after termination of the
filtering process on the horizontal boundary for the adjacent macro block. As an
example, Fig. 9 shows that filtering need determination process Jvl,o for macro block
MB1 depends on results of filtering processes FHO,l and FH0,3 for macro block MI30
and filtering need determination process J v I ,f~o r macro block MBl depends on a
30 result of filtering process FH0,3 for macro block MBO.
[0068]
The existing technique involves the dependency between processes and
therefore provides parallel processing of the deblocking filter to a very limited extent
even if the technique proposed in JCTVC-A119 is used.
[0069]
5 Fig. 10 is an explanatory diagram illustrating a sequence of deblocking filter
processes according to an existing technique. The example assumes that the
deblocking filter is supplied with an image having the size of 32x32 pixels. The
input image includes four macro blocks MBO through MB3 each having the size of
16x 16 pixels.
i
10 [0070]
In Fig. 10, each broken-line frame represents a process to be performed in
parallel. For example, the first step performs, in parallel, filtering need
determination processes Jvo,o, Jvo,~J,V 0,2a nd Jv0,3 on four vertical boundaries in
macro block MBO. The second step performs, in parallel, filtering processes Fvo,o,
15 Fvo,~F, vo2 and FV0,3 on four vertical boundaries in macro block MBO. After
termination of the second step, the third step performs, in parallel, filtering need
determination processes JHO,oJ, HO,J~H, Oa,~nd JH0,3 on four horizontal boundaries in
macro block MBO. The fourth step performs, in parallel, filtering processes
FHO,i, FH0,2 and FH0,3 on four horizontal boundaries in macro block MBO. After
20 termination of the fourth step, processes (fifth to eighth steps) for macro block MB1
are performed successively. After termination of the processes on macro block
MB1, processes (ninth to twelfth steps) for macro block MB2 are performed
successively. After termination of the processes on macro block MB2, processes
(thirteenth to sixteenth steps) for macro block MB3 are performed successively.
Such parallel processing within the limited extent cannot satisfactorily solve
the problem of delay or data rate degradation due to a large processing amount when
the deblocking filter is applied. Three working examples described below further
improve parallel processing when the definition is applied.
30 [0072]
<3. First Working Example>
[3- 1. Deblocking Filter Configuration Example]
The following describes example configurations of the deblocking filter 24a
for the image encoding device 10 shown in Fig. 1 and the deblocking filter 24b for
the image decoding device 60 shown in Fig. 2 according to the first working example.
The configurations of the deblocking filter 24a and the deblocking filter 24b may be
common to each other. In the following description, the deblocking filter 24a and
the deblocking filter 24b are generically referred to as a deblocking filter 24 when
there is no need for distinction between them.
[0073]
(1) Dependency between new processes
According to the working example, processes using the deblocking filter 24
also include two types of processes, namely, filtering need detemnination processes
and filtering processes. However, the deblocking filter 24 uses values of a
reference pixel different from the existing technique to determine whether to apply
the deblocking filter to vertical boundaries and horizontal boundaries. Specifically,
for the determination on vertical boundaries, the deblocking filter 24 uses a reference
pixel, i.e., a pixel that is included in pixels of adjacent blocks around the vertical
boundary and belongs to a row to which the deblocking filter for horizontal
boundaries is not applied. For the determination on horizontal boundaries, the
deblocking filter 24 uses another reference pixel, i.e., a pixel that is included in
pixels of adjacent blocks around the horizontal boundary and belongs to a row to
which the deblocking filter for vertical boundaries is not applied. Also in the
following description, the deblocking filter 24 performs processes based on a block
size of 8x8 pixels as a processing unit, for example.
[0074]
Fig. 11 is an explanatory diagram illustrating reference pixels during the
filtering need determination processes performed by the deblocking filter 24 on
vertical boundaries. With reference to Fig. 11, macro block MBO has the size of
16x16 pixels. The deblocking filter 24 determines whether to apply the filtering to
four vertical boundaries of macro block MBO using the reference pixel that belongs
to at least one of the fourth and fifth rows (L4 and L5) of each block. The
deblocking filter for horizontal boundaries is not applied to these two rows (see Fig.
9). This configuration solves the dependency between the filtering processes on
horizontal boundaries and the filtering need determination processes on vertical
boundaries.
[0075]
Fig. 12 is an explanatory diagram illustrating reference pixels during the
filtering need determination processes performed by the deblocking filter 24 on
horizontal boundaries. Fig. 12 also shows macro block MBO. he deblocking
filter 24 determines whether to apply the filtering to four horizontal boundaries of
*
macro block MBO using the reference pixel that belongs to at least one of the fourth
and fifth columns (C4 and C5) of each block. The deblocking filter for vertical
boundaries is not applied to these two columns (see Fig. 7 or 8). , This configuration
solves the dependency between the filtering processes on vertical boundaries and the
filtering need determination processes on horizontal boundaries.
Solving the dependency between processes can consequently parallelize
filtering need determination processes for vertical boundaries and horizontal
boundaries within one macro block. Processes can be parallelized between macro.
blocks. Filtering need determination processes can be performed in parallel on
vertical boundaries and horizontal boundaries of all macro blocks within an input
image.
Fig. 13 is an explanatory diagram illustrating a first example of process
sequence available on the deblocking filter 24. The example also assumes that the
deblocking filter is supplied with an image having the size of 32x32 pixels. The
input image includes four macro blocks MI30 through MB3 each having the size of
16x 16 pixels.
[0078]
In Fig. 13, each broken-line frame represents a process to be performed in
parallel. While the example in Fig. 10 requires 16 process steps for a sequence of
processes, the example in Fig. 13 aggregates the same number of processes into three
process steps. The first step performs, in parallel, filtering need determination
processes Jvo,o through Jv3,3 and JHO,0th rough JmT3o n all vertical boundaries and all
horizontal boundaries of all macro blocks MBO through MB3. The second step
performs, in parallel, filtering processes FvoSot hrough F v ~o,n~ 1 6 vertical boundaries
of all macro blocks MBO through MB3. The third step performs, in parallel,
filtering processes FHO,0 through FH3,3 on 16 horizontal boundaries of all macro
blocks MBO through MB3. The second step and the third step may be performed in
the reverse order.
[0079]
<
The example in Fig. 13 maximizes parallelism (the quantity of processes
performed in parallel) based on parallel processing between macro blocks.
According to the example in Fig. 14, however, the deblocking filter 24 can perform a
process on each macro block.
[OOSO]
The example in Fig. 14 aggregates the same number of processes illustrated
in Figs. 10 and 13 into 12 process steps. The first step performs, in parallel,
filtering need determination processes Jvoso through JVo,3 and JHOt,h~r ough k 0 , 3 on
four vertical boundaries and four horizontal boundaries of macro block MBO. The
second step performs, in parallel, filtering processes Fvo,0 through F v ~o,n~ f our
vertical boundaries in macro block MBO. The third step performs, in parallel,
filtering need determination processes Jvl,o through JV1,3 and JH~,toh rough JH1,3o n
four vertical boundaries and four horizontal boundaries of macro block MB1. The
fourth step performs, in parallel, filtering processes FVI,Oth rough Fvl,3 on 'four
vertical boundaries in macro block MBl. The fifth step performs, in parallel,
filtering processes FHO,tOh rough FH0,3 on four horizontal boundaries in macro block
MBO. The sixth step performs, in parallel, filtering need determination processes
Jv2,o through J v ~a,n~d JH2,0 through JH2,3 on four vertical boundaries and four
horizontal boundaries of macro block MB2. The seventh step performs, in pafallel,
filtering processes FV2,0t hrough F v ~o,n~ f our vertical boundaries in macro block
MB2. The eighth step performs, in parallel, filtering processes FHI,Oth rough FH1,3
on four horizontal boundaries in macro block MBl. The ninth step performs, in
parallel, filtering need determination processes JV3,0 through JV3,3 and Jm,o through
JW,3 on four vertical boundaries and four horizontal boundaries of macro block MB3.
The tenth step performs, in parallel, filtering processes FV3,0 through FV3,3 on four
vertical boundaries in macro block MB3. The eleventh step performs, in parallel,
5 filtering processes FH2,0 through FH2,3 on four horizontal boundaries in macro block
MB2. The twelfth step performs, in parallel, filtering processes FH3,0 through FH3,3
on four horizontal boundaries in macro block MB3. In this case, the deblocking
filter 24 can perform a process on the entire input image using process steps fewer
than those of the existing technique while the parallelism is lower than the example
i
10 in Fig. 13
(2) Basic configuration of deblocking filter
Fig. 15 is a block diagram illustrating a detailed configuration of the
deblocking filter 24 according to the first working example for performing the
15 above-described parallel processes. With reference to Fig. 15, the deblocking filter
24 includes a determination block 110, a horizontal filtering block 130, a vertical
filtering block 140, and a parallelization control section 150.
[0082]
(2-1) Determination block
20 The determination block 110 includes vertical boundary determination
sections 112-1 through 112-n and horizontal boundary determination sections 114-1
through 1 14-n. The vertical boundary determination sections 1 12 and the horizontal
boundary determination sections 114 are supplied with images input to the
deblocking filter 24 and determination information used to determine the necessity of
25 filtering.
[0083]
The vertical boundary determination sections 112 determine whether to
apply the deblocking filter to vertical boundaries using pixel values of reference
pixels belonging to a row to which the deblocking filter for horizontal boundaries is
30 not applied as illustrated in Fig. 11. In this example, a pixel value of the reference
pixel is input to the deblocking filter 24. The vertical boundary determination
sections 112 output, to the horizontal filtering block 130, information indicating a
determination result about each vertical boundary (for example, binary information
of which value " 1 " indicates a determination result that the deblocking filter needs to
be applied).
[0084]
The horizontal boundary determination sections 114 determine whether to
apply the deblocking filter to horizontal boundaries using pixel values of reference
pixels belonging to a row to which the deblocking filter for vertical boundaries is not
applied as illustrated in Fig. 12. In this example, a pixel value of the reference pixel
is also input to the deblocking filter 24. The determihation process performed by
each of the horizontal boundary determination sections 114 is performed in parallel
to the determination process performed by each of the, vertical boundary
determination sections 112. The horizontal boundary determination sections 114
output, to the vertical filtering block 140, information indicating a determination
result about each horizontal boundary.
[0085]
Fig. 16 is a block diagram illustrating a detailed configuration of each of the
vertical boundary determination sections 112 and the horizontal boundary
determination sections 114. With reference to Fig. 16, each determination section
includes a tap constitution section 121, a calculation section 122, a threshold
comparison section 123, a distortion evaluation section 124, and a filtering
determination section 125.
[0086]
The tap constitution section 121 acquires a reference pixel value from pixel
values of two neighboring blocks around a focused boundary in the input image and
constitutes a tap (a set of reference pixel values) for determining determination
condition B for the above-described luma component. For example, a vertical
boundary may be focused in the blocks each of which has the size of 8x8 pixels. In
this case, the tap constitution section 121 constitutes a tap from a pixel value
belonging to the fourth and/or fifth rows of two blocks at the right and left. If a
horizontal boundary is focused, the tap constitution section 121 constitutes a tap from
a pixel value belonging to the fourth andlor fifth columns of two blocks at the top
and bottom. The calculation section 122 assigns the tap constituted by the tap
constitution section 121 to the left-hand side of the determination expression in
determination condition B and calculates an edge value to be compared with edge
determination threshold value P. The threshold comparison section 123 compares
the value calculated by the calculation section 122 with edge determination threshold
value p and outputs a comparison result to the filtering determination section 125.
[0087]
The distortion evaluation section 124 evaluates determination condition A of
the above-described luma component using mode information (MB mode), transform
coefficient information, and motion vector information supplied as the determination
information. The distortion evaluation section 124 outputs an, evaluation result to
the filtering determination section 125. The distortion evaluation section 124
evaluates only determination condition A1 of a chroma component based on the
mode information.
[OOSS]
The filtering determination section 125 determines whether to apply the
deblocking filter to a focused boundary based on a comparison result of
determination condition B supplied from the threshold comparison section 123 and
an evaluation result of determination condition A supplied from the distortion
evaluation section 124. The filtering determination section 125 outputs information
indicating the determination result.
(2-2) Horizontal filtering block
Returning back to Fig. 15, the configuration of the deblocking filter 24 will
be described further. The horizontal filtering block 130 includes horizontal filtering
sections 132-1 through 132-n. The horizontal filtering sections 132 are supplied
with an input image and a determination result concerning each vertical boundary
from the determination block 1 10.
[0090]
The horizontal filtering sections- 132 apply the deblocking filter for vertical
?
boundaries to right and left pixels around the corresponding vertical boundary if the
determination result from the vertical boundary determination section 1 12 indicates
that the filter needs to be applied. The horizontal filtering sections 132 output, to
the vertical filtering block 140, a pixel value after the filtering in terms of the filtered
pixel or a pixel value of the input image in terms of the other pixels.
[009 11
(2-3) Vertical filtering block
The vertical filtering block 140 includes vertical filtering sections 142- 1
through 142-11. The vertical filtering sections 142 are supplied with an input image
k
and a determination result concerning each horizontal boundary from the
determination block 11 0.
COO921
The vertical filtering sections 142 apply the deblocking filter for horizontal
boundaries to top and bottom pixels around the corresponding horizontal boundary if
the determination result from the horizontal boundary determination section 114
indicates that the filter needs to be applied. The vertical filtering sections 142
output a pixel value after the filtering in terms of the filtered pixel or a pixel value
supplied from the horizontal filtering block 130 in terms of the other pixels. An
output from each of the vertical filtering sections 142 may be contained in an output
image from the deblocking filter 24.
[0093]
(2-4) Parallelization control section
The parallelization control section 150 controls the parallelism of filtering
need determination processes in the determination block 110 and the parallelism of
filtering processes in the horizontal filtering block 130 and the vertical filtering block
140.
[0094]
For example, the parallelization control section 150 may control the
parallelism of processes for each block based on an input image size. More
specifically, the parallelization control section 150 increases the parallelism of
processes for each block if the input image size is relztively large. This can
adaptively prevent delay or data,rate degradation due to a processing amount that
increases according to image sizes. For example, the parallelization control section
150 may control the parallelism of processes for each block based on a sequence
parameter set, a picture parameter set, or parameters contained in the slice header.
This enables to flexibly configure the parallelism according to requirements of users
who develop apparatuses. For example the parallelism may be configured
according to restrictions on the installation environment such as the number of
processor cores or the number of software threads.
[0095]
The working example can parallelize processes between macro blocks.
This signifies that any sequence of processes on blocks within an image has no effect
on a finally output result. Accordingly, the parallelization co~trosl ection 150 can
control a sequence of filtering need determination processes in the determination
block 110 and a sequence of filtering processes in the horizontal filtering block 130
and the vertical filtering block 140 on a block basis.
[0096]
More specifically, the parallelization control section 150 may control a
sequence of filtering processes according to the dependency of the filtering processes
between macro blocks. According to an existing technique, for example, the
dependency of processes between neighboring macro blocks around a slice boundary
may delay parallel processes on each slice within an image. However, the
parallelization control- section -150 according to the working example can -perform
filtering processes on neighboring macro blocks around the slice boundary prior to
the other macro blocks.
[0097]
For example, Fig. 17 illustrates eight macro blocks MBlO through MB13
and MB20 through MB23 around a slice boundary. Macro blocks MBlO through
ME! 13 belong to slice SL1. Macro blocks MB20 through MB23 belong to slice SL2.
Concerning these macro blocks, the filtering processes for horizontal boundaries on
macro block MB20 in slice SL2 depend on the filtering processes for vertical
boundaries on macro block MB12 in slice SLl. Similarly, the filtering processes
--
for horizontal boundaries on macro block MB21 in slice SL2 depend on the filtering
processes for vertical boundaries on macro block MB 13 in slice SL1.
[0098]
According to an example in Fig. 18 under these conditions, the
parallelization control section 150 performs filtering processes on the vertical
boundaries of macro blocks MB 12 and MB 13 out of filtering processes for slice SL1
in preference to processes on the other boundaries. The result is to prevent a large
delay from occurring in filtering processes on the horizontal boundaries of macro
blocks MI320 and MB21 out of filtering processes for slice SL2. An example in Fig.
19 initially performs filtering processes in parallel on vertical boundaries for all
macro blocks included in slice SLl. Also in this case, no delay occurs in the
filtering process on the horizontal boundaries of macro blocks MB20 and MB21 in
slice SL2.
[0099]
[3-2. Process Flow]
With reference to Figs. 20 through 22, a process flow for the deblocking
filter 24 will be described.
[O 1001
(1) First scenario
Fig. 20 is a flowchart illustrating a process flow example for the deblocking
filter 24 according to the first scenario. The first scenario corresponds to the
example of large parallelism as shown in Fig. 13.
With reference to Fig. 20, the vertical boundary determination sections 112-
1 through 112-n determine in parallel whether filtering is needed for all vertical
boundaries included in macro blocks within an input image (step S102). The
horizontal boundary determination sections 1 14- 1 through 1 14-n determine in
parallel whether filtering is needed for all horizontal boundaries included in niacro
blocks within an input image (step S104). Steps S102 and S104 are also performed
in parallel.
[O 1 021
The horizontal filtering sections 132-1 through 132-n apply the deblocking
filter in parallel to all vertical boundaries determined at step S102 to require the
deblocking filter to be applied (step S110). The vertical filtering sections 142-1
through 142-n apply the deblocking filter in parallel to all horizontal boundaries
5 determined at step S 104 to require the deblocking filter to be applied (step S 120).
[0 1 031
(2) Second scenario
Fig. 21 is a flowchart illustrating a process flow example for the deblocking
filter 24 according to the second scenario. The second scenario corresponds to the
10 example of smaller parallelism as shown in Fig. 14.
[0 1 041
With reference to Fig. 2 1, the vertical boundary determination sections 11 2-
1 through 112-n determine in parallel whether filtering is needed for all vertical
boundaries included in a focused macro block within an input image (step S202).
15 The horizontal boundary determination sections 114-1 through 114-n determine in
parallel whether filtering is needed for all horizontal boundaries included in the
focused macro block (step S204). Steps S202 and S204 are also performed in
parallel.
[0 1051
20 The horizontal filtering sections 132-1 through 132-n apply the deblocking
filter in parallel to vertical boundaries in the focused macro block determined at step
S202 to require the deblocking filter to be applied (step S210).
[0 1 061
The process at step S220 aims at a focused macro block in the most recent
25 loop. The process at step S220 may be skipped for the first focused macro block.
The vertical filtering sections 142-1 through 142-n apply the deblocking filter in
parallel to horizontal boundaries determined, at step S204 in the most recent loop, to
require the deblocking filter to be applied (step S220).
[0 1071
30 The process at steps S202 through S220 is repeated for a newly focused
macro block if focused macro blocks remain unprocessed in the input image (step
S230).
[0 1 081
If there remains no focused macro block unprocessed, the vertical filtering
sections 142-1 through 142-n apply the deblocking filter in parallel to horizontal
5 boundaries determined to require the deblocking filter to be applied in the focused
macro block for the last loop (step S240). The process then terminates.
[0 1 091
While there have been described the two typical scenarios to parallelize
processes in units of images and macro blocks, the two scenarios are mere examples
10 for the description. Processes of the deblocking filter 24 may be parallelized in
various units such as a given number of macro blocks (two or four macro blocks) or
a group of horizontally or vertically placed blocks, for example.
[OllO]
(3) Filtering need determination processes
15 Fig. 22 is a flowchart illustrating a flow of filtering need determination
processes corresponding to steps S102 and S104 in Fig. 21 and steps S202 and S204
in Fig. 22.
[Olll]
With reference to Fig. 22, the distortion evaluation section 124 evaluates
20 distortion of each boundary based on the mode information, the transform coefficient
information, and the motion vector information (step S130). The process proceeds
to step S134 if the evaluation results in the presence of distortion (determination
condition A is true). The process proceeds to step S140 if the evaluation results in
the absence of distortion (step S 132).
25 [O112]
At step S134, the calculation section 122 calculates an edge value based on
a reference pixel tap constituted by the tap constitution section 121 (step S134).
The threshold comparison section 123 compares the calculated value with edge
determination threshold value P (step S136). The process proceeds to step S 138 if
30 the edge value is smaller than threshold value P (determination condition B is true).
The process proceeds to step S140 if the edge value is not smaller than threshold
value p.
[0113]
At step S138, the filtering determination section 125 determines to apply the
deblocking filter to a boundary to be determined (step S138). At step S140, the
5 filtering determination section 125 determines not to apply the deblocking filter to a
boundary to be determined (step S140).
[0114]
<4. Second Working Example>
The first working example performs the filtering need determination process
10 on a given block using the pixel value of a pixel not updated by the filtering
processes on the other blocks. By contrast, the second working example described
below provides memory to store pixel values input to the deblocking filter and
thereby enables to eliminate restrictions on the filtering need determination processes
and use more versatile determination conditions.
15 [0115]
[4- 1. Deblocking Filter Configuration Example]
(I) Description of sections
Fig. 23 is a block diagram illustrating a detailed configuration of the
deblocking filter 24 according to the second working example. With reference to
20 Fig. 23, the deblocking filter 24 includes line memory 208, a determination block
210, the horizontal filtering block 130, the vertical filtering block 140, and the
parallelization control section 150.
[0116]
The line memory 208 stores pixel values for an input image supplied to the
25 deblocking filter 24. Filtering processes in the horizontal filtering block 130 and
the vertical filtering block 140 do not update pixel values stored in the line memory
208. Filtering need determination processes performed by sections described below
in the determination block 21 0 reference pixel values stored in the line memory 208.
The apparatus includes another memory for purposes different from processes of the
30 deblocking filter 24. This memory may be reused (shared) as the line memory 208.
[0117]
The determination block 2 10 includes vertical boundary determination
sections 2 12- 1 through 2 12-n and horizontal boundary determination sections 2 14- 1
through 2 14-n. The vertical boundary determination sections 2 12 and the
horizontal boundary determination sections 214 are supplied with pixel values stored
5 in the line memory 208 for an image input to the deblocking filter 24 and
determination information used to determine the need for filtering.
[0118]
The vertical boundary determination sections 2 12 use pixel values input to
the deblocking filter 24 to determine whether to apply the deblocking filter to each
10 vertical boundary. The vertical boundary determination sections 212 output, to the
horizontal filtering block 130, information indicating a determination result about
each vertical boundary.
[0119]
The horizontal boundary determination sections 214 also use pixel values
15 input to the deblocking filter 24 to determine whether to apply the deblocking filter
to each horizontal boundary. The horizontal boundary determination sections 214
perform determination processes in parallel to determination processes performed by
the vertical boundary determination sections 2 12. The horizontal boundary
determination sections 214 output, to the vertical filtering block 140, information
20 indicating a determination result about each horizontal boundary.
[O 1201
(2) Versatile determination conditions
Similarly to the existing technique as shown in Fig. 4, the vertical boundary
determination sections 212 according to the working example may reference pixels
25 on the third and sixth rows of a block to determine the necessity of filtering on a
vertical boundary of each block. In this case, however, pixel values to be
referenced are stored in the line memory 208 and are attributed to an image input to
the deblocking filter 24. Similarly, the horizontal boundary determination sections
214 may reference pixels on the third and sixth rows of a block to determine the
30 necessity of filtering on a horizontal boundary of each block. In this case, the
configuration according to the working example can be easily provided without
changing determination conditions for the filtering need determination processes
installed on an existing apparatus.
[0121]
The vertical boundary determination sections 212 may reference pixels of
5 three or more rows in a block during the determination. Similarly, the horizontal
boundary determination sections 214 may reference pixels of three or more columns
in a block during the determination. The vertical boundary determination sections
212 and the horizontal boundary determination sections 214 may use determination
condition expressions different from the existing technique. With reference to Figs.
10 24 through 26, the following describes six examples of the determination technique
according to the working example.
[O 1221
(2-1) First example
Fig. 24 is an explanatory diagram illustrating first and second examples of
15 the determination technique. In the first and second examples, the filtering need
determination processes (particularly the determination using determination
condition B for luma components) for vertical boundaries references pixels of all
rows L1 through L8 from the first to the eighth in each block. The filtering need
determination processes for horizontal boundaries also references pixels of all
20 columns from the first to the eighth in each block.
[0 1231
The first example may define determination conditions for luma
components as follows.
Determination condition of luma component (Luma) ... The deblocking
25 filter is applied if conditions A and B are both true.
Condition A:
(Al) Block Ba or Bb enters the intra prediction mode;
(A2) Block Ba or Bb has a nonzero orthogonal transform coefficient; or
(A3) IMVAx-MVBxl24 or IMVAy-MVByl24
30 Condition B:
iDo=lp20-2plo+p00J+Iq20-2q10+q001+IP277+-~20 71+lq27-2q1~+q07I
~ D ~ = I P ~ I - ~l+~lq~21I-2+qPl l+oqI01 l+l~26-2~16+~061+lq26-2q16+q061
iD2=1p22-2p12+p021+lq22-2q12+q021+IP2~-2p1~+~051+lq25-2q1~+q0~1
iD3=l~23-2p13+~o31+1423-2q13+q03(+1P24-2~l4+~~l+lq24-~ql4+qo4l
iDave=(iDo+iDl+ iD2+iD3)>>2
Under this condition, iD,

l>
15 Under this condition, iDa,

1>
Under this condition, iDave

[5- 1. Overview]
According to the second working example, the filtering need determination
15 processes on vertical boundaries and horizontal boundaries reference a pixel value
input to the deblocking filter and thereby eliminating the dependency between the
filtering need determination processes and making the parallelization of
determination processes possible. The third working example described below
applies the conkept of the second working example also to the filtering process.
20 The filtering processes for vertical boundaries and horizontal boundaries also filter
pixel values input to the deblocking filter.
Y
101561
Fig. 29 is an explanatory diagram illustrating an overview of the working
example. At the bottom left of Fig. 29, there is shown a shape representing input
25 pixels (also referred to as reconstruct pixels) before being processed by the
deblocking filter. The working example allows pixel values input to the deblocking
filter to be referenced from not only filtering need determination processes for
vertical boundaries and horizontal boundaries but also filtering processes for vertical
boundaries and filtering processes for horizontal boundaries. Therefore, the
30 dependency between the two filtering processes is eliminated. The two filtering
processes are performed in parallel.
[0157]
The filtering processes for vertical boundaries and the filtering processes for
horizontal boundaries may update values of duplicate pixels. Filled pixels in Fig.
29 illustrate positions of the pixels likely to be duplicated. The deblocking filter
according to the working example calculates one output pixel value from two filter
outputs in terms of pixels that are duplicately updated by two filters operating in
parallel.
[0158]
[5-2. Deblocking Filter Configuration Example]
Fig. 30 is a block diagram illustrating a detailed. configuration of the
deblocking filter 24 according to the third working example. With reference to Fig.
30, the deblocking filter 24 includes the line memory 208, the determination block
210, a horizontal filtering block 330, a vertical filtering block 340, the parallelization
control section 150, a calculation section 360. The determination block 210
includes vertical boundary determination sections 2 12- 1 through 2 12-n and
horizontal boundary determination sections 2 14- 1 through 2 14-11. The vertical
boundary determination sections 212 and the horizontal boundary determination
sections 214 may determine the necessity of filtering on boundaries according to the
various determination conditions as described in the second working example.
[0159]
The horizontal filtering block 330 includes horizontal filtering sections 332-
1 through 332-n. The horizontal filtering sections 332 are supplied with an input
image value from the line memory 208 and a determination result concerning each
vertical boundary from the determination block 2 10.
[0 1 601
The horizontal filtering sections 332 apply the deblocking filter for vertical
boundaries to right and left pixels around the corresponding vertical boundary if the
determination result from the vertical boundary determination section 212 indicates
that the filter needs to be applied. The horizontal filtering sections 332 output, to
the calculation section 360, a pixel value after the filtering in terms of the filtered
pixel or an input pixel value in terms of the other pixels.
[0161]
The vertical filtering block 340 includes vertical filtering sections 342-1
through 342-n. The vertical filtering sections 342 are supplied with an input pixel
value from the line memory 208 and a determination result concerning each
5 horizontal boundary from the determination block 2 10.
[0 1 621
The vertical filtering sections 342 apply the deblocking filter for horizontal
boundaries to top and bottom pixels around the corresponding horizontal boundary if
the determination result from the horizontal boundary determination section 214
10 indicates that the filter needs to be applied. Filtering processes of the vertical
filtering sections 342-1 through 342-n are performed in parallel to filtering processes
of the horizontal filtering sections 332-1 through 332-n. The vertical filtering
sections 342 output, to the calculation section 360, a pixel value after the filtering in
terms of the filtered pixel or an input pixel value in terms of the other pixels.
15 [0163]
The calculation section 360 is supplied with an output pixel value from the
horizontal filtering block 330 and an output pixel value from the vertical filtering
block 340 in parallel. Further, the calculation section 360 is supplied with
determination results from the vertical boundary determination section 212 and the
20 horizontal boundary determination section 214. According to a determination result,
the calculation section 360 calculates output pixel values for pixels filtered from the
horizontal filtering block 330 and the vertical filtering block 340 based on filter
outputs from the horizontal filtering block 330 and the vertical filtering block 340.
[0 1 641
2 5 According to the working example, for example, the calculation section 360
calculates an average of two filter outputs for duplicately filtered pixels. The
calculation section 360 may calculate a simple average of two filter outputs.
Instead, the calculation section 360 may calculate a weighted average of two filter
outputs. For example, the calculation section 360 may determine a weight for
30 weighted averages of pixels according to the distance from each pixel to the vertical
., boundary and to the horizontal boundary.
[0 1651
Fig. 31 is an explanatory diagram illustrating determination of a weight for
weighted average calculated by the calculation section 360. Fig. 31 shows focused
pixel Pz in black corresponding to one of the duplicated positions illustrated in Fig.
29. There are three pixels corresponding to distance Dv between focused pixel PZ
and nearest vertical boundary VZ. There are two pixels corresponding to distance
DH between focused pixel Pz and nearest horizontal boundary Hz. Distance DH is
smaller than distance Dv. In this case, the calculation section 360 may set a weight
for output from the deblocking filter applied to horizontal boundary Hz to be larger
than a weight for output from the deblocking filter applied to~verticalb oundary Vz.
The example in Fig. 3 1 assumes that a ratio of filter output VOufto r vertical boundary
VZ to filter output HOufto r horizontal boundary HZ is 2:3.
[0 1 661
As seen from Fig. 3 1, calculating a weighted average of two filter outputs
can consequently provide each focused pixel with an output pixel value similar to the
case of applying one two-dimensional filter having a filter tap along the horizontal
direction and a filter tap along the vertical direction. Parallelizing filtering
processes on the vertical boundary and the horizontal boundary can also
appropriately reduce block distortion appearing on the vertical boundary and the
horizontal boundary. As another working example, the deblocking filter 24 may
include one two-dimensional filter that simultaneously calculates horizontal filtering,
vertical filtering, and a weighted-average. In this case, however, the installation is
very complicated because filter coefficients need to be variously changed
correspondingly to pixels. On the other hand, the third working example performs
two one-dimensional filters in parallel and then calculates a weighted average. This
can easily provide processes substantially equal to a two-dimensional filter while
ensuring the functionality of existing deblocking filters.
[O 1671
Fig. 32 is an explanatory diagram illustrating an example of the weight for
weighted averages calculated based on the example in Fig. 31. Fig. 32 shows 36
pixels (6x6) around an intersection between the vertical boundary and the horizontal
boundary. The pixels correspond to the above-described duplicated positions. The.
ratio of the weight for filter output Vout to the weight for filter output Hout is 1 : 1 (2:2
or 3:3) for pixels positioned at an equal distance from the vertical boundary and the
horizontal boundary. The weight for filter output Vout is larger than the weight for
5 filter output Hout for pixels nearer to the vertical boundary. For example, the ratio of
weights for pixel PI is Vout:Hou=t 3:l. The weight for filter output Vouti s smaller
than the weight for filter output Hout for pixels nearer to the horizontal boundary.
For example, the ratio of weights for pixel P2 is Vout:Hou=t 1: 3.
[0168]
10 The block distortion can be more effectively suppressed and the image
quality can be improved by varying the weight for weighted averages depending on
the distance between each pixel and the boundary.
[0 1 691
The above-described weights are mere examples. For example, the
15 calculation section 360 may determine the weight of weighted averages for pixels
according to the edge strengths of the vertical boundary and the horizontal boundary
corresponding to each pixel instead of or in addition to the distance between each
pixel and the boundary. The edge strength may be represented with a parameter
such as an edge Value calculated from the calculation section 122 as shown in Fig. 16,
20 for example. In this case, the weight for filer output on a boundary having a
stronger edge may be set to be larger than the weight for filer output on a boundary
having a weaker edge. Varying the weight-of weighted averages according to the
edge strength can adaptively improve the effect of the deblocking filter at a boundary
that remarkably causes block distortion.
25 [0170]
The calculation section 360 selects outputs from actually filtered blocks in
terms of pixels filtered by one of the horizontal filtering block 330 and the vertical
filtering block 340. The calculation section 360 directly outputs an input pixel
value to be output to the deblocking filter 24 in terms of pixels not filtered by the
30 horizontal filtering block 330 or the vertical filtering block 340. A table in Fig. 33
lists output pixel values from the calculation section 360 according to results of the
determination whether to require the filtering.
[0171]
[5-3. Process Sequence Example]
The following describes two examples of process sequences available for
5 the deblocking filter 24 according to the working example. The example also
assumes that the deblocking filter is supplied with an image having the size of 32x32
pixels. The input image includes four macro blocks MBO through MB3 each
having the size of 16x 16 pixels.
[0 1721
10 (I) First example
For comparison, Fig. 34 illustrates a process sequence when the dependency
remains between a filtering process on the vertical boundary and a filtering process
on the horizontal boundary. The process sequence in Fig. 34 substantially equals
the process sequence in Fig. 13 according to the first working example. In Fig. 34,
15 the first step performs, in parallel, filtering need determination processes Jvo,~
through JV3,3 and JHO,0 through JH3,3 on all vertical boundaries and all horizontal
boundaries of all the four macro blocks MBO through MB3. The second step
performs filtering processes Fvo,t~h rough Fv3,3 on 16 vertical boundaries of the four
macro blocks MBO through MB3. The third step performs filtering processes FHO,0
20 through FH3,3 on 16 horizontal boundaries of the four macro blocks MBO through
MB3. The fourth step (omitted from Fig. 13) stores pixel values after the filtering
process on the horizontal boundary in the memory used for outputs from the
deblocking filter 24.
[0 1731
2 5 Fig. 35 illustrates a first example of process sequence provided by the
working example. In Fig. 35, the first step performs, in parallel, filtering need
determination processes Jvo,o through JV3,3 and JHOt,h~r ough JH3,3 on all vertical
boundaries and all horizontal boundaries of the four macro blocks MBO through
MB3. The second step performs, in parallel, filtering processes Fvo,t~h rough FV3,3
30 and FHOt,h~ro ugh FH3,3o n all vertical boundaries and all horizontal boundaries of the
four macro blocks MBO through MB3. Actually, the second step filters only a
boundary determined to require the filtering. The third step stores pixel values in
the memory used for outputs from the deblocking filter 24. A weighted average of
two filter outputs may be calculated as an output pixel value in terms of pixels
filtered by the horizontal filtering block 330 and the vertical filtering block 340.
[0 1741
(2) Second example
While the first example maximizes the parallelism, the deblocking filter 24
according to the second example can also perform a process for each macro block.
[0175]
For comparison, Fig. 36 illustrates a process sequence for each macro block
when the dependency remains between a filtering process on the vertical boundary
and a filtering process on the horizontal boundary. The process sequence in Fig. 36
substantially equals the process sequence in Fig. 14 according to the first working
example. Fig. 36 explicitly shows the four process steps (sixth, tenth, 14th, and
16th) that store pixel values in the memory for output and are omitted from Fig. 14
for simplicity. Sixteen process steps including the four process steps configure the
process in Fig. 36.
[0 1 761
Fig. 37 illustrates a second example of process sequence provided by the
working example. In Fig. 37, the first step performs, in parallel, filtering need
determination processes Jvo,~th rough Jvoz and JHO,0t hrough JH0,3o n four vertical
Z
boundaries and four horizontal boundaries of macro block MBO. The second step
performs, in parallel, filtering processes FVOt,h~ro ugh FVOa,n~d FHO,tOhr ough FHOo,~n
four vertical boundaries and four horizontal boundaries of macro block MBO. The
third step stores pixel values of macro block MBO in the memory used for outputs
from the deblocking filter 24. A weighted average of two filter outputs may be
calculated as an output pixel value in terms of pixels duplicately filtered by two
filters. The fourth to sixth steps similarly process macro block MBl. The seventh
to ninth steps similarly process macro block MB2. The tenth to twelfth steps
similarly process macro block MB3. The process in Fig. 37 includes twelve process
steps fewer than those of the process in Fig. 36.
[0 1 771
The third working example eliminates the dependency between filtering
processes for vertical boundaries and a filtering process for horizontal boundaries.
The process of the deblocking filter 24 can be performed using fewer process steps
than those used for the first and second working examples. One of advantages of
allowing a filtering process to reference only pixels input to the deblocking filter is
that any configuration of filter taps causes no dependency between filtering processes
for vertical boundaries and filtering processes for horizontal boundaries. The third
working example can improve the image quality by configuring a filter tap using
more pixels than used for existing techniques. For example, .the existing technique
uses a filter tap of three pixels for each side of each boundary as described with
reference to Fig. 7. The working example causes no dependency between processes
even if a filter tap of five pixels or more is used at each boundary. No dependency
occurs between processes even by further decreasing the block size as a process unit
for the deblocking filter.
[0178]
Also in the third working example as well as the first and second working
examples, the parallelization control section 150 may control the parallelism and
sequence of processes in the deblocking filter 24.
[0 1 791
[5-4. Process Flow]
Fig. 38 is a flowchart illustrating an example of a process flow for the
deblocking filter according to the third working example. Fig. 39 is a flowchart
illustrating a flow of the pixel value calculation process shown in Fig. 38.
[0180] .
With reference to Fig. 38, the vertical boundary determination sections 2 12-
1 through 212-11 determine in parallel whether filtering is needed for all vertical
boundaries within an input image or a macro block (step S302). The horizontal
boundary determination sections 2 14- 1 through 2 14-n determine in parallel whether
filtering is needed for all horizontal boundaries within the input image or the macro
block (step S304). Steps S302 and S304 are also performed in parallel.
[0181]
The horizontal filtering sections 332-1 through 332-n apply the deblocking
filter in parallel to all vertical boundaries determined at step S302 to require the
deblocking filter to be applied (step S306). The vertical filtering sections 342-1
through 342-11 apply the deblocking filter in parallel to all horizontal boundaries
determined at step S304 to require the deblocking filter to be applied (step S308).
Steps S306 and S308 are also performed in parallel.
[0 1 821
The calculation section 360 then performs the pixel value calculation
process as shown in Fig. 39 (step S310). With reference to Fig. 39, the process
from step S3 14 to step S326 loops for each pixel to be processed (step S3 12).
[0 1831
At step S314, the calculation section 360 determines whether two filters for
vertical boundaries and horizontal boundaries have filtered a focused pixel (step
S3 14). The process proceeds to step S322 if the two filters have filtered the focused
pixel. The process proceeds to step S316 if the two filters have not filtered the
focused pixel.
[0 1 841
At step's3 16, the calculation section 360 determines whether one of the two
filters for vertical boundaries and horizontal boundaries has filtered the focused pixel
(step S316). The process proceeds to step S320 if one of the two filters has filtered
'$
the focused pixel. The process proceeds to step S318 if none of the filters has
filtered the focused pixel.
[0185]
At step S3 18, the calculation section 360 acquires an input pixel value to the
deblocking filter 24 (step S3 18). At step S320, the calculation section 360 acquires
a filter output from the filter that actually filters the focused pixel (step S320).
[0 1 861
At step S322, the calculation section 360 determines weight values for
calculating a weighted average of filter outputs from the two filters concerning the
focused pixel according to distances from the focused pixel to the vertical boundary
and the horizontal boundary or the edge strengths of the vertical boundary and the
horizontaI boundary corresponding to the focused pixel (step S322). The
calculation section 360 calculates a weighted average of filter outputs from the two
filters using the determined weight (step S324).
5 [0187]
The calculation section 360 stores the pixel value of the focused pixel in the
memory while the pixel value is acquired at step S3 18 or S320 or is calculated at step
S324 (step S326). The sequences of processes as shown in Figs. 38 and 39
terminate when the process is performed on all pixels to be processed.
10 [0188]
<6. Application to Various Codecs>
The technology according to the disclosure is applicable to various codecs
related to image encoding and decoding. The following describes examples of
applying the technology according to the disclosure to multiview codec and scalable
15 codec.
[0189]
[6-1. Multiview Codec]
The multiview codec is an image encoding system that encodes and decodes
multiple-perspe~tive video. Fig. 40 is an explanatory diagram illustrating the
20 multiview codec. Fig. 40 illustrates sequences of frames for three views captured at
three observing points. ,Each view is provided with a view ID (view - id). One of
the views is specified as a base view. Views other than the base view are referred to
as non-base views. The example in Fig. 40 represents a base view with view ID "0"
and two non-base views with view ID "1" or "2." Encoding multiview image data
25 may compress the data size of the encoded stream as a whole by encoding frames of
the non-base view based on encoding information about frames of the base view.
[0 1 901
The deblocking filter may be applied to each view during the encoding
process and the decoding process according to the multiview codec described above.
30 When the deblocking filter is applied to each view, filtering need determination
processes on vertical block boundaries and horizontal block boundaries for each-view
may be parallelized based on the technology according to the disclosure. A filtering
need determination process and a filtering process for each view may be parallelized.
The parameter (such as the one described in the preceding paragraph 0094) to control
a filtering need determination process or a filtering process may be provided for each
5 view. A parameter provided for the base view may be reused for the non-base view.
[0191]
Filtering need determination processes or filtering processes may be
parallelized over views. Views may share the parameter (such as the one described
in the preceding paragraph 0094) to control filtering need determination processes or
10 filtering processes. It may be advantageous to additionally specify a flag indicating
whether views share the parameter.
[0 1 921
Fig. 41 is an explanatory diagram illustrating an image encoding process
applied to the multiview codec described above. Fig. 41 shows a configuration of a
15 multiview encoding device 710 as an example. The multiview encoding device 710
includes a first encoding section 720, a second encoding section 730, and a
multiplexing section 740.
[0 1931
The firit encoding section 720 encodes a base view image and generates an
20 encoded stream for the base view. The second encoding section 730 encodes a nonbase
view image and generates an encoded stream for the non-base view. The
multiplexing section 740 multiplexes an encoded stream for the base view generated
from the, first encoding section 720 and one or more encoded streams for the nonbase
view generated from the second encoding section 730 to generate a multiplexed
25 stream for multiview.
[0 1941
The first encoding section 720 and the second encoding section 730
illustrated in Fig. 41 are configured similarly to the image encoding device 10
according to the above-described embodiment. Applying the deblocking filter to
30 views enables to parallelize filtering need determination processes for vertical block
boundaries -and horizontal block boundaries or parallelize filtering need
determination processes and filtering processes. A parameter to control these
processes may be inserted into a header area of the encoded stream for each view or
into a common header area in the multiplexed stream.
[0 1 951
5 Fig. 42 is an explanatory diagram illustrating an image decoding process
applied to the multiview codec described above. Fig. 42 shows a configuration of a
multiview decoding device 760 as an example. The multiview decoding device 760
includes a demultiplexing section 770, a first decoding section 780, and a second
decoding section 790.
10 [0196]
The demultiplexing section 770 demultiplexes a multiplexed stream for
multiview into an encoded stream for the base view and an encoded stream for one or
more non-base views. The first decoding section 780 decodes a base view image
from an encoded stream for the base view. The second decoding section 730
15 decodes a non-base view image from an encoded stream for the non-base view.
[0 1 971
The first decoding section 780 and the second decoding section 790
illustrated in Fig. 42 are configured similarly to the image decoding device 60
according to the above-described embodiment. Applying the deblocking filter to
20 views enables to parallelize filtering need determination processes for vertical block
boundaries and horizontal block boundaries or parallelize filtering need
determination processes and filtering processes, A parameter to control these
processes may be acquired from a header area of the encoded stream for each view or
from a common header area in the multiplexed stream.
25 [0198]
16-2. Scalable Codec]
The scalable codec is an image encoding system to provide hierarchical
encoding. Fig. 43 is an explanatory diagram illustrating the scalable codec. Fig.
43 illustrates frame sequences for three layers of different space resolutions, time
30 resolutions, or.image qualities. Each layer is provided with a layer ID (layer - id).
These layers include a base layer having the lowest resolution (or image quality).
Layers other than the base layer are referred to as enhancement , layers. The
example in Fig. 43 represents a base layer with layer ID "0" and two enhancement
layers with layer ID "I" or "2." Encoding multi-layer image data may compress the
data size of the encoded stream as a whole by encoding frames of the enhancement
5 layer based on encoding information about frames of the base layer.
[0 1991
The deblocking filter may be applied to each layer during the encoding
process and the decoding process according to the scalable codec described above.
When the deblocking filter is applied to each .view, filtering need determination
10 processes on vertical block boundaries and horizontal block boundaries for each
layer may be parallelized based on the technology according to the disclosure. A
filtering need determination process and a filtering process for each layer may be
parallelized. The parameter (such as the one described in the preceding paragraph
0094) to control a filtering need determination process or a filtering process may be
15 provided for each layer. A parameter provided for the base layer may be reused for
the enhancement layer.
[0200]
Filtering need determination processes or filtering processes may be
parallelized ove; layers. Layers may share the parameter (such as the one described
20 in the preceding paragraph 0094) to control filtering need determination processes or
filtering processes. It qay be advantageous to additionally specify a flag indicating
whether layers share the parameter.
[0201]
Fig. 44 is an explanatory diagram illustrating an image encoding process
25 applied to the scalable codec described above. Fig. 44 shows a configuration of a
scalable encoding device 810 as an example. The scalable encoding device 810
includes a first encoding section 820, a second encoding section 830, and a
multiplexing section 840.
[0202]
30 The first encoding section 820 encodes a base layer image and generates an
encoded stream for the base layer. The second encoding section 830 encodes an
enhancement layer image and generates an encoded stream for the enhancement layer.
!
The multiplexing section 840 multiplexes an encoded stream for the base layer
generated from the first encoding section 820 and one or more encoded streams for
the enhancement layer generated from the second encoding section 830 to generate a
5 multiplexed stream for multi-layer.
[0203]
The first encoding section 820 and the second encoding section 830
illustrated in Fig. 44 are configured similarly to the image encoding device 10
according to the above-described embodiment. Applying the deblocking filter to
10 layers enables to parallelize filtering need determination processes for vertical block
boundaries and horizontal block boundaries or parallelize filtering need
determination processes and filtering processes. A parameter to control these
processes may be inserted into a header area of the encoded stream for each layer or
into a common header area in the multiplexed stream.
15 102041
Fig. 45 is an explanatory diagram illustrating an image decoding process
applied to the scalable codec described above. Fig. 45 shows a configuration of a
scalable decoding device 860 as an example. The scalable decoding device 860
includes a dem;ltiplexing section 870, a first decoding section 880, and a second
20 decoding section 890.
[0205] The demultiplexing section 870 demultiplexes a multiplexed stream for
multi-lay.er into an encoded stream for the base layer and an encoded stream for one
or more enhancement layers. The first decoding section 880 decodes a base layer
25 image from an encoded stream for the base layer. The second decoding section 830
decodes an enhancement layer image from an encoded stream for the enhancement
layer.
[0206]
The first decoding section 880 and the second decoding section 890
30 illustrated in Fig. 45 are configured similarly to the image decoding device 60
according to the above-described embodiment. Applying the deblocking filter to
, layers enables to parallelize filtering need determination processes for vertical block
boundaries and horizontal block boundaries or parallelize filtering need
determination processes and filtering processes. A parameter to control these
processes may be acquired from a header area of the encoded stream for each layer
5 or from a common header area in the multiplexed stream.
[0207]
<7. Example Application>
The image encoding device 10 and the image decoding device 60 according
to the embodiment described above may be applied to various electronic appliances
10 such as a transmitter and a receiver for satellite broadcasting, cable broadcasting such
as cable TV, distribution on the Internet, distribution to terminals via cellular
. communication, and the like, a recording device that records images in a medium
such as an optical disc, a magnetic disk or a flash memory, a reproduction device that
reproduces images from such storage medium, and the like. Four example
15 applications will be described below.
[0208]
[7-1. First Example Application]
Fig. 46 is a block diagram showing an example of a schematic configuration
of a television adopting the embodiment described above. A television 900 includes
20 an antenna 901, a tuner 902, a demultiplexer 903, a decoder 904, an video signal
processing section 905, adisplay section 906, an audio signal processing section 907,
a speaker 908, an external interface 909, a control section 910, a user interface 91 1,
and a bus 912.
[0209]
25 The tuner 902 extracts a signal of a desired channel from broadcast signals
received via the antenna 901, and demodulates the extracted signal. Then, the tuner
902 outputs an encoded bit stream obtained by demodulation to the demultiplexer
903. That is, the tuner 902 serves as transmission means of the televisions 900 for
receiving an encoded stream in which an image is encoded.
30 [0210]
The demultiplexer 903 separates a video stream and an audio stream of a
program to be viewed from the encoded bit stream, and outputs each stream which
has been separated to the decoder 904. Also, the demultiplexer 903 extracts
auxiliary data such as an EPG (Electronic Program Guide) from the encoded bit
stream, and supplies the extracted data to the control section 910. Additionally, the
demultiplexer 903 may perform descrambling in the case the encoded bit stream is
scrambled.
[02 1 11
The decoder 904 decodes the video stream and the audio stream input from
the demultiplexer 903. Then, the decoder 904 outputs video data generated by the
decoding process to the video signal processing section 905. .Also, the decoder 904
outputs the audio data generated by the decoding process to the audio signal
processing section 907.
[02 121
The video signal processing section 905 reproduces the video data input
from the decoder 904, and causes the display section 906 to display the video. The
video signal processing section 905 may also cause the display section 906 to display
an application screen supplied via a network. Further, the video signal processing
section 905 may perform an additional process such as noise removal, for example,
on the video data according to the setting. Furthermore, the video signal processing
section 905 may generate an image of a GUI (Graphical User Interface) such as a
menu, a button, a cursor or the like, for example, and superimpose the generated
image on an output image. -
[0213j ,
The display section 906 is driven by a drive signal supplied by the video
signal processing section 905, and displays a video or an image on an video screen of
a display device (for example, a liquid crystal display, a plasma display, an OLED, or
the like).
102 141
The audio signal processing section 907 performs reproduction processes
such as DIA conversion and amplification on the audio data input from the decoder
904, and outputs audio from the speaker 908. Also; the audio signal processing
section 907 may perform an additional process such as noise removal on the audio
data.
[02 151
The external interface 909 is an interface for connecting the television 900
5 and an external appliance or a network. For example, a video stream or an audio
stream received via the external interface 909 may be decoded by the decoder 904.
That is, the external interface 909 also serves as transmission means of the
televisions 900 for receiving an encoded stream in which an image is encoded.
[02 161
10 The control section 910 includes a processor such as a CPU (Central
Processing Unit), and a memory such as an RAM (Random Access Memory), an
ROM (Read Only Memory), or the like. The memory stores a program to be
executed by the CPU, program data, EPG data, data acquired via a network, and the
like. The program stored in the memory is read and executed by the CPU at the
15 time of activation of the television 900, for example. The CPU controls the
operation of the television 900 according to an operation signal input from the user
interface 91 1, for example, by executing the program.
[02 1 71
The user interface 911 is connected to the control section 910. The user
20 interface 91 1 includes a button and a switch used by a user to operate the television
900, and a receiving seftion for a remote control signal, for example. The user
interface 9 11 detects an operation of a user via these structural elements, generates an
operation signal, and outputs the generated operation signal to the control section 910.
[02 1 81
25 The bus 912 interconnects the tuner 902, the demultiplexer 903, the decoder
904, the video signal processing section 905, the audio signal processing section 907,
the external interface 909, and the control section 910.
[02 191
In the television 900 configured in this manner, the decoder 904 has a
30 function of the image decoding device 60 according to the embodiment described
above. Accordingly, also in the case of the image decoding in the television 900, it
is possible to enhance the parallelism of deblocking filter processes and ensure highspeed
processing.
[0220]
[7-2. Second Example Application]
5 Fig. 47 is a block diagram showing an example of a schematic configuration
of a mobile phone adopting the embodiment described above. A mobile phone 920
includes an antenna 921, a communication section 922, an audio codec 923, a
speaker 924, a microphone 925, a camera section 926, an image processing section
927, a demultiplexing section 928, a recording/reproduction section 929, a display
10 section 930, a control section 931, an operation section 932, and a bus 933.
[022 11
The antenna 921 is connected to the communication section 922. The
speaker 924 and the microphone 925 are connected to the audio codec 923. The
operation section 932 is connected to the control section 931. The bus 933
15 interconnects the communication section 922, the audio codec 923, the camera
section 926, the image processing section 927, the demultiplexing section 928, the
recording/reproduction section 929, the display section 930, and the control section
93 1.
[0222]
20 The mobile phone 920 performs operation such as transmission~receptiono f
audio signal, transmissi,onlreception of emails or image data, image capturing,
recording of data, and the like, in various operation modes including an audio
communication mode, a data communication mode, an image capturing mode, and a
videophone mode.
25 [0223]
In the audio communication mode, an analogue audio signal generated by
the microphone 925 is supplied to the audio codec 923. The audio codec 923
converts the analogue audio signal into audio data, and AD converts and compresses
the converted audio data. Then, the audio codec 923 outputs the compressed audio
30 data to the communication section 922. The communication section 922 encodes
and modulates the audio data, and generates a transmission signal. Then, the
communication section 922 transmits the generated transmission signal to a base
station (not shown) via the antenna 921. Also, the communication section 922
amplifies a wireless signal received via the antenna 921 and converts the frequency
of the wireless signal, and acquires a received signal. Then, the communication
5 section 922 demodulates and decodes the received signal and generates audio data,
and outputs the generated audio data to the audio codec 923. The audio codec 923
extends and DIA converts the audio data, and generates an analogue audio signal.
Then, the audio codec 923 supplies the generated audio signal to the speaker 924 and
causes the audio to be output.
10 [0224]
Also, in the data communication mode, the control section 931 generates
text data that makes up an email, according to an operation of a user via the operation
section 932, for example. Moreover, the control section 931 causes the text to be
displayed on the display section 930. Furthermore, the control section 931
15 generates email data according to a transmission instruction of the user via the
operation section 932, and outputs the generated email data to the communication
section 922. Then, the communication section 922 encodes and modulates the
email data, and generates a transmission signal. Then, the communication section
922 transmits thk generated transmission signal to a base station (not shown) via the
20 antenna 921. Also, the communication section 922 amplifies a wireless signal
received via the antennay921 and converts the frequency of the wireless signal, and
acquires a received signal. Then, the communication section 922 demodulates and
decodes the received signal, restores the email data, and outputs the restored email
data to the control section 931. The control section 93 1 causes the display section
25 930 to display the contents of the email, and also, causes the email data to be stored
in the storage medium of the recordinglreproduction section 929.
[0225 1
The recording/reproduction section 929 includes an arbitrary readable and
writable storage medium. For example, the storage medium may be a built-in
30 storage medium such as an RAM, a flash memory or the like, or an externally
mounted storage medium such as a hard disk, a magnetic disk, a magneto-optical
disk, an optical disc, an USB memory, a memory card, or the like.
[0226]
Furthermore, in the image capturing mode, the camera section 926 captures
an image of a subject, generates image data, and outputs the generated image data to
5 the image processing section 927, for example. The image processing section 927
encodes the image data input from the camera section 926, and causes the encoded
stream to be stored in the storage medium of the recording/reproduction section 929.
[0227]
Furthermore, in the videophone mode, the demultiplexing section 928
10 multiplexes a video stream encoded by the image processing section 927 and an
audio stream input from the audio codec 923, and outputs the multiplexed stream to
the communication section 922, for example. The communication section 922
encodes and modulates the stream, and generates a transmission signal. Then, the
communication section 922 transmits the generated transmission signal to a base
15 station (not shown) via the antenna 921. Also, the communication section 922
amplifies a wireless signal received via the antenna 921 and converts the frequency
of the wireless signal, and acquires a received signal. These transmission signal and
received signal may include an encoded bit stream. Then, the communication
section 922 deniodulates and decodes the received signal, restores the stream, and
20 outputs the restored stream to the demultiplexing section 928. The demultiplexing
section 928 separates a video stream and an audio stream from the input stream, and
outputs the video stream to the image processing section 927 and the audio stream to
the audio codec 923. The image processing section 927 decodes the video stream,
and generates video data. The video data is supplied to the display section 930, and
25 a series of images is displayed by the display section 930. The audio codec 923
extends and D/A converts the audio stream, and generates an analogue audio signal.
Then, the audio codec 923 supplies the generated audio signal to the speaker 924 and
causes the audio to be output.
[0228]
30 In the mobile phone 920 configured in this manner, the image processing
section 927 has a function of the image encoding device 10 and the image decoding
device 60 according to the embodiment described above. Accordingly, also in the
case of the image decoding and encoding in the mobile phone 920, it is possible to
enhance the parallelism of deblocking filter processes and ensure high-speed
processing.
5 [0229]
[7-3. Third Example Application]
Fig. 48 is a block diagram showing an example of a schematic configuration
of a recording/reproduction device adopting the embodiment described above. A
recording/reproduction device 940 encodes, and records in a recording medium,
10 audio data and video data of a received broadcast program, for example. The
recording/reproduction device 940 may also encode, and record in the recording
medium, audio data and video data acquired from another device, for example.
Furthermore, the recording/reproduction device 940 reproduces, using a monitor or a
speaker, data recorded in the recording medium, according to an instruction of a user,
15 for example. At this time, the recording/reproduction device 940 decodes the audio
data and the video data.
[0230]
The recording/reproduction device 940 includes a tuner 941, an external
interface 942, an encoder 943, an HDD (Hard Disk Drive) 944, a disc drive 945, a
20 selector 946, a decoder 947, an OSD (On-Screen Display) 948, a control section 949,
and a user interface 950.
[023 11
The tuner 941 extracts a signal of a desired channel from broadcast signals
received via an antenna (not shown), and demodulates the extracted signal. Then,
25 the tuner 941 outputs an encoded bit stream obtained by demodulation to the selector
946. That is, the tuner 941 serves as transmission means of the
recording/reproduction device 940.
[0232]
The external interface 942 is an interface for connecting the
30 recording/reproduction device 940 and an external appliance or a network. For
example, the external interface 942 may be an IEEE 1394 interface, a network
interface, an USB interface, a flash memory interface, or the like. , For example,
video data and audio data received by the external interface 942 are input to the
encoder 943. That is, the external interface 942 serves as transmission means of the
recording/reproduction device 940.
5 [0233]
In the case the video data and the audio data input from the external
interface 942 are not encoded, the encoder 943 encodes the video data and the audio
data. Then, the encoder 943 outputs the encoded bit stream to the selector 946.
[0234]
10 The HDD 944 records in an internal hard disk an encoded bit stream, which
is compressed content data of a video or audio, various programs, and other pieces of
data. Also, the HDD 944 reads these pieces of data from the hard disk at the time
of reproducing a video or audio.
[023 51
15 The disc drive 945 records or reads data in a recording medium that is
mounted. A recording medium that is mounted on the disc drive 945 may be a
DVD disc (a DVD-Video, a DVD-RAM, a DVD-R, a DVD-RW, a DVD+, a
DVD+RW, or the like), a Blu-ray (registered trademark) disc, or the like, for example.
[023 61
20 The selector 946 selects, at the time of recording a video or audio, an
encoded bit stream input from the tuner 941 or the encoder 943, and outputs the
Y
Selected encoded bit stream to the HDD 944 or the -disc drive 945. Also, the
selector 946 outputs, at the time of reproducing a video or audio, an encoded bit
stream input from the HDD 944 or the disc drive 945 to the decoder 947.
The decoder 947 decodes the encoded bit stream, and generates video data
and audio data. Then, the decoder 947 outputs the generated video data to the OSD
948. Also, the decoder 904 outputs the generated audio data to an external speaker.
[023 81
3 0 The OSD 948 reproduces the video data input from the decoder 947, and
displays a video. Also, the OSD 948 may superimpose an image of a GUI, such as
a menu, a button, a cursor or the like, for example, on a displayed video.
[0239]
The control section 949 includes a processor such as a CPU, and a memory
such as an RAM or an ROM. The memory stores a program to be executed by the
5 CPU, program data, and the like. A program stored in the memory is read and
executed by the CPU at the time of activation of the recording/reproduction device
940, for example. The CPU controls the operation of the recording/reproduction
device 940 according to an operation signal input from the user interface 950, for
example, by executing the program.
The user interface 950 is connected to the control section 949. The user
interface 950 includes a button and a switch used by a user to operate the
recording/reproduction device 940, and a receiving section for a remote control - .
signal, for example. The user interface 950 detects an operation of a user via these
15 structural elements, generates an operation signal, and outputs the generated
operation signal to the control section 949.
[024 11
In the recording/reproduction device 940 configured in this manner, the
encoder 943 has a function of the image encoding device 10 according to the
20 embodiment described above. Also, the decoder 947 has a function of the image
decoding device 60 according to the embodiment described above. Accordingly,
'4
also. in the case of the image decoding and encoding in the recording/reproduction
device 940, it is possible to enhance the parallelism of deblocking filter processes
and ensure high-speed processing.
25 [0242]
[7-4. Fourth Example Application]
Fig. 49 is a block diagram showing an example of a schematic configuration
of an image capturing device adopting the embodiment described above. An image
capturing device 960 captures an image of a subject, generates an image, encodes the
30 image data, and records the image data in a recording medium.
The image capturing device 960 includes an optical block 961, an image
capturing section 962, a signal processing section 963, an image processing section
964, a display section 965, an external interface 966, a memory 967, a media drive
968, an OSD 969, a control section 970, a user interface 971, and a bus 972.
5 [0244]
The optical block 961 is connected to the image capturing section 962.
The image capturing section 962 is connected to the signal processing section 963.
The display section 965 is connected to the image processing section 964. The user
interface 971 is connected to the control section 970. The bus 972 interconnects the
10 image processing section 964, the external interface 966, the memory 967, the media
drive 968, the OSD 969, and the control section 970.
[0245]
The optical block 961 includes a focus lens, an aperture stop mechanism,
and the like. The optical block 961 forms an optical image of a subject on an image
15 capturing surface of the image capturing section 962. The image capturing section
962 includes an image sensor such as a CCD, a CMOS or the like, and converts by
photoelectric conversion the optical image formed on the image capturing surface
into an image signal which is an electrical signal. Then, the image capturing
section 962 outputs the image signal to the signal processing section 963.
20 [0246]
The signal processing section 963 performs various camera signal processes,
such as knee correction, gamma correction, color correction and the like, on the
image signal input from the image capturing section 962. The signal processing
section 963 outputs the image data after the camera signal process to the image
25 processing section 964.
[0247]
The image processing section 964 encodes the image data input from the
signal processing section 963, and generates encoded data. Then, the image
processing section 964 outputs the generated encoded data to the external interface
30 966 or the media drive 968. Also, the image processing section 964 decodes
encoded data input from the external interface 966 or the media drive 968, and
generates image data. Then, the image processing section 964 outputs the
generated image data to the display section 965. Also, the image processing section
964 may output the image data input from the signal processing section 963 to the
display section 965, and cause the image to be displayed. Furthermore, the image
5 processing section 964 may superimpose data for display acquired from the OSD 969
on an image to be output to the display section 965.
[0248]
The OSD 969 generates an image of a GUI, such as a menu, a button, a
cursor or the like, for example, and outputs the generated image to the image
10 processing section 964.
[0249]
The external interface 966 is configured as an USB inputloutput terminal,
for example. The external interface 966 connects the image capturing device 960
and a printer at the time of printing an image, for example. Also, a drive is
15 connected to the external interface 966 as necessary. A removable medium, such as
a magnetic disk, an optical disc or the like, for example, is mounted on the drive, and
a program read from the removable medium may be installed in the image capturing
device 960. Furthermore, the external interface 966 may be configured as a
network interfade to be connected to a network such as a LAN, the Internet or the
20 like. That is, the external interface 966 serves as transmission means of the image
capturing device 960.
[0250]
A recording medium to be mounted on the media drive 968 may be an
arbitrary readable and writable removable medium, such as a magnetic disk, a
25 magneto-optical disk, an optical disc, a semiconductor memory or the like, for
example. Also, a recording medium may be fixedly mounted on the media drive
968, configuring a non-transportable storage section such as a built-in hard disk drive
or an SSD (Solid State Drive), for example.
[025 11
30 The control section 970 includes a processor such as a CPU, and a memory
such as an RAM or an ROM. The memory stores a program to be executed by the
CPU, program data, and the like. A program stored in the memory is read and
executed by the CPU at the time of activation of the image capturing device 960, for
example. The CPU controls the operation of the image capturing device 960
according to an operation signal input from the user interface 971, for example, by
5 executing the program.
102521
The user interface 971 is connected to the control section 970. The user
interface 971 includes a button, a switch and the like used by a user to operate the
image capturing device 960, for example. The user interface 971 detects an
10 operation of a user via these structural elements, generates an operation signal, and
outputs the generated operation signal to the control section 970.
[0253]
In the image capturing device 960 configured in this manner, the image
processing section 964 has a function of the image encoding device 10 and the image
15 decoding device 60 according to the embodiment described above. Accordingly, in
the case of the image decoding and encoding in the image capturing device 960, it is
possible to enhance the parallelism of deblocking filter processes and ensure highspeed
processing.
[0254]
With reference to Figs. 1 through 49, there have been described three
Y
working examples of the deblocking filters for the image encoding device 10 and the
image decoding device 60 according to an embodiment. The three working
examples use pixel values input to the deblocking filters to determine whether the
25 deblocking filters for a vertical boundary and a horizontal boundary need to be
applied or not. The determination processes can be performed in parallel
independently of filtering process results. The dependency of processes between
macro blocks is eliminated to enable parallelization of processes on macro blocks.
As a result, it is possible to avoid delay or data rate degradation due to a large
30 processing amount of the deblocking filters and ensure high-speed processing. The
parallelism and sequences of deblocking filter processes can be flexibly configured
according to various conditions such as image sizes or installation environment.
[0255]
According to the first working example, the determination of whether to
apply the deblocking filter to vertical boundaries uses pixel values of pixels
5 belonging to rows to which the definition for horizontal boundaries is not applied.
The determination of whether to apply the deblocking filter to horizontal boundaries
uses pixel values of pixels belonging to rows to which the definition for vertical
boundaries is not applied. The filtering process on a block does not update a pixel
value used for the filtering need determination process on another block. Pixel
10 values input to the deblocking filter need not be stored in extra memory even if a
filtering need determination process on a given block follows a filtering process on
another block. This can save hardware costs needed for the apparatus installation.
[0256]
According to the second working example, the memory not updated by the
15 filtering process stores a pixel value input to the deblocking filter. The filtering
need determination processes reference the input pixel value. In this case, the
filtering need determination processes reference a pixel whose position is free from
restrictions. This enables to use flexible determination conditions appropriate to
various purposes such as more accurate determination of filtering necessity or the
20 determination with reduced processing costs.
[0257]
According to the third working example, filtering processes for vertical
boundaries and horizontal boundaries filter pixels input to the deblocking filter.
This configuration can parallelize filtering processes for vertical boundaries and
25 horizontal boundaries each other. This can further accelerate processes of the
deblocking filter. An output pixel value is calculated based on two filter outputs in
terms of a pixel updated by two filtering processes performed in parallel.
Parallelizing two filtering processes can also appropriately reduce block distortion
appearing on the vertical boundary and the horizontal boundary. An output pixel
30 value can be calculated as a weighted average of two filter outputs. This can allow
the deblocking- filter to more effectively eliminate the block distortion and further
improve the image quality.
The specification has mainly described examples where filtering processes
for vertical boundaries precedes filtering processes for horizontal boundaries. In
5 addition, the above-described effects of the technology according to the disclosure
are equally available to a case where filtering processes for horizontal boundaries
precede filtering processes for vertical boundaries. The deblocking filter processing
unit or the macro block may be sized otherwise than described in the specification.
An available technique may omit the filtering need determination processes and
10 parallelize application of the deblocking filter to vertical boundaries and horizontal
boundaries.
[0259]
A technique of transmitting information used for deblocking filter process
parallelization from the encoding side to the decoding side is not limited to the
15 technique of multiplexing the information into the encoded stream header. For
example, the information may not be multiplexed into an encoded bit stream but may
be transmitted or recorded as separate data associated with the encoded bit stream.
The term "association" signifies ensuring possibility of linking an image (or part of
an image such as a slice or a block) contained in the bit stream with information
20 corresponding to the image. Namely, the information may be transmitted over a
transmission path different from that used for images (or bit streams). The
information may be recorded on a recording medium (or a different recording area on
the same recording medium) different from that used for images (or bit streams).
The information and the image (or bit stream) may be associated with each other
25 based on any units such as multiple frames, one frame, or part of a frame.
[0260]
The preferred embodiments of the present invention have been described
above with reference to the accompanying drawings, whilst the present invention is
not limited to the above examples, of course. A person skilled in the art may find
30 various alternations and modifications within the scope of the appended claims, and
it should be understood that they will naturally come under the technical scope of the
present invention.
[026 11
The specification represents filtering processes for vertical boundaries as
"horizontal filtering" and filtering processes for horizontal boundaries as "vertical
5 filtering." Generally, filtering processes for vertical boundaries uses horizontally
positioned filter taps. Filtering processes for horizontal boundaries uses vertically
positioned filter taps. For this reason, the above-described nomenclature is used for
the filtering processes.
10 Reference Signs List
[0262]
10,60 image processing device
1 12-1 to 1 12-n, 2 12-1 to 212-n first determination section (vertical boundary
determination section)
15 11 4-1 to 11 4-n, 2 14- 1 to 2 14-n second determination section (horizontal
boundary determination section)
132-1 to 132-11, 332-1 to 332-n first filtering section (horizontal filtering section)
142-1 to 142-n, 342-1 to 342-n second filtering section (vertical filtering section)
150 parallelization control section
20 208 line memory (memory)
360 calculation section

Claim 1 (Amended)
An image processing device comprising:
a decoding section configured to decode an encoded stream to generate an
image; and
5 a determination section configured to perform horizontal filtering
determination processes in parallel per a parallel processing unit of a plurality of
vertical block boundaries among a plurality of blocks within the image generated by
the decoding section, each horizontal filtering determination process determining
whether to apply a deblocking filter to neighboring blocks neighboring across each
10 of the plurality of vertical block boundaries.
Claim 2 (Amended)
The image processing device according to claim 1, further comprising a
control section configured to cause the determination section to set the plurality of
15 vertical block boundaries as the parallel processing unit in the horizontal filtering
determination processes.
Claim 3 (Amended)
The &age processing device according to claim 2, wherein the control
20 section causes the determination section to perform the horizontal filtering
determination process fqr each vertical block boundary without depending on the
horizontal filtering determination processes for other vertical block boundaries.
Claim 4 (Amended)
2 5 The image processing device according to claim 3, wherein the control
section causes the determination section to use a block size of each block, the block
size posing no dependency between the horizontal filtering determination processes
for the vertical block boundaries different from each other.
30 Claim 5 (Amended)
The image processing device according to claim 4, wherein the
determination processes in parallel and thereafter causes the determination section to
perform the vertical filtering determination processes in parallel.
Claim 11 (Amended)
5 The image processing device according to claim 10,
wherein the encoded stream is encoded per a unit of hierarchized block, and
wherein the decoding section decodes the encoded stream per the unit of
hierarchized block.
10 Claim 12 (Amended)
An image processing method comprising:
decoding an encoded stream to generate an image; and
performing horizontal filtering determination processes in parallel per a
parallel processing unit of a plurality of vertical block boundaries among a plurality
15 of blocks within a generated image, each horizontal filtering determination process
determining whether to apply a deblocking filter to neighboring blocks neighboring
across each of the plurality of vertical block boundaries. .
Claim 13 (~ddeh)
20 An image processing device comprising:
a determinatio? section configured to perform horizontal filtering
determination processes in parallel per a parallel processing unit of a plurality of
vertical block boundaries among a plurality of blocks within an image locally
decoded when encoding an image to be encoded, each horizontal filtering
25 determination process determining whether to apply a deblocking filter to
neighboring blocks neighboring across each of the plurality of vertical block
boundaries; and
an encoding section configured to encode the image to be encoded using an
image filtered based on results of the horizontal filtering determination processes.
30
Claim 14 (Added)
The image processing device according to claim 13, further , comprising a
control section configured to cause the determination section to set the plurality of
vertical block boundaries as the parallel processing unit in the horizontal filtering
determination processes.
5
Claim 15 (Added)
The image processing device according to claim 14, wherein the control
section causes the determination section to perform the horizontal filtering
determination process for each vertical block boundary without depending on the
10 horizontal filtering determination processes for other vertical block boundaries.
Claim 16 (Added)
The image processing device according to claim 15, wherein the control
section causes the determination section to use a block size of each block, the block
15 size posing no dependency between the horizontal filtering determination processes
for the vertical block boundaries different from each other.
Claim 17 (Added)
The &age processing device according to claim 16, wherein the
20 determination section performs vertical filtering determination processes in parallel
per a parallel processing unit of a plurality of horizontal block boundaries among a
plurality of blocks, each vertical filtering determination process determining whether
to apply. a deblocking filter to neighboring blocks neighboring across each of the
plurality of horizontal block boundaries.
2 5
Claim 18 (Added)
The image processing device according to claim 17, wherein the control
section causes the determination section to set the plurality of horizontal block
boundaries as a parallel processing unit in the vertical filtering determination
30 processes.
Claim 19 (Added)
The image processing device according to claim 18, wherein the control
section causes the determination section to perform the vertical filtering
determination process for each horizontal block boundary without depending on the
5 vertical filtering determination processes for other horizontal block boundaries.
Claim 20 (Added)
The image processing device according to claim 19, wherein the control
section causes the determination section to use a block size of each block, the block
10 size posing no dependency between the vertical filtering deterhination processes for
the horizontal block boundaries different from each other.
Claim 21 (Added)
The image processing device according to claim 20, wherein the control
15 section causes the determination section to use 8x8 pixels as the block size.
Claim 22 (Added)
The image processing device according to claim 21, wherein the control
section causes the determination section to perform the horizontal filtering
20 determination processes in parallel and thereafter causes the determination section to
perform the vertical filtering determination processes in parallel.
Claim 23 (Added)
The image processing device according to claim 22,
2 5 wherein the encoding section encodes the image to be encoded per a unit of
hierarchized block, and
wherein the image to be encoded is locally decoded per the unit of
hierarchized block.
30 Claim 24 (Added)
An image processing method comprising:
performing horizontal filtering determination processes in parallel per a
parallel processing unit of a plurality of vertical block boundaries among a plurality
of blocks within an image locally decoded when encoding an image to be encoded,
each horizontal filtering determination process determining whether to apply a
5 deblocking filter to neighboring blocks neighboring across each of the plurality of
vertical block boundaries; and
encoding the image to be encoded using an image filtered based on results
of the horizontal filtering determination processes: