Decoder, Encoder and Method for Informed Loudness Estimation
in Object-Based Audio Coding Systems
Description
The present invention relates to audio signal encoding, processing and decoding , and, in
particular, to a decoder, an encoder and method for informed loudness estimation in
object-based audio coding systems.
Recently, parametric techniques for bitrate-efficient transmission/storage of audio scenes
comprising multiple audio object signals have been proposed in the field of audio coding
[BCC, JSC, SAOC, SAOC1 , SAOC2] and informed source separation [ISS1 , ISS2, ISS3,
ISS4, ISS5, ISS6]. These techniques aim at reconstructing a desired output audio scene
or audio source object based on additional side information describing the
transmitted/stored audio scene and/or source objects in the audio scene. This
reconstruction takes place in the decoder using an informed source separation scheme.
The reconstructed objects may be combined to produce the output audio scene.
Depending on the way the objects are combined, the perceptual loudness of the output
scene may vary.
In TV and radio broadcast, the volume levels of the audio tracks of various programs may
be normalized based on various aspects, such as the peak signal level or the loudness
level. Depending on the dynamic properties of the signals, two signals with the same peak
level may have a widely differing level of perceived loudness. Now switching between
programs or channels the differences in the signal loudness are very annoying and have
been to be a major source for end-user complaints in broadcast.
In the prior art, it has been proposed to normalize all the programs on all channels
similarly to a common reference level using a measure based on perceptual signal
loudness. One such recommendation in Europe is the EBU Recommendation R 128 [EBU]
(later referred to as R 128).
The recommendation says that the "program loudness", e.g. , the average loudness over
one program (or one commercial, or some other meaningful program entity) should equal
a specified level (with small allowed deviations). When more and more broadcasters
comply with this recommendation and the required normalization, the differences in the
average loudness between programs and channels should be minimized.
Loudness estimation can be performed in several ways. There exist several mathematical
models for estimating the perceptual loudness of an audio signal. The EBU
recommendation R 128 relies on the model presented in ITU-R BS. 1770 (later referred to
as BS. 1770) see [ITU]) for the loudness estimation.
As stated before, e.g ., according to the EBU Recommendation R 128, the program
loudness, e.g. , the average loudness over one program should equal a specified level with
small allowed deviations. However, this leads to significant problems when audio
rendering is conducted, unsolved until now in the prior art. Conducting audio rendering on
the decoder side has a significant effect on the overall/total loudness of the received audio
input signal. However, despite scene rendering is conducted, the total loudness of the
received audio signal shall remain the same.
Currently, no specific decoder-side solution exists for this problem.
EP 2 146 522 A 1 ([EP]), relates to concepts for generating audio output signals using
object based metadata. At least one audio output signal is generated representing a
superposition of at least two different audio object signals, but does not provide a solution
for this problem.
WO 2008/035275 A2 ([BRE]) describes an audio system comprising an encoder which
encodes audio objects in an encoding unit that generates a down-mix audio signal and
parametric data representing the plurality of audio objects. The down-mix audio signal and
parametric data is transmitted to a decoder which comprises a decoding unit which
generates approximate replicas of the audio objects and a rendering unit which generates
an output signal from the audio objects. The decoder furthermore contains a processor for
generating encoding modification data which is sent to the encoder. The encoder then
modifies the encoding of the audio objects, and in particular modifies the parametric data,
in response to the encoding modification data. The approach allows manipulation of the
audio objects to be controlled by the decoder but performed fully or partly by the encoder.
Thus, the manipulation may be performed on the actual independent audio objects rather
than on approximate replicas thereby providing improved performance.
EP 2 146 522 A 1 ([SCH]) discloses an apparatus for generating at least one audio output
signal representing a superposition of at least two different audio objects comprises a
processor for processing an audio input signal to provide an object representation of the
audio input signal, where this object representation can be generated by a parametrically
guided approximation of original objects using an object downmix signal. An object
manipulator individually manipulates objects using audio object based metadata referring
to the individual audio objects to obtain manipulated audio objects. The manipulated audio
objects are mixed using an object mixer for finally obtaining an audio output signal having
one or several channel signals depending on a specific rendering setup.
WO 2008/046531 A 1 ([ENG]) describes an audio object coder for generating an encoded
object signal using a plurality of audio objects includes a downmix information generator
for generating downmix information indicating a distribution of the plurality of audio objects
into at least two downmix channels, an audio object parameter generator for generating
object parameters for the audio objects, and an output interface for generating the
imported audio output signal using the downmix information and the object parameters.
An audio synthesizer uses the downmix information for generating output data usable for
creating a plurality of output channels of the predefined audio output configuration.
It would be desirable to have an accurate estimate of the output average loudness or the
change in the average loudness without a delay and when the program does not change
or the rendering scene is not changed, the average loudness estimate should also remain
static.
The object of the present invention is to provide improved audio signal encoding,
processing and decoding concepts. The object of the present invention is solved by a
decoder according to claim 1, by an encoder according to claim 15 , by a system according
to claim 18, by a method according to claim 19 , by a method according to claim 20 and by
a computer program according to claim 2 1.
An informed way for estimating the loudness of the output in an object-based audio coding
system is provided. The provided concepts rely on information on the loudness of the
objects in the audio mixture to be provided to the decoder. The decoder uses this
information along with the rendering information for estimating the loudness of the output
signal. This allows then, for example, to estimate the loudness difference between the
default downmix and the rendered output. It is then possible to compensate for the
difference to obtain approximately constant loudness in the output regardless of the
rendering information. The loudness estimation in the decoder takes place in a fully
parametric manner, and it is computationally very light and accurate in comparison to
signal-based loudness estimation concepts.
Concepts for obtaining information on the loudness of the specific output scene using
purely parametric concepts are provided, which then allows for loudness processing
without explicit signal-based loudness estimation in the decoder. Moreover, the specific
technology of Spatial Audio Object Coding (SAOC) standardized by MPEG [SAOC] is
described, but the provided concepts can be used in conjunction with other audio object
coding technologies, too.
A decoder for generating an audio output signal comprising one or more audio output
channels is provided. The decoder comprises a receiving interface for receiving an audio
input signal comprising a plurality of audio object signals, for receiving loudness
information on the audio object signals, and for receiving rendering information indicating
whether one or more of the audio object signals shall be amplified or attenuated.
Moreover, the decoder comprises a signal processor for generating the one or more audio
output channels of the audio output signal. The signal processor is configured to
determine a loudness compensation value depending on the loudness information and
depending on the rendering information. Furthermore, the signal processor is configured
to generate the one or more audio output channels of the audio output signal from the
audio input signal depending on the rendering information and depending on the loudness
compensation value.
According to an embodiment, the signal processor may be configured to generate the one
or more audio output channels of the audio output signal from the audio input signal
depending on the rendering information and depending on the loudness compensation
value, such that a loudness of the audio output signal is equal to a loudness of the audio
input signal, or such that the loudness of the audio output signal is closer to the loudness
of the audio input signal than a loudness of a modified audio signal that would result from
modifying the audio input signal by amplifying or attenuating the audio object signals of
the audio input signal according to the rendering information.
According to another embodiment, each of the audio object signals of the audio input
signal may be assigned to exactly one group of two or more groups, wherein each of the
two or more groups may comprise one or more of the audio object signals of the audio
input signal. In such an embodiment, the receiving interface may be configured to receive
a loudness value for each group of the two or more groups as the loudness information,
wherein said loudness value indicates an original total loudness of the one or more audio
object signals of said group. Furthermore, the receiving interface may be configured to
receive the rendering information indicating for at least one group of the two or more
groups whether the one or more audio object signals of said group shall be amplified or
attenuated by indicating a modified total loudness of the one or more audio object signals
of said group. Moreover, in such an embodiment, the signal processor may be configured
to determine the loudness compensation value depending on the modified total loudness
of each of said at least one group of the two or more groups and depending on the original
total loudness of each of the two or more groups. Furthermore, the signal processor may
be configured to generate the one or more audio output channels of the audio output
signal from the audio input signal depending on the modified total loudness of each of said
at least one group of the two or more groups and depending on the loudness
compensation value.
In particular embodiments, at least one group of the two or more groups may comprise
two or more of the audio object signals.
Moreover an encoder is provided. The encoder comprises an object-based encoding unit
for encoding a plurality of audio object signals to obtain an encoded audio signal
comprising the plurality of audio object signals. Furthermore, the encoder comprises an
object loudness encoding unit for encoding loudness information on the audio object
signals. The loudness information comprises one or more loudness values, wherein each
of the one or more loudness values depends on one or more of the audio object signals.
According to an embodiment, each of the audio object signals of the encoded audio signal
may be assigned to exactly one group of two or more groups, wherein each of the two or
more groups comprises one or more of the audio object signals of the encoded audio
signal. The object loudness encoding unit may be configured to determine the one or
more loudness values of the loudness information by determining a loudness value for
each group of the two or more groups, wherein said loudness value of said group
indicates an original total loudness of the one or more audio object signals of said group.
Furthermore, a system is provided. The system comprises an encoder according to one of
the above-described embodiments for encoding a plurality of audio object signals to
obtain an encoded audio signal comprising the plurality of audio object signals, and for
encoding loudness information on the audio object signals. Moreover, the system
comprises a decoder according to one of the above-described embodiments for
generating an audio output signal comprising one or more audio output channels. The
decoder is configured to receive the encoded audio signal as an audio input signal and
the loudness information. Moreover, the decoder is configured to further receive rendering
information. Furthermore, the decoder is configured to determine a loudness
compensation value depending on the loudness information and depending on the
rendering information. Moreover, the decoder is configured to generate the one or more
audio output channels of the audio output signal from the audio input signal depending on
the rendering information and depending on the loudness compensation value.
Moreover, a method for generating an audio output signal comprising one or more audio
output channels is provided. The method comprises:
Receiving an audio input signal comprising a plurality of audio object signals.
Receiving loudness information on the audio object signals.
- Receiving rendering information indicating whether one or more of the audio object
signals shall be amplified or attenuated.
Determining a loudness compensation value depending on the loudness
information and depending on the rendering information. And:
Generating the one or more audio output channels of the audio output signal from
the audio input signal depending on the rendering information and depending on
the loudness compensation value.
Furthermore, a method for encoding is provided. The method comprises:
Encoding an audio input signal comprising a plurality of audio object signals. And:
Encoding loudness information on the audio object signals, wherein the loudness
information comprises one or more loudness values, wherein each of the one or
more loudness values depends on one or more of the audio object signals.
Moreover, a computer program for implementing the above-described method when being
executed on a computer or signal processor is provided.
Preferred embodiments are provided in the dependent claims.
In the foliowing, embodiments of the present invention are described in more detail with
reference to the figures, in which:
Fig. 1 illustrates a decoder for generating an audio output signal comprising one
or more audio output channels according to an embodiment,
Fig. 2 illustrates an encoder according to an embodiment,
Fig. 3 illustrates a system according to an embodiment,
Fig. 4 illustrates a Spatial Audio Object Coding system comprising an SAOC
encoder and a SAOC decoder,
Fig. 5 illustrates an SAOC decoder comprising a side information decoder, an
object separator and a renderer,
Fig. 6 illustrates a behavior of output signal loudness estimates on a loudness
change,
Fig. 7 depicts informed loudness estimation according to an embodiment,
illustrating components of an encoder and a decoder according to an
embodiment,
Fig. 8 illustrates an encoder according to another embodiment,
Fig. 9 illustrates an encoder and a decoder according to an embodiment related
to the SAOC-Dialog Enhancement, which comprises bypass channels,
Fig. 10 depicts a first illustration of a measured loudness change and the result of
using the provided concepts for estimating the change in the loudness in a
paramedical manner,
Fig . 11 depicts a second illustration of a measured loudness change and the result
of using the provided concepts for estimating the change in the loudness in
a parametrical manner, and
Fig . 12 illustrates another embodiment for conducting loudness compensation.
Before preferred embodiments are described in detail, loudness estimation, Spatial Audio
Object Coding (SAOC) and Dialogue Enhancement (DE) are described.
At first, loudness estimation is described.
As already stated before, the EBU recommendation R 1 8 relies on the model presented
in ITU-R BS. 770 for the loudness estimation. This measure will be used as an example,
but the described concepts below can be applied also for other loudness measures.
The operation of the loudness estimation according to BS. 1770 is relatively simple and it
is based on the following main steps [ITU]:
The input signal x (or signals in the case of multi-channel signal) is filtered with a Kfilter
(a combination of a shelving and a high-pass filters) to obtain the signal(s) yt .
The mean squared energy z of the signal y t is calculated.
In the case of multi-channel signal, channel weighting G is applied, and the
weighted signals are summed. The loudness of the signal is then defined to be
= c + \ \ g G, ,
with the constant value c - - 0.6 1. The output is then expressed in the units of
"LKFS" (Loudness, K-weighted, relative to Full Scale) which scales similarly to the
decibel scale.
In the above formula, G, may. for example, be equal to 1 for some of the channels, while
G, may, for example, be 1.41 for some other channels. For example, if a left channel, a
right channel, a center channel, a left surround channel and a right surround channel is
considered, the respective weights G, may, for example, be 1 for the left, right and center
channel, and may, for example, be 1 4 1 for the left surround channel and the right
surround channel, see [ITU].
It can be seen, that the loudness value L is closely related to the logarithm of the signal
energy.
n the following, Spatial Audio Object Coding is described.
Object-based audio coding concepts allow for much flexibility in the decoder side of the
chain. An example of an object-based audio coding concept is Spatial Audio Object
Coding (SAOC).
Fig. 4 illustrates a Spatial Audio Object Coding (SAOC) system comprising an SAOC
encoder 4 10 and an SAOC decoder 420.
The SAOC encoder 4 10 receives N audio object signals S ¾ as the input. Moreover,
the SAOC encoder 4 10 further receives instructions "Mixing information D" how these
objects should be combined to obtain a downmix signal comprising M downmix channels
X \, X - The SAOC encoder 4 10 extracts some side information from the objects and
from the downmixing process, and this side information is transmitted and/or stored along
with the downmix signals.
A major property of an SAOC system is that the downmix signal X comprising the
downmix channels X \ , XM forms a semantically meaningful signal. In other words, it is
possible to listen to the downmix signal. If, for example, the receiver does not have the
SAOC decoder functionality, the receiver can nonetheless always provide the downmix
signal as the output.
Fig. 5 illustrates an SAOC decoder comprising a side information decoder 5 10 . an object
separator 520 and a renderer 530. The SAOC decoder illustrated by Fig. 5 receives, e.g.,
from an SAOC encoder, the downmix signal and the side information. The downmix signal
can be considered as an audio input signal comprising the audio object signals, as the
audio object signals are mixed within the downmix signal (the audio object signals are
mixed within the one or more downmix channels of the downmix signal).
The SAOC decoder may, e.g. , then attempt to (virtually) reconstruct the original objects,
e.g. , by employing the object separator 520, e.g. , using the decoded side information.
These (virtual) object reconstructions S , ..., SN , e.g. , the reconstructed audio object
signals, are then combined based on the rendering informat ion e.g. , a rendering matrix .
to produce K audio output channels Y Y of an audio output signal Y.
In SAOC, often, audio object signals are, for example, reconstructed, e.g. , by employing
covariance information, e.g. , a signal covariance matrix E, that is transmitted from the
SAOC encoder to the SAOC decoder.
For example, the following formula may be employed to reconstruct the audio object
signals on the decoder side:
S= GX with G E D'7 (D E D") "
wherein
N number of audio object signals,
a ies number of considered samples of an audio object signal
M number of downmix channels,
downmix audio signal, size M x N Samp i ,
D downmixing matrix, size M N
E signal covariance matrix, size N x N defined as E
parametrically reconstructed N audio object signals, size N NS ies
self-adjoint (Hermitian) operator which represents the conjugate transpose
f (-)
Then, a rendering matrix R may be applied on the reconstructed audio object signals S to
obtain the audio output channels of the audio output signal Y . e.g., according to the
formula:
R
wherein
number of the audio output channels Y\, . . .. ¾ of the audio output signal Y .
R rendering matrix of size K x N
Y audio output signal comprising the K audio output channels,
Size K X N s
In Fig. 5 , the process of object reconstruction, e.g., conducted by the object separator
520, is referred to with the notion "virtual", or "optional", as it may not necessarily need to
take place, but the desired functionality can be obtained by combining the reconstruction
and the rendering steps in the parametric domain (i.e., combiring the equations).
In other words, instead of reconstructing the audio object signals using the mixing
information D and the covariance information E first, and then applying the rendering
information R on the reconstructed audio object signals to obtain the audio output
channels Y 7 both steps may be conducted in a single step, so that the audio
output channels Y, Y are directly generated from the downmix channels.
For example, the following formula may be employed:
Y = RGX with G ¾E D (D E " ) ~'< .
In principle, the rendering information R may request any combination of the original audio
object signals. In practice, however, the object reconstructions may comprise
reconstruction errors and the requested output scene may not necessarily be reached. As
a rough general rule covering many practical cases, the more the requested output scene
differs from the downmix signal, the more there will be audible reconstruction errors
In the following, dialogue enhancement (DE) is described. The SAOC technology may for
example by employed to realize the scenario. It should be noted, that even though the
name "Dialogue enhancement" suggests focusing on dialogue-oriented signals, the same
principle can be used with other signal types, too.
In the DE-scenario, the degrees of freedom in the system are limited from the general
case.
For example, the audio object signals S ,...,SN S are grouped (and possibly mixed) into
two meta-objects of a foreground object (FGO) and a background object (BGO)
GO
Moreover, the output scene Y ..., YK = Y resembles the downmix signal , ..., = X .
More specifically, both signals have the same dimensionalities, i.e., K = M , and the enduser
can only control the relative mixing levels of the two meta-objects FGO and BGO. To
be more exact, the downmix signal is obtained by mixing the FGO and BGO with some
scalar weights
X =
and the output scene is obtained similarly with some scalar weighting of the FGO and
BGO:
= S S
Depending on the relative values of the mixing weights, the balance between the FGO
and BGO may change. For example, with the setting
{ S
S
it is possible to increase the relative level of the FGO in the mixture. If the FGO is the
dialogue, this setting provides dialogue enhancement functionality.
As a use-case example, the BGO can be the stadium noises and other background sound
during a sports event and the FGO is the voice of the commentator. The DE-functionality
allows the end-user to amplify or attenuate the level of the commentator in relation to the
background.
Embodiments are based on the finding that utilizing the SAOC-technology (or similar) in a
broadcast scenario allows providing the end-user extended signal manipulation
functionality. More functionality than only changing the channel and adjusting the playback
volume is provided.
One possibility to employ the DE-technoiogy is briefly described above. If the broadcast
signal, being the downmix signal for SAOC, is normalized in level, e.g. , according to
R128, the different programs have similar average loudness when no (SAOC-)processing
is applied (or the rendering description is the same as the downmixing description).
However, when some (SAOC-)processing is applied, the output signal differs from the
default downmix signal and the loudness of the output signal may be different from the
loudness of the default downmix signal. From the point of view of the end-user, this may
lead into a situation in which the output signal loudness between channels or programs
may again have the un-desirable jumps or differences. In other words, the benefits of the
normalization applied by the broadcaster are partially lost.
This problem is not specific for SAOC or for the DE-scenario only, but may occur also with
other audio coding concepts that allow the end-user to interact with the content. However,
in many cases it does not cause any harm if the output signal has a different loudness
than the default downmix.
As stated before, a total loudness of an audio input signal program should equal a
specified level with small allowed deviations. However, as already outlined, this leads to
significant problems when audio rendering is conducted, as rendering may have a
significant effect on the overall/total loudness of the received audio input signal. However,
despite scene rendering is conducted, the total loudness of the received audio signal shall
remain the same.
One approach would be to estimate the loudness of a signal while it is being played, and
with an appropriate temporal integration concept, the estimate may converge to the true
average loudness after some time. The time required for the convergence is however
problematic from the point of view of the end-user. When the loudness estimate changes
even when no changes are applied on the signal, the loudness change compensation
should also react and change its behavior. This would lead into an output signal with
temporally varying average loudness, which can be perceived as rather annoying .
Fig. 6 illustrates a behavior of output signal loudness estimates on a loudness change.
Inter alia, a signal-based output signal loudness estimate is depicted, which illustrates the
effect of a solution as just described. The estimate approaches the correct estimate quite
slowly. Instead of a signal-based output signal loudness estimate, an informed output
signal loudness estimate, that immediately determines the output signal loudness
correctly would be preferable.
In particular, in Fig. 6, the user input, e.g., the level of the dialogue object, changes at time
instant T by increasing in value. The true output signal level, and correspondingly the
loudness, changes at the same time instant. When the output signal loudness estimation
is performed from the output signal with some temporal integration time, the estimate will
change gradually and reach the correct value after a certain delay. During this delay, the
estimate values are changing and cannot reliably be used for further processing the
output signal, e.g., for loudness level correction.
As already stated, it would be desirable to have an accurate estimate of the output
average loudness or the change in the average loudness without a delay and when the
program does not change or the rendering scene is not changed, the average loudness
estimate should also remain static. In other words, when some loudness change
compensation is applied, the compensation parameter should change only when either
the program changes or there is some user interaction.
The desired behavior is illustrated in the lowest illustration of Fig. 6 (informed output
signal loudness estimate). The estimate of the output signal loudness shall change
immediately when the user input changes.
Fig. 2 illustrates an encoder according to an embodiment.
The encoder comprises an object-based encoding unit 2 10 for encoding a plurality of
audio object signals to obtain an encoded audio signal comprising the plurality of audio
object signals.
Furthermore, the encoder comprises an object loudness encoding unit 220 for encoding
loudness information on the audio object signals. The loudness information comprises one
or more loudness values, wherein each of the one or more loudness values depends on
one or more of the audio object signals.
According to an embodiment, each of the audio object signals of the encoded audio signal
is assigned to exactly one group of two or more groups, wherein each of the two or more
groups comprises one or more of the audio object signals of the encoded audio signal.
The object loudness encoding unit 220 is configured to determine the one or more
loudness values of the loudness information by determining a loudness value for each
group of the two or more groups, wherein said loudness value of said group indicates an
original total loudness of the one or more audio object signals of said group.
Fig. 1 illustrates a decoder for generating an audio output signal comprising one or more
audio output channels according to an embodiment.
The decoder comprises a receiving interface 110 for receiving an audio input signal
comprising a plurality of audio object signals, for receiving loudness information on the
audio object signals, and for receiving rendering information indicating whether one or
more of the audio object signals shall be amplified or attenuated.
Moreover, the decoder comprises a signal processor 120 for generating the one or more
audio output channels of the audio output signal. The signal processor 20 is configured
to determine a loudness compensation value depending on the loudness information and
depending on the rendering information. Furthermore, the signal processor 120 is
configured to generate the one or more audio output channels of the audio output signal
from the audio input signal depending on the rendering information and depending on the
loudness compensation value.
According to an embodiment, the signal processor 10 is configured to generate the one
or more audio output channels of the audio output signal from the audio input signal
depending on the rendering information and depending on the loudness compensation
value, such that a loudness of the audio output signal is equal to a loudness of the audio
input signal, or such that the loudness of the audio output signal is closer to the loudness
of the audio input signal than a loudness of a modified audio signal that would result from
modifying the audio input signal by amplifying or attenuating the audio object signals of
the audio input signal according to the rendering information.
According to another embodiment, each of the audio object signals of the audio input
signal is assigned to exactly one group of two or more groups, wherein each of the two or
more groups comprises one or more of the audio object signals of the audio input signal.
In such an embodiment, the receiving interface 10 is configured to receive a loudness
value for each group of the two or more groups as the loudness information wherein said
loudness value indicates an original total loudness of the one or more audio object signals
of said group. Furthermore, the receiving interface 110 is configured to receive the
rendering information indicating for at least one group of the two or more groups whether
the one or more audio object signals of said group shall be amplified or attenuated by
indicating a modified total loudness of the one or more audio object signals of said group.
Moreover, in such an embodiment, the signal processor 120 is configured to determine
the loudness compensation value depending on the modified total loudness of each of
said at least one group of the two or more groups and depending on the original total
loudness of each of the two or more groups. Furthermore, the signal processor 120 is
configured to generate the one or more audio output channels of the audio output signal
from the audio input signal depending on the modified total loudness of each of said at
least one group of the two or more groups and depending on the loudness compensation
value.
In particular embodiments, at least one group of the two or more groups comprises two or
more of the audio object signals.
A direct relationship exists between the energy e, of an audio object signal and the
loudness L of the audio object signal i according to the formulae:
+ 1 1og , _ 1 »
wherein c is a constant value.
Embodiments are based on the following findings: Different audio object signals of the
audio input signal may have a different loudness and thus a different energy. If, e.g, a
user wants to increase the loudness of one of the audio object signals, the rendering
information may be correspondingly adjusted, and the increase of the loudness of this
audio object signal increases the energy of this audio object. This would lead to an
increased loudness of the audio output signal. To keep the total loudness constant, a
loudness compensation has to be conducted. In other words, the modified audio signal
that would result from applying the rendering information on the audio input signal would
have to be adjusted. However, the exact effect of the amplification of one of the audio
object signals on the total loudness of the modified audio signal depends on the original
loudness of the amplified audio object signal, e.g., of the audio object signal, the loudness
of which is increased. If the original loudness of this object corresponds to an energy, that
was quite low, the effect on the total loudness of the audio input signal will be minor. If,
however, the original loudness of this object corresponds to an energy, that was quite
high, the effect on the total loudness of the audio input signal will be significant.
Two examples may be considered. In both examples, an audio input signal comprises two
audio object signal, and in both examples, by applying the rendering information, the
energy of a first one of the audio object signals is increased by 50 %.
In the first example, the first audio object signal contributes 20 % and the second audio
object signal contributes 80 % to the total energy of the audio input signal. However, in the
second example, the first audio object, the first audio object signal contributes 40 % and
the second audio object signal contributes 60 % to the total energy of the audio input
signal. In both examples these contributions are derivable from the loudness information
on the audio object signals, as a direct relationship exists between loudness and energy.
In the first example, an increase of 50 % of the energy of the first audio object results in
that a modified audio signal that is generated by applying the rendering information on the
audio input signal has a total energy 1.5 x 20 % + 80 % = 110 % of the energy of the
audio input signal.
In the second example, an increase of 50 % of the energy of the first audio object results
in that the modified audio signal that is generated by applying the rendering information on
the audio input signal has a total energy .5 x 40 % + 60 % = 120 % of the energy of the
audio input signal.
Thus, after applying the rendering information on the audio input signal, in the first
example, the total energy of the modified audio signal has to be reduced by only 9 %
( 10 / 110 ) to obtain equal energy in both the audio input signal and the audio output
signal, while in the second example, the total energy of the modified audio signal has to
be reduced by 17 % ( 20 / 120 ) . For this purpose, a loudness compensation value may be
calculated.
For example, the loudness compensation value may be a scalar that is applied on all
audio output channels of the audio output signal.
According to an embodiment, the signal processor is configured to generate the modified
audio signal by modifying the audio input signal by amplifying or attenuating the audio
object signals of the audio input signal according to the rendering information. Moreover,
the signal processor is configured to generate the audio output signal by applying the
loudness compensation value on the modified audio signal, such that the loudness of the
audio output signal is equal to the loudness of the audio input signal, or such that the
loudness of the audio output signal is closer to the loudness of the audio input signal than
the loudness of the modified audio signal.
For example, in the first example above, the loudness compensation value lev, may, for
example, be set to a value lev = 10/1 , and a multiplication factor of 10/1 1 may be applied
on all channels that result from rendering the audio input channels according to the
rendering information.
Accordingly, for example, in the second example above, the loudness compensation value
lev, may, for example, be set to a value lev = 10/1 2 = 5/6, and a multiplication factor of 5/6
may be applied on all channels that result from rendering the audio input channels
according to the rendering information.
In other embodiments, each of the audio object signals may be assigned to one of a
plurality of groups, and a loudness value may be transmitted for each of the groups
indicating a total loudness value of the audio object signals of said group. If the rendering
information specifies that the energy of one of the groups is attenuated or amplified, e.g.,
amplified by 50 % as above, a total energy increase may be calculated and a loudness
compensation value may be determined as described above.
For example, according to an embodiment, each of the audio object signals of the audio
input signal is assigned to exactly one group of exactly two groups as the two or more
groups. Each of the audio object signals of the audio input signal is either assigned to a
foreground object group of the exactly two groups or to a background object group of the
exactly to groups. The receiving interface 1 0 is configured to receive the original total
loudness of the one or more audio object signals of the foreground object group.
Moreover, the receiving interface 110 is configured to receive the original total loudness of
the one or more audio object signals of the background object group. Furthermore, the
receiving interface 1 0 is configured to receive the rendering information indicating for at
least one group of the exactly two groups whether the one or more audio object signals of
each of said at least one group shall be amplified or attenuated by indicating a modified
total loudness of the one or more audio object signals of said group.
In such an embodiment, the signal processor 0 is configured to determine the loudness
compensation value depending o the modified total loudness of each of said at least one
group, depending on the original total loudness of the one or more audio object signals of
the foreground object group and depending on the original total loudness of the one or
more audio object signals of the background object group. Moreover, the signal processor
120 is configured to generate the one or more audio output channels of the audio output
signal from the audio input signal depending on the modified total loudness of each of said
at least one group and depending on the loudness compensation value.
According to some embodiments, each of the audio object signals is assigned to one of
three or more groups, and the receiving interface may be configured to receive a loudness
value for each of the three or more groups indicating the total loudness of the audio object
signals of said group.
According to an embodiment, to determine the total loudness value of two or more audio
object signals, for example, the energy value corresponding to the loudness value is
determined for each audio object signal, the energy values of all loudness values are
summed up to obtain an energy sum, and the loudness value corresponding to the energy
sum is determined as the total loudness value of the two or more audio object signals. For
example, the formulae
L = + 101og10 , e = 10 '
may be employed.
In some embodiments, loudness values are transmitted for each of the audio object
signals, or each of the audio object signals is assigned to one or two or more groups,
wherein for each of the groups, a loudness value is transmitted.
However, in some embodiments, for one or more audio object signals or for one or more
of the groups comprising audio object signals, no loudness value is transmitted. Instead,
the decoder may, for example, assume that these audio object signals or groups of audio
object signals, for which no loudness value is transmitted, have a predefined loudness
value. The decoder may, e.g., base all further determinations on this predefined loudness
value.
According to an embodiment, the receiving interface 110 is configured to receive a
downmix signal comprising one or more downmix channels as the audio input signal,
wherein the one or more downmix channels comprise the audio object signals, and
wherein the number of the audio object signals is smaller than the number of the one or
more downmix channels. The receiving interface 1 0 is configured to receive downmix
information indicating how the audio object signals are mixed within the one or more
downmix channels. Moreover, the signal processor 1 0 is configured to generate the one
or more audio output channels of the audio output signal from the audio input signal
depending on the downmix information, depending on the rendering information and
depending on the loudness compensation value. In a particular embodiment, the signal
processor 120 may, for example, be configured to calculate the loudness compensation
value depending on the downmix information.
For example, the downmix information may be a downmix matrix. In embodiments, the
decoder may be an SAOC decoder. In such embodiments, the receiving interface 110
may, e.g., be further configured to receive covariance information, e.g. , a covariance
matrix as described above.
With respect to the rendering information indicating whether one or more of the audio
object signals shall be amplified or attenuated, it should be noted that for example,
information that indicates how one or more of the audio object signals shall be amplified or
attenuated, is rendering information. For example, a rendering matrix R, e.g., a rendering
matrix of SAOC, is rendering information.
Fig. 3 illustrates a system according to an embodiment.
The system comprises an encoder 3 10 according to one of the above-described
embodiments for encoding a plurality of audio object signals to obtain an encoded audio
signal comprising the plurality of audio object signals.
Moreover, the system comprises a decoder 320 according to one of the above-described
embodiments for generating an audio output signal comprising one or more audio output
channels. The decoder is configured to receive the encoded audio signal as an audio
input signal and the loudness information. Moreover, the decoder 320 is configured to
further receive rendering information. Furthermore, the decoder 320 is configured to
determine a loudness compensation value depending on the loudness information and
depending on the rendering information. Moreover, the decoder 320 is configured to
generate the one or more audio output channels of the audio output signal from the audio
input signal depending on the rendering information and depending on the loudness
compensation value.
Fig. 7 illustrates informed loudness estimation according to an embodiment. On the left of
transport stream 730, components of an object-based audio coding encoder are
illustrated. In particular, an object-based encoding unit 7 10 ("object-based audio encoder")
and an object loudness encoding unit 720 is illustrated ("object loudness estimation").
The transport stream 730 itself comprises loudness information L , downmixing
information I) and the output of the object-based audio encoder 710 B.
On the right of transport stream 730. components of a signal processor of an object-based
audio coding decoder are illustrated. The receiving interface of the decoder is not
illustrated. An output loudness estimator 740 and an object-based audio decoding unit
750 is depicted. The output loudness estimator 740 may be configured to determine the
loudness compensation value. The object-based audio decoding unit 750 may be
configured to determine a modified audio signal from an audio signal, being input to the
decoder, by applying the rendering information R. Applying the loudness compensation
value on the modified audio signal to compensate a total loudness change caused by the
rendering is not shown in Fig. 7 .
The input to the encoder consists of the input objects S in the minimum. The system
estimates the loudness of each object (or some other loudness-related information, such
as the object energies) , e.g., by the object loudness encoding unit 720, and this
information L is transmitted and/or stored. (It is also possible, the loudness of the objects
is provided as an input to the system, and the estimation step within the system can be
omitted).
In the embodiment of Fig. 7 , the decoder receives at least the object loudness information
and, e.g. , the rendering information R describing the mixing of the objects into the output
signal. Based on these, e.g., the output loudness estimator 740, estimates the loudness of
the output signal and provides this information as its output.
The downmixing information D may be provided as the rendering information, in which
case the loudness estimation provides an estimate of the downmix signal loudness. It is
also possible to provide the downmixing information as an input to the object loudness
estimation, and to transmit and/or store it along the object loudness information. The
output loudness estimation can then estimate simultaneously the loudness of the downmix
signal and the rendered output and provide these two values or their difference as the
output loudness information. The difference value (or its inverse) describes the required
compensation that should be applied on the rendered output signal for making its
loudness similar to the loudness of the downmix signal. The object loudness information
can additionally include information regarding the correlation coefficients between various
objects and this correlation information can be used in the output loudness estimation for
a more accurate estimate.
In the following, a preferred embodiment for dialogue enhancement application is
described.
In the dialogue enhancement application, as described above, the input audio object
signals are grouped and partially downmixed to form two meta-objects, FGO and BGO,
which can then be trivially summed for obtaining the final downmix signal.
Following the description of SAOC [SAOC], N input object signals are represented as a
matrix S of the size N x NSa te , and the downmixing information as a matrix I) of the size
M x N. The downmix signals can then be obtained as X = DS.
The downmixing information D can now be divided into two parts
= ^FGO ^BGO
for the meta-objects.
As each column of the matrix D corresponds to an original audio object signal, the two
component downmix matrices can be obtained by setting the columns, which correspond
to the other meta-object into zero (assuming that no original object may be present in both
meta-objects). In other words, the columns corresponding to the meta-object BGO are set
to zero in Dro , and vice versa.
These new downmixing matrices describe the way the two meta-objects can be obtained
from the input objects, namely:
G ~ G r d = S,
and the actual downmixing is simplified to
FGO ^BGO
It can be also considered that the object (e.g ., SAOC) decoder attempts to reconstruct the
meta-objects:
S G0 * S 0 and G :^BGO
and the DE-specific rendering can be written as a combination of these two meta-object
reconstructions:
The object loudness estimation receives the two meta-objects S and S G as the input
and estimates the loudness of each of them: L PG0 being the (total/overall) loudness of
S , and being the (total/overall) loudness of S 0 . These loudness values are
transmitted and/or stored.
As an alternative, using one of the meta-objects, e.g., the FGO, as reference, it is possible
to calculate the loudness difference of these two objects, e.g., as
This single value is then transmitted and/or stored.
Fig. 8 illustrates an encoder according to another embodiment. The encoder of Fig. 8
comprises an object downmixer 8 1 and an object side information estimator 8 12 .
Furthermore, the encoder of Fig. 8 further comprises an object loudness encoding unit
820. Moreover, the encoder of Fig. 8 comprises a meta audio object mixer 805.
The encoder of Fig 8 uses intermediate audio meta-objects as an input to the object
loudness estimation. In embodiments, the encoder of Fig. 8 may be configured to
generate two audio meta-objects. In other embodiments, the encoder of Fig. 8 may be
configured to generate three or more audio meta-objects.
Inter alia, the provided concepts provide the new feature that the encoder may, e.g. ,
estimates the average loudness of all input objects. The objects may, e.g. , be mixed into a
downmix signal that is transmitted. The provided concepts moreover provide the new
feature that the object loudness and the downmixing information may. e.g. , be included in
the object-coding side information that is transmitted.
The decoder may. e.g., use the object-coding side information for (virtual) separation of
the objects and re-combines the objects using the rendering information.
Furthermore the provided concepts provide the new feature that either the downmixing
information can be used to estimate the loudness of the default downmix signal, the
rendering information and the received object loudness can be used for estimating the
average loudness of the output signal, and/or the loudness change can be estimated from
these two values. Or, the downmlxing and rendering information can be used to estimate
the loudness change from the default downmix, another new feature of the provided
concepts.
Furthermore, the provided concepts provide the new feature that the decoder output can
be modified to compensate for the change in the loudness so that the average loudness of
the modified signal matches the average loudness of the default downmix.
A specific embodiment related to SAOC-DE is illustrated in Fig. 9 The system receives
the input audio object signals, the downmixing information, and the information of the
grouping of the objects to meta-objects. Based on these, the meta audio object mixer 905
forms the two meta-objects SFG0 and S O . It is possible, that the portion of the signal
that is processed with SAOC, does not constitute the entire signal. For example, in a 5 .1
channel configuration, SAOC may be deployed on a sub-set of channels, like on the front
channel (left, right, and center), while the other channels (left surround, right surround,
and low-frequency effects) are routed around, (by-passing) the SAOC and delivered as
such. These channels not processed by SAOC are denoted with X B SS The possible by¬
pass channels need to be provided for the encoder for more accurate estimation of the
loudness information.
The by-pass channels may be handled in various ways.
For example, the by- ass channels may, e.g. , form an independent meta-object. This
allows defining the rendering so that all three meta-objects are scaled independently.
Or, for example, the by-pass channels may. e.g., be combined with one of the other two
meta-objects. The rendering settings of that meta-object control also the by-pass channel
portion. For example, in the dialogue enhancement scenario, it may be meaningful to
combine the by-pass channels with the background meta-object: X g = S + X YP SS .
Or, for example, the by-pass channels may, e.g., be ignored.
According to embodiments, the object-based encoding unit 2 10 of the encoder is
configured to receive the audio object signals, wherein each of the audio object signals is
assigned to exactly one of exactly two groups, wherein each of the exactly two groups
comprises one or more of the audio object signals. Moreover, the object-based encoding
unit 210 is configured to downmix the audio object signals, being comprised by the exactly
two groups, to obtain a downmix signal comprising one or more downmix audio channels
as the encoded audio signal, wherein the number of the one or more downmix channels is
smaller than the number of the audio object signals being comprised by the exactly two
groups. The object loudness encoding unit 220 is assigned to receive one or more further
by-pass audio object signals, wherein each of the one or more further by-pass audio
object signals is assigned to a third group, wherein each of the one or more further by¬
pass audio object signals is not comprised by the first group and is not comprised by the
second group, wherein the object-based encoding unit 210 is configured to not downmix
the one or more further by-pass audio object signals within the downmix signal.
In an embodiment, the object loudness encoding unit 220 is configured to determine a first
loudness value, a second loudness value and a third loudness value of the loudness
information, the first loudness value indicating a total loudness of the one or more audio
object signals of the first group, the second loudness value indicating a total loudness of
the one or more audio object signals of the second group, and the third loudness value
indicating a total loudness of the one or more further by-pass audio object signals of the
third group. In an another embodiment, the object loudness encoding unit 220 is
configured to determine a first loudness value and a second loudness value of the
loudness information, the first loudness value indicating a total loudness of the one or
more audio object signals of the first group, and the second loudness value indicating a
total loudness of the one or more audio object signals of the second group and of the one
or more further by-pass audio object signals of the third group.
According to an embodiment, the receiving interface 110 of the decoder is configured to
receive the downmix signal. Moreover, the receiving interface 110 is configured to receive
one or more further by-pass audio object signals, wherein the one or more further by-pass
audio object signals are not mixed within the downmix signal. Furthermore, the receiving
interface 1 0 is configured to receive the loudness information indicating information on
the loudness of the audio object signals which are mixed within the downmix signal and
indicating information on the loudness of the one or more further by-pass audio object
signals which are not mixed within the downmix signal. Moreover, the signal processor
120 is configured to determine the loudness compensation value depending on the
information on the loudness of the audio object signals which are mixed within the
downmix signal, and depending on the information on the loudness of the one or more
further by-pass audio object signals which are not mixed within the downmix signal.
Fig. 9 illustrates an encoder and a decoder according to an embodiment related to the
SAOC-DE, which comprises by-pass channels. Inter alia, the encoder of Fig. 9 comprises
an SAOC encoder 902.
In the embodiment of Fig. 9 , the possible combining of the by-pass channels with the
other meta-objects takes place in the two "bypass inclusion" blocks 9 13, 914, producing
the meta-objects X and X with the defined parts from the by-pass channels
included.
The perceptual loudness LB SS , LFG0 , and LBG0 of both of these meta-objects are
estimated in the loudness estimation units 921 , 922, 923. This loudness information is
then transformed into an appropriate encoding in a meta-object loudness information
estimator 925 and then transmitted and/or stored.
The actual SAOC en- and decoder operate as expected extracting the object side
information from the objects, creating the downmix signal , and transmitting and/or
storing the information to the decoder. The possible by-pass channels are transmitted
and/or stored along the other information to the decoder.
The SAOC-DE decoder 945 receives a gain value "Dialog gain" as a user-input. Based
on this input and the received downmixing information, the SAOC decoder 945
determines the rendering information. The SAOC decoder 945 then produces the
rendered output scene as the signal Y . In addition to that, it produces a gain factor (and a
delay value) that should be applied on the possible by-pass signals BYPASS
The 'bypass inclusion" unit 955 receives this information along with the rendered output
scene and the by-pass signals and creates the full output scene signal. The SAOC
decoder 945 produces also a set of meta-object gain values, the amount of these
depending on the meta-object grouping and desired loudness information form.
The gain values are provided to the mixture loudness estimator 960 which also receives
the meta-object loudness information from the encoder.
The mixture loudness estimator 960 is then able to determine the desired loudness
information, which may include but is not limited to, the loudness of the downmix signal,
the loudness of the rendered output scene, and/or the difference in the loudness between
the downmix signal and the rendered output scene.
In some embodiments, the loudness information itself is enough, while in other
embodiments, it is desirable to process the full output depending on the determined
loudness information. This processing may, for example, be compensation of any possible
difference in the loudness between the downmix signal and the rendered output scene.
Such a processing, e.g., by a loudness processing unit 970, would make sense in the
broadcast scenario, as it would reduce the changes in the perceived signal loudness
regardless of the user interaction (setting of the input "dialog gain").
The loudness-related processing in this specific embodiment comprises the a plurality of
new features. Inter alia, the FGO, BGO, and the possible by-pass channels are pre-mixed
into the final channel configuration so that the downmixing can be done by simply adding
the two pre-mixed signals together (e.g. , downmix matrix coefficients of 1) , which
constitutes a new feature. Moreover, as a further new feature, the average loudness of the
FGO and BGO are estimated, and the difference is calculated. Furthermore, the objects
are mixed into a downmix signal that is transmitted. Moreover, as a further new feature,
the loudness difference information is included to the side information that is transmitted
(new) Furthermore, the decoder uses the side information for (virtual) separation of the
objects and re-combines the objects using the rendering information which is based on
the downmixing information and the user input modification gain. Moreover, as another
new feature, the decoder uses the modification gain and the transmitted loudness
information for estimating the change in the average loudness of the system output
compared to the default downmix.
In the following, a formal description of embodiments is provided.
Assuming that the object loudness values behave similar to the logarithm of energy values
when summing the objects, i.e. , the loudness values must be transformed into linear
domain, added there, and finally transformed back to the logarithmic domain. Motivating
this through the definition of BS. 1770 loudness measure will now be presented (for
simplicity, the number of channels is set to one, but the same principle can be applied on
multi-channel signals with appropriate summing over channels).
The loudness of the i t K-filtered signal zi with the mean-squared energy e is defined as
= + 0 log ,
wherein c is an offset constant. For example, c may be -0.691 . From this follows that the
energy of the signal can be determined from the loudness with
et = - w
The energy of the sum of N uncorrelated signals = - , is then
(£,-c)/10
su = , =<
= 1 (=1
and the loudness of this sum signal is then
If the signals are not uncorrelated, the correlation coefficients C,. . must be taken into
account when approximating the energy of the sum signal as
=<
wherein the cross-energy e, . between the z' and ' objects is defined as
wherein - 1< , < 1 is the correlation coefficient between the two objects i and j .
When two objects are uncorrelated, the correlation coefficient equals to 0 , and when the
two objects are identical, the correlation coefficient equals to 1.
Further extending the model with mixing weights g, to be applied on the signals in the
mixing process, i.e., zSUM =g,z , the energy of the sum signal will be
= SiSj iJ ,
and the loudness of the mixture signal can be obtained from this, as earlier, with
L SUM = c + & e suM
The difference between the loudness of two signals can be estimated as
If the definition of loudness is now used as earlier, this can be written as
L (i, j ) = L , - Lj
= c + 101og 0 , ) - ( c + 10 1og10 e ) ,
which can be observed to be a function of signal energies. If it is now desired to estimate
the loudness difference between two mixtures
Z A = S,z and = ,
with possibly differing mixing weights g , and h this can be estimated with
L A B ) = \ l g
10 log
:10 lOg,
n the case the objects are uncorrected ( C,.
difference estimate becomes
,
= 101og i=1
In the following, differential encoding is considered.
It is possible to encode the per-object loudness values as differences from the loudness of
a selected reference object:
K j = L —L R ,
wherein L REF is the loudness of the reference object. This encoding is beneficial if no
absolute loudness values are needed as the result, because it is now necessary to
transmit one value less, and the loudness difference estimation can be written as
AL (A, B ) = l l g l
or in the case of uncorrelated objects
,
In the following, a dialogue enhancement scenario is considered.
Considering again the application scenario of dialogue enhancement. The freedom of
defining the rendering information in the decoder is limited only into changing the levels of
the two meta-objects. Let us furthermore assume that the two meta-objects are
uncorrelated, i.e., CF O G = 0 . If the downmixing weights of the meta-objects are h 0
and hBG0 , and they are rendered with the gains f FG0 and f BG0 , the loudness of the
output relative to the default downmix is
This is then also the required compensation if it is desired to have the same loudness in
the output as in the default downmix.
AL(A, B) may be considered as a loudness compensation value, that may be transmitted
by the signal processor 20 of the decoder. &L(A, B) can also be named as a loudness
change value and thus the actual compensation value can be an inverse value. Or is it ok
to use the "loudness compensation factor" name for it, too? Thus, the loudness
compensation value lev mentioned earlier in this document would correspond the value
g i a below.
-AL(A,B)
For example, g =10 / 2° 1 / AL(A, B) may be applied as a multiplication factor on
each channel of a modified audio signal that results from applying the rendering
information on the audio input signal. This equation for g t works in the linear domain. In
the logarithmic domain, the equation would be different such as 1 / AL(A, B) and applied
accordingly.
If the downmixing process is simplified such that the two meta-objects can be mixed with
unity weights for obtaining the downmix signal, i.e., FG0 = h l , and now the
rendering gains for these two objects are denoted with gFG0 and g BG0 . This simplifies the
equation for the loudness change into
= 10 1og110
10 +
i /io
10 +10
Again, ) may be considered as a loudness compensation value that is determined
by the signal processor 120.
In general, g GO may be considered as a rendering gain for the foreground object FGO
(foreground object group), and gBGo may be considered as a rendering gain for the
background object BGO (background object group).
As mentioned earlier, it is possible to transmit loudness differences instead of absolute
loudness. Let us define the reference loudness as the loudness of the FGO meta-object
REF ~ FG0 , i.e. , K FG0 = LFG0 - LREF = 0 and BGO = LBG0 —L F = LBG0 - LFG0 . Now, the
loudness change is
It may also be, as the case in the SAOC-DE is, that two meta-objects do not have
individual scaling factors, but one of the objects is left un-modified, while the other is
attenuated to obtain the correct mixing ratio between the objects. In this rendering setting,
the output will be lower in loudness than the default mixture, and the change in the
loudness is
10
AL (A, B) = l l g g G +
+ i o ,°
w th
This form is already rather simple, and is rather agnostic regarding the loudness measure
used. The only real requirement is, that the loudness values should sum in the
exponential domain. It is possible to transmit/store values of signal energies instead of
loudness values, as the two have close connection.
In each of the above formulae, AL(A, B ) may be considered as a loudness compensation
value, that may be transmitted by the signal processor 120 of the decoder.
In the following, example cases are considered. The accuracy of the provided concepts is
illustrated through two example signals. Both signals have a 5 .1 downmix with the
surround and LFE channels by-passed from the SAOC processing .
Two main approaches are used: one ("3-term") with three meta-objects: FGO, BGO, and
by-pass channels, e.g.,
FG BYPASS '
And another one ("2-term") with two meta-objects, e.g.
F O + BG
In the 2-term approach, the by-pass channels may, e.g. , be mixed together with the BGO
for the meta-object loudness estimation. The loudness of both (or all three) objects as well
as the loudness of the downmix signal are estimated, and the values are stored.
The rendering instructions are of form
Y BYPASS
and
for the two approaches respectively.
The gain values are, e.g. , determined according to:
wherein the FGO gain gFG0 is varied between -24 to +24 dB
The output scenario is rendered, the loudness is measured, and the attenuation from the
loudness of the downmix signal is calculated.
This result is displayed in Fig. 10 and Fig. 11 with the blue line with circle markers. Fig. 10
depicts a first illustration and Fig. 11 depicts a second illustration of a measured loudness
change and the result of using the provided concepts for estimating the change in the
loudness in a purely parametrical manner.
Next, the attenuation from the downmix is estimated parametrically employing the stored
meta-object loudness values and the downmixing and rendering information. The estimate
using the loudness of three meta-objects is illustrated with the green line with square
markers, and the estimate using the loudness of two meta-objects is illustrated with the
red line with star markers.
It can be seen from the figures, that the 2- and 3-term approaches provide practically
identical results, and they both approximate the measured value quite well.
The provided concepts exhibit a plurality of advantages. For example, the provided
concepts allow estimating the loudness of a mixture signal from the loudness of the
component signals forming the mixture. The benefit of this is that the component signal
loudness can be estimated once, and the loudness estimate of the mixture signal can be
obtained parametrically for any mixture without the need of actual signal-based loudness
estimation. This provides a considerable improvement in the computational efficiency of
the overall system in which the loudness estimate of various mixtures is needed. For
example, when the end-user changes the rendering settings, the loudness estimate of the
output is immediately available.
In some applications, such as when conforming with the EBU R128 recommendation, the
average loudness over the entire program is important. If the loudness estimation in the
receiver, e.g. , in a broadcast scenario, is done based on the received signal, the estimate
converges to the average loudness only after the entire program has been received.
Because of this, any compensation of the loudness wi l have errors or exhibit temporal
variations. When estimating the loudness of the component objects as proposed and
transmitting the loudness information, it is possible to estimate the average mixture
loudness in the receiver without a delay.
If it is desired that the average loudness of the output signal remains (approximately)
constant regardless of the changes in the rendering information, the provided concepts
allow determining a compensation factor for this reason. The calculations needed for this
in the decoder are from their computational complexity negligible, and the functionality is
thus possible to be added to any decoder.
There are cases in which the absolute loudness level of the output is not important, but
the importance lies in determining the change in the loudness from a reference scene. In
such cases the absolute levels of the objects are not important, but their relative levels
are. This allows defining one of the objects as the reference object and representing the
loudness of the other objects in relation to the loudness of this reference object. This has
some benefits considering the transport and/or storage of the loudness information.
First of all, it is not necessary to transport the reference loudness level. In the application
case of two meta-objects, this halves the amount of data to be transmitted. The second
benefit relates to the possible quantization and representation of the loudness values.
Since the absolute levels of the objects can be almost anything, the absolute loudness
values can also be almost anything. The relative loudness values, on the other hand, are
assumed to have a 0 mean and a rather nicely formed distribution around the mean. The
difference between the representations allows defining the quantization grid of the relative
representation in a way with potentially greater accuracy with the same number of bits
used for the quantized representation.
Fig. 12 illustrates another embodiment for conducting loudness compensation. In Fig. 12,
loudness compensation may be conducted, e.g. , to compensate the loss in loudness, For
this purpose, e.g. , the values DE_loudness_diff_dialogue (= KFGO) and
DE udness diff background (= K co) from E ontr l iiifo may be used. Here,
E control info may specify Advanced Clean Audio "Dialogue Enhancement" (DE)
control information
The loudness compensation is achieved by applying a gain value "g" on the SAOC-DE
output signal and the by-passed channels (in case of a multichannel signal).
the embodiment of Fig. 12 , this is done as follows:
A limited dialogue modification gain value mG is used to determine the effective gains for
the foreground object (FGO, e.g , dialogue) and for the background object (BGO, e.g.,
ambiance). This is done by the "Gain mapping" block 1220 which produces the gain
values m Q and mB60 .
The Output loudness estimator" block 230 uses the loudness information K G and
G
d r effective gain values m G and m G0 to estimate this possible change in
the loudness compared to the default downmix case. The change is then mapped into the
"Loudness compensation factor" which is applied on the output channels for producing the
final "Output signals".
The following steps are applied for loudness compensation:
Receive the limited gain value m from the SAOC-DE decoder (as defined in
clause 12.8 "Modification range control for SAOC-DE" [DE]), and determine the
applied FGO/BGO gains:
mFG0 = m , and mBG0 = 1 if m < 1
mFG0 = 1, and mBG0 = rn if m > 1
Obtain the meta-object loudness information K FG0 and K o .
Calculate the change in the output loudness compared to the default downmix with
10 / + 10
-0.05 Calculate the loudness compensation gain — 0
Calculate the scaling factors g = wherein
g if channel i belongs to SAOC-DE output
g , = and N is the total
BG0 if channel i is a by-pass channel
number of output channels. In Fig. 12, the gain adjustment is divided into two
steps: the gain of the possible "by-pass channels" is adjusted with m prior to
combining them with the "SAOC-DE output channels ' , and then a common gain
is then applied on all the combined channels. This is only a possible re¬
ordering of the gain adjustment operations, while g here combines both gain
adjustment steps into one gain adjustment.
- Apply the scaling values g on the audio channels Y consisting of the "SAOCDE
output channels" Y S 0 C and the possible time-aligned "by-pass channels'' Y P SS :
Y
FULL
= Y
SAOC
u Y
BYPASS
Applying the scaling values g on the audio channels . is conducted by the gain
adjustment unit 1240.
AL as calculated above may be considered as a loudness compensation value. In
general, m indicates a rendering gain for the foreground object FGO (foreground
object group), and MBGO indicates a rendering gain for the background object EGO
(background object group).
Although some aspects have been described in the context of an apparatus, it is clear that
these aspects also represent a description of the corresponding method, where a block or
device corresponds to a method step or a feature of a method step. Analogously, aspects
described in the context of a method step also represent a description of a corresponding
block or item or feature of a corresponding apparatus.
The inventive decomposed signal can be stored on a digital storage medium or can be
transmitted on a transmission medium such as a wireless transmission medium or a wired
transmission medium such as the Internet.
Depending on certain implementation requirements, embodiments of the invention can be
implemented in hardware or in software. The implementation can be performed using a
digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an
EPROM, an EEPROM or a FLASH memory, having electronically readable control signals
stored thereon, which cooperate (or are capable of cooperating) with a programmable
computer system such that the respective method is performed.
Some embodiments according to the invention comprise a non-transitory data carrier
having electronically readable control signals, which are capable of cooperating with a
programmable computer system, such that one of the methods described herein is
performed.
Generally, embodiments of the present invention can be implemented as a computer
program product with a program code, the program code being operative for performing
one of the methods when the computer program product runs on a computer. The
program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods
described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a computer program
having a program code for performing one of the methods described herein, when the
computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital
storage medium, or a computer-readable medium) comprising, recorded thereon, the
computer program for performing one of the methods described herein.
A further embodiment of the inventive method is, therefore, a data stream or a sequence
of signals representing the computer program for performing one of the methods
described herein. The data stream or the sequence of signals may for example be
configured to be transferred via a data communication connection, for example via the
Internet.
A further embodiment comprises a processing means, for example a computer, or a
programmable logic device, configured to or adapted to perform one of the methods
described herein.
A further embodiment comprises a computer having installed thereon the computer
program for performing one of the methods described herein.
In some embodiments, a programmable logic device (for example a field programmable
gate array) may be used to perform some or all of the functionalities of the methods
described herein. In some embodiments, a field programmable gate array may cooperate
with a microprocessor in order to perform one of the methods described herein. Generally,
the methods are preferably performed by any hardware apparatus.
The above described embodiments are merely illustrative for the principles of the present
invention. It is understood that modifications and variations of the arrangements and the
details described herein will be apparent to others skilled in the art. It is the intent,
therefore, to be limited only by the scope of the impending patent claims and not by the
specific details presented by way of description and explanation of the embodiments
herein.
References
[BCC] C. Faller and F. Baumgarte, "Binaural Cue Coding - Part II : Schemes and
applications," IEEE Trans on Speech and Audio Proc, vol. 11, no. 6 , Nov. 2003.
[EBU] EBU Recommendation R 128 'Loudness normalization and permitted maximum
level of audio signals", Geneva, 20 1.
[JSC] C. Faller, "Parametric Joint-Coding of Audio Sources", 120th AES Convention,
Paris, 2006.
[ISS 1] M. Parvaix and L. Girin: "Informed Source Separation of underdetermined
instantaneous Stereo Mixtures using Source Index Embedding", IEEE ICASSP,
201 0.
[ISS2] M. Parvaix, L . Girin, J.-M. Brassier: "A watermarking-based method for informed
source separation of audio signals with a single sensor", IEEE Transactions on
Audio, Speech and Language Processing, 201 0 .
[ISS3] A. Liutkus and J. Pinel and R. Badeau and L. Girin and G. Richard: "Informed
source separation through spectrogram coding and data embedding", Signal
Processing Journal, 201 1.
[ISS4] A. Ozerov. A. Liutkus, R. Badeau, G. Richard: "Informed source separation:
source coding meets source separation", IEEE Workshop on Applications of
Signal Processing to Audio and Acoustics, 20 1.
[ISS5] S. Zhang and L. Girin : "An Informed Source Separation System for Speech
Signals", INTERSPEECH, 201 1.
L . Girin and J Pinel: "Informed Audio Source Separation from Compressed
Linear Stereo Mixtures", AES 42nd International Conference: Semantic Audio,
201 1.
[ITU] International Telecommunication Union: "Recommendation ITU-R BS. 1770-3 -
Algorithms to measure audio programme loudness and true-peak audio level",
Geneva, 201 2 .
[SAOC1 ] J. Herre, S. Disch, J . Hiipert O. Helimuth: "From SAC To SAOC - Recent
Developments in Parametric Coding of Spatial Audio", 22nd Regional UK AES
Conference, Cambridge, UK, April 2007.
[SAOC2] J. Engdegard, B. Resch, C. Falch, 0 . Helimuth J . Hiipert, A. Holzer, L .
Terentiev, J. Breebaart, J . Koppens, E. Schuijers and W. Oomen: "Spatial Audio
Object Coding (SAOC) - The Upcoming MPEG Standard on Parametric Object
Based Audio Coding", 124th AES Convention, Amsterdam 2008.
[SAOC] ISO/IEC, "MPEG audio technologies - Part 2: Spatial Audio Object Coding
(SAOC)," ISO/IEC JTC1 /SC29/WG1 1 (MPEG) International Standard 23003-2.
[EPl EP 2 146522 A 1: S. Schreiner, W. Fiesel, M. Neusinger, O. Helimuth, R.
Sperschneider, "Apparatus and method for generating audio output signals
using object based metadata", 201 0.
[DE] ISO/IEC, "MPEG audio technologies - Part 2: Spatial Audio Object Coding
(SAOC) - Amendment 3, Dialogue Enhancement," ISO/IEC 23003-2:201 0/DAM
3 , Dialogue Enhancement.
[BRE] WO 2008/035275 A2.
[SCH] EP 2 146 522 A 1.
[ENG] WO 2008/046531 A 1.
Claims
A decoder for generating an audio output signal comprising one or more audio
output channels, wherein the decoder comprises:
a receiving interface ( 110) for receiving an audio input signal comprising a plurality
of audio object signals, for receiving loudness information on the audio object
signals, and for receiving rendering information indicating whether one or more of
the audio object signals shall be amplified or attenuated, and
a signal processor ( 120) for generating the one or more audio output channels of
the audio output signal,
wherein the signal processor (120) is configured to determine a loudness
compensation value depending on the loudness information and depending on the
rendering information, and
wherein the signal processor ( 120) is configured to generate the one or more audio
output channels of the audio output signal from the audio input signal depending
on the rendering information and depending on the loudness compensation value.
A decoder according to claim 1, wherein the signal processor ( 120) is configured to
generate the one or more audio output channels of the audio output signal from
the audio input signal depending on the rendering information and depending on
the loudness compensation value, such that a loudness of the audio output signal
is equal to a loudness of the audio input signal, or such that the loudness of the
audio output signal is closer to the loudness of the audio input signal than a
loudness of a modified audio signal that would result from modifying the audio
input signal by amplifying or attenuating the audio object signals of the audio input
signal according to the rendering information.
A decoder according to claim 2 ,
wherein the signal processor ( 120) is configured to generate the modified audio
signal by modifying the audio input signal by amplifying or attenuating the audio
object signals of the audio input signal according to the rendering information , and
wherein the signal processor ( 120) is configured to generate the audio output
signal by applying the loudness compensation value on the modified audio signal,
such that the loudness of the audio output signal is equal to the loudness of the
audio input signal, or such that the loudness of the audio output signal is closer to
the loudness of the audio input signal than the loudness of the modified audio
signal.
A decoder according to one of the preceding claims,
wherein each of the audio object signals of the audio input signal is assigned to
exactly one group of two or more groups, wherein each of the two or more groups
comprises one or more of the audio object signals of the audio input signal,
wherein the receiving interface ( 110) is configured to receive a loudness value for
each group of the two or more groups as the loudness information,
wherein the signal processor ( 120) is configured to determine the loudness
compensation value depending on the loudness value of each of the two or more
groups, and
wherein the signal processor ( 120) is configured to generate the one or more audio
output channels of the audio output signal from the audio input signal depending
on the loudness compensation value.
A decoder according to claim 4 , wherein at least one group of the two or more
groups comprises two or more of the audio object signals.
A decoder according to one of claims 1 to 3 ,
wherein each of the audio object signals of the audio input signal is assigned to
exactly one group of more than two groups, wherein each of the more than two
groups comprises one or more of the audio object signals of the audio input signal,
wherein the receiving interface ( 110) is configured to receive a loudness value for
each group of the more than two groups as the loudness information,
wherein the signal processor ( 120) is configured to determine the loudness
compensation value depending on the loudness value of each of the more than
two groups, and
wherein the signal processor ( 20) is configured to generate the one or more audio
output channels of the audio output signal from the audio input signal depending
on the loudness compensation value.
A decoder according to claim 6, wherein at least one group of the more than two
groups comprises two or more of the audio object signals.
A decoder according to one of claims 4 to 7 ,
wherein the signal processor ( 120) is configured to determine the loudness
compensation value according to the formula
or according to the formula
= lOlog
wherein A.L is the loudness compensation value,
wherein indicates an z'-th audio object signal of the audio object signals,
wherein Li is a loudness of the ;'-th audio object signal,
wherein is a first mixing weight for the z-th audio object signal,
wherein h, is a second mixing weight for the z'-th audio object signal,
wherein c is a constant value, and
wherein ' is a number.
A decoder according to one of claims 4 to 7,
wherein the signal processor ( 120) is configured to determine the loudness
compensation value according to the formula
wherein AL is the loudness compensation value,
wherein i indicates an z'-th audio object signal of the audio object signals,
wherein g, is a first mixing weight for the z'-th audio object signal,
wherein h, is a second mixing weight for the z'-th audio object signal,
wherein N is a number, and
wherein K, is defined according to
wherein is a loudness of the z'-th audio object signal, and
wherein LRE is the loudness of a reference object.
A decoder according to claim 4 or 5,
wherein each of the audio object signals of the audio input signal is assigned to
exactly one group of exactly two groups as the two or more groups,
wherein each of the audio object signals of the audio input signal is either assigned
to a foreground object group of the exactly two groups or to a background object
group of the exactly two groups,
wherein the receiving interface ( 1 10) is configured to receive the loudness value of
the foreground object group,
wherein the receiving interface ( 1 10) is configured to receive the loudness value of
the background object group,
wherein the signal processor (120) is configured to determine the loudness
compensation value depending on the loudness value of the foreground object
group, and depending on the loudness value of the background object group, and
wherein the signal processor (120) is configured to generate the one or more audio
output channels of the audio output signal from the audio input signal depending
on the loudness compensation value.
A decoder according to claim 10,
wherein the signal processor (120) is configured to determine the loudness
compensation value according to the formula
wherein is the loudness compensation value,
wherein KFG0 indicates the loudness value of the foreground object group,
wherein KBGO indicates the loudness value of the background object group,
wherein m o indicates a rendering gain of the foreground object group, and
wherein m indicates a rendering gain of the background object group.
12 . A decoder according to claim 10 ,
wherein the signal processor ( 120) is configured to determine the loudness
compensation value according to the formula
wherein A L is the loudness compensation value,
wherein L G indicates the loudness value of the foreground object group,
wherein GO indicates the loudness value of the background object group,
wherein gF o indicates a rendering gain of the foreground object group, and
wherein §BGO indicates a rendering gain of the background object group.
13 . A decoder according to one of the preceding claims,
wherein the receiving interface (110) is configured to receive a downmix signal
comprising one or more downmix channels as the audio input signal, wherein the
one or more downmix channels comprise the audio object signals, and wherein the
number of the one or more downmix channels is smaller than the number of the
audio object signals,
wherein the receiving interface ( 1 10) is configured to receive downmix information
indicating how the audio object signals are mixed within the one or more downmix
channels, and
wherein the signal processor ( 120) is configured to generate the one or more audio
output channels of the audio output signal from the audio input signal depending
on the downmix information, depending on the rendering information and
depending on the loudness compensation value.
. A decoder according to claim 13
wherein the receiving interface ( 1 10) is configured to receive one or more further
by-pass audio object signals, wherein the one or more further by-pass audio object
signals are not mixed within the downmix signal,
wherein the receiving interface ( 110) is configured to receive the loudness
information indicating information on the loudness of the audio object signals which
are mixed within the downmix signal and indicating information on the loudness of
the one or more further by-pass audio object signals which are not mixed within
the downmix signal, and
wherein the signal processor ( 120) is configured to determine the loudness
compensation value depending on the information on the loudness of the audio
object signals which are mixed within the downmix signal, and depending on the
information on the loudness of the one or more further by-pass audio object
signals which are not mixed within the downmix signal.
15. An encoder, comprising:
an object-based encoding unit (21 0; 7 10) for encoding a plurality of audio object
signals to obtain an encoded audio signal comprising the plurality of audio object
signals, and
an object loudness encoding unit (220; 720; 820) for encoding loudness
information on the audio object signals,
wherein the loudness information comprises one or more loudness values, wherein
each of the one or more loudness values depends on one or more of the audio
object signals.
16 . An encoder according to claim 15 ,
wherein each of the audio object signals of the encoded audio signal is assigned to
exactly one group of two or more groups, wherein each of the two or more groups
comprises one or more of the audio object signals of the encoded audio signal,
wherein the object loudness encoding unit (220; 720; 820) is configured to
determine the one or more loudness values of the loudness information by
determining a loudness value for each group of the two or more groups, wherein
said loudness value of said group indicates an total loudness of the one or more
audio object signals of said group.
An encoder according to claim 15 ,
wherein the object-based encoding unit (21 0; 7 10) is configured to receive the
audio object signals, wherein each of the audio object signals is assigned to
exactly one of exactly two groups, wherein each of the exactly two groups
comprises one or more of the audio object signals,
wherein the object-based encoding unit (21 0; 7 10) is configured to downmix the
audio object signals, being comprised by the exactly two groups, to obtain a
downmix signal comprising one or more downmix audio channels as the encoded
audio signal, wherein the number of the one or more downmix channels is smaller
than the number of the audio object signals being comprised by the exactly two
groups,
wherein the object loudness encoding unit (220; 720; 820) is assigned to receive
one or more further by-pass audio object signals, wherein each of the one or more
further by-pass audio object signals is assigned to a third group, wherein each of
the one or more further by-pass audio object signals is not comprised by the first
group and is not comprised by the second group, wherein the object-based
encoding unit (2 10 ; 7 0) is configured to not downmix the one or more further by¬
pass audio object signals within the downmix signal, and
wherein the object loudness encoding unit (220; 720; 820) is configured to
determine a first loudness value, a second loudness value and a third loudness
value of the loudness information, the first loudness value indicating a total
loudness of the one or more audio object signals of the first group, the second
loudness value indicating a total loudness of the one or more audio object signals
of the second group, and the third loudness value indicating a total loudness of the
one or more further by-pass audio object signals of the third group, or is configured
to determine a first loudness value and a second loudness value of the loudness
information, the first loudness value indicating a total loudness of the one or more
audio object signals of the first group, and the second loudness value indicating a
total loudness of the one or more audio object signals of the second group and of
the one or more further by-pass audio object signals of the third group.
A system comprising:
an encoder (31 0) according to one of claims 15 to 17 for encoding a plurality of
audio object signals to obtain an encoded audio signal comprising the plurality of
audio object signals, and
a decoder (320) according to one of claims 1 to 14 , for generating an audio output
signal comprising one or more audio output channels,
wherein the decoder (320) is configured to receive the encoded audio signal as an
audio input signal and to receive the loudness information
wherein the decoder (320) is configured to further receive rendering information,
wherein the decoder (320) is configured to determine a loudness compensation
value depending on the loudness information and depending on the rendering
information, and
wherein the decoder (320) is configured to generate the one or more audio output
channels of the audio output signal from the audio input signal depending on the
rendering information and depending on the loudness compensation value.
19. A method for generating an audio output signal comprising one or more audio
output channels, wherein the method comprises:
receiving an audio input signal comprising a plurality of audio object signals,
receiving loudness information on the audio object signals,
receiving rendering information indicating whether one or more of the audio object
signals shall be amplified or attenuated,
determining a loudness compensation value depending on the loudness
information and depending on the rendering information, and
generating the one or more audio output channels of the audio output signal from
the audio input signal depending on the rendering information and depending on
the loudness compensation vaiue.
A method for encoding, comprising:
encoding an audio input signal comprising a plurality of audio object signals, and
encoding loudness information on the audio object signals, wherein the loudness
information comprises one or more loudness values, wherein each of the one or
more loudness values depends on one or more of the audio object signals.
A computer program for implementing the method of claim 19 or 20 when being
executed on a computer or signal processor.