FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
(See section 10, rule 13)
“SIGNAL PROCESSING DEVICE, SIGNAL PROCESSING METHOD,
AND PROGRAM”
SONY CORPORATION, of 1-7-1, Konan, Minato-ku, Tokyo
108-0075, Japan
The following specification particularly describes the invention and the manner in which it is to
be performed.
1
SYP329357WO01
DESCRIPTION
SIGNAL PROCESSING DEVICE, SIGNAL PROCESSING METHOD, AND
PROGRAM
5 TECHNICAL FIELD
[0001]
The present technology relates to a signal
processing device that performs signal processing on
signals from a plurality of microphones, a method
10 thereof, and a program, and particularly relates to a
technique to compensate for a signal of a clipped
microphone when performing an echo cancellation process
on signals of a plurality of microphones.
15 BACKGROUND ART
[0002]
In recent years, devices called smart speakers and
the like in which a plurality of microphones and a
speaker are provided in the same casing have become
20 widespread. Some devices of this type estimate a speech
direction of a user or speech content (voice recognition)
on the basis of signals from a plurality of microphones.
Operations such as directing the front of the device to
the user speech direction on the basis of the estimated
25 speech direction, having a conversation with the user on
the basis of a voice recognition result, and the like
have been achieved.
[0003]
In this type of device, the positions of the
30 plurality of microphones are usually closer to the
speaker compared to the position of the user, and during
2
SYP329357WO01
loud sound reproduction by the speaker, in a process of
A/D converting a signal of a microphone, a phenomenon
called a clip occurs in which quantized data sticks to a
maximum value.
5 [0004]
Note that as a related conventional technique,
Patent Document 1 below discloses a technique that
achieves, in a system for recording signals from a
plurality of microphones, clip compensation by replacing
10 the waveform of a clipped portion in a signal of a
clipped microphone with the waveform of a signal of a
non-clipped microphone.
CITATION LIST
15 PATENT DOCUMENT
[0005]
Patent Document 1: Japanese Patent Application Laid-Open
No. 2010-245657
20 SUMMARY OF THE INVENTION
PROBLEMS TO BE SOLVED BY THE INVENTION
[0006]
Here, in the device such as a smart speaker, an
echo cancellation process may be performed to suppress an
25 output signal component of the speaker included in
signals from a plurality of microphones. By performing
such an echo cancellation process, it is possible to
improve accuracy of speech direction estimation and voice
recognition under sound output performed by the speaker.
30 [0007]
The present technology has been made in view of the
3
SYP329357WO01
above circumstances, and an object thereof is to increase
compensation accuracy with respect to clip compensation
in a case where signals from a plurality of microphones
are subjected to an echo cancellation process.
5
SOLUTIONS TO PROBLEMS
[0008]
A signal processing device according to an
embodiment of the present technology includes an echo
10 cancellation unit that performs an echo cancellation
process of canceling an output signal component from a
speaker on signals from a plurality of microphones, a
clip detection unit that performs a clip detection for
signals from the plurality of microphones, and a clip
15 compensation unit that compensates for a signal after the
echo cancellation process of clipped one of the
microphones on the basis of a signal of non-clipped one
of the microphones.
[0009]
20 In a case where the echo cancellation process is
performed on signals from the plurality of microphones,
when the clip compensation is performed on a signal
before the echo cancellation process, the clip
compensation is performed in a state that an output
25 signal component of the speaker and other components
including a target sound are difficult separate, and thus
clip compensation accuracy tends to decrease. By
performing the clip compensation on the signal after the
echo cancellation process as described above, it is
30 possible to perform the clip compensation on a signal in
which the output signal component of the speaker is
4
SYP329357WO01
suppressed to some extent.
[0010]
In the signal processing device described above
according to the present technology, it is desirable that
5 the clip compensation unit compensates for a signal of
the clipped microphone by suppressing the signal.
[0011]
By employing a compensation method of suppressing
the signal of the clipped microphone, it is possible to
10 prevent phase information of the signal of the clipped
microphone from being lost by the compensation.
[0012]
In the signal processing device described above
according to the present technology, it is desirable that
15 the clip compensation unit suppresses a signal of the
clipped microphone on the basis of an average power ratio
between a signal of the non-clipped microphone and a
signal of the clipped microphone.
[0013]
20 Thus, power of the signal of the clipped microphone
can be appropriately suppressed to power after the echo
cancellation process that has to be obtained in a case
where it is not clipped.
[0014]
25 In the signal processing device described above
according to the present technology, it is desirable that
the clip compensation unit uses, as the average power
ratio, an average power ratio with a signal of the
microphone having a minimum average power among the
30 signals of the non-clipped microphones is used.
[0015]
5
SYP329357WO01
The microphone with the minimum average power can
be restated as the microphone in which it is most
difficult for clipping to occur.
[0016]
5 In the signal processing device described above
according to the present technology, it is desirable that
the clip compensation unit adjusts a suppression amount
of a signal of the clipped microphone according to a
speech level in a case where a user speech is present and
10 a speaker output is present.
[0017]
In what is called a double talk section in which a
user speech is present and a speaker output is present,
if the speech level of the user is high, the speech
15 component is also included in a large amount even in the
noise superposed section due to clipping (note that the
double talk mentioned here means that the user speech and
the speaker output overlap in time as illustrated in Fig.
9). On the other hand, in a case where the speech level
20 is low, the speech component tends to be buried in large
clipping noise. Accordingly, in the double talk section,
the suppression amount of the signal of the clipped
microphone is adjusted according to the speech level.
Thus, if the speech level of the user is high, it
25 is possible to reduce the suppression amount of the
signal to prevent the speech component from being
suppressed, and when the speech level of the user is low,
it is possible to increase the suppression amount of the
signal to suppress the clipping noise.
30 [0018]
In the signal processing device described above
6
SYP329357WO01
according to the present technology, it is desirable that
the clip compensation unit suppresses a signal of the
clipped microphone by a suppression amount according to a
characteristic of a voice recognition process in a
5 subsequent stage in a case where a user speech is present
and no speaker output is present.
[0019]
The case where a user speech is present and no
speaker output is present is a case where a cause of a
10 clip is estimated to be the user speech. With the above
configuration, in the case where the cause of the clip is
estimated to be the user speech, for example, it is
possible to perform the clip compensation with an
appropriate suppression amount according to
15 characteristics of the voice recognition process in the
subsequent stage such that the voice recognition accuracy
can be maintained better in a case where there is a
certain degree of speech level even if clipping noise is
superposed than in a case where the speech component is
20 suppressed, or the like.
[0020]
In the signal processing device described above
according to the present technology, it is desirable that
the clip compensation unit does not perform the
25 compensation for the clipped microphone signal in a case
where a user speech is present and no speaker output is
present.
[0021]
In the case where the user speech is present and
30 the speaker output is not present, that is, a case where
the cause of the clip is estimated to be the user speech,
7
SYP329357WO01
it is empirically known that not suppressing the signal
can result in a more favorable voice recognition result
in the subsequent stage. In such a case, it is possible
to improve the voice recognition accuracy by not
5 performing the clip compensation as described above.
[0022]
In the signal processing device described above
according to the present technology, it is desirable to
further includes a drive unit that changes a position of
10 at least one of the plurality of microphones or the
speaker, and a control unit that changes the position of
at least one of the plurality of microphones or the
speaker by the drive unit in response to detection of a
clip by the clip detection unit.
15 [0023]
Thus, if a clip is detected, it is possible to
change the positional relationship among the respective
microphones and the speaker, or move the positions of the
plurality of microphones or the speaker to a position
20 where wall reflection or the like is small.
[0024]
Further, a signal processing method according to
the present technology includes an echo cancellation
procedure to perform an echo cancellation process of
25 canceling an output signal component from a speaker on
signals from a plurality of microphones, a clip detection
procedure to perform a clip detection for signals from
the plurality of microphones, and a clip compensation
procedure to compensate for a signal after the echo
30 cancellation process of clipped one of the microphones on
the basis of a signal of non-clipped one of the
8
SYP329357WO01
microphones.
[0025]
Also with such a signal processing method,
operations similar to those of the signal processing
5 device described above according to the present
technology can be obtained.
[0026]
Moreover, a program according to the present
technology is a program executed by an information
10 processing device, the program causing the information
processing device to implement functions including an
echo cancellation function to perform an echo
cancellation process of canceling an output signal
component from a speaker on signals from a plurality of
15 microphones, a clip detection function to perform a clip
detection for signals from the plurality of microphones,
and a clip compensation function to compensate for a
signal after the echo cancellation process of clipped one
of the microphones on the basis of a signal of non20
clipped one of the microphones.
[0027]
The signal processing device according to such
present technology described above is achieved by a
program according to the present technology.
25
EFFECTS OF THE INVENTION
[0028]
With the present technology, it is possible to
increase compensation accuracy with respect to clip
30 compensation in a case where signals from a plurality of
microphones are subjected to an echo cancellation
9
SYP329357WO01
process.
Note that the effect described here is not
necessarily limited, and may be any effect described in
the present disclosure.
5
BRIEF DESCRIPTION OF DRAWINGS
[0029]
Fig. 1 is a perspective view illustrating an
external appearance configuration example of a signal
10 processing device as an embodiment according to the
present technology.
Fig. 2 is an explanatory diagram of a microphone
array included in the signal processing device as the
embodiment.
15 Fig. 3 is a block diagram for explaining an
electrical configuration example of the signal processing
device as the embodiment.
Fig. 4 is a block diagram illustrating an internal
configuration example of a voice signal processing unit
20 included in the signal processing device as the
embodiment.
Fig. 5 is a diagram illustrating an image of a
clip.
Fig. 6 is a flowchart for explaining an operation
25 of the signal processing device as the embodiment.
Fig. 7 is a diagram for explaining a basic concept
of an echo cancellation process.
Fig. 8 is a diagram illustrating an internal
configuration example of an AEC processing unit included
30 in the signal processing device as the embodiment.
Fig. 9 is an explanatory diagram of a double talk.
10
SYP329357WO01
Fig. 10 is an explanatory diagram for selectively
executing a process related to clip compensation in each
case.
Fig. 11 is a diagram illustrating a behavior of a
5 sigmoid function employed in the embodiment.
Fig. 12 is a diagram schematically representing a
clip compensation method in a conventional technique.
Fig. 13 is an explanatory diagram of a problem in
the conventional technique.
10 Fig. 14 is a flowchart illustrating a specific
processing procedure to be executed to implement the clip
compensation method as the embodiment.
MODE FOR CARRYING OUT THE INVENTION
15 [0030]
Hereinafter, an embodiment according to the present
technology will be described in the following order with
reference to the accompanying drawings.
<1. External appearance configuration of signal
20 processing device>
<2. Electrical configuration of signal processing
device>
<3. Operation of signal processing device>
<4. Echo cancellation method in embodiment>
25 <5. Clip compensation method as embodiment>
<6. Processing procedure>
<7. Modification example>
<8. Summary of embodiment>
<9. Present technology>
30 [0031]
<1. External appearance configuration of signal
11
SYP329357WO01
processing device>
Fig. 1 is a perspective view illustrating an
external appearance configuration example of a signal
processing device 1 as an embodiment according to the
5 present technology.
As illustrated in the diagram, the signal
processing device 1 includes a substantially columnar
casing 11 and a substantially columnar movable unit 14
located above the casing 11.
10 The movable unit 14 is supported by the casing 11
so as to be rotatable in the direction indicated by an
outline double-headed arrow in the diagram (rotation in a
pan direction). The casing 11 does not rotate in
conjunction with the movable unit 14, for example, in a
15 state of being placed on a predetermined position of a
table, a floor, or the like, and forms what is called a
fixed portion.
The movable unit 14 is rotationally driven by a
servo motor 21 (described later with reference to Fig. 3)
20 incorporated in the signal processing device 1 as a drive
unit.
[0032]
A microphone array 12 is provided at an upper end
of the casing 11.
25 As illustrated in Fig. 2, the microphone array 12
is configured by arranging a plurality of (eight in the
example of Fig. 2) microphones 13 on a circumference at
substantially equal intervals.
Since the microphone array 12 is provided on the
30 casing 11 side rather than on the movable unit 14 side,
the position of each microphone 13 remains unchanged even
12
SYP329357WO01
when the movable unit 14 rotates. That is, the position
of each microphone 13 in the space 100 does not change
even when the movable unit 14 rotates.
[0033]
5 The movable unit 14 is provided with a display unit
15 including, for example, a liquid crystal display
(LCD), an electro-luminescence (EL) display, or the like.
In this example, a picture of a face is displayed on the
display unit 15, and the direction in which the face
10 faces is a front direction of the signal processing
device 1. As will be described later, the movable unit
14 is rotated so that the display unit 15 faces the
speech direction, for example.
[0034]
15 Further, in the movable unit 14, a speaker 16 is
housed on a back side of the display unit 15. The
speaker 16 outputs sounds such as a message and music to
the user.
[0035]
20 The signal processing device 1 as described above
is arranged in, for example, a space 100 such as a room.
The signal processing device 1 is incorporated in,
for example, a smart speaker, a voice agent, a robot, or
the like, and has a function of estimating the speech
25 direction of a voice when the voice is emitted from a
surrounding sound source (for example, a person). The
estimated direction is used to direct the front of the
signal processing device 1 toward the speech direction.
[0036]
30 <2. Electrical configuration of signal processing
device>
13
SYP329357WO01
Fig. 3 is a block diagram for explaining an
electrical configuration example of the signal processing
device 1.
As illustrated in the diagram, the signal
5 processing device 1 includes, together with the
microphone array 12, the display unit 15, and the speaker
16 illustrated in Fig. 1, a voice signal processing unit
17, a control unit 18, a display drive unit 19, a motor
drive unit 20, and a voice drive unit 22.
10 [0037]
The voice signal processing unit 17 can include,
for example, a digital signal processor (DSP), or a
computer device having a central processing unit (CPU),
or the like, and processes a signal from each microphone
15 13 in the microphone array 12.
Note that although not illustrated, the signal from
each microphone 13 is analog-digital converted by an A-D
converter and then input to the voice signal processing
unit 17.
20 [0038]
The voice signal processing unit 17 includes an
echo component suppression unit 17a and a voice
extraction processing unit 17b, and a signal from each
microphone 13 is input to the voice extraction processing
25 unit 17b via the echo component suppression unit 17a.
The echo component suppression unit 17a performs an
echo cancellation process for suppressing an output
signal component from the speaker 16 included in the
signal of each microphone 13, using an output voice
30 signal Ss described later as a reference signal. Note
that the echo component suppression unit 17a of this
14
SYP329357WO01
example performs clip compensation for the signal from
each microphone 13, which will be described later.
[0039]
The voice extraction processing unit 17b performs
5 extraction of a target sound (voice extraction) by
estimating the speech direction, emphasizing the signal
of the target sound, and suppressing noise on the basis
of the signal of each microphone 13 input via the echo
component suppression unit 17a. The voice extraction
10 processing unit 17b outputs an extracted voice signal Se
to the control unit 18 as a signal obtained by extracting
the target sound. Further, the voice extraction
processing unit 17b outputs information indicating the
estimated speech direction to the control unit 18 as
15 speech direction information Sd.
Note that details of the voice extraction
processing unit 17b will be described again.
[0040]
The control unit 18 includes a microcomputer
20 having, for example, a CPU, a read only memory (ROM), a
random access memory (RAM), and the like, and performs
overall control of the signal processing device 1 by
executing a process according to a program stored in the
ROM.
25 For example, the control unit 18 performs control
related to display of information by the display unit 15.
Specifically, an instruction is given to the display
drive unit 19 having a driver circuit for driving display
of the display unit 15 to cause the display unit 15 to
30 execute display of various types of information.
[0041]
15
SYP329357WO01
Further, the control unit 18 of this example
includes a voice recognition engine that is not
illustrated, and performs a voice recognition process on
the basis of the extracted voice signal Se input from the
5 voice signal processing unit 17 (voice extraction
processing unit 17b) by the voice recognition engine, and
also determines a process to be executed on the basis of
the result of the voice recognition process.
Note that in a case where the control unit 18 is
10 connected to a cloud 60 via the Internet or the like and
a voice recognition engine exists in the cloud 60, the
voice recognition engine can be used to perform the voice
recognition process.
[0042]
15 Further, when the control unit 18 inputs the speech
direction information Sd from the voice signal processing
unit 17 accompanying detection of a speech, the control
unit 18 calculates a rotation angle of the servo motor 21
necessary for directing the front of the signal
20 processing device 1 in the speech direction, and outputs
information indicating the rotation angle to the motor
drive unit 20 as rotation angle information.
The motor drive unit 20 includes a driver circuit
or the like for driving the servo motor 21, and drives
25 the servo motor 21 on the basis of the rotation angle
information input from the control unit 18.
[0043]
Moreover, the control unit 18 controls sound output
by the speaker 16. Specifically, the control unit 18
30 outputs a voice signal to the voice drive unit 22
including a driver circuit (including a D-A converter, an
16
SYP329357WO01
amplifier, and the like) and the like for driving the
speaker 16, so as to cause the speaker 16 to execute
voice output according to the voice signal.
Note that hereinafter, the voice signal output by
5 the control unit 18 to the voice drive unit 22 in this
manner will be referred to as an “output voice signal
Ss”.
[0044]
Fig. 4 is a block diagram illustrating an internal
10 configuration example of the voice signal processing unit
17.
As illustrated, the voice signal processing unit 17
includes the echo component suppression unit 17a and the
voice extraction processing unit 17b illustrated in Fig.
15 3, and the echo component suppression unit 17a includes a
clip detection unit 30, a fast Fourier transformation
(FFT) processing unit 31, an acoustic echo cancellation
(AEC) processing unit 32, a clip compensation unit 33,
and an FFT processing unit 34, and the voice extraction
20 processing unit 17b includes a speech section estimation
unit 35, a speech direction estimation unit 36, a voice
emphasis unit 37, and a noise suppression unit 38.
[0045]
In the echo component suppression unit 17a, the
25 clip detection unit 30 performs clip detection on the
signal from each microphone 13.
Fig. 5 illustrates an image of a clip. The clip
means a phenomenon in which quantized data sticks to the
maximum value during A-D conversion.
30 In response to detection of the clip, the clip
detection unit 30 outputs information indicating the
17
SYP329357WO01
channel of the microphone 13 in which the clip is
detected to the clip compensation unit 33.
[0046]
In the echo component suppression unit 17a, the
5 signal from each microphone 13 is input to the FFT
processing unit 31 via the clip detection unit 30. The
FFT processing unit 31 performs orthogonal transformation
by FFT on the signal from each microphone 13 input as a
time signal to convert the signal into a frequency
10 signal.
Further, the FFT processing unit 34 performs
orthogonal transformation by FFT on the output voice
signal Ss input as a time signal to convert the signal
into a frequency signal.
15 Here, the orthogonal transformation is not limited
to the FFT, and for example, other techniques such as
discrete cosine transformation (DCT) can also be
employed.
[0047]
20 To the AEC processing unit 32, the signals from the
respective microphones 13 converted into frequency
signals respectively by the FFT processing unit 31 and
the FFT processing unit 34 and the output voice signal Ss
are input.
25 The AEC processing unit 32 performs processing of
canceling the echo component included in the signal from
each microphone 13 on the basis of the input output voice
signal Ss. That is, the voice output from the speaker 16
may be delayed by a predetermined time, and may be picked
30 up by the microphone array 12 as an echo mixed with other
voices. The AEC processing unit 32 uses the output voice
18
SYP329357WO01
signal Ss as a reference signal and performs processing
so as to cancel the echo component from the signal of
each microphone 13.
Further, the AEC processing unit 32 of this example
5 performs a process related to double talk evaluation as
described later, which will be described again.
[0048]
The clip compensation unit 33 performs, for the
signal of each microphone 13 after the echo cancellation
10 process by the AEC processing unit 32, clip compensation
based on a detection result by the clip detection unit 30
and the output voice signal Ss as a frequency signal
input via the FFT processing unit 34.
In the present example, to the clip compensation
15 unit 33, a double talk evaluation value Di generated by
the AEC processing unit 32 performing the evaluation
related to a double talk is input, and the clip
compensation unit 33 performs clip compensation on the
basis of the double talk evaluation value Di, which will
20 be explained again.
[0049]
In the voice extraction processing unit 17b, the
signal from each microphone 13 via the clip compensation
unit 33 is input to each of the speech section estimation
25 unit 35, the speech direction estimation unit 36, and the
voice emphasis unit 37.
[0050]
The speech section estimation unit 35 performs a
process of estimating a speech section (a section of a
30 speech in the time direction) on the basis of the input
signal from each microphone 13, and outputs the speech
19
SYP329357WO01
section information Sp that is information indicating the
speech section to the speech direction estimation unit 36
and the voice emphasis unit 37.
Note that various methods, for example, methods
5 using artificial intelligence (AI) technology (such as
deep learning) and the like are conceivable as a specific
method for estimating the speech section, and because
these methods are not directly related to the present
technology, a description of specific processing is
10 omitted.
[0051]
The speech direction estimation unit 36 estimates
the speech direction on the basis of the signal from each
microphone 13 and the speech section information Sp. The
15 speech direction estimation unit 36 outputs information
indicating the estimated speech direction as the speech
direction information Sd.
Note that as a method of estimating the speech
direction, various methods such as an estimation method
20 on the basis of Multiple Signal Classification (MUSIC)
method, specifically, MUSIC method using generalized
eigenvalue decomposition can be mentioned, for example.
However, the method for estimating the speech direction
is not directly related to the present technology, and a
25 description of a specific process will be omitted.
[0052]
The voice emphasis unit 37 emphasizes a signal
component corresponding to a target sound (speech sound
here) among signal components included in the signal from
30 each microphone 13 on the basis of the speech direction
information Sd output by the speech direction estimation
20
SYP329357WO01
unit 36 and the speech section information Sp output by
the speech section estimation unit 35. Specifically, a
process of emphasizing the component of a sound source
existing in the speech direction is performed by beam
5 forming.
[0053]
The noise suppression unit 38 suppresses a noise
component (mainly a stationary noise component) included
in the output signal from the voice emphasis unit 37.
10 The output signal from the noise suppression unit
38 is output from the voice extraction processing unit
17b as the extracted voice signal Se described above.
[0054]
<3. Operation of signal processing device>
15 Next, an operation of the signal processing device
1 will be described with reference to a flowchart in Fig.
6.
Note that in Fig. 6, operations related to echo
cancellation by the AEC processing unit 32 and clip
20 compensation by the clip compensation unit 33 are
omitted.
[0055]
In Fig. 6, first, in step S1, the microphone array
12 inputs a voice. That is, a voice generated by a
25 speaking person is input.
In step S2, the speech direction estimation unit 36
executes a speech direction estimation process.
In step S3, the voice emphasis unit 37 emphasizes a
signal. That is, a voice component in a direction
30 estimated as the speech direction is emphasized.
Moreover, in step S4, the noise suppression unit 38
21
SYP329357WO01
suppresses the noise component and improves the signalto-
noise ratio (SNR).
[0056]
In step S5, the control unit 18 (or an external
5 voice recognition engine existing in the cloud 60)
performs a process of recognizing a voice. That is, the
process of recognizing a voice is performed on the basis
of the extracted voice signal Se input from the voice
signal processing unit 17. Note that the recognition
10 result is converted into a text as necessary.
[0057]
In step S6, the control unit 18 determines an
operation. That is, an operation corresponding to
content of the recognized voice is determined. Then, in
15 step S7, the control unit 18 controls the motor drive
unit 20 to drive the movable unit 14 by the servo motor
21.
Moreover, in step S8, the control unit 18 causes
the voice drive unit 22 to output the voice from the
20 speaker 16.
[0058]
Thus, for example, when a greeting such as “hi” is
recognized from the speaking person, the movable unit 14
is rotated in the direction of the speaking person, and a
25 greeting such as “hi, how are you?” is sent to the
speaking person from the speaker 16.
[0059]
<4. Echo cancellation method in embodiment>
Here, prior to description of clip compensation as
30 an embodiment, first, an echo cancellation method that is
assumed in the embodiment will be described.
22
SYP329357WO01
A basic concept of an echo cancellation process
will be described with reference to Fig. 7.
First, an output signal (output voice signal Ss)
from the speaker 16 in a certain time frame n is referred
5 to as a reference signal x(n). The reference signal x(n)
is output from the speaker 16 and then input to the
microphone 13 through the space. At this time, the
signal (sound collection signal) obtained by the
microphone 13 is referred to as a microphone input signal
10 d(n).
[0060]
A spatial transfer characteristic h until an output
sound from the speaker 16 reaches the microphone 13 is
unknown, and in the echo cancellation process, this
15 unknown spatial transfer characteristic h is estimated,
and the reference signal x(n) considering the estimated
spatial transfer characteristic is subtracted from the
microphone input signal d(n). The estimated spatial
transfer characteristic will be referred to as an
20 estimated transfer characteristic w(n) below.
[0061]
The output sound of the speaker 16 that reaches the
microphone 13 includes a component having a certain time
delay, such as a sound that directly arrives is reflected
25 on a wall or the like and returns, and thus when a target
delay time in the past is represented by a tap length L,
the microphone input signal d(n) and the estimated
transfer characteristic w(n) can be represented as the
following [Formula 1] and [Formula 2].
30 [Mathematical Formula 1]
23
SYP329357WO01
... [Formula 1]
... [Formula 2]
In [Formula 1], T represents transposition.
[0062]
5 In practice, the number of frequency bins N that
has been subjected to fast Fourier transformation for the
time frame n is estimated. In a case where a general
least mean square (LMS) method is used, an echo
cancellation process at a frequency k (k = 1 to N) is
10 performed with the following [Formula 3] and [Formula 4].
[Mathematical Formula 2]
... [Formula 3]
... [Formula 4]
H represents a Hermitian transposition and *
15 represents a complex conjugate. μ is a step size that
determines the learning speed, and normally a value
between 0 < μ ≤ 2 is selected.
As illustrated in [Formula 3], an error signal
e(k,n) is obtained by subtracting an estimated sneak
20 signal obtained as a reference signal (x) for L tap
lengths convolving an estimated transfer characteristic
w(k,n) from a microphone input signal d(k,n).
As can be seen from Fig. 7, this error signal
e(k,n) corresponds to an output signal of the echo
25 cancellation process.
In the LMS method, w is sequentially updated so
that the average power of the error signal e(k,n) is
24
SYP329357WO01
minimized.
Note that in addition to the LMS method, there are
methods such as normalized LMS (NLMS) obtained by
normalizing an update-type reference signal, affine
5 projection algorithm (APA), recursive least square (RLS),
and the like. In any of the methods, the reference
signal x is used to learn the estimated transfer
characteristic.
[0063]
10 Here, the AEC processing unit 32 is usually
configured to reduce the learning speed during the double
talk by a configuration as illustrated in Fig. 8 in order
to avoid erroneous learning during a double talk.
The double talk mentioned here means that a user
15 speech and a speaker output are temporally overlapped, as
illustrated in Fig. 9.
[0064]
In Fig. 8, the AEC processing unit 32 includes an
echo cancellation processing unit 32a and a double talk
20 evaluation unit 32b.
Here, in the following description, the notations
of time n and frequency bin number k will be omitted
unless time information and frequency information are
handled in the description.
25 [0065]
The double talk evaluation unit 32b calculates a
double talk evaluation value Di representing certainty of
whether or not it is during the double talk on the basis
of the output voice signal Ss by a frequency signal input
30 via the FFT processing unit 34, that is, the reference
signal x, and the signal (error signal e) of each
25
SYP329357WO01
microphone 13 that has undergone the echo cancellation
process by the echo cancellation processing unit 32a.
[0066]
The echo cancellation processing unit 32a
5 calculates the error signal e according to [Formula 3]
described above on the basis of the signal from each
microphone 13 input via the FFT processing unit 31, that
is, the microphone input signal d, and the output voice
signal Ss input via the FFT processing unit 34 (that is,
10 the reference signal x).
Further, the echo cancellation processing unit 32a
sequentially learns the estimated transfer characteristic
w according to [Formula 6] described later, on the basis
of the error signal e, the reference signal x, and the
15 double talk evaluation value Di input from the double
talk evaluation unit 32b.
[0067]
Here, various methods for evaluating double talk
have been proposed, but as a typical method, there is a
20 method using fluctuations of average power of the
reference signal x and instantaneous signal power after
an echo cancellation process (Wiener type double talk
determination unit). In this method, the double talk
evaluation value Di becomes a value close to “1” during
25 normal learning and behaves so as to approach “0” during
the double talk.
[0068]
Specifically, in this example, the double talk
evaluation value Di is calculated by the following
30 [Formula 5].
[Mathematical Formula 3]
26
SYP329357WO01
... [Formula 5]
In [Formula 5], “Pref^¯” (note that “^¯” means that
“¯” is written above “Pref”) is “Pref^¯ = E[xxH]”, and
means the average power of the reference signal x
5 (however, E[⋅ ] represents an expected value). Further,
“β” is a sensitivity adjustment constant.
[0069]
During the double talk, the error signal e
increases due to the influence of the speech component.
10 Therefore, according to [Formula 5], the double talk
evaluation value Di becomes small during the double talk.
Conversely, if it is during a non-double talk and the
error signal e is small, the double talk evaluation value
Di becomes large.
15 [0070]
The echo cancellation processing unit 32a learns
the estimated transfer characteristic w according to
following [Formula 6] on the basis of the double talk
evaluation value Di as described above.
20 [Mathematical Formula 4]
... [Formula 6]
Thus, during the double talk in which the double
talk evaluation value Di becomes small, the learning
speed by an adaptive filter is reduced, and erroneous
25 learning during the double talk is suppressed.
[0071]
<5. Clip compensation method as embodiment>
Next, a clip compensation method as an embodiment
will be described.
27
SYP329357WO01
First, as a premise, when a signal clipped by a
time signal is decomposed into frequency components by
Fourier transformation, a signal that originally does not
exist during transmission in the space appears as noise
5 at each frequency (clipping noise). This clipping noise
cannot be removed by a linear echo canceller as used in
this example, and an erasure residue in large volume
occurs only at the moment of clipping. This erasure
residue component is generated over a wide area and
10 becomes a factor that deteriorates accuracy of voice
recognition in a subsequent stage.
In the present embodiment, clip compensation is
performed in consideration of such a premise.
[0072]
15 In the present embodiment, the clip compensation
unit 33 (see Fig. 4) determines whether or not there is a
channel in which a clip has occurred (a channel of the
microphone 13) on the basis of the detection result of
the clip detection unit 30. Then, if there is a channel
20 in which a clip has occurred, a clip compensation process
described below is applied to the signal after the echo
cancellation process for this channel.
[0073]
In the present embodiment, the clip compensation
25 process is performed on the basis of the signal of the
microphone 13 that is not clipped. Specifically, it is
performed by suppressing the signal of the clipped
microphone 13 on the basis of the average power ratio
between the signal of the non-clipped microphone 13 and
30 the signal of the clipped microphone 13.
In the following example, as the average power
28
SYP329357WO01
ratio described above, the ratio to the minimum average
power among non-clipped channels is used.
[0074]
In the present embodiment, the clip compensation
5 process is basically performed by the method represented
by the following [Formula 7].
Here, in the following, a signal after clip
compensation is expressed as “ei^~” (note that “^~” means
that “~” is written above “ei”).
10 [Mathematical Formula 5]
... [Formula 7]
In [Formula 7], “ei” represents an instantaneous
signal after the echo cancellation process of an i
channel (clipped channel), and “eMin” represents an
15 instantaneous signal after the echo cancellation process
of the channel with the minimum average power among the
non-clipped channels.
Further, “Pi^¯” (“^¯” means that “¯” is written
above “Pi”) is “Pi^¯ = E[eiei
H]”, and represents the
20 average power of the signal after the echo cancellation
process for i channel, and “PMin^¯” (“^¯” means that “¯”
is written above “PMin”) means the minimum average power
among the non-clipped channels.
The average power here means the average power in a
25 section where a speaker output is present and no clipping
is present.
[0075]
The basic concept of the clip compensation
according to [Formula 7] can be explained as follows.
29
SYP329357WO01
That is, only phase information is extracted from
the signal of the clipped channel (i), and the signal
power is replaced with the instantaneous power of the
non-clipped channel (in this example, the channel with
5 the minimum average power). However, if left as it is,
the signal power after the echo cancellation process that
has to be output in a case where no clipping has occurred
will not be achieved, and thus the replaced signal power
is corrected using a signal power ratio between channels
10 that has been sequentially obtained.
In other words, the clipping compensation according
to [Formula 7] can be represented as to suppress a nonlinear
component that is an erasure residue after the
echo cancellation process, and perform gain correction on
15 the signal of the clipped channel to an estimated
suppression level when it is not clipped, on the basis of
the microphone input signal information of the nonclipped
channel.
[0076]
20 Here, the fact that only the phase information is
extracted from the signal of the clipped channel as
described above is expressed by the terms “1/eiei
H” and
“ei” in [Formula 7].
Further, the point that the signal power is
25 replaced with the instantaneous power of the non-clipped
channel is expressed by the term “eMineH
Min” in [Formula
7].
Moreover, the point that the replaced signal power
is corrected using the signal power ratio between
30 channels that has been sequentially obtained is expressed
by the term “Pi^¯/PMin^¯” in [Formula 7].
30
SYP329357WO01
[0077]
Note that the reason for a difference to occur in
the signal power ratio between channels is that a
difference occurs between signals of respective channels
5 due to a directivity characteristic of the speaker 16, a
transmission path in the space, microphone sensitivity
variation, and stationary noise having directivity, or
the like.
[0078]
10 In the clip compensation of the present embodiment,
regarding the clipped channel, the waveform itself of the
signal is not replaced with the waveform of another
channel, and the phase information is left. By doing so,
the phase relationship among the microphones 13 is
15 prevented from being destroyed due to the clip
compensation. Since the phase relationship among the
microphones 13 is important in the speech direction
estimation process, the present method can prevent speech
direction estimation accuracy from being deteriorated due
20 to the clip compensation. That is, beamforming by the
voice emphasis unit 37 is less likely to fail, and the
voice recognition accuracy by the voice recognition
engine in the subsequent stage can be improved.
[0079]
25 Here, average powers as “Pi^¯” and “PMin^¯” are
sequentially calculated by the clip compensation unit 33
in a section in which no clip has occurred and a speaker
output is present. At this time, the clip compensation
unit 33 identifies the section in which no clip has
30 occurred and a speaker output is present on the basis of
the detection result by the clip detection unit 30, and
31
SYP329357WO01
the output voice signal Ss (reference signal x) input
through the FFT processing unit 34.
[0080]
As the clip compensation, the compensation by
5 [Formula 7] can always be performed at least for a user
speech section, but in this example, dividing into cases
as illustrated in next Fig. 10 is performed, and a
process related to the clip compensation is selectively
executed corresponding to each of the cases.
10 Specifically, in a case where both the speaker
output and the user speech are “present”, which is
represented as “Case 1” in the diagram, the suppression
amount in the clip compensation is adjusted according to
the user speech while performing the clip compensation.
15 Further, in a case where the speaker output is
“present” and the user speech is “none” as “Case 2”, the
clip compensation is performed.
In a case where the speaker output is “none” and
the user speech is “present” as “Case 3”, a process
20 corresponding to the voice recognition engine is
performed.
In a case where both the speaker output and the
user speech are “none” as “case 4”, the clip compensation
is not performed. In this case, the signal after the
25 echo cancellation process is discarded before voice
recognition.
Note that a cause of clipping in Case 1 can be
presumed to be a double talk as illustrated in the
diagram. Further, it can be estimated that the causes of
30 clipping in Case 2, Case 3, and Case 4 are sneaking into
speaker, user speech, and noise, respectively.
32
SYP329357WO01
[0081]
First, the clip compensation that is performed in
the case of Case 1 and that involves the suppression
amount adjustment according to the user speech level will
5 be described.
In a case where the user speech level is high,
information of the target sound (speech sound) tends to
be mostly included also in a superposition section of
clipping noise, and thus the signal suppression amount in
10 the clip compensation is preferred to be reduced for the
voice recognition process in the subsequent stage. On
the contrary, in a case where the user speech level is
low, the speech component tends to be buried in large
clipping noise, and thus increasing the signal
15 suppression amount in the clip compensation is preferred
for the voice recognition process in the subsequent
stage.
[0082]
Accordingly, in Case 1, the clip compensation
20 involving adjustment of the suppression amount according
to the user speech level is performed by the following
[Formula 8].
[Mathematical Formula 6]
... [Formula 8]
25 In [Formula 8], “αdt” is a suppression amount
correction coefficient, the signal suppression amount is
maximum when αdt is “1”, and the signal suppression
amount is reduced as αdt becomes larger than “1”.
[0083]
33
SYP329357WO01
In Case 1, the value of the suppression amount
correction coefficient αdt is adjusted according to the
speech level.
The following [Formula 9] illustrates an example of
5 an adjustment formula of the suppression amount
correction coefficient αdt. [Formula 9] exemplifies an
adjustment formula using a sigmoid function, where “a” is
a sigmoid function inclination constant and “c” is a
sigmoid function center correction constant.
10 [Mathematical Formula 7]
... [Formula 9]
In [Formula 9], “Pdti^¯” (“^¯” means that “¯” is
written above “Pdti”) is “Pdti^¯ = E[eiei
H]” and represents
the average power of the signal after the echo
15 cancellation processing of an i channel during the double
talk and in a non-clipped section. Such “Pdti^¯” can be
treated as an estimated value of the user speech level.
“Max” is a value represented by the following
[Formula 10] and [Formula 11], and means the maximum
20 value of the suppression amount correction coefficient
αdt. That is, it is a value that makes “ei^~” calculated
by [Formula 8] the same power as “ei” input from the AEC
processing unit 32, in other words, a value that cancels
the clip compensation (or that brings the signal
25 suppression amount into a maximally lowered state).
[Mathematical Formula 8]
... [Formula 10]
34
SYP329357WO01
... [Formula 11]
[0084]
Fig. 11 illustrates a behavior of the sigmoid
function according to [Formula 9].
5 According to the adjustment formula represented by
[Formula 9], the value of the suppression amount
correction coefficient αdt changes from “1” to “Max”
accompanying that the magnitude of “Pdti^¯” as a user
speech level estimated value changes. Specifically, in a
10 case where the speech level estimated value “Pdti^¯” is
large, the value of the suppression amount correction
coefficient αdt approaches “Max”, thereby decreasing the
signal suppression amount according to [Formula 8]. On
the contrary, in a case where the speech level estimated
15 value “Pdti^¯” is small, the value of the suppression
amount correction coefficient αdt approaches “1”, thereby
increasing the signal suppression amount according to
[Formula 8].
[0085]
20 Note that as described above, the clip compensation
unit 33 estimates the speech level of the user on the
basis of the average power during the double talk in the
non-clipped section of the signal of the clipped
microphone 13 (the signal after the echo cancellation
25 process).
Therefore, the speech level of the signal of the
clipped microphone 13 can be appropriately obtained at a
time when clipping occurs.
[0086]
35
SYP329357WO01
Here, in the clip compensation unit 33, it is
necessary to determine whether or not it is during the
double talk in order to sequentially calculate “Pdti^¯” as
the user speech level estimated value. The determination
5 as to whether or not it is during the double talk is
performed on the basis of the output voice signal Ss
(reference signal x) input via the FFT processing unit
34, the double talk evaluation value Di, and a double
talk determination threshold γ.
10 Specifically, presence or absence of the speaker
output is determined on the basis of the output voice
signal Ss, and as a result, if it is determined that a
speaker output is present and it is determined that the
double talk evaluation value Di is equal to or less than
15 the double talk determination threshold γ, a
determination result that it is during the double talk is
obtained.
[0087]
The description is returned to Fig. 10.
20 As the clip compensation for Case 2, clip
compensation is performed by the method represented by
[Formula 7].
[0088]
Further, as the process corresponding to the voice
25 recognition engine in Case 3, clip compensation is
performed in which the value of the suppression amount
correction coefficient αdt in [Formula 8] is made to
correspond to characteristics of the voice recognition
engine (characteristics of the voice recognition
30 process). As the value of the suppression amount
correction coefficient αdt at this time, for example, a
36
SYP329357WO01
fixed value that is predetermined according to the voice
recognition engine in the control unit 18 (or the cloud
60) is used.
[0089]
5 Note that Case 3 is not limited to executing the
process corresponding to the voice recognition engine as
described above, and the clip compensation may be omitted
as illustrated in parentheses in Fig. 10.
In a case where a user speech is present and no
10 speaker output is present as in Case 3, that is, a case
where the cause of the clip is estimated to be the user
speech, it is empirically known that not suppressing the
signal can result in a more favorable voice recognition
result in the subsequent stage. In such a case, it is
15 possible to improve the voice recognition accuracy by not
performing the clip compensation.
[0090]
It has been described above that the clip
compensation unit 33 selectively executes the process
20 related to the clip compensation corresponding to
dividing into cases depending on presence or absence of
the speaker output and presence or absence of the user
speech. However, at this time, determination of the
presence or absence of the user speech is performed on
25 the basis of the double talk evaluation value Di.
Specifically, the clip compensation unit 33 obtains, for
example, a determination result that a user speech is
present if the double talk evaluation value Di is equal
to or smaller than a predetermined value, or a
30 determination result that no user speech is present if
the double talk evaluation value Di is larger than the
37
SYP329357WO01
predetermined value.
Note that as described in [Formula 5], the double
talk evaluation value Di is an evaluation value that
increases during the double talk in which a user speech
5 is present.
[0091]
Here, a difference between the clip compensation
method as the embodiment represented by [Formula 7] or
[Formula 8] and the conventional technique will be
10 described with reference to Figs. 12 and 13.
Fig. 12 schematically represents the clip
compensation method described in Patent Document 1
described above as a conventional technique.
In the method described in Patent Document 1, a
15 signal (division signal m1b) between zero cross points
including a clip portion of a clipped signal (voice
signal Mb) is replaced with a signal (division signal
m1a) between corresponding zero cross points in a nonclipped
signal (voice signal Ma).
20 [0092]
An example of Fig. 12 illustrates an example in
which the division signal m1a, which corresponds to the
clip portion, in the non-clipped voice signal Ma arrives
later in time than the clip portion, but in this case,
25 according to the method of Patent Document 1, the clip
compensation cannot be performed in real time at a clip
timing illustrated as time t1 in Fig. 13.
[0093]
On the other hand, according to the clip
30 compensation method as the embodiment represented by
[Formula 7] or [Formula 8], it is not necessary to wait
38
SYP329357WO01
for the arrival of the waveform section corresponding to
the clip portion in the non-clipped signal, and the clip
compensation can be performed in real time at the timing
of occurrence of the clip.
5 [0094]
<6. Processing procedure>
A specific processing procedure to be executed in
order to achieve the clip compensation method as the
embodiment described above will be described with
10 reference to a flowchart in Fig. 14.
The clip compensation unit 33 repeatedly executes a
process illustrated in Fig. 14 for every time frame.
Note that the clip compensation unit 33 executes,
apart from the process illustrated in Fig. 14, a process
15 of sequentially calculating “Pdti^¯” as the average power
of every channel of the microphone 13 (the average power
after the echo cancellation process in a section where a
speaker output is present and no clipping has occurred)
and as the user speech level estimated value.
20 [0095]
First, the clip compensation unit 33 determines in
step S101 whether or not a clip is detected. That is,
presence or absence of a channel in which a clip has
occurred is determined on the basis of the detection
25 result of the clip detection unit 30.
If it is determined that no clip is detected, the
clip compensation unit 33 determines in step S102 whether
or not a termination condition is satisfied. Note that
the termination condition here is a condition
30 predetermined as a processing termination condition, such
as power-off of the signal processing device 1, for
39
SYP329357WO01
example.
If the termination condition is not satisfied, the
clip compensation unit 33 returns to step S101, or if the
termination condition is satisfied, the series of
5 processes illustrated in Fig. 14 is terminated.
[0096]
If it is determined in step S101 that a clip has
been detected, the clip compensation unit 33 proceeds to
step S103 and acquires the average power ratio between a
10 clipping channel and a minimum power channel. That is,
out of the average powers of the respective channels
calculated sequentially, the ratio (“Pi^¯/PMin^¯”) of the
average power of the clipped channel and the average
power of the channel with the minimum average power is
15 acquired by calculation.
[0097]
In subsequent step S104, the clip compensation unit
33 calculates a suppression coefficient of the clipping
channel. Here, the suppression coefficient means a
portion that excludes the terms “eMineH
20 Min” and “ei” on the
right side of [Formula 7].
[0098]
Then, in step S105, the clip compensation unit 33
determines whether or not a speaker output is present.
25 This determination process corresponds to determining
which of a set of Case 1 and Case 2 and a set of Case 3
and Case 4 illustrated in Fig. 10 is applicable.
If it is determined that a speaker output is
present, the clip compensation unit 33 determines in step
30 S106 whether or not a user speech is present.
[0099]
40
SYP329357WO01
If it is determined in step S106 that a user speech
is present (that is, corresponding to Case 1), the clip
compensation unit 33 proceeds to step S107 and updates
the suppression coefficient according to the estimated
5 speech level. That is, first, the suppression amount
correction coefficient αdt is calculated with the above
[Formula 9] on the basis of the speech level estimated
value “Pdti^¯”. Then, the suppression coefficient is
updated by multiplying the suppression coefficient
10 obtained in step S104 by the calculated suppression
amount correction coefficient αdt.
[0100]
Then, the clip compensation unit 33 executes a
clipping signal suppression process of step S108, and
15 returns to step S101. As the clipping signal suppression
process in step S108, a process of calculating “ei^~”
with [Formula 8] is performed using the suppression
coefficient updated in step S107.
[0101]
20 Further, if it is determined in step S106 that a
user speech is present (that is, corresponding to Case
2), the clip compensation unit 33 proceeds to step S109
to execute the clipping signal suppression process, and
returns to step S101. As the clipping signal suppression
25 process in step S109, a process of calculating “ei^~”
with [Formula 7] using the suppression coefficient
obtained in step S104.
[0102]
Further, if it is determined in step S105 that no
30 speaker speech is present (Case 3 or Case 4), the clip
compensation unit 33 determines in step S110 whether or
41
SYP329357WO01
not a user speech is present.
If it is determined in step S110 that a user speech
is present (Case 3), the clip compensation unit 33
proceeds to step S111, and performs a process of updating
5 to the suppression coefficient according to the
recognition engine. That is, the suppression coefficient
is updated by multiplying the suppression coefficient
obtained in step S104 by the suppression amount
correction coefficient αdt determined according to the
10 characteristics of the voice recognition engine.
Then, the clip compensation unit 33 performs the
process of calculating “ei^~” with [Formula 8] using the
suppression coefficient updated in step S111 as the
clipping signal suppression process of step S112, and
15 returns to step S101.
[0103]
Further, if it is determined in step S110 that no
user speech is present (Case 4), the clip compensation
unit 33 returns to step S101. That is, in this case, the
20 clip compensation is not performed.
[0104]
<7. Modification example>
Here, the embodiment is not limited to the specific
examples described above, and various modifications can
25 be made without departing from the scope of the present
technology.
For example, in the foregoing, the example in which
the plurality of microphones 13 is arranged on the
circumference has been described, but an arrangement
30 other than the arrangement on the circumference, such as
a linear arrangement, may be employed.
42
SYP329357WO01
[0105]
Further, in the embodiment, the example has been
described in which the signal processing device 1
includes the servo motor 21 to be capable of changing the
5 orientation of the speaker 16, that is, capable of
changing the positions of the respective microphones 13
with respect to the speaker 16. However, in a case of
employing such a configuration, for example, the clip
compensation unit 33 or the control unit 18 can be
10 configured to instruct the motor drive unit 20 to change
the position of the speaker 16 in response to detection
of a clip. Thus, the position of the speaker 16 can be
moved to a position where wall reflection or the like is
small, and the possibility of clipping to occur can be
15 decreased and clipping noise can be reduced.
Note that the signal processing device 1 may employ
a configuration in which the side of the microphones 13
is displaced instead of the speaker 16, and even in this
case, effects similar to those described above can be
20 obtained by displacing the microphones 13 in response to
detection of a clip similarly to as described above.
Further, the displacement of the speaker 16 and the
microphones 13 is not limited to a displacement caused by
rotation. For example, the signal processing device 1
25 may employ a configuration including wheels and a drive
unit thereof, or the like to be capable of moving by
itself. In this case, the drive unit may be controlled
so that the signal processing device 1 itself is moved in
response to detection of a clip. Thus, also by the
30 signal processing device 1 itself moving in this manner,
it is possible to move the positions of the speaker 16
43
SYP329357WO01
and the microphones 13 to positions where wall reflection
or the like is small, and effects similar to those
described above can be obtained.
Note that the configuration in which the speaker 16
5 and the microphones 13 are displaced according to
detection of a clip as described above can be applied
even in a case where the clip compensation represented by
[Formula 7] or [Formula 8] is not performed.
[0106]
10 <8. Summary of embodiment>
As described above, a signal processing device
(same 1) as the embodiment includes an echo cancellation
unit (AEC processing unit 32) that performs an echo
cancellation process of canceling an output signal
15 component from a speaker (same 16) on signals from a
plurality of microphones (same 13), a clip detection unit
(same 30) that performs a clip detection for signals from
the plurality of microphones, and a clip compensation
unit (same 33) that compensates for a signal after the
20 echo cancellation process of clipped one of the
microphones on the basis of a signal of non-clipped one
of the microphones.
[0107]
In a case where the echo cancellation process is
25 performed on signals from the plurality of microphones,
when the clip compensation is performed on a signal
before the echo cancellation process, the clip
compensation is performed in a state that an output
signal component of the speaker and other components
30 including a target sound are difficult separate, and thus
clip compensation accuracy tends to decrease. By
44
SYP329357WO01
performing the clip compensation on the signal after the
echo cancellation process as described above, it is
possible to perform the clip compensation on a signal in
which the output signal component of the speaker is
5 suppressed to some extent.
Therefore, the clip compensation accuracy can be
improved.
[0108]
Further, in the signal processing device as the
10 embodiment, the clip compensation unit compensates for a
signal of the clipped microphone by suppressing the
signal.
[0109]
By employing a compensation method of suppressing
15 the signal of the clipped microphone, it is possible to
prevent phase information of the signal of the clipped
microphone from being lost by the compensation.
Therefore, it is possible to prevent the phase
relationship among the respective microphones from being
20 destroyed by the compensation.
In the configuration in which voice recognition is
performed by performing speech direction estimation and
beamforming (voice emphasis) in the subsequent stage of
the clip compensation as in the embodiment, accuracy of
25 speech direction estimation is improved because the phase
relationship among the respective microphones is not
destroyed, a target speech component can be appropriately
extracted by beamforming, and voice recognition accuracy
can be improved.
30 [0110]
Moreover, in the signal processing device as the
45
SYP329357WO01
embodiment, the clip compensation unit suppresses a
signal of the clipped microphone on the basis of an
average power ratio between a signal of the non-clipped
microphone and a signal of the clipped microphone.
5 [0111]
Thus, power of the signal of the clipped microphone
can be appropriately suppressed to power after the echo
cancellation process that has to be obtained in a case
where it is not clipped.
10 Therefore, the accuracy of the clip compensation
can be improved.
[0112]
Furthermore, in the signal processing device
according to the embodiment, the clip compensation unit
15 uses, as the average power ratio, an average power ratio
with a signal of the microphone having a minimum average
power among the signals of the non-clipped microphones is
used.
[0113]
20 The microphone with the minimum average power can
be restated as the microphone in which it is most
difficult for clipping to occur.
Therefore, it is possible to maximize certainty
that the compensation is performed for the signal of the
25 clipped microphone.
[0114]
Further, in the signal processing device as the
embodiment, the clip compensation unit adjusts a
suppression amount of a signal of the clipped microphone
30 according to a speech level in a case where a user speech
is present and a speaker output is present.
46
SYP329357WO01
[0115]
In what is called a double talk section in which a
user speech is present and a speaker output is present,
in a case where the speech level of the user is high, the
5 speech component is also included in a large amount even
in the noise superposed section due to clipping. On the
other hand, in a case where the speech level is low, the
speech component tends to be buried in large clipping
noise. Accordingly, in the double talk section, the
10 suppression amount of the signal of the clipped
microphone is adjusted according to the speech level.
Thus, if the speech level of the user is high, it
is possible to reduce the suppression amount of the
signal to prevent the speech component from being
15 suppressed, and when the speech level of the user is low,
it is possible to increase the suppression amount of the
signal to suppress the clipping noise.
Therefore, when voice recognition is performed in a
subsequent stage of the clip compensation as in the
20 embodiment, the voice recognition accuracy can be
improved.
[0116]
Moreover, in the signal processing device as the
embodiment, the clip compensation unit suppresses a
25 signal of the clipped microphone by a suppression amount
according to a characteristic of a voice recognition
process in a subsequent stage in a case where a user
speech is present and no speaker output is present.
[0117]
30 The case where a user speech is present and no
speaker output is present is a case where a cause of a
47
SYP329357WO01
clip is estimated to be the user speech. With the above
configuration, in the case where the cause of the clip is
estimated to be the user speech, for example, it is
possible to perform the clip compensation with an
5 appropriate suppression amount according to
characteristics of the voice recognition process in the
subsequent stage such that the voice recognition accuracy
can be maintained better in a case where there is a
certain degree of speech level even if clipping noise is
10 superposed than in a case where the speech component is
suppressed, or the like.
Therefore, the voice recognition accuracy can be
improved.
[0118]
15 Furthermore, in the signal processing device as the
embodiment, the clip compensation unit does not perform
the compensation for the clipped microphone signal in a
case where a user speech is present and no speaker output
is present.
20 [0119]
In the case where the user speech is present and
the speaker output is not present, that is, a case where
the cause of the clip is estimated to be the user speech,
it is empirically known that not suppressing the signal
25 can result in a more favorable voice recognition result
in the subsequent stage. In such a case, it is possible
to improve the voice recognition accuracy by not
performing the clip compensation as described above.
[0120]
30 Further, the signal processing device as the
embodiment further includes a drive unit (servo motor 21)
48
SYP329357WO01
that changes a position of at least one of the plurality
of microphones or the speaker, and a control unit (clip
compensation unit 33 or control unit 18) that changes the
position of at least one of the plurality of microphones
5 or the speaker by the drive unit in response to detection
of a clip by the clip detection unit.
[0121]
Thus, if a clip is detected, it is possible to
change the positional relationship among the respective
10 microphones and the speaker, or move the positions of the
plurality of microphones or the speaker to a position
where wall reflection or the like is small.
Therefore, in order to reduce the possibility of a
clip to occur or reduce clipping noise so as to respond
15 to a case where the clip is chronically generated or a
case where large clipping noise is generated, or the
like, the positional relationship of the plurality of
microphones and the speaker, or the positions of the
plurality of microphones themselves or the position of
20 the speaker itself can be changed, and the accuracy of
voice recognition in the subsequent stage can be
improved.
[0122]
Further, a signal processing method according to
25 the embodiment includes an echo cancellation procedure to
perform an echo cancellation process of canceling an
output signal component from a speaker on signals from a
plurality of microphones, a clip detection procedure to
perform a clip detection for signals from the plurality
30 of microphones, and a clip compensation procedure to
compensate for a signal after the echo cancellation
49
SYP329357WO01
process of clipped one of the microphones on the basis of
a signal of non-clipped one of the microphones.
[0123]
With the signal processing method as such an
5 embodiment, operation and effect similar to those of the
signal processing device as the embodiment described
above can be obtained.
[0124]
Here, the functions of the voice signal processing
10 unit 17 as has been described (particularly the functions
related to echo cancellation, clip detection, and clip
compensation) can be achieved as software processes by
CPU or the like. The software processes are executed on
the basis of a program, and the program is stored in a
15 storage device readable by a computer device (information
processing device) such as a CPU.
[0125]
The program as an embodiment is a program executed
by an information processing device, the program causing
20 the information processing device to implement functions
including an echo cancellation function to perform an
echo cancellation process of canceling an output signal
component from a speaker on signals from a plurality of
microphones, a clip detection function to perform a clip
25 detection for signals from the plurality of microphones,
and a clip compensation function to compensate for a
signal after the echo cancellation process of clipped one
of the microphones on the basis of a signal of nonclipped
one of the microphones.
30 [0126]
With such a program, the signal processing device
50
SYP329357WO01
as the embodiment described above can be achieved.
[0127]
Note that effects described in the present
description are merely examples and are not limited, and
5 other effects may be provided.
[0128]
<9. Present technology>
Note that the present technology can also have
configurations as follows.
10 (1)
A signal processing device including:
an echo cancellation unit that performs an echo
cancellation process of canceling an output signal
component from a speaker on signals from a plurality of
15 microphones;
a clip detection unit that performs a clip
detection for signals from the plurality of microphones;
and
a clip compensation unit that compensates for a
20 signal after the echo cancellation process of clipped one
of the microphones on the basis of a signal of nonclipped
one of the microphones.
(2)
The signal processing device according to (1)
25 above, in which
the clip compensation unit compensates for a signal
of the clipped microphone by suppressing the signal.
(3)
The signal processing device according to (2)
30 above, in which
the clip compensation unit suppresses a signal of
51
SYP329357WO01
the clipped microphone on the basis of an average power
ratio between a signal of the non-clipped microphone and
a signal of the clipped microphone.
(4)
5 The signal processing device according to (3)
above, in which
the clip compensation unit uses, as the average
power ratio, an average power ratio with a signal of the
microphone having a minimum average power among the
10 signals of the non-clipped microphones is used.
(5)
The signal processing device according to any one
of (1) to (4) above, in which
the clip compensation unit adjusts a suppression
15 amount of a signal of the clipped microphone according to
a speech level in a case where a user speech is present
and a speaker output is present.
(6)
The signal processing device according to any one
20 of (1) to (5) above, in which
the clip compensation unit suppresses a signal of
the clipped microphone by a suppression amount according
to a characteristic of a voice recognition process in a
subsequent stage in a case where a user speech is present
25 and no speaker output is present.
(7)
The signal processing device according to any one
of (1) to (5) above, in which
the clip compensation unit does not perform the
30 compensation for the clipped microphone signal in a case
where a user speech is present and no speaker output is
52
SYP329357WO01
present.
(8)
The signal processing device according to any one
of (1) to (7) above, further including:
5 a drive unit that changes a position of at least
one of the plurality of microphones or the speaker; and
a control unit that changes the position of at
least one of the plurality of microphones or the speaker
by the drive unit in response to detection of a clip by
10 the clip detection unit.
REFERENCE SINGS LIST
[0129]
1 Signal processing device
15 11 Casing
12 Microphone array
13 Microphone
14 Movable unit
15 Display unit
20 16 Speaker
30 Clip detection unit
32 AEC processing unit
32a Echo cancellation processing unit
32b Double talk evaluation unit
25 33 Clip compensation unit
35 Speech section estimation unit
36 Speech direction estimation unit
37 Voice emphasis unit
38 Noise suppression unit
53
SYP329357WO01
CLAIMS
1. A signal processing device comprises:
an echo cancellation unit that performs an echo
5 cancellation process of canceling an output signal
component from a speaker on signals from a plurality of
microphones;
a clip detection unit that performs a clip
detection for signals from the plurality of microphones;
10 and
a clip compensation unit that compensates for a
signal after the echo cancellation process of clipped one
of the microphones on a basis of a signal of non-clipped
one of the microphones.
15
2. The signal processing device according to claim 1,
wherein
the clip compensation unit compensates for a signal
of the clipped microphone by suppressing the signal.
20
3. The signal processing device according to claim 2,
wherein
the clip compensation unit suppresses a signal of
the clipped microphone on a basis of an average power
25 ratio between a signal of the non-clipped microphone and
a signal of the clipped microphone.
4. The signal processing device according to claim 3,
wherein
30 the clip compensation unit uses, as the average
power ratio, an average power ratio with a signal of the
54
SYP329357WO01
microphone having a minimum average power among the
signals of the non-clipped microphones is used.
5. The signal processing device according to claim 1,
5 wherein
the clip compensation unit adjusts a suppression
amount of a signal of the clipped microphone according to
a speech level in a case where a user speech is present
and a speaker output is present.
10
6. The signal processing device according to claim 1,
wherein
the clip compensation unit suppresses a signal of
the clipped microphone by a suppression amount according
15 to a characteristic of a voice recognition process in a
subsequent stage in a case where a user speech is present
and no speaker output is present.
7. The signal processing device according to claim 1,
20 wherein
the clip compensation unit does not perform the
compensation for the clipped microphone signal in a case
where a user speech is present and no speaker output is
present.
25
8. The signal processing device according to claim 1,
further comprising:
a drive unit that changes a position of at least
one of the plurality of microphones or the speaker; and
30 a control unit that changes the position of at
least one of the plurality of microphones or the speaker
55
SYP329357WO01
by the drive unit in response to detection of a clip by
the clip detection unit.
9. A signal processing method comprising:
5 an echo cancellation procedure to perform an echo
cancellation process of canceling an output signal
component from a speaker on signals from a plurality of
microphones;
a clip detection procedure to perform a clip
10 detection for signals from the plurality of microphones;
and
a clip compensation procedure to compensate for a
signal after the echo cancellation process of clipped one
of the microphones on a basis of a signal of non-clipped
15 one of the microphones.
10. A program executed by an information processing
device, the program causing the information processing
device to implement functions comprising:
20 an echo cancellation function to perform an echo
cancellation process of canceling an output signal
component from a speaker on signals from a plurality of
microphones;
a clip detection function to perform a clip
25 detection for signals from the plurality of microphones;
and
a clip compensation function to compensate for a
signal after the echo cancellation process of clipped one
of the microphones on a basis of a signal of non-clipped
30 one of the microphones.