FORM 2
THE PATENTS ACT, 1970
(39 of 1970)
&
THE PATENTS RULES, 2003
COMPLETE SPECIFICATION
[See section 10, Rule 13]
WORK SUPPORT SYSTEM AND WORK SUPPORT METHOD;
HITACHI SYSTEMS, LTD., A CORPORATION ORGANIZED AND EXISTING
UNDER THE LAWS OF JAPAN, WHOSE ADDRESS IS 1-2-1, OSAKI,
SHINAGAWA-KU, TOKYO 1418672, JAPAN
THE FOLLOWING SPECIFICATION PATICULARLY DESCRIBES THE
INVENTION AND THE MANNER IN WHICH IT IS TO BE PERFORMED.
[DESCRIPTION]
[FIELD OF THE INVENTION]
[0001]
The present invention relates to a work support system and a work support
method.
[BACKGROUND ART]
[0002]
Before a worker actually performs various works such as maintenance and
inspection at a work site, training is conducted to allow the worker to acquire the
necessary skills and knowledge. The training is performed, for example, using the
work performed by the expert as an example, so that the work skill of the expert is
transferred to the worker (see, for example, Patent Document 1).
[0003]
In Patent Document 1, a technique for generating training information for
training is disclosed, in which, the determination criteria of the expert are acquired
based on the perception information such as the content of the on-site work of the
expert, the facility noise during the work, and the gaze of the expert, and the worker is
trained based on the criteria.
[0004]
On the other hand, in the on-site work, even a trained person needs various
information such as work procedure information, work target device information, and
work tool information to be used. A support system that supports the on-site work by
providing necessary information to a worker who performs the on-site work is already
known (see, for example, Patent Document 2).
[0005]
In Patent Document 2, a work support apparatus is disclosed, which has: a
measurement unit for grasping a situation where the work target is placed; an
information processing unit that controls the measurement unit, performs an
appropriate information processing with respect to the obtained information, and
further constructs visual information to be provided to the worker; and a display unit
for providing the visual information superimposed on the target object, wherein the
work to be performed is instructed to the worker at the work site.
[CITATION LIST]
[PATENT LITERATURE]
[0006]
[Patent Document 1] Japanese Unexamined Patent Application Publication No.
18-132556
[Patent Document2] Japanese Unexamined Patent Application Publication No.
2009-069954
[SUMMARY OF THE INVENTION]
[TECHNICAL PROBLEM]
[0007]
However, although the conventional work support apparatus described in
Patent Document 2 and the like helps to transfer the advanced work skills of the expert
to the worker, there has been no consideration for determining the success or failure of
the work by the worker with low work skill so as to prevent the mistakes in the work.
In actual work sites, there are an increasing number of cases where foreign workers
who have low work skills and are not accustomed to Japanese language perform the
work, so that there are increasing number of mistakes in the work, occurrences of
forgetting the work, and the accidents in the work. The social impact of accidents is
large, so that in addition to improving work quality as a measure to prevent recurrence,
confirmation by two-person work, refinement of work checklists and work reports may
be required. In addition, conventionally, since a checklist is generated and a work
report is created by the worker or the like, it is troublesome and inefficient.
[0008]
The present invention has been made in order to solve the above-described
problems, and it is the object according to the present invention to provide a work
support system and work support method capable of preventing work mistakes by
reliably performing work checks even for workers with low work skills.
[MEANS TO SOLVE THE PROBLEM]
[0009]
In order to achieve the above object, a work support 5 system according to the
present invention is (1) a work support system (1, 1A), comprising: an information
acquisition device (120) that acquires information from a worker; a determination
device (100) that determines success or failure of a work; and an information output
device (23) for that outputs work support information based on a determination result
by the determination device, wherein the determination device determines the success
or failure of the work according to a predetermined determination criterion based on
information from the worker acquired by the information acquisition device.
[0010]
In the present invention, the determination device determines the success or
failure of the work according to a predetermined determination criterion based on
information from the worker such as video and voice acquired by the information
acquisition device, and the information output device outputs the work support
information based on the determination result by the determination device. By this
configuration, even a worker with low work skills can prevent work mistakes by
reliably determining the success or failure of the work based on information from the
worker and feeding it back to the worker.
[0011]
In order to achieve the above object, the work support system according to the
present invention (2) comprises: a first video acquisition device (11) that acquires video
of work performed by a first worker (2); a voice acquisition device (12) that acquires
voice of the first worker during the work; a work check information generation device
(340) that generates work check information (70) that serves as a reference for a work
check based on the work video acquired by the first video acquisition device and the
voice information acquired by the voice acquisition device; a second video acquisition
device (21) that acquires video of work performed by a second worker (3); a
determination device (338) that determines success or failure of the work of the second
worker; a support information output device (23) that outputs the determination result
by the determination device as work support information, wherein the determination
device determines the success or failure of the work by comparing the 5 work video of the
first worker and the work video of the second worker based on the work check
information.
[0012]
By this configuration, the determination device determines the success or
failure of the work by comparing the work video of the expert (first worker) with the
work video of the worker on site (second worker) based on the work check information,
so that a work check (determination of work success / failure) can be reliably performed.
It is also possible to guide the user to redo the work by pointing out the work mistake by
voice or video based on the result of the work check. Therefore, work mistakes can be
reliably prevented by performing work checks even for workers with low work skills.
[0013]
Further, the work support system according to the present invention may be so
configured that (3) the determination device determines whether or not the work is
successful depending on whether or not a specific object included in the work video of
the first worker is included in the work video of the second worker.
[0014]
By this configuration, it is possible to reliably detect forgetting work by
determining the success or failure of the work based on the presence or absence of a
specific object essential for the work.
[0015]
Further, the work support system according to the present invention may be so
configured that (4) the determination device determines whether or not the work is
successful based on whether or not an object not included in the work video of the first
worker is included in the work video of the second worker.
[0016]
By this configuration, it is possible to reliably detect erroneous work (wrong
work contents) by determining the success or failure of the work based on the presence
or absence of objects unnecessary for the work.
[0017]
Further, the work support system according to the present invention may be so
configured that (5) the determination device determines whether or not the work is
successful or failure depending on whether or not a specific object included in the work
video of the first worker is included in a predetermined sequence in the work video of
the second worker.
[0018]
By this configuration, it is possible to reliably detect a procedure mistake by
determining the success or failure of a work based on whether or not objects essential
for the work appear in a predetermined sequence.
[0019]
Further, the work support system according to the present invention may be so
configured that (6) the work check information generation device divides the work
video of the first worker into element works based on the voice information of the first
worker, and the determination device determines the success or failure of the work for
each element work.
[0020]
By this configuration, it is possible to determine the success or failure of the
work more accurately by dividing the long-time work into element works having an
appropriate required time.
[0021]
Further, the work support system according to the present invention may be so
configured that (7) the support information output device can selectively output the
work support information by at least one of voice and moving image.
[0022]
By this configuration, it is possible to perform work support by selecting an
optimal method (voice only / moving image only / voice and moving image) according
to the work environment.
[0023]
Further, the work support system according to the present 5 invention may (8)
further comprise a language conversion device (314) that converts a voice that
transmits the work support information output from the support information output
device into a specified language.
[0024]
By this configuration, it is possible to perform work support to foreign workers
who do not understand Japanese.
[0025]
Further, the work support system according to the present invention may be (9)
further comprising a work report creation device (341) that creates a work report
including a determination result by the determination device.
[0026]
By this configuration, a work report is automatically created when the work on
site is completed, so that the labor of creating a work report can be saved.
[0027]
Further, the work support system according to the present invention may be
(10) further comprising an attention call control device (338) that causes the support
information output device to output attention information as the work support
information before a specific stage during the work based on the work check
information and the temporal information of the work of the second worker.
[0028]
By this configuration, since the attention call control device outputs attention
information before a specific stage that requires attention calling based on work check
information and temporal information (for example, elapsed work time or work stage),
work mistakes and accidents can be effectively prevented, thereby making it possible to
improve the work quality.
[0029]
Further, the work support system according to the present invention may be so
configured that (11) the work check information generation device determines the
specific stage based on voice information during work of the first 5 worker as the work
check information.
[0030]
This configuration makes it possible to easily specify a specific stage that
requires attention.
[0031]
Further, the work support system according to the present invention may be so
configured that (12)the temporal information is information on an elapsed time from
the start of work of the second worker.
[0032]
By this configuration it is possible to make it easy to call attention.
[0033]
Further, the work support system according to the present invention may be so
configured that (13) the temporal information is work stage information that is
recognized from a work video of the second worker.
[0034]
By this configuration, it is possible to makes it easy to call attention at a more
appropriate time.
[0035]
Further, the work support system according to the present invention may be
(14) further comprising: a server (30) including at least the work check information
generation device; a first terminal (10) having the first video acquisition device and the
voice acquisition device and connected to the server; and a second terminal (20) having
the second video acquisition device and the support information output device and
connected to the server.
[0036]
By this configuration, it is possible to simplify the configuration of the second
terminal carried by the worker on the site through use of the server.
[0037]
The work support method according to the present invention 5 (15) comprises: a
first video acquisition step to acquire a video of a work performed by a first worker
(2);a voice acquisition step to acquire the voice of the first worker during the work; a
work check information generation step to generate work check information (70)
serving as a reference for the work check based on the work video acquired in the first
video acquisition step and the voice information acquired in the voice acquisition step;
a second video acquisition step to acquire a video of the work performed by a second
worker (3);a determination step to determine success or failure of the work of the
second worker; and a support information output step to output the determination result
in the determination step as work support information, wherein the determination step
determines the success or failure of the work by comparing the video of the work of the
first worker and the video of the work of the second worker based on the work check
information.
[0038]
By this configuration, since the success / failure of the work is determined by
comparing the work video of the expert (first worker) with the work video of the worker
on the site (second worker) based on the work check information in the determination
step, the work check (determination of work success / failure) can be reliably
performed. It is also possible to guide the user to redo the work by pointing out the work
mistake with voice or video based on the result of the work check. Therefore, work
mistakes can be prevented by reliably performing work checks even for workers with
low work skills.
[0039]
The work support system according to the present invention may be so
configured that (16)the determination device includes: a work instruction unit (102) to
have instruction information for instructing the work to be outputted from the
information output device; an information detection unit (105) to detect response
information in response to the instruction information from the information acquired by
the information acquisition device, after the instruction information is outputted by the
information output device; and a determination unit (112) that determines 5 success or
failure of the work based on the response information detected by the information
detection unit.
[0040]
By this configuration, at the start of work, the work instruction unit and the
information output device transmit the work instruction information to the worker by
voice or screen display, and at the end of the work, the information acquisition unit
acquires the information from the worker by voice or screen operation or the like. At
that time, the information detection unit detects response information in response to the
instruction information, and the determination unit determines the success or failure of
15 the work from the content of the response information. For example, if the content of
the response information indicates “Completed.” or “Finished”, it is determined that
the operation has succeeded. If the information indicates information other than
“Completed.” or “Finished”, it is determined that the operation has failed or has not
been completed. Thus, the worker’s work is supported through, for example,
performing a dialogue by voice or screen between the work support system and the
worker .The work support system can be constructed at a low cost with a simple
configuration, and the work support system is easy to use because the worker can use
both hands freely, and work mistakes can be efficiently prevented, since the success or
failure of the work is determined based on information from the worker such as the
utterance and screen operation of the worker.
[0041]
The work support system according to the present invention may be so
configured that (17) the determination device further includes an inquiry confirmation
unit (107) that outputs inquiry information for confirming completion of work from the
information output device after the determination unit determines that the work is
successful, and the determination unit determines the success or failure of the work
based on the response information detected by the information detection unit, after the
inquiry information is outputted by the information output device.
[0042]
By this configuration, after the determination unit determines the success of
the work, the inquiry confirmation unit and the information output device make an
inquiry to confirm the completion of the work, so that it is possible to increase the
possibility that the worker will notice a work mistake, thereby making it possible to
effectively reduce the work mistakes.
[0043]
The work support system according to the present invention may be so
configured that (18) the determination device further includes an attention information
output unit (103) that causes a work attention information to be outputted from the
information output device.
[0044]
By this configuration, when the work starts, the attention information output
unit outputs work attention information from the information output device, so that the
worker’s attention can be called, and work mistakes and work accidents can be
effectively reduced.
[0045]
The work support system according to the present invention may be so
configured that (19)an image acquisition device(122) to acquire an image of a work
target is further provided, the determination device further includes an image
information acquisition unit (109) that acquires image information of the work target
acquired by the image acquisition device, the determination unit determines that the
work is incomplete, when the image information acquisition unit has not acquired the
image information of the work target, and the work instruction unit causes instruction
information for requiring the image information to be outputted from the information
12
output device, when the determination unit determines that the work is incomplete,
on an account that the image information has not been acquired.
[0046]
By this configuration, the determination unit determines that the work is
incomplete when the image information acquisition unit has 5 not acquired the image
information (evidence photograph) of the work target, so that the worker performs the
operation more carefully, thereby making it possible to more effectively reduce work
mistakes. Further, traceability can also be realized by acquiring image information
(evidence photographs) of work target, and furthermore, the image information can
also be used as teacher data in machine learning of work comparison models.
[0047]
The work support system according to the present invention may be so
configured that (20) the determination unit includes: a first determination unit (114)
that determines the success or failure of the work based on the response information
detected by the information detection unit, after the instruction information is
outputted by the information output device; a second determination unit (115) that
determines the success or failure of the work based on the response information
detected by the information detection unit, after the inquiry information is outputted
by the information output device; and a third determination unit (116) that
determines that the work is incomplete when the image information acquisition unit
has not acquired the image information of the work target, and determines the success
or failure of the work based on the response information detected by the information
detection unit after the instruction information is outputted by the information output
device, when the image information of the work object is acquired, and the
determination unit selects and uses one of the first determination unit, the second
determination unit and the third determination unit, in accordance with importance
of the work.
[0048]
By this configuration, since the determination unit selects and uses one of the
first to third determination units in accordance of the importance of the work, flexible
handling is possible in such a way that the first determination with a low
determination criterion is used for less important work, and the second and third
determination units with strict judgment criteria are used 5 for work with high
importance, so that the work mistakes can be effectively reduced without lowering the
efficiency of the work.
[0049]
The work support system according to the present invention may be so
configured that (21)the determination unit determines whether the image information
acquired by the image information acquisition unit is appropriate, andthe work
instruction unit causes instruction information requiring image information to be
outputted from the information output device, when the determination unit
determines that the image information is inappropriate.
15 [0050]
By this configuration, the determination unit determines whether the image
information (evidence photograph) acquired by the image information acquisition unit
is appropriate from the viewpoint of whether the image information is captured at an
appropriate photographing position or an appropriate photographing direction, so that
acquisition of inappropriate image information (evidence photograph) can be
prevented. Further, the worker's attention can be called, thereby making it possible to
reduce work mistakes.
[0051]
The work support system according to the present invention may be so
configured that (22) the determination unit determines completion of the work based
on the image information acquired by the image information acquisition unit.
[0052]
By this configuration, the determination unit determines the completion of
the work based on the image information (evidence photograph) acquired by the image
information acquisition unit, so that the completion of the operation can be
determined more accurately.
[0053]
The work support system according to the present invention may be so
configured that (23) a video acquisition device (21) that acquires 5 a video of the work
performed by the worker is further provided, the determination device further
includes a video information acquisition unit (109) that acquires video information
acquired by the video acquisition device, and the determination unit determines the
completion of the work based on the video of the work by the worker acquired by the
video acquisition unit.
[0054]
By this configuration, the determination unit determines the completion of
the work based on the video information (evidence video) acquired by the video
information acquisition unit, so that the completion of the work can be determined
more accurately. Traceability can also be realized by acquiring work video information
(evidence video), which can also be used as teacher data in machine learning of a work
comparison model.
[0055]
The work support system according to the present invention may be so
configured that (24) a sensor device that detects information related to the work is
further provided, the determination device further includes a sensor information
acquisition unit (123) that acquires the information related to the work detected by
the sensor device, and the determination unit determines the completion of the work
based on the information related to the work acquired by the sensor information
acquisition unit.
[0056]
By this configuration, the determination unit determines the completion of
the work based on information related to the work acquired by the sensor device and
the sensor information acquisition unit, so that the completion of work can be
determined more accurately.
[0057]
The work support system according to the present invention may be so
configured that (25)the work instruction unit causes information indicating that the
work is incomplete to be outputted in real time from the information 5 output device,
when the determination unit determines that the work is incomplete.
[0058]
By this configuration, when the work is not completed due to a work mistake
or the like, the work instruction unit and the information output device notify the
worker of information in real time, so that the work mistake can be prevented more
effectively.
[0059]
The work support system according to the present invention may (26)further
comprising: a data acquisition unit (118) that acquires data related to the work, when
15 the determination unit determines that the work has been normally completed; and a
work document generation unit (119) that generates a work document based on the
data related to the work acquired by the data acquisition unit.
[0060]
By this configuration, work documents such as checklists and work reports
are automatically generated by the work document generation unit, which reduces the
burden on the worker, prevents description mistakes and omissions, and makes it
easier for the creation work. Man-hour required is reduced.
[0061]
The work support system according to the present invention may be so configured
that (27) the work document generation unit generates the work document, when the
determination unit determines that the work is completed based on the response
information detected by the information detection unit.
[0062]
16
By this configuration, work documents such as checklists and work reports
are automatically generated by the work document generation unit triggered by
response information indicating “Completed.” from the worker, reducing the burden
on the worker, preventing mistakes and omissions, and the man-hours required for the
creation 5 work are reduced.
[0063]
The work support system according to the present invention may be so
configured that (28)the data acquisition unit acquires information on the result of the
work success / failure by the determination unit and information on work time
required for the work, and the work document generation unit generates the work
document inclusive of the determination result and the information on the work time.
[0064]
By this configuration, the work document generation unit generates the work
document including the result of the work success / failure and the work time, so that
a more appropriate work document can be created.
[0065]
The work support system according to the present invention may be so
configured that (29) the determination device changes a scope of the work to be
determined by the determination unit in accordance with a skill level of the worker.
[0066]
By this configuration, the scope of the work determined by the determination
unit is changed according to the skill level of the worker, so that work support can be
performed more efficiently. For example, a worker with a high skill level can perform
work without interruption between the work A and the work B by determining the
success or failure of the work within the scope of the work A and the work B combined.
For example, a worker with a low skill level can be provided with detailed work
support by determining the success or failure of the work A and the work B.
[0067]
The work support system according to the present invention may be so
configured that (30)the information acquisition device acquires voice information
during the work, and the information output device outputs information to be
outputted by voice.
5 [0068]
By this configuration, since both hands of the worker can be used freely, the
workability is improved, and the work can be efficiently supported to the worker by
voice dialogue.
[0069]
The work support system according to the present invention may be so
configured that (31)the information acquisition device acquires information on a
screen operation of the worker, and the information output device displays
information to be outputted on a screen.
[0070]
By this configuration, it is possible to enable efficient work support with a
simple configuration on the screen of the worker's smartphone or the like.
[0071]
The work support system according to the present invention may be so
configured that (32) the information acquisition device acquires the voice information
during the work and the information on the screen operation of the worker, and the
information output device outputs information to be outputted by voice and displays
the information on the screen.
[0072]
By this configuration, it is possible to provide the worker with work support
by screen display such as a smartphone and work support by voice dialogue at the
same time, so that reliable and convenient work support can be realized.
[EFFECTOFTHEINVENTION]
[0073]
According to the present invention, it is possible to provide a work support
system and a work support method capable of preventing work mistakes by reliably
performing work checks even for workers with low work skills.

[BRIEF DESCRIPTION 5 OF THE DRAWINGS]
[0074]
FIG. 1 is a configuration diagram of a work support system according to the first
embodiment according to the present invention.
FIG. 2is a block diagram of the expert terminal shown in FIG.1.
FIG. 3 is a configuration diagram of the worker terminal shown in FIG.1.
FIG. 4 is a configuration diagram of the work information generation server shown in
FIG.1.
FIG. 5 is a configuration diagram of the expert work operation information record
database shown in FIG.4.
Fig. 6 is a configuration diagram of the on-site worker work information record
database shown in FIG. 4.
FIG. 7 is a configuration diagram of the work target device information record
database shown in FIG 4.
FIG. 8 is an explanatory drawing which shows the method of dividing an expert's work
video and work voice.
FIG. 9 is an explanatory drawing which shows the method of producing and
generating work check information.
FIG. 10A is an explanatory diagram showing a method of work check (work success /
failure determination).
FIG. 10B is an explanatory diagram showing a method of work check (work success /
failure determination).
FIG. 11 is an explanatory diagram showing another method of work check (work
success / failure determination).
FIG. 12 is an explanatory diagram showing another method of work check (work
success / failure determination).
FIG. 13 is an explanatory diagram showing a method of attention calling.
FIG. 14 is an explanatory drawing which shows another method of attention calling.
FIG. 15 is a flowchart of a work support method according to an embodiment according
to the 5 present invention.
FIG. 16 is a flowchart of the work check information generation step shown in FIG. 15.
FIG. 17 is a flowchart of the work check execution step shown in FIG. 15.
FIG. 18 is a diagram showing a schematic configuration of a work support system
according to the second embodiment according to the present invention.
FIG. 19 is a diagram showing functional blocks of the work support system according
to the second embodiment according to the present invention.
FIG. 20 is a diagram showing a schematic configuration of the control device shown in
FIG. 19.
FIG. 21 is a diagram illustrating work support pattern 1.
FIG. 22 is a diagram illustrating work support pattern 2.
FIG. 23 is a diagram illustrating work support pattern 3.
FIG. 24 is a diagram illustrating work support pattern 4.
FIG. 25 is a diagram for explaining how to use image data and video data.
FIG. 26 is a diagram showing an example of a display screen of the worker terminal.
FIG. 27 is a diagram showing an example of a display screen of the worker terminal.
FIG. 28 is a diagram showing an example of a display screen of the worker terminal.
FIG. 29 is a diagram showing an example of the display screen of the worker terminal.
FIG. 30 is a diagram showing an example of the display screen of the worker terminal.
FIG. 31 is a diagram showing an example of the display screen of theworker terminal.
FIG. 32 is a diagram showing an example of the display screen of the worker terminal.
[DETAILED DESCRIPTION OF EMBODIMENTS]
[0075]
Hereinafter, a work support system according to an embodiment according to
the present invention will be described with reference to the drawings.
[0076]
(First Embodiment)
The work support system 1 according to the first embodiment according to the
present invention is a system that prevents a work mistake by determining in real
time whether or not a work performed by the worker 3 is successful. 5 To be specific, the
work support system 1 generates work check information based on the work record
(work video / work voice) of the expert 2, and the work video of the expert 2 and the
worker 3 based on the work check information. The success or failure of the work of
the worker 3 is determined by comparing the work videos. The work to be supported is
not limited to maintenance and inspection work, and may be any work performed on
site.
[0077]
FIG. 1 is a configuration diagram showing a configuration of a work support
system 1 according to the first embodiment according to the present invention. As
shown in FIG. 1, the work support system 1 includes an expert terminal 10, a worker
terminal 20, and a work information generation server 30, which are connected to
each other via a network 4.
[0078]
A video acquisition device (for example, a camera) 11 and a voice acquisition
device (for example, a microphone) 12 are connected to the expert terminal 10 by wired
or proximity wireless communication. The worker terminal 20 is connected to a video
acquisition device (for example, a camera) 21, a voice acquisition device (for example, a
microphone) 22, and a support information output device (for example, an earphone, a
speaker, a display, and the like) 23 by wired or proximity wireless communication.
The work information generation server 30 is connected to a storage device 31 that
stores various databases.
[0079]
In the present embodiment, the video acquisition device 11 and the voice
acquisition device 12 are connected to the expert terminal 10, but the present
invention is not limited to this. The video acquisition device 11 and the voice
acquisition device 12 may be connected to the work information generation server 30
via the network 4. In this case, the expert terminal 10 is not always necessary.
[0080]
In the present embodiment, the video acquisition device 5 21, the voice
acquisition device 22, and the support information output device 23 are connected to
the worker terminal 20. However, the present invention is not limited to this. The
video acquisition device 21, the voice acquisition device 22, and the support
information output device 23 may be connected to the work information generation
server 30 via the network 4. In this case, the worker terminal 20 is not always
necessary.
[0081]
The expert 2, the worker 3, the expert terminal 10, and the worker terminal
of the present embodiment correspond to the first worker, the second worker, the
first terminal, and the second terminal according to the present invention,
respectively. Further, the video acquisition device 11 and the video acquisition device
21 of the present embodiment correspond to the first video acquisition device and the
second video acquisition device according to the present invention, respectively.
[0082]
(Expert Terminal)
First, the expert terminal 10 will be described.
The expert terminal 10 is, for example, a smartphone, a tablet terminal, a
head mounted display (HMD), a notebook PC, or the like. Further, the expert terminal
acquires a work video (work video) from a video acquisition device (for example,
ear-mounted camera) 11 attached to the expert 2 and acquires the voice (work voice)
uttered by the expert 2 during the work from a voice acquisition device (for example, a
microphone) 12 attached to the expert 2. The video acquisition device 11 is a portable
ear camera, for example, but may be a fixed camera that is used by being fixed at an
arbitrary position.
[0083]
Either one of the video acquisition device 11 and the voice acquisition device
12 or both may be incorporated in the expert terminal 10.
[0084]
FIG. 2 is a configuration diagram showing the configuration 5 of the expert
terminal 10. As shown in FIG. 2, the expert terminal 10 includes a control unit 14, a
memory 15 that stores various processing programs, a storage 16 that stores various
databases, a communication unit 17, a video acquisition unit 11a, a voice acquisition
unit 12a, and a display unit. 13 and a power supply unit 18.
[0085]
The control unit 14 may be configured to use a computer having a CPU
(Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access
Memory), an input / output interface, an external storage device, and the like. All or a
part of the processing can be realized by executing various processing programs stored
in the memory 15 by the CPU.
[0086]
The communication unit 17 communicates with the work information
generation server 30 via the network 4.
[0087]
The voice acquisition unit 12a acquires the voice of the expert 2 at the time of
work from the voice acquisition device 12 such as a microphone.
[0088]
The video acquisition unit 11a acquires a work video of the expert 2
photographed by the video acquisition device 11 such as an ear camera.
[0089]
The display unit 13 is for presenting work information and the like to the
expert 2, and to be specific, includes a display unit such as a smartphone or a tablet
terminal.
[0090]
The power supply unit 18 supplies power necessary for driving the expert
terminal 10.
[0091]
The memory 15 stores various processing programs constituting a video
record processing unit 15b, a voice record processing unit 15a, 5 and an information
transmission and reception processing unit 15c. These processing programs are
executed by the CPU of the control unit 14 so that the respective processing functions
can be realized.
[0092]
The video record processing unit 15b is configured to execute a process of
recording (recording) a work video by driving and controlling the video acquisition
unit 11a by the CPU of the control unit 14 executing a processing program. The work
video is a moving image, but may be a still image. In any case, each image constituting
a moving image or a still image and temporal information (time, elapsed time from the
start time, and the like) are associated with each other and recorded in the expertwork
operation information record database 16a of the storage 16.
[0093]
The voice record processing unit 15a is configured to execute a process of
recording a voice during work by driving and controlling the voice acquisition unit 12a
when a processing program is executed by the CPU of the control unit 14. The voice
information is recorded in an expert work operation information record database 16a
of the storage 16 in association with temporal information (time, elapsed time from
the start time, and the like).
[0094]
The information transmission and reception processing unit 15 is configured
to execute processing for driving and controlling the communication unit 17 to
transmit and receive information to and from the work information generation server
by executing a processing program by the CPU of the control unit 14.
[0095]
The storage 16 stores the expert work operation information record database
16a.
[0096]
FIG. 5 is a configuration diagram showing a configuration example of the
expert work operation information record database 16a. As shown 5 in FIG. 5, items
recorded in each record (row) of the expert work operation information record
database 16a are, for example, the worker No. of the expert 2, the work No., the
recorded video, the recording voice, the work time, and the like.
[0097]
The worker number is an identification number uniquely assigned to each
worker. The work No. is a number uniquely assigned to identify the work. The
recorded video is the name of a moving image file that records the work. The recorded
voice is the name of the voice file that records the voice during work. The work time is
the time required for the work.
[0098]
The record (row) data is added to the expert work operation information
record database 16a every time the expert 2 performs the work specified by the work
No.
[0099]
When acquiring the work video and the work voice of the expert 2 while
transmitting those data to the work information generation server 30 and storing
them in the storage device 31 of the work information generation server 30, the expert
work operation information record database 16a may not be stored in the expert
terminal 10.
[0100]
(Worker terminal)
Next, the worker terminal 20 will be described.
The worker terminal 20 is, for example, a smartphone, a tablet terminal, a
head mounted display (HMD), a notebook PC, or the like. Further, the worker
terminal 20 acquires a video of the work video (work video) from a video acquisition
device (for example, ear-mounted camera) 21 attached to the worker 3, and acquires
the voice (work voice) uttered by the worker 3 during the work from a voice acquisition
device (for example, a pin microphone) 22 attached to the worker 3. The worker
terminal 20 outputs work support information to a support information 5 output device
(for example, an earphone, a speaker, and the like).
[0101]
Any one or two or all of the video acquisition device 21, the voice acquisition
device 22, and the support information output device 23 may be incorporated in the
worker terminal 20.
[0102]
FIG. 3 is a configuration diagram showing the configuration of the worker
terminal 20. As shown in FIG. 3, the worker terminal 20 includes a control unit 24, a
memory 25 storing various processing programs, a storage 26 storing various
databases, a communication unit 27, a voice acquisition unit 22a, a video acquisition
unit 21a, and a display unit. 23a, a voice / video output unit 23b, and a power supply
unit 28. The display unit 23a also functions as a support information output device
when the work support information is displayed as a moving image or a still image.
[0103]
Since the control unit 24, the memory 25 storing various processing programs,
the communication unit 27, the voice acquisition unit 22a, the video acquisition unit
21a, the display unit 23a, and the power supply unit 28 of the worker terminal 20 have
the same function as those corresponding units of the expert terminal 10, detailed
description thereof is omitted.
[0104]
The voice / video output unit 23b outputs voice and/or data to the support
information output device 23 (earphone, speaker, display, and the like) in order to
transmit work support information (success / failure of work, attention calling, and
the like) to the worker.
[0105]
The memory 25 stores various processing programs constituting the video
record processing unit 25b, the voice record processing unit 25a, and the information
transmission / reception processing unit 25c. These processing programs are executed
by the CPU of the control unit 24 so that each processing function 5 can be realized.
Since the processing programs realize the same function as the corresponding
processing programs of the expert terminal 10, detailed description thereof is omitted.
[0106]
The storage 26 is provided with an on-site worker work information record
database 26a.
[0107]
FIG. 6 is a configuration diagram showing a configuration example of the
on-site worker work information record database 26a. As shown in FIG. 6, the items
recorded in each record (row) of the on-site worker work information record database
26a are, for example, worker No, work date / time, work No, element work No, check
item, work time, work result, recorded video, and recorded voice, and the like.
[0108]
The worker No. is an identification number uniquely assigned to each worker.
The work date / time is the date / time when the worker 3 performed the work. The
work No. is a number uniquely assigned to identify the work. The element work No is
a number that identifies the element work when the work is divided into element
works. The check item indicates check contents. The work time is the time required for
the element work by the worker 3. The work result is a success / failure determination
result of the work. The recorded video is a video file name in which the work is
25 recorded. The recorded voice is the name of the voice file that records the voice during
the work.
[0109]
Record data (row) is added to the on-site worker work information record
database 26a every time the worker 3 performs the work specified by the work No.
[0110]
Note that the on-site worker work information record database 26a may be
stored in one or both of the storage 26 of the worker terminal 20 and the storage 302 of
the work information generation server 30.
[0111]
(Work information generation server)
Next, the work information generation server 30 will be described.
[0112]
FIG. 4 is a configuration diagram of the work information generation server
. As shown in FIG. 4, the work information generation server 30 includes a control
unit 300, a memory 301 storing various processing programs, a storage 302 storing
various databases, a communication unit 307, a display unit 303, an authentication
unit 308, and an external device connection unit 311, a power supply unit 312, a
worker terminal management unit 330, and a work analysis unit 335.
[0113]
The control unit 300 may be configured to use a computer having a CPU, a
ROM, a RAM, an input / output interface, an external storage device, and the like, and
a part or all of the functions of the work information generation server 30 can be
realized by executing various processing programs which are stored in the memory
301 on the CPU.
[0114]
The communication unit 307 communicates with the expert terminal 10 and
the worker terminal 20 via the network 4.
[0115]
The display unit 303 is constituted by a liquid crystal display or the like, and
can display work videos, system management statuses, and the like of the expert 2
and /or the worker 3.
[0116]
The authentication unit 308 confirms a user's usage qualification using a
known authentication method, and only the user having the usage qualification can
use the work information generation server 30.
[0117]
The external device connection unit 311 can be connected to an external
device such as a monitor device, a printer device, or an external 5 storage device as
necessary.
[0118]
The power supply unit 312 supplies power necessary to drive the work
information generation server 30.
[0119]

The memory 301 stores various processing programs constituting the video
data processing unit 313, the voice data processing unit 314, the information
superimposition processing unit 315, the information transmission / reception
processing unit 316, the positional data processing unit 318, and the space data
processing unit 319. These processing programs are executed by the CPU of the
control unit 300 so that the respective processing functions can be realized.
[0120]
The video data processing unit 313 performs various data processing on the
work video data transmitted from the expert terminal 10 and the worker terminal 20
when the processing program is executed by the CPU of the control unit 300. For
example, the video data processing unit 313 can perform image recognition processing
to recognize the work target device, the arms and hands of the expert 2 and the worker
3, the work tool, and the like. The video data processing unit 313 extracts feature
points and calculates feature amount for each image included in the work video
(moving image), for example. Based on the information on these feature points and
feature amount, it is possible to determine the similarity of two images, the presence /
absence of an object in the image, and the like.
[0121]
The voice data processing unit 314 performs various data processing on the
work voice data transmitted from the expert terminal 10 and the worker terminal 20
when the processing program is executed by the CPU of the control unit 300. To be
specific, the voice data processing unit 314 includes a voice recognition processing unit,
a language conversion processing 5 unit, and the like.
[0122]
The voice recognition processing unit performs voice recognition processing
on the voice data of the expert 2 transmitted from the expert terminal 10 or the voice
data of the worker 3 transmitted from the worker terminal 20 and converts the voice
data to obtain text data. The language conversion processing unit converts
(translates) the Japanese text data obtained by performing the voice recognition
processing on the voice data of the expert 2 into a designated foreign language. The
support information output device 23 outputs the text data that conveys the work
support information, which has been translated into the foreign language, by voice
and/or video. Note that the voice data processing unit 314 of the present embodiment
corresponds to the language conversion device according to the present invention.
[0123]
The information superimposition processing unit 315 is configured to execute
a process of superimposing and displaying work procedure information on an
augmented reality screen, for example, by executing a processing program by the CPU
of the control unit 300.
[0124]
The information transmission / reception processing unit 316 executes a
process of transmitting and receiving information to and from the expert terminal 10
and the worker terminal 20 by driving and controlling the communication unit 307
when the processing program is executed by the CPU of the control unit 300.
[0125]
The positional data processing unit 318 executes various data processing on
the positional data transmitted from the expert terminal 10 and the worker terminal
when the processing program is executed by the CPU of the control unit 300.
[0126]
The space data processing unit 319 executes various data processing on the
space (distance) data transmitted from the expert terminal 10 and the worker
terminal 20 by executing a processing program by the CPU of the 5 control unit 300.
[0127]

The worker terminal management unit 330 includes a work support
information distribution unit 331 and a device state management unit 332.
[0128]
The work support information distribution unit 331 is configured to distribute
the work support information to the worker terminal 20. To be specific, the work
support information distribution unit 331 transmits work support information to the
worker terminal 20 such as a determination result of the success or failure of the work
performed by the worker 3 and an attention call message given to the worker 3 before
the stage in which attention is required. The worker terminal 20 transmits the
received work support information data to the support information output device 23
via the voice / video output unit 23b. The support information output device 23 can
selectively output the work support information by at least one of voice and video
(moving image). The video data of the work support information may be displayed on
the display unit 23a. In this case, the display unit 23a functions as the support
information output device 23.
[0129]
As a result of the success / failure of the work, the voice or video message
transmitted when the work is unsuccessful is, for example, “The screw to be removed
is wrong!”,in a work of removing the screw. If the result of the work success / failure
determination is that the work is successful, it is not necessary to transmit a message,
or a message such as “The work to remove the screw has been successfully completed.”
may be transmitted. As a voice or video message for attention calling for example,
“Beware of electric shock!” or the like, in a parts replacement work.
[0130]
As work support information, for example, work procedures, environment
information, and status information may be included. As an explanation of the work
procedure, for example, there is a message of “Work 1: 5 Remove Screw”. The
environmental information includes, for example, the temperature and humidity of
the work site environment. Status information includes work progress, elapsed time,
and the like.
[0131]
The device state management unit 332 stores and manages information on
device specifications and states related to work including work target device and
information on the environment of the work site.
[0132]

The work analysis unit 335 includes a work operation record aggregation unit
336, a work record analysis unit 338, an analysis result output unit 339, a work check
information generation unit 340, and a work report creation unit 341. Note that the
work record analysis unit 338 of the present embodiment corresponds to the
determination device and the attention call control device according to the present
invention.
[0133]
The work operation record aggregation unit 336 acquires and aggregates the
work operation records (work video, work voice) by the expert 2 or the worker 3.The
information on the work operation of the expert 2 is stored in the expert work
operation information record database 323 stored in the storage 302, and the
information on the work operation of the worker 3 is stored in the on-site worker work
information record database 324.
[0134]
Based on the work check information, the work record analysis unit 338
compares and analyzes the work record of the worker 3 (work operation information
such as work video and work voice) with the work record of the expert 2 (work
operation information such as work video and work voice), and determines the success
or failure of the work of the worker 3.
[0135]
In addition, the work record analysis unit 338 executes a control to have the
support information output device 23 output attention calling information as work
support information before a specific stage during work, based on the work check
information and temporal information (elapsed time or work stage information) of the
work of the worker 3.
[0136]
The analysis result output unit 339 outputs the analysis result of the work
record to the work support information distribution unit 331 of the worker terminal
management unit 330, the work report creation unit 341, and the like.
[0137]
The work check information generation unit 340 generates the work check
information 70 based on the work video and work voice of the expert 2 stored in the
expert work operation information record database 323. The generated work check
information is stored in an appropriate location in the storage 302 such as the on-site
worker work information record database 324 and the expert work operation
information record database 323, for example.
[0138]
The work report creation unit 341 creates a work report in the form of an
electronic file including information such as the result of the success or failure of each
work of the worker 3 by the work record analysis unit 338 and the time required for
each work. The data is stored in an appropriate place in the storage 302 such as the
on-site worker work information record database 324. In the work report, a worker No.,
a worker name, a work date and time, a work No. (work content), and the like are also
recorded. The work report is transmitted to the worker terminal 20 as necessary, so
that the worker 3 can view the contents of the work report.
[0139]

The storage 302 stores an expert work information record database 323, an
on-site worker work information record database 324, and a 5 work target device
information record database 325. The storage 302 corresponds to the storage device 31
of FIG.1.
[0140]
The expert work operation information record database 323 and the on-site
worker work information record database 324 are the same as the corresponding
database configuration of the expert terminal 10 or the worker terminal 20, and thus
detailed description thereof is omitted. The work video information stored in the
expert work operation information record database 323 and the on-site worker work
information record database 324 may be information such as obtained feature points
and feature amount obtained by performing image processing on each image
constituting the video instead of the original video data.
[0141]
FIG. 7 is a configuration diagram showing the configuration of the work
target device information record database 325. As shown in FIG. 7, items recorded in
each record (row) of the work target device information record database 325 are a
device No., a work No., a specification, and a video.
[0142]
The device No. is an identification number that identifies a target device in
the work specified by the work No. The work No. is a number uniquely assigned to
identify the work. The specification column has a file name in which information on
the specification (spec) of the device specified by the device No. is recorded. The video
column has the image file name of the device identified by the device No..
[0143]
Next, a method for dividing video information and voice information in the
generation of work check information will be described with reference to FIG.8.
[0144]
The work check information generation unit 340 divides the work video and
work voice of the expert 2 into element works based on the information of the work
voice of the expert 2, and the work record analysis unit 338 5 determines whether the
work is successful for each element work.
[0145]
To be specific, first, the video acquisition device 11 acquires the work video of
the expert 2, and the voice acquisition device 12 acquires the work voice of the expert 2.
The expert terminal 10 receives the acquired work video and work voice data and
stores them in the expert work operation information record database 16a. The work
video is a moving image file 51 and includes a series of images taken at a
predetermined time interval. The work voice is a voice file 52 and includes voice
uttered by the expert 2. The expert terminal 10 sends the work video (moving image
file 51) and work voice (voice file 52) of the expert 2 stored in the expert work operation
information record database 16a to the work information generation server 30 via the
communication unit 17.
[0146]
The work information generation server 30 stores the work video and work
voice of the expert 2 in the expert work operation information record database 323.
Next, the voice data processing unit 314 performs voice recognition process on the
work voice data of the expert 2. Next, the work check information generation unit 340
finds a message such as “Work 1 Perform XXX.” from the result of the voice recognition
process, and identifies the start time of each element work (work 1, work 2,...). The
work check information generation unit 340 divides the work video and work voice
files into each of element works based on the information on the start time of each
element work, and stores it in an appropriate place in the storage 302.
[0147]
Next, generation processing of work check information 70 by the work check
information generation unit 340 will be described with reference to FIG.9.
[0148]
The generation process of the work check information 70 is performed
simultaneously with the process of dividing the work video and work voice of the
expert 2, but is not limited thereto, and may be performed before or 5 after the division
process.
[0149]
The work check information generation unit 340 performs voice recognition
process on the work voice data of the expert 2 stored in the expert work operation
information record database 323 by using the voice data processing unit 314, identifies
the work to be performed (work check item) and the item to be noted (attention item),
and stores the identified items as work check information 70 in an appropriate
location in the storage 302.
[0150]
In order to specify the work check item, for example, a work start message (for
example, “Work 1 Remove Screw.”, “Work 2 Remove Lid.”, and the like) including
specific words (for example, “Work 1”, “Work 2”, and the like) indicating the start of
each work is specified, and the message content and time (elapsed time from a
reference time) are acquired.
[0151]
As shown in FIG. 9, for example, the work check information generation unit
340 records the information specified in the respective columns of the work No., the
work time, the work check item of the work check information 70, in the case that the
work start message such as “Work 1 Remove Screw”, “Work 2 Remove Lid.”, “Work 3
Replace Part”, “Work 4 Close Lid.”, “Work 5 Fasten Screw” or the like has been
identified from the work voice of the expert 2..
[0152]
The work check item may be acquired not only from the work start message
but also from the message issued by the expert 2 during each work. For example, from
the message “Put the replaced part into the collection box” from the expert 2, the
content “Put the replaced part into the collection box” is recorded in the column of the
work check item, and the time is also recorded.
[0153]
In addition, the work check information generation unit 5 340, for example,
identifies an attention message to call for attention in order to identify an attention
item from the work voice of the expert 2, and acquires the message content and the
elapsed time (standard time (for example, work time) from the start)).
[0154]
10 For example, as shown in FIG. 9, when the work check information
generation unit 340 identifies an attention message such as “Beware of Electric Shock.”
or “Beware of Finger Pinching” from the work voice of the expert 2, information
specified in each column of the attention item and the elapsed time is recorded as the
work check information 70.
15 [0155]

Next, the work success / failure determination method 1 in the work check
will be described with reference to FIGS. 10A and 10B.
[0156]
20 The work record analysis unit 338 determines the success or failure of the
work based on whether or not a specific target included in the work video of the expert
2 is included in the work video of the worker 3.
[0157]
To be specific, the work record analysis unit 338 compares the work video of
25 the worker 3 with the work video of the expert 2 recorded in the expert work operation
information record database 323. This comparison process is performed in real time
while acquiring a work video of the worker 3 at the work site. As long as the real-time
property is not impaired, the work video of the worker 3 at the work site may be
temporarily recorded in the on-site worker work information record database 324, and
37
the work video data of the worker 3 recorded in this database may be used.
[0158]
As shown in FIG. 10A, for example, when the work check item is “Remove
Screw” of work 1, there exists an image A in which the work target screw is removed
from the target device in the work video of the expert 2 of work 1. Therefore, 5 if there
exists no image A in which the screw has been removed from the target device in the
work video of the worker 3, it can be determined that the work of “Remove Screw” has
not been performed. Therefore, it is checked whether or not there is an image
corresponding to the image A in which the screw has been removed in the work video
10 of the worker 3, and if it exists, it is determined that the work is successful (FIG. 10A),
and if not, it is determined that the work has failed (FIG. 10B).
[0159]
Whether or not an image corresponding to the image A of the object in the
specific state exists in the work video of the worker 3 can be determined, for example,
15 by whether or not the image to be compared has the same feature point and/or feature
amount as the image A exists.
[0160]
Whether or not the image A exists in the work video of the worker 3 is
determined by checking whether or not an image corresponding to the image A exists
20 in the work video of the worker within a range of ± Δt of the time t1 at which the image
A of the expert 2 appears. By limiting the image range to be compared in this way, it
is possible to quickly determine success or failure of the work.
[0161]
In the present embodiment, the image range to be compared in the work video
25 of the worker 3 is limited to a predetermined range according to the elapsed time, but
is not limited thereto. A comparison determination may be made for all images from
the first image S to the last image E of the work video of the worker 3. Further, when it
is found that an image corresponding to the image A exists during the comparison, the
comparison may be cut short for the subsequent images.
38
[0162]
As shown in FIG. 10B, when there is no image corresponding to the image A
in the work video of the worker 3 within the time t1 ± Δt, after the time t1 + Δt from
the start time ts of the work 1 has elapsed, attention is called with a voice and/or video
message, such as “Remove Screw” at an appropriate 5 point in time.
[0163]
Here, a method for determining the work start / work end of the worker 3 will
be described. As shown in FIGS. 10A and 10B, an image S obtained by photographing
a specific target exists at the top of the work video of the expert 2 in the work 1, and an
10 image E exists at the end. For example, if an image corresponding to the image S
appears in the work video of the worker 3, it is determined that the work 1 has started,
and if an image corresponding to the image E appears in the work video of the worker
3, the work 1 is determined to have ended. In this way, it can be determined whether
the worker 3 has started or ended the work (or element work) uniquely identified by
15 the work No..
[0164]
As another method, when the voice uttered by the worker 3 to the voice
acquisition device 22 (for example, a microphone) is subjected to voice recognition
processing by the voice data processing unit 314 and a message voice such as “Start
20 work 1” is detected, for example, it may be determined that the work 1 has started in
the work video, signaled by the message voice.
[0165]

Next, another work success / failure determination method 2 in the work
25 check will be described with reference to FIG. 11.
[0166]
In the work success / failure determination method 2, the work record
analysis unit 338 determines the success / failure of the work depending on whether or
not an object which is not included in the work video of the expert 2 is included in the
39
work video of the worker 3.
[0167]
To be specific, the work record analysis unit 338 compares the work video of
the worker 3 with the work video of the expert 2 recorded in the expert work operation
information record database 323. This comparison process is performed 5 in real time
while acquiring a work video of the worker 3 at the site. As long as the real-time
property is not impaired, the work video of the worker 3 at the work site may be
temporarily recorded in the on-site worker work information record database 324, and
the work video data of the worker 3 recorded in this database may be used.
10 [0168]
As shown in FIG. 11, for example, when the work check item is “Remove
Screw” of work 1, an image B in which a screw other than the work target screw is
removed from the target device in the work video of the expert 2 of work 1does not
exist. Therefore, if there exists the image B in which the screw other than the screw is
15 removed from the target device in the work video of the worker 3, it can be determined
that the operation of “Remove Screw” has not been performed successfully. Therefore,
it can be determined whether or not there is an image that does not correspond to any
of the images of the work video of the expert 2 in the work video of the worker 3, and if
it does not exist, it can be determined that the work is successful, if it exists, it is
20 determined that the work has failed (FIG. 11).
[0169]
Whether or not there exists an image that does not correspond to any of the
images of the work video of the expert 2 in the work video of the worker 3 can be
determined by, for example, whether or not the feature points and/or feature amount
25 of the images are similar.
[0170]
Whether or not an image corresponding to the image in the work video of the
expert 2 exists in the work video of the worker 3 may be determined by whether or not
an image corresponding to the work video of the expert 2 exists within a range of ± Δt
40
of the time t1 when the work video image of the worker 3 appears. By limiting the
image range to be compared in this way, it is possible to quickly determine success or
failure.
[0171]
As shown in FIG. 11, for example, when there exists an 5 image B that does not
correspond to any of the images of the work video of the expert 2 in the work video of
the worker 3, attention can be called by voice and/or video, for example, “Screw to
Remove is wrong”.
[0172]
10
Next, another work success / failure determination method 3 in the work
check will be described with reference to FIG. 12.
[0173]
In the work success / failure determination method 3, the work record
15 analysis unit 338 determines whether or not the work is successful by whether or not
specific objects are included in the work video of the expert 2 is included in the work
video of the worker 3 in a predetermined sequence.
[0174]
To be specific, the work record analysis unit 338 compares the work video of
20 the worker 3 with the work video of the expert 2 recorded in the expert work operation
information record database 323. This comparison process is performed in real time
while acquiring a work video of the worker 3 at the work site. As long as the
real-time property is not impaired, the work video of the worker 3 at the work site may
be temporarily recorded in the on-site worker work information record database 324,
25 and the work video data of the worker 3 recorded in the database may be used.
[0175]
As shown in FIG. 12, for example, when it is known from the work video
and/or work voice of the expert 2 that the work should proceed in the sequence of work
procedure 1, work procedure 2, and work procedure 3, the success / failure
41
determination of work 1 is determined as follows: first, the images A, B, and C
corresponding to the work procedures 1, 2, and 3 are specified from the work video of
the expert 2 respectively; next, it is checked whether or not images corresponding to
the images A, B, and C exist in the work video of the worker 3 and appear in that
sequence: if the images corresponding to the images A, B, and C exist 5 in that sequence
in the work video of the worker 3, it is determined that the work is successful; and if
the images do not exist in that sequence, then it is determined that the work is
unsuccessful. (FIG. 12).
[0176]
10 Whether or not the images corresponding to the images A, B, and C of the
work video of the expert 2 exist in the work video of the worker 3 can be determined by,
for example, whether or not the feature points and/or the feature amountof both
corresponding images are similar.
[0177]
15 Whether or not the image corresponding to the image A, B, or C in the work
video of the expert 2 exists in the work video of the worker 3 may be determined by
whether or not a corresponding image exists in the work video of the worker 3within
the range of ± Δt of the time t1 at which the image A, B or C appears in the work video
of the expert 2 By limiting the image range to be compared in this way, it is possible
20 to quickly determine success or failure.
[0178]
As shown in FIG. 12, when images corresponding to images A, B, and C do not
exist in that sequence in the work video of the worker 3, attention is called, for
example, by a voice or video such as “Work sequence is wrong”.
25 [0179]

Next, an attention call method 1 will be described with reference to FIG. 13.
[0180]
In this attention call method 1, the work record analysis unit 338 causes the
42
support information output device to output attention information as work support
information before a specific stage during the work based on the work check
information and the elapsed temporal information of the work of the worker 3.
[0181]
Here, the “specific stage” is a stage where attention should 5 be called during
the work. Examples of “the specific stage” include the stage of “grip the part” for
replacement in the “replace part” work 3 and the stage of “put the lid on the main body”
in the “close the lid” work 4. The work check information generation unit 340
determines the specific stage based on the work voice information of the expert 2 as
10 work check information.
[0182]
In the present embodiment, “temporal information” is information on the
elapsed time from the start of work by the worker 3. The work check information
generation unit 340 analyzes the work voice of the expert 2, extracts, for example, an
15 attention item such as “Beware of Electric Shock”, and records the time when the
message is issued (elapsed time from the start of the work).
[0183]
As shown in FIG. 13, the work record analysis unit 338 monitors the elapsed
time from the start of the work of the worker 3, and calls attention, for example, by a
20 voice or video such as “Beware of Electric Shock” at a time t1, which is before the time
t2 at the specific stage (corresponding to the image D). Here, t1 = t2 – Δt.
[0184]

Next, another attention call method 2 will be described with reference to FIG.
25 14.
[0185]
In the present embodiment, attention call is performed based on the elapsed
time from the start of work by the worker 3, but is not limited to this configuration,
and the attention call may be made based on information on the work stage identified
43
from the work video of the worker 3.
[0186]
As shown in FIG. 14, the work check information generation unit 340
analyzes the work video of the expert 2 in advance, and confirms and records that the
work stages 1,2 and 3 (corresponding to images A, B and C, respectively) 5 exist in this
sequence prior to the specific stage to call attention (corresponding to the image D).
Then, the work record analysis unit 338 analyzes the work video in real time during
the work of the worker 3, monitors the appearance of the images corresponding to the
images A, B, and C, and makes an attention call, for example, “Beware of Electric
10 Shock” by voice and/or video when the image corresponding to the image C is detected.
This means that the attention is securely called at the work stage before the specific
stage corresponding to the image D.
[0187]
Next, a work support method according to the present embodiment will be
15 described.
[0188]
FIG. 15 is a flowchart of the work support method according to the present
embodiment.
In the work information generation server 30, the work check information
20 generation unit 340 preliminarily divides the work video and work voice of the expert
2 into element works based on the work voice of the expert 2, and generates the work
check information 70 based on the work voice (S1). When the worker 3 is working, the
work record analysis unit 338 in the work information generation server 30 conducts a
work check on the work performed by the worker 3 in real time based on the work
25 check information 70 (S2). After the work is completed, the work report creation unit
341 creates a work report based on the result of the work check in the work
information generation server 30 (S3).
[0189]
Next, a method for generating the work check information 70 will be
44
described.
[0190]
FIG. 16 is a flowchart showing a method for generating the work check
information 70.
First, the video/voice during work of the expert 2 is acquired 5 (S10). To be
specific, the video acquired by the video acquisition device (for example, ear-mounted
camera) 11 worn by the expert 2 and the voice acquired by the voice acquisition device
(for example, microphone) 12 are taken into the expert terminal 10 and transmitted to
the work information creation server 30 via the network 4. The work video and the
10 work voice received by the work information generation server 30 are stored in the
expert work operation information record database 323.
[0191]
Next, the work video and work voice files of the expert 2 are divided into
element works based on the work voice of the expert 2 (S11). To be specific, in the work
15 information generation server 30, the voice data processing unit 314 performs voice
recognition processing on the work voice of the expert 2. As a result of the voice
recognition processing, a message for specifying an element work such as “Work 1 is
executed” is detected, and the start time of each element work (work 1, work 2,...) is
identified. Then, based on the start time information of each element work, the work
20 video and work voice files are divided into each of the element works.
[0192]
Next, work check items are extracted from the work voice of the expert 2 (S12).
To be specific, the work check item is extracted from the result of the voice recognition
process performed on the work voice of the expert 2 in the work information
25 generation server 30. For example, if a message such as “Work 1 Remove Screw”,
“Work 2 Remove Lid”, “Work 3 Replace Part”, “Work 4 Close Lid”, or “Work 5 tighten
screw” As check items, operation 1 “Remove Screw”, operation 2 “Remove Lid”,
operation 3 “Replace Part”, operation 4 “Close Lid”, Operation 5 “Fasten Screw”, and
the like are extracted. The work check information generation unit 340 stores the
45
work check items as work check information 70 together with the work time required
for the work.
[0193]
Further, when the work check information generation unit 340 detects an
attention calling message such as “Beware of Electric Shock” and 5 “Beware of Finger
Pinching”, the work check information generation unit 340 recognizes the message as
an attention item, and stores the reference time (for example, at the start of work) as
the work check information 70 together with the elapsed time information.
[0194]
10 Next, with reference to FIG. 17, a method for performing a work check on the
work performed by the worker 3 will be described.
[0195]
First, the work record analysis unit 338 of the work information generation
server 30 acquires the first work check item from the work check information 70 (S20).
15 To be specific, information such as a work check item and work time corresponding to
the work 1 of the work check information 70 stored in the storage 302 is extracted.
[0196]
Next, the work record analysis unit 338 determines whether or not there is an
attention item corresponding to the work 1 in the work check information 70, that is,
20 whether or not an attention calling is required (S21). If attention calling is necessary
(Yes in S21), the work record analyzing unit 338 causes the support information
output device 23 to call attention by voice and/or video when a predetermined elapsed
time has elapsed from the start of the task (S22). For example, when 60 seconds have
elapsed from the start of the work 3, the attention call “Beware of Electric Shock” is
25 given to the work 3 by voice and/or video. If the attention calling is not required (No in
S21), the process proceeds to step S23.
[0197]
In step S23, the work record analysis unit 338 conducts a work check on the
work of the worker 3 by comparing the work video of the expert 2 and the work video of
46
the worker 3. To be specific, the success / failure of the work is determined by using
any one of the above-described work success / failure determination methods 1 to 3 or
any combination thereof. The work success / failure determination method may vary
depending on the work content.
5 [0198]
If the work check item is unacceptable (failure) as a result of the
determination (No in S24), the work record analysis unit 338 causes the support
information output device 23 to warn the worker 3 by voice or video (S25). For example,
when the work check item “Fasten Screw” of the work 5 is not successfully completed,
10 a message “Please fasten the screw” is sent to the worker 3 via the support
information output device 23 by voice and/or video. The worker 3 who has received the
warning reworks the work (S26), and the process returns to step S23, and a work
check is conducted.
[0199]
15 If the work check item has passed (successful) as a result of the determination
(Yes in S24), the process proceeds to step S27, and the work record analysis unit 338
records information such as the pass of the work check item and work time on the
on-site worker work information record database 324 (S27)in the storage 302. If there
is another work check item to be checked in the work check information 70 (Yes in S28),
20 the process returns to step S20 and the above steps are repeated. If there is no other
work check item to be checked in the work check information 70 (No in S28), the work
check is terminated.
[0200]
Next, functions and effects will be described.
25 [0201]
As described above, in the work support system 1 according to the present
embodiment, the work check information generation unit 340 generates the work
check information 70 serving as a reference for the work check based on the voice
information acquired by the voice acquisition device 12. At the same time, the work
47
record analysis unit 338 determines the success or failure of the work by comparing
the work video of the expert 2 and the work video of the worker 3 on the site based on
the work check information.
[0202]
By this configuration, the work record analysis unit 5 338 determines the
success or failure of the work by comparing the work video of the expert 2 and the
work video of the worker 3 on the site based on the work check information 70, so that
the work check (work success / failure determination) can be securely accomplished. It
is also possible to guide the worker to rework the work by pointing out the work
10 mistake by voice or video based on the result of the work check. Therefore, work
mistakes can be reliably prevented by performing work checks even for workers with
low work skills.
[0203]
In the work support system 1 according to the present embodiment, the work
15 record analysis unit 338 determines whether the work is successful based on whether
a specific object included in the work video of the expert 2 is included in the work video
of the worker 3. By this configuration, it is possible to reliably detect forgetting work
by determining success or failure of the work based on the presence or absence of a
specific object essential for the work.
20 [0204]
Further, in the work support system 1 of the present embodiment, the work
record analysis unit 338 determines whether the work is successful based on whether
or not an object not included in the work video of the expert 2 is included in the work
video of the worker 3. By this configuration, it is possible to reliably detect an
25 erroneous operation (incorrect operation content) by determining the success or
failure of the operation based on the presence or absence of an object unnecessary for
the operation.
[0205]
Further, in the work support system 1 of the present embodiment, the work
48
record analysis unit 338 may determine the success or failure of the work based on
whether a specific object included in the work video of the expert 2 is included in the
work video of the worker 3 in a predetermined sequence. By this configuration, it is
possible to reliably detect a procedure mistake by determining the success or failure of
a work based on whether or not an object essential for the 5 work appears in a
predetermined sequence.
[0206]
Further, in the work support system 1 of the present embodiment, the work
check information generation unit 340 divides the work video of the expert 2 into
10 element works based on the voice information of the expert 2, and the work record
analysis unit 338 determines the success or failure of the work per each of the element
works. By this configuration, even if the work is performed for a long time, the success
or failure of the work can be determined more accurately by dividing the work into
element works each having an appropriate required time.
15 [0207]
Further, in the work support system 1 of the present embodiment, the support
information output device 23 can selectively output the work support information at
least one of voice and video (moving image or still image). By this configuration, it is
possible to conduct work support by selecting an optimum method (voice only / video
20 only / voice and video) in accordance with the work environment.
[0208]
In addition, the work support system 1 of the present embodiment further
includes a voice data processing unit 314 that translates a voice that conveys the work
support information output from the support information output device 23 into a
25 specified language. By this configuration, a foreign worker who does not understand
Japanese can be provided with work support in a foreign language that the foreigner
can understand.
[0209]
In addition, the work support system 1 of the present embodiment further
49
includes a work report creation unit 341 that creates a work report including a work
determination result by the work record analysis unit 338.By this configuration, since
the work report is automatically created when the work on the site is completed, the
labor for the worker 3 to create the work report can be saved.
5 [0210]
Further, in the work support system 1 of the present embodiment, the work
record analysis unit 338 causes the support information output device 23 to output the
attention information as the work support information prior to the specific stage
during the work, based on the work check information 70 and the information on the
10 elapsed time from the start of the work by the worker 3.By this configuration, the
work record analysis unit 338 outputs attention information prior to the specific stage
requiring attention based on the work check information 70 and the elapsed time, so
that work mistakes and accidents can be effectively prevented, thereby making it
possible to improve the quality of work.
15 [0211]
Also, in the work support system 1 of the present embodiment, the work check
information generation unit 340 determines the specific stage based on the voice
information during work of the expert 2 as work check information. By this
configuration, it is possible to easily specify a specific stage requiring attention.
20 [0212]
Further, in the work support system 1 of the present embodiment, the work
record analysis unit 338 may cause the support information output device 23 to output
the attention information as the work support information prior to the specific stage
during the work, based on the work check information 70 and the work stage
25 information recognized from the work video of the worker 3. By this configuration, it is
possible to call attention at a more appropriate timing.
[0213]
The work support system 1 according to the present embodiment includes: the
work information generation server 30 including at least the work check information
50
generation unit 340; the expert terminal 10 having the video acquisition device 11 and
the voice acquisition device 12 and connected to the work information generation
server 30; and the worker terminal 20 having the video acquisition device 21 and the
support information output device 23 and connected to the work information
generation server 30.By this configuration, by using the server, 5 the configuration of
the worker terminal 20 that the worker carries at the work site can be simplified.
[0214]
In the present embodiment, the work information generation server 30
includes a work record analysis unit 338, an expert work operation information record
10 database 323, and the like, and performs a work check (success / failure
determination) of the worker 3. However, it is not limited to this configuration. The
work support system according to the present invention may be so configured that the
worker terminal 20 includes a work record analysis unit 338, an expertwork operation
information record database 323, an on-site worker work information record database
15 324, and the like, wherein the worker terminal 20 performs a work check (success /
failure determination) on the worker 3.Further, in the present embodiment, the work
information generation server 30 having a work report creation unit 341 creates a
work report. However, it is not limited to this configuration. The work support
system according to the present invention may be so configured that the worker
20 terminal 20 having a work report creation unit 341 creates a work report on the
worker terminal 20.
[0215]
Further, in the work check information generation step S1, the work support
method according to the present embodiment generates the work check information 70
25 serving as a reference for the work check based on the information of the voice file 52
acquired in the voice acquisition step S10. In the determination step S2, the success or
failure of the work is determined by comparing the work video of the expert 2 and the
work video of the worker 3 on the site based on the work check information 70.
[0216]
51
By this configuration, the success / failure of the work is determined by
comparing the work video of the expert 2 and the work video of the worker 3 in the
work site based on the work check information 70 in the determination step S2, so that
the work check (work success / failure determination) can be securely performed. It is
also possible to guide the user to redo the work by pointing out the 5 work mistake with
voice or video based on the result of the work check. Therefore, work mistakes can be
prevented by performing work checks with certainty even for workers with low work
skills.
[0217]
10 (Second Embodiment)
Next, a second embodiment according to the present invention will be
described with reference to the drawings. In the following description, the same
components as those of the first embodiment are denoted by the same reference
numerals, and detailed description thereof is omitted as appropriate.
15 [0218]
FIG. 18 is a diagram showing a schematic configuration of a work support
system 1A according to the second embodiment according to the present invention.
The work support system 1A is a system that supports the work of the worker 3 by
interacting with the worker 3 in the work site, for example, by voice or a terminal
20 screen. To be specific, as illustrated in FIG. 18, the work support system 1A includes a
worker terminal 20 and a work support server 30A, which are connected to each other
via a network 4. The work support server 30A may include the configuration of the
work information generation server 30.
[0219]
25 The worker terminal 20 has: the video acquisition device (for example, an
ear-mounted camera) 21 that acquires a video of the operation; the voice acquisition
device (for example, a microphone) 22 as an information acquisition device that
acquires voice during the work; the support information output device(for example, an
earphone, a speaker, a display, an AR (Augmented Reality) glass) 23 that outputs the
52
work support information; the image acquisition device (for example, a camera) 122
for acquiring an image (for example, a photograph) of a work target; and the sensor
device 123 that senses information related to the work, connected therewith by wired
or proximity wireless communication. The work support server 30A is connected
with the storage device 31 having various database stored 5 therein. The support
information output device 23 according to the present embodiment corresponds to the
information output device according to the present invention.
[0220]
The worker terminal 20 is, for example, a portable terminal such as a
10 smartphone or a tablet terminal, and any or all of the voice acquisition device 22, the
image acquisition device 122, the video acquisition device 21, and the support
information output device 23 may be built in the worker terminal 20. For example,
the support information output device 23 may be a speaker or a display built in the
worker terminal 20 such as a smartphone or a tablet terminal. The worker terminal 20
15 may include a screen operation acquisition device (for example, a touch panel) 121 as
an information acquisition device that acquires information on a screen operation by
the worker (see FIG. 19). Regardless of whether the various devices connected to the
worker terminal 20 are built-in or external devices, only those necessary are equipped
and used depending on the work content, the work environment, the worker skill level,
20 and the work support pattern to be described later, and the like.
[0221]
The representative devices of the worker 3 are not limited, but exemplified by,
for example, (i) only a smartphone as the worker terminal 20, (ii) a smartphone as the
worker terminal 20, and an earphone as the support information output device 23, (iii)
25 a smartphone as worker terminal 20 and an AR glass as the support information
output device 23, (iv) a smartphone as the worker terminal 20, an earphone as the
support information output device 23 and an ear camera as the video acquisition
device 21 (V) a smartphone as the worker terminal 20, an earphone as the support
information output device 23, a body sensor as the sensor device 123, and the like.
53
[0222]
In the device case of (i), the worker 3 performs the work by carrying a
smartphone with a touch panel, and receives work support by interacting on the
screen with the work support system 1A through screen display and screen operation
of the touch panel of the smartphone. In this device case, a work support 5 system can be
constructed at a low cost.
[0223]
In the device case of (ii), the worker 3 performs the work while carrying the
smartphone and wearing the earphone, and receives work support through conducting
10 a voice dialogue with the work support system 1A using the earphone and the
microphone built in the smartphone. In this device case, it is possible to construct the
work support system at a low cost, and it is not necessary for the worker 3 to operate
the smartphone during the work, so that the workability is good because both hands
can be used freely.
15 [0224]
In the device case of (iii), the worker 3 carries the smartphone and performs
out the work with the AR glass, and receives work support through the video displayed
on the AR glass. In this device case, both hands of the worker 3 can be used freely, so
that workability is good, and support information can be displayed superimposed on
20 the work target by AR technology, so that highly functional support is possible.
[0225]
In the device case (iv), the worker 3 carries a smartphone and wears an
earphone and an ear-mounted camera to perform the work, transmits work video
information from the ear-mounted camera to the work support system 1A, and
25 receives work support from the earphone by voice. In this device case, since both
hands of the worker 3 can be used freely, the workability is good, and the success or
failure of the work can be determined more accurately based on the video information
of the work. .
[0226]
54
In the device case of (v), the worker 3 carries a smartphone and wears an
earphone and a body sensor to perform work, and transmits body information (for
example, action, fingertip sensation, fingertip pressure, and the like) obtained from
the body sensor to the work support system A1, while receiving support from the ear
phone by voice. In this device case, both hands of the worker 3 can 5 be used freely, so
the workability is good, and the success or failure of the work can be determined more
accurately based on the physical information of the worker 3, thereby enabling
high-performance support.
[0227]
10 In each of the above device cases, a smartphone with any or all of camera
function, speaker function, microphone function, display function, and touch sensor
function as necessary is used. Of course, the smartphone as the worker terminal 20 of
each of the device cases may be another portable terminal such as a tablet terminal.
[0228]
15 FIG. 19 is a diagram showing functional blocks of the work support system 1A
according to the second embodiment according to the present invention. As shown in
FIG. 19, the work support system 1A includes: a control device 100 and a storage 302
on a work support server 30A side; and a support information output device 23, an
information acquisition device 120, an image acquisition device 122, a video
20 acquisition device 21 and a sensor device 123 on a worker 3 side. The information
acquisition device 120 acquires information such as voice and screen operation from
the worker 3, and includes a voice acquisition device 22 and a screen operation
acquisition device 121. Note that the control device 100 according to the present
embodiment corresponds to the determination device according to the present
25 invention.
[0229]
The control device 100 determines success / failure of the work according to a
predetermined determination criterion based on the information from the worker 3
acquired by the information acquisition device 120.The control device 100 includes a
55
dialogue unit 101, an information input / output unit 108, a determination unit 112,
and a document creation unit 117.
[0230]
The dialogue unit 101 performs an audio dialogue with the worker 3 or a
dialogue through such as a touch panel attached to the worker 5 terminal 20 as the
support information output device 23 (hereinafter also simply referred to as “screen”),
and includes a work instruction unit 102, attention information output unit 103,
information detection unit 104, and an inquiry confirmation unit 107. The information
detection unit 104 detects information acquired by the information acquisition device
10 120, and includes an utterance detection unit 105 and a screen operation detection
unit 106.
[0231]
The work instruction unit 102 is configured to cause the support information
output device 23 to output instruction information for instructing the work, based on
15 the work check information stored in the work check information recording unit 130 of
the storage 302. The output from the support information output device 23 may be any
output method of voice, screen display, and voice + screen display (the same applies
hereinafter). For example, at the start of the work, the work instruction unit 102
causes the support information output device 23 to audibly output an instruction
20 message “Please perform a screw fastening work”.
[0232]
The work instruction unit 102 may cause the support information output
device 23 output the attention information to indicate that the work has not been
completed in real time by voice or the like, if the determination unit 112 determines
25 that the work has not been completed based on the work video acquired by the video
acquisition device 21, the work information acquired by the sensor device 123,and the
like. When the work is not completed due to a work mistake or the like, the work
instruction unit 102 notifies the worker 3 of the attention information in real time by
voice, so that the work mistake can be prevented more effectively.
56
[0233]
The attention information output unit 103 causes the work attention
information to be outputted from the support information output device 23 based on
the work check information stored in the work check information recording unit 130 of
the storage 302. The output from the support information output 5 device 23 may be in
any output method of voice, screen display, and voice + screen display. For example,
the attention information output unit 103 outputs an attention message “Do not hold
the screw and screwdriver by both hands” following the instruction message “Please
perform the screw fastening operation” from the support information output device 23
10 by voice. As described above, at the start of the work or the like, the attention
information output unit 103 can caused the attention information to be outputted from
the support information output device 23 by voice, so that attention of the worker 3
can be called, thereby making it possible to effectively reduce the work mistakes and
the work accidents.
15 [0234]
The information detection unit 104 detects response information responding
to the instruction information from the information acquired by the information
acquisition device 120 after the support information output device 23 outputs the
instruction information.
20 [0235]
The utterance detection unit 105 detects the response information in voice in
response to the instruction information from the voice information acquired by the
voice acquisition device 22 after the instruction information is outputted by the
support information output device 23 using a known voice recognition technique. For
25 example, the utterance detection unit 105 detects the word “Completed.” uttered by
the worker 3 at the end of the work or at the confirmation of the inquiry. The voice
information indicating the response information detected by the utterance detection
unit 105 is stored in a voice storage unit 133 provided in the storage 302 as a work
evidence.
57
[0236]
The specific utterance of the worker 3 detected by the utterance detection unit
105 is not limited to “Completed.”, but may be other words, word combinations,
phrases, sentences, coined words, and the like.
5 [0237]
The screen operation acquisition device 121, serving as the support
information output device 23, acquires information on the operation (for example, tap,
double tap, soft key input, and the like) of the worker 3 on the screen such as a touch
panel attached to the worker terminal 20. Hereinafter, the screen operation performed
10 by the worker 3 is referred to as “screen operation”.
[0238]
The inquiry confirmation unit 107 causes the inquiry information for
confirming completion of work to be outputted from the support information output
device 23 after the determination unit 112 described later determines that the work is
15 successful. The output from the support information output device 23 may be in any
output method of voice, screen display, and voice + screen display. For example, the
inquiry confirmation unit 107 causes an inquiry confirmation message “Fastened
Completely?” to be outputted from the support information output device 23 by voice.
[0239]
20 Further, the information input / output unit 108 inputs information from the
image acquisition device 122, the video acquisition device 21, the sensor device 123
and the like built in or connected to the worker terminal 20,outputs various support
information to the support information output device 23, and includes an image / video
information acquisition unit 109, a sensor information acquisition unit 110, and a
25 support information output unit 111.
[0240]
The image / video information acquisition unit 109 acquires the image
information of the work target acquired by the image acquisition device 122 and /or
acquires the video information acquired by the video acquisition device 21. The
58
image information and the video information acquired by the image / video
information acquisition unit 109 are stored in an image / video storage unit 131
provided in the storage 302. The work image information and video information stored
in the image / video storage unit 131 are used as work evidence or for machine
learning of a work comparison model based on a known AI (artificial 5 intelligence)
technique. The image / video information acquisition unit 109 of this embodiment
corresponds to the image information acquisition unit and the video information
acquisition unit according to the present invention.
[0241]
10 The sensor information acquisition unit 110 acquires the information
relevant to the work detected by the sensor device 123.The sensor device 123is, for
example, exemplified by: an operation sensor that is attached to the worker 3 to detect
the operation of the worker 3; a sensor that is attached to the fingertip of the worker 3
to detect the sensation and pressure of the fingertip; an odor sensor that senses an
15 on-site odor; a temperature sensor; a humidity sensor; a taste sensor; a sound sensor
that detects a sound of the facilities and the audio in the work sit; a vibration sensor
that detects vibrations; an acceleration sensor; line-of-sight sensor that detects the
line of sight of the worker 3; and the like. The determination unit 112 may determine
completion of the work based on the information acquired by the sensor information
20 acquisition unit 110. By doing so, completion of the work can be determined more
correctly.
[0242]
The support information output unit 111 outputs work support information to
the support information output device 23 based on the determination result by the
25 determination unit 112 of the control device 100. The output from the support
information output device 23 may be in any output method of voice, screen display,
and voice + screen display.
[0243]
The determination unit 112 determines the success or failure of the work
59
based on the response information detected by the information detection unit 104, and
includes a determination switching unit 113, a first determination unit 114, a second
determination unit 115, and a third determination unit 116.
[0244]
The determination switching unit 113 selects one of the 5 first determination
unit 114, the second determination unit 115, and the third determination unit 116,
according to the importance of the work. The determination unit determines the
success or failure of the work.
[0245]
10 The first determination unit 114 determines the success or failure of the work
based on the response information detected by the information detection unit 104,
after the instruction information is outputted by the work instruction unit 102 and the
support information output device 23. For example, after the support information
output device 23 outputs an instruction message “Please perform screw fastening
15 operation” by voice, the first determination unit 114 determines the success or failure
of the work, based on the utterance (response information) of the worker 3 detected by
the utterance detection unit 105 of the information detection unit 104. In this case,
the first determination unit 114 determines that the work has been completed if the
utterance of the worker 3 is, for example, “Completed.”, and determines that the work
20 has not been completed if the utterance of the worker 3 is, other than “Completed.”.
[0246]
The second determination unit 115 determines the success or failure of the
work based on the response information detected by the information detection unit
104, after the inquiry information is outputted by the inquiry confirmation unit 107
25 and the support information output device 23. For example, the second determination
unit 115 determines the success or failure of the work based on the utterance
(response information) of the worker 3 detected by the utterance detection unit 105 of
the information detection unit 104, after outputting the inquiry message “Fastened
Completely?” by voice by the support information output device 23. In this case, the
60
second determination unit 115 determines that the work has been completed if the
utterance of the worker 3 is “Completed.”, for example, and determines that the work
has not been completed if the utterance of the worker 3 is other than “Completed.”.The
inquiry message here may not only be the above expression that confirms the
completion of work, but also the color of the lamp and the input 5 numerical value (for
example, “What is the color of the lamp? What is the input numerical value?), or
confirming the completion state of work (for example, “Which direction is the valve
lever?”) may be uttered. Moreover, the inquiry message may be repeating them several
times.
10 [0247]
The third determination unit 116 determines that the work is incomplete
when the image / video information acquisition unit 109 has not acquired image
information such as a photograph of the work target, and the third determination unit
106 determines the success or failure of the work based on the response information
15 detected by the information detection unit 104, after outputting the inquiry message
by the support information output device 23. For example, when , the image / video
information acquisition unit 109 acquires the image information such as a photograph
of the work object after the support information output device 23 outputs an
instruction message “Please perform a screw fastening work” by voice, the third
20 determination unit 116 determines the success or failure of the work based on the
utterance (response information) of the worker 3 detected by the utterance detection
unit 105 of the information detection unit 104. In this case, the third determination
unit 116 determines that the work has been completed if the utterance of the worker 3
is, for example, “Completed.”, and determines that the work is not completed if it is
25 other than “Completed.”.
[0248]
The third determination unit 116 may determine whether the image
information (evidence photograph) acquired by the image / video information
acquisition unit 109 is appropriate based on whether the image information (evidence
61
photograph) was taken at an appropriate shooting position or an appropriate shooting
direction. When the third determination unit 116 determines that the image
information is inappropriate or has not been uploaded, the work instruction unit 102
may output the instruction information for requesting the image information from the
support information output device 23 by voice or the like. Thereby, 5 acquisition of
inappropriate image information (evidence photograph) can be prevented. Moreover, it
will also call attention of the worker 3, so that work mistakes can be reduced.
[0249]
In the present embodiment, since the determination unit 112 selects and uses
10 one of the first to third determination units 114, 115, and 116 according to the
importance of the work, it is possible to handle flexibly, such as using, for example, the
first determination unit 114 having a low determination criteria with respect to the
work having a low degree of importance, while using, for example, the second
determination unit 115 or the third determination unit 116 having a high
15 determination criteria with respect to the work having a high degree of importance,
thereby making it possible to effectively reduce work mistakes without lowering the
work efficiency.
[0250]
Further, the document creation unit 117 has a function to create documents
20 necessary for the work (for example, a check list, a work report, and the like), and
includes a data acquisition unit 118 and a work document generation unit 119.
[0251]
The data acquisition unit 118 acquires data related to the work, when the
determination unit 112 determines that the work has been normally completed.To be
25 specific, the data acquisition unit 118 acquires, for example, information on the
determination result of the success or failure of the work by the determination unit
112 and the work time required for the work.
[0252]
The work document generation unit 119 generates work documents such as a
62
check list and a work report based on data related to the work acquired by the data
acquisition unit 118. To be specific, for example, when the work document generation
unit 119 determines that the work is completed based on the response information
such as an utterance or a screen operation detected by the information detection unit
104, the document work generation unit 119 generates the work 5 documents such as
check list. Thus, the work documents such as the check list are automatically
generated by the work document generation unit 119 at a trigger of related
information indicating “completion” from the worker 3, the burden on the worker 3 is
reduced, mistakes and omissions of description can be prevented, and the man-hours
10 required for the creation work can be reduced. Further, the work document
generation unit 119 generates the work documents such as the check list inclusive of
the information of determination result by the determination unit 112 and the work
time, thereby making it possible to create more appropriate work documents.
[0253]
15 The storage 302 is a storage device that stores work-related information,
various databases, and the like, and includes a work check information storage unit
130, a voice storage unit 133, an image / video storage unit 131, and a work document
storage unit 132.
[0254]
20 The work check information storage unit 130 stores work check information
70 that serves as a basis for work check (work success / failure determination).
[0255]
The voice storage unit 133 stores voice information of the utterance of the
worker 3 detected by the utterance detection unit 105 of the information detection unit
25 104 in association with information such as a work item and a work date as work
evidence. The voice information stored in the voice storage unit 133 contributes to
ensuring traceability.
[0256]
The image / video storage unit 131 stores the image information and video
63
information acquired by the image / video information acquisition unit 109 in
association with information such as work items and work dates. These image
information and video information contribute to ensuring traceability as evidence of
work, and can also be used for machine learning of a work comparison model based on
5 AI technology.
[0257]
The work document storage unit 132 stores work document information such
as the check list and the work report generated by the work document generation unit
119.
10 [0258]
The control device 100 according to the present embodiment may vary the
scope of work for which success or failure is determined by the determination unit 112
in accordance with the skill level of the worker. Thus, work support can be performed
more efficiently. For example, a worker with a high skill level can perform work
15 without interruption between the work A and the work B by determining the success
or failure of the work within the range of the work A and the work B combined. For
example, a worker with a low skill level can provide detailed work support by
determining the success or failure of the work A and the work B separately.
[0259]
20 FIG. 20 is a diagram illustrating a schematic configuration of the control
device 100 according to the present embodiment. As shown in FIG. 20, the control
device 100 includes a CPU 140, a ROM 141, a RAM 142, an input / output I / F 144, and
a communication I / F 145 connected to each other via a bus 143. Each functional block
of the control device 100 shown in FIG. 19 is realized by, for example, reading various
25 programs stored in the ROM 141 or the storage 302 on to the RAM 142 and executing
them by the CPU 140.
[0260]
Next, the work support pattern (method) in this embodiment will be described
with reference to the drawings. In the following, work support by voice will be
64
described first, and then work support by screen display will be described.
[0261]

In the case of work support by voice, the work support system 1A includes, for
example, the work support server 30A including the control device 5 100, the worker
terminal 20 connected to the work support server 30A via the network 4, and the
support information output device 23 that outputs the support information. In the
following paragraphs, the work support patterns 1-4 will be explained in which a
smartphone (corresponding to the worker terminal 20), an earphone (corresponding to
10 the support information output device 23) connected to the smartphone by Bluetooth
(registered trademark), and a microphone (corresponding to the voice acquisition
device 22) built in the smartphone are used.
In this configuration, a hybrid type that uses not only voice but also a screen of a
smart device such as a smartphone may be used. In this case, the smart device may
15 display the details of the work, the work animation, the position of the current work in
the whole work, the contact of the inquiry about unknown points, and the like.
[0262]
[Work Support Pattern 1]
FIG. 21 is a diagram illustrating the work support pattern 1 according to the
20 present embodiment. The broken line indicates work support for work B. Hereinafter,
each step of the work support will be sequentially described.
[0263]
(1) Work Instruction Step: At the start of work B, the work instruction unit
102 of the work support server 30A in the work support system 1A transmits
25 instruction information for instructing work to the worker terminal 20 (smartphone),
and the support information output device 23 (earphone) outputs the instruction
information by voice. The instruction information is, for example, a voice instruction
message “Please perform screw fastening work”. Further, the contents of the work
may be explained.
65
[0264]
(2) Call Attention Step: Following the output of the instruction information,
the attention information output unit 103 of the work support system 1A transmits
the attention information in the work to the worker terminal 20 (smartphone), and
outputs attention information to the worker 3 by voice 5 through the support
information output device 23 (earphone).The attention information is, for example, an
attention message by voice saying “Do not hold the screw and driver by both hands”.
This attention calling may be directed to work ingenuity and tips regarding work
mistakes, injury / accident prevention and work efficiency.
10 [0265]
(3) Completion Notification Step: When the work is completed, the worker 3
gives a completion notification by voice. To be specific, the worker 3 utters
“Completed.”, the voice acquisition device 22 (smartphone built-in microphone)
acquires the voice of the utterance, and the utterance detection unit 105 detects the
15 voice information “Completed.”, which is response information in response to the
instruction information, from the voice information acquired by the voice acquisition
device 22. Here, the completion notification is not limited to the completion state
such as “Completed.”, but may be a completion notification that To be specific,
responds such as the lamp color is “blue” or the bulb direction is “right”.
20 [0266]
Then, the determination unit 112 determines the success or failure of the
work based on the utterance information detected by the utterance detection unit 105.
For example, if the content of the utterance of the worker 3 is “Completed.”, it is
determined that the work is successful and the process proceeds to the next work C. If
25 it is other than “Completed.”, the work is determined to be unsuccessful or incomplete.
[0267]
As described above, in the work support pattern 1, the work support system
1A can be provided at a low cost with a simple configuration, since the work of the
worker 3 is supported by the work support system 1A and the worker 3 interacting
66
with each other by voice, the work support system 1A is easy to use, since the worker 3
can use both hands freely, and it is possible to effectively prevent a work mistake,
since the success / failure of the work is determined based on the utterance of the
worker 3.
5 [0268]
[Work Support Pattern 2]
FIG. 22 is a diagram illustrating the work support pattern 2 according to the
present embodiment. The broken line indicates work support for work B. Hereinafter,
each step of the work support will be sequentially described.
10 [0269]
(1) Work Instruction Step: At the start of work B, the work instruction unit
102 of the work support server 30A in the work support system 1A transmits
instruction information for instructing work to the worker terminal 20 (smartphone),
and the support information output device 23 (earphone) outputs the instruction
15 information by voice. The instruction information is, for example, a voice instruction
message “Please perform screw fastening work”. At this time, the contents of the
above work may be additionally described.
[0270]
(2) Call Attention Step: Following the output of the instruction information,
20 the attention information output unit 103 of the work support system 1A transmits
the attention information in the work to the worker terminal 20 (smartphone), and the
support information output device 23 (earphone)outputs the attention information to
the worker 3 by voice. The attention information is, for example, an attention message
by voice saying “Do not hold the screw and driver by both hands”. This attention call
25 may be directed to work ingenuity and tips regarding work mistakes, injury / dead
prevention and work efficiency.
[0271]
(3) Completion Notification Step: When the work is completed, the worker 3
gives a completion notification by voice. To be specific, the worker 3 utters
67
“Completed.”, the voice acquisition device 22 (smartphone built-in microphone)
acquires the voice of the utterance, and the utterance detection unit 105 detects the
voice information “Completed.”, which is response information in response to the
instruction information acquired by the voice acquisition device 22.
5 [0272]
(4) Inquiry Confirmation Step: After the completion notification, the inquiry
confirmation unit 107 of the work support system 1A makes an inquiry confirmation to
the worker 3 by voice. To be specific, when the determination unit 112 determines that
the work is successful, the inquiry confirmation unit 107 transmits inquiry
10 information for confirming the completion of the work to the worker terminal 20
(smartphone), and the inquiry information is outputted by voice from the support
information output device 23 (earphone). The inquiry information may be, for example,
a specific inquiry message with a voice such as “Fastened Completely?”. “What is the
color of the lamp?” or “Which is the direction of the valve?”.
15 [0273]
(5) Completion Notification Step: After receiving the inquiry message, the
worker 3 gives a completion notification by voice. To be specific, the worker 3 utters
“Completed.”, the voice acquisition device 22 (smartphone built-in microphone)
acquires the voice of the utterance, and the utterance detection unit 105 detects the
20 voice information “Completed.”, which is the response information in response to the
instruction information. Further, a specific state or situation may be uttered
according to the inquiry message. Furthermore, (4) and (5) may be repeated a
plurality of times.
[0274]
25 Then, the determination unit 112 determines the success or failure of the
work based on the utterance information of the worker 3 detected by the utterance
detection unit 105 after the inquiry voice is outputted by the support information
output device 23. For example, if the content of the utterance of the worker 3 is
“Completed.”, it is determined that the work is successful and the process proceeds to
68
the next work C. If it is other than “Completed.”, the work is determined to be
unsuccessful or incomplete.
[0275]
As described above, in the work support pattern 2, after the determination
unit 112 determines the success of the work, the inquiry confirmation 5 unit 107 and the
support information output device 23 make an inquiry to confirm the completion of the
work. The possibility of noticing mistakes can be increased, so that the work mistakes
can be effectively reduced.
[0276]
10 [Work Support Pattern 3]
FIG. 23 is a diagram illustrating the work support pattern 3 according to the
present embodiment. The broken line indicates work support for work B. In the
work support pattern 3, an image acquisition device 122 (a camera built in a
smartphone) is also used. Hereinafter, each step of the work support wi ll be
15 sequentially described.
[0277]
(1) Work Instruction Step: At the start of work B, the work instruction unit
102 of the work support server 30A in the work support system 1A transmits
instruction information for instructing work to the worker terminal 20 (smartphone),
20 and the support information output device 23 (earphone) outputs the instruction
information by voice. The instruction information is, for example, a voice instruction
message such as “Please perform the screw fastening work. Please upload a
photograph of the target part after completion”. At this time, the contents of the work
may also be mentioned.
25 [0278]
(2) Call Attention Step: Following the output of the instruction information,
the attention information output unit 103 of the work support system 1A transmits
the work attention information to the worker terminal 20 (smartphone), and the
support information output device 23 (earphone) to output attention calling
69
information to the worker 3 by voice. The attention information is, for example, an
attention message by voice saying “Do not hold the screw and driver by both hands”.
This attention call may be directed to work ingenuity and tips regarding work
mistakes, injury / accident prevention and work efficiency.
5 [0279]
(3) Photograph Uploading Step: When the work is completed, the worker 3
takes a photograph of a target portion of the work with the image acquisition device
122 (built-in camera), and the image / video information acquisition unit 109 acquires
the photographed photograph data.
10 [0280]
(4) Completion notification step: After uploading the photograph, the worker
3 gives a completion notification by voice. To be specific, the worker 3 utters
“Completed.”, the voice acquisition device 22 (smartphone built-in microphone)
acquires the voice of the utterance, and the utterance detection unit 105 detects the
15 voice information “Completed.”, which is response information in response to the
instruction information acquired by the voice acquisition device 22. At this time, the
sent photograph may be compared with a correct image, so that positive or negative is
confirmed, and the result is notified. At this time, if the sent photograph is different
from the correct image, the target work may be reworked from the beginning.
20 [0281]
Then, the determination unit 112 determines the success or failure of the
work based on the utterance information of the worker 3 detected by the utterance
detection unit 105. For example, if the content of the utterance of the worker 3 is
“Completed.”, it is determined that the work is successful and the process proceeds to
25 the next work C. If it is other than “Completed.”, the work is determined to be
unsuccessful or incomplete.
[0282]
(5) Re-Request Step: The determination unit 112 determines that the work is
incomplete when the image / video information acquisition unit 109 has not acquired
70
the image information (photograph data) of the work target, and the work instruction
unit 102 causes the support information output device to output the instruction
information for re-requesting the image information (photograph data) by voice. This
instruction information is, for example, a re-request message by voice such as
“Photograph is not uploaded. 5 Please upload again”.
Also, the sent image in this block may be compared with the correct image
stored in advance, and the result may be sent. In this case, when it is determined that
the image is different from the correct image, the target work may be controlled to
start again.
10 [0283]
(6) Completion Notification Step: After receiving the re-request message, the
worker 3 gives a completion notification by voice. To be specific, the worker 3 utters
“Completed.”, the voice acquisition device 22 (smartphone built-in microphone)
acquires the voice of the utterance, and the utterance detection unit 105 detects the
15 voice information “Completed.”, which is the response information in response to the
instruction acquired by the voice acquisition device 22.
[0284]
Then, the determination unit 112 determines the success or failure of the
work based on the utterance information of the worker 3 detected by the utterance
20 detection unit 105 after the re-request voice is outputted by the support information
output device 23. For example, if the content of the utterance of the worker 3 is
“Completed.”, it is determined that the work is successful and the process proceeds to
the next work C. If it is other than “Completed.”, the work is determined to be
unsuccessful or incomplete.
25 [0285]
Thus, in the work support pattern 3, the determination unit 112 determines
that the work is incomplete when the image / video information acquisition unit 109
has not acquired the image information (evidence photograph) of the work target.
Therefore, the worker 3 performs the work more carefully, and can reduce work
71
mistakes more effectively. Traceability can also be realized by acquiring the image
information (evidence photographs) of work objects, and furthermore, the image
information can be used as teacher data in machine learning of work success / failure
models.
5 [0286]
In the work support pattern 3, the determination unit 112 may determine the
completion of the work based on the image information (evidence photograph)
acquired by the image / video information acquisition unit 109. In this way, completion
of the work can be determined more accurately.
10 [0287]
When photographing the work of the worker 3 by the image acquisition device
122 (camera), the work image is stored as evidence in the image / video storage unit
131, and the path is described along with the voice data in the work report described
later.
15 [0288]
[Work Support Pattern 4]
FIG. 24 is a diagram illustrating the work support pattern 4 according to the
present embodiment. In the work support pattern 4, for example, the sensor device
123 that detects physical information such as the movement of the worker 3 and the
20 feeling of the fingertips and the video acquisition device 21 (ear-hook camera) are used.
Hereinafter, each step of the work support will be sequentially described.
[0289]
(1) Work Instruction Step: At the start of work B, the work instruction unit
102 of the work support server 30A in the work support system 1A transmits
25 instruction information for instructing work to the worker terminal 20 (smartphone),
and the support information output device 23 (earphone) outputs the instruction
information by voice. The instruction information is, for example, a voice instruction
message “Please perform screw fastening work.”.
[0290]
72
(2) Call Attention Step: Following the output of the instruction information,
the attention information output unit 103 of the work support system 1A transmits
the work attention information to the worker terminal 20 (smartphone), and the
support information output device 23 (earphone) outputs the attention information to
the worker 3 by voice. The attention information is, for example, 5 an attention message
by voice saying “Do not hold the screw and driver by both hands”.
[0291]
(3) Sensor Information Acquisition Step: During the work, the sensor
information acquisition unit 110 acquires the physical information of the worker 3
10 acquired by the sensor device 123.
[0292]
(4) Pointing OutStep: The determination unit 112 recognizes the work status
based on the physical information of the worker 3 acquired by the sensor information
acquisition unit 110 and determines completion of the work. When the work is not
15 normally performed, the support information output unit 111 transmits the attention
information indicating that the work is not normal to the worker terminal 20
(smartphone), and outputs the attention information to the worker3 by voice from
the support information output device 23 (earphone). The attention information is,
for example, an attention message by voice “Are you performing screw fastening
20 work?”.
[0293]
(5) Normal Automatic Confirmation Step: The determination unit 112
performs video recognition based on the work video of the worker 3 acquired by the
image / video information acquisition unit 109, and automatically determines whether
25 or not the work has been performed normally (work completion), thereby making it
possible to determine the completion of the work more accurately.
[0294]
To be specific, the work support system 1A may be so configured that the work
video of the worker 3 is captured as a moving image by the image acquisition device 21,
73
stored in the image / video storage unit 131 via the image / video information
acquisition unit 109, compared with originally acquired model work video, and the
work is determined to be a mistake and the work mistake is pointed out to the worker
3 by the support information output device 23, if the captured image significantly
differs from the model work video. This may be done during or after 5 the work. The
worker's work progress may be recognized by an image or a body sensor.
Further, abnormal noise or the like may be detected and pointed out to the
worker by comparing the abnormal noise with stored normal noise data.
Still further, information of vibration sensors and surrounding environment
10 sensors (weather, temperature, humidity, wind direction, brightness, time) and the
like may be recognized, and work mistakes and determinations may be pointed out in
accordance with a predetermined algorithm.
[0295]
(6) Finish Notice Step: Instead of (5), the worker 3 may give a finish
15 notification by voice. To be specific, the worker 3 utters (repeals) “Screw fastening
work is finished.”, the voice acquisition device 22 (smartphone built-in microphone)
acquires the voice of the utterance, and the voice information acquired by the voice
acquisition device 22and the utterance detection unit 105 detects the voice
information “Screw fastening work is finished.”, which is the response information in
20 response to the instruction acquired by the voice acquisition device 22.
[0296]
Then, the determination unit 112 determines the success or failure of the
work based on the utterance information of the worker 3 detected by the utterance
detection unit 105. For example, if the content of the utterance of the worker 3 is
25 “Screw fastening work is finished.”, it is determined that the work is successful and
the process proceeds to the next work C. If it is other than “Screw fastening work is
finished.”, the work is determined to be unsuccessful or incomplete.
[0297]
(7) When the determination unit 112 determines that the work has been
74
normally completed from the sensor information acquired by the sensor information
acquisition unit 110 and the video information acquired by the image / video
information acquisition unit 109, the process may proceed to the work C without
making a finish notification. In the work C, the work instruction step and the call
attention step performed at the start of the work may be omitted. 5 By doing so, the
work can be performed without being interrupted between the work B and the work C.
[0298]
Next, a method of using image data and video data will be described with
reference to FIG. 25.
10 [0299]
The image (photograph) data and video data acquired by the image / video
information acquisition unit 109 are sent to and stored in the image / video storage
unit 131 of the storage 302. Traceability can be realized by acquiring the photograph
data (evidence photograph) and the video data of the work target as evidence. In
15 addition, the photograph data and video data stored in the image / video storage unit
131 are compared with the photograph data (evidence photograph) and video data of
the work completion state originally stored, so that work mistakes can be pointed out
in real time.
[0300]
20 The photograph data and video data stored in the image / video storage unit
131 are used as teacher data in the machine learning of the work comparison model
136. To be specific, a person makes a success / failure determination on the
photographic data and video data stored in the image / video storage unit 131, and
labels each photographic data or video data with success / failure based on the
25 determination result. To bespecific, for example, the labeling is performed from the
viewpoint of (i) whether the photographing position is correct or (ii) whether the
photographing is normally completed. Machine learning of the work comparison
model 136 is performed using the labeled photograph data and video data (teacher
data). In machine learning, photographic data and video data taken when an expert
75
performs work are also used as teacher data.
[0301]

Next, work support by screen display will be described.
5 [0302]
The above-described work support by voice can be performed by voice alone,
but the work support by the screen of the support information output device 23 (touch
panel built in the smartphone) can be used in combination. In the following, work
support using a screen used in combination with the voice work support patterns 1 to 4
10 will be described.
[0303]
[Work Support Pattern by Screen 1]
FIG. 26 is a diagram showing an example of a display screen of the support
information output device 23 (touch panel) in the above-described work support
15 pattern 1 by voice, and shows a work guidance screen 151.
[0304]
The work guidance screen 151 includes a work instruction area 152, a model
work area 153, and an attention calling area 154, and is provided with a complete soft
key 155, a start over soft key 156, a work detail soft key 157, and a work navigation
20 soft key 158, and an inquiry soft key, as soft keys.
[0305]
In the work instruction area 152, a work instruction message from the work
support system 1A to the worker 3 is displayed. For example, a work instruction
message such as “Fasten side screw. Press Complete when finished” is displayed.
25 [0306]
The model work area 153 is an area where a video of a model work by an
expert or the like is displayed for reference by the worker 3. This model work video is
usually played back when the model work area 153 is tapped or when the work starts.
Further, the video of the model work may be played back in synchronization with the
76
work of the worker 3 when the work of the worker 3 is recognized. When this area is
tapped, the video may be enlarged and displayed. In addition, the video of the model
work may be played back with the video position adjusted, so that the current working
stage of the worker 3 and the working stage of the model work match (synchronize).
The current work status of the worker 3 is recognized based on video 5 information and
the voice information acquired by the video acquisition device 21, the voice acquisition
device 22 and the like.
[0307]
In the attention calling area 154, for example, an attention message such as
10 “Hold the driver by both hands not to drop it.” is displayed.
[0308]
The complete soft key 155 is displayed, for example, as “Complete.”, and when
the worker 3 presses the complete soft key 155 at the end of the work, the completion
information is notified to the work support system 1A.
15 [0309]
The start over soft key 156 is displayed, for example, as “Start Over”., and
when the worker 3 presses the start over soft key 156, the video of the model work
displayed in the model work area is played back from the beginning.
[0310]
20 When the work detail soft key 157 is pressed by the worker 3, technical
material such as a work manual is displayed.
[0311]
When the work navigation soft key 158 is pressed by the worker 3, the current
work position in the entire series of works is displayed. For example, the current
25 position of the work may be highlighted by increasing the size or changing the color of
the characters indicating the current work in a flow chart of a series of works.
[0312]
When the worker 3 presses the inquiry soft key 159, information on a contact
point is displayed.
77
[0313]
The above-mentioned soft keys are selected and pressed by the worker 3, but
in addition to this, the voice uttered by the worker 3 may be selected by a voice
recognition technology. For example, when the worker 3 utters “Inquiry”, the
information on the contact point 5 may be displayed.
[0314]
[Work Support Pattern by Screen 2]
FIG. 27 is a diagram showing an example of a display screen of the support
information output device 23 (touch panel) in the above-described work support
10 pattern 2 by voice, and shows a screen at the time of inquiry confirmation.
[0315]
A screen at the time of inquiry confirmation as shown in FIG. 27 is a screen
displayed after the worker 3 presses the complete soft key 155 at the end of the work
on the work guidance screen 151. In the inquiry confirmation area 160, for example,
1an inquiry confirmation message “Fastened Completely?” is displayed, and when the
worker 3 presses the response soft key 161 displayed as “Yes”, the completion
information is notified to the work support system 1A.
[0316]
[Work Support Pattern by Screen 3]
FIG. 28 is a diagram showing an example of a display screen of the support
information output device 23 (touch panel) in the above-described work support
pattern 3 by voice and shows a work guidance screen 151.
[0317]
In the work instruction area 152, a work instruction message from the work
support system 1A to the worker 3 is displayed. For example, a work instruction
message such as “Fasten side screw. Upload photo of target portion when completed.”
is displayed.
[0318]
In the attention calling area 154, for example, an attention message such as
“Hold the driver by both hands not to drop it” is displayed.
[0319]
For example, “Photo” is displayed on the photo soft key 163, and when the
worker 3 presses the photo soft key 163, a camera function screen is displayed, so that
a photograph of the work target can be taken. The photographed 5 data is uploaded to
the work support system 1A.
[0320]
The worker 3 performs the work according to the work instruction message,
and when the work is completed, taps the photo soft key 163 to take a photograph of
the work target and uploads the photograph data to the work support system 1A. Next,
the worker 3 taps the complete soft key 155 to notify the work support system 1A of
the completion.
[0321]
FIG. 29 is a diagram showing an example of a display screen of the support
information output device 23 (touch panel) in the work support pattern 3, and shows a
screen when re-requesting uploading of photograph data.
[0322]
The screen for re-requesting the uploading of the photograph data shown in
FIG. 29 is displayed in the case that the determination unit 112 cannot confirm the
upload of the photograph data or the data is not an appropriate photograph data, after
the worker 3 presses the complete soft key 155 on the work guidance screen 151 shown
in FIG. 28. When a re-request message, for example, “Photo is not uploaded. Upload
photo again.” is displayed in the re-request area 164, and a re-take photo soft key 165
displayed as “Re-Take Photo” is pressed by the worker 3, the screen changes to the
camera function screen, so that a photograph of the work target can be taken. The
photographed data is uploaded to the work support system 1A. Next, the worker 3
presses the complete soft key 155 to notify the work support system 1A of the
completion.
[0323]
[Work Support Pattern by Screen 4]
FIG. 30 is a diagram showing an example of a display screen of the support
information output device 23 (touch panel) in the work support pattern 4 by voice
described above, and shows a screen when a work mistake is pointed out.
[0324]
The screen when a work mistake is pointed out as shown in FIG. 30 is a
screen displayed when the determination unit 112 recognizes a work based on the
sensor information acquired by the sensor device 123 and is determined to be a work
mistake. In a point out area 166, for example, a point out message “In the Fasten
Screw work ?” is displayed. At the end of the work, the worker 3 taps a work end soft
key 167 displayed as, for example, “End of screw tightening work”, in order to notify
the work support system 1A of the work end.
[0325]
As described above, the work support system 1A according to the present
embodiment can perform work support by using both work support by voice dialogue
and work support by dialogue on the screen, but is not limited thereto. It is also
possible to perform work support only by voice dialogue and work support only by
dialogue on the screen. When performing work support only through dialogue on the
screen, the worker 3 may carry a smartphone (worker terminal 20) including at least a
touch panel (support information output device 23). Further, when both voice
dialogue and dialogue on the screen are used together, the completion notification by
the worker 3 may be performed by voice or by screen operation.
[0326]

The work document generation unit 119 of the work support server 30A
automatically creates work documents such as a checklist, a work report, and a work
analysis document, based on the data related to each work acquired by the data
acquisition unit 118 after the completion of the entire work, and stores them in the
storage 302. The work documents stored in the storage 302 can be displayed on the
display device at the worker terminal 20 or the work support server as necessary. The
format and description items of the work document created by the document creation
unit 117 can be arbitrarily set.
[0327]
FIG. 31 shows a work document screen 168 displayed 5 on the support
information output device 23 (touch panel). The work document screen 168 shows a
work report that also serves as a check list. In the work report, information identifying
the work such as work date / time, work location, work content, work person in charge
is written in an upper portion of the document, while columns showing work ID, work
name, work time, completion check box, evidence, quality, retry / detail confirmation
are provided in a lower portion of the document.
[0328]
The column of work time includes standard time and work time. The
“standard time” indicates a standard work time obtained when extracting (generating)
work check information or a standard work time set in advance, while the “work time”
in the same column indicates an actually measured work time in the current work.
[0329]
In the complete column, the result of the work success / failure determined by
the determination unit 112 is shown in a check box. For example, the check box in the
complete column shows: the determination result of work success / failure based on
the completion notification by voice in response to the work instruction by voice (work
support pattern 1); the determination result of work success / failure based on the
completion notification by voice in response to the inquiry confirmation by voice(work
support pattern 2); the determination result of work success / failure based on the
completion notification by the uploaded photograph and voice in response to the work
instruction by voice (work support pattern 3); the determination result of work success
/ failure based on the end notification by voice in response to the work instruction by
voice and the pointing out of the work mistake (work support pattern 4) ; and the like.
[0330]
The evidence column contains a link to: the completion notification voice data
file (work support pattern 1) responded to the work instruction; the completion
notification voice data file (work support pattern 2) responded to the inquiry
confirmation; the uploaded image (photograph) data file (work support pattern 3);or
the voice data file (work support pattern 4) of the end notification 5 in response to the
work instruction and the pointing out of the work mistake.
[0331]
The quality column contains, for example, the quality of the work (classified
in the order of A, B, and C from the top) determined based on the work video and work
time acquired by the video acquisition device 21 (ear camera).
[0332]
Theretry / detail confirmation column contains the number of times of retry or
detail confirmation in the work is described.
[0333]
FIG. 32 shows a work analysis form screen 169 displayed on the support
information output device 23 (touch panel). The work analysis form screen 169 is a
screen for displaying a work analysis form indicating a result of analyzing the work.
In the work analysis form, information identifying the work such as work date and
time, work location, work contents, work person in charge is written in the upper part,
20 and in the lower part, the work time and the standard work time are displayed in a
comparable manner so that the difference between the work time and the standard
time of each work ID can be understood, and in addition, the ratio (%) of the work time
to the standard time of all the works is shown. The number of retries / detail
confirmations is displayed for each work ID. A “detail” soft key is displayed near the
retry / detail confirmation number display, and when tapped, the inquiry content and
the detail confirmation history are displayed. Then, the ratio of the actual work time
to the standard work time and the overall work level determined based on the number
of retries / detail confirmation (classified in the order of A, B, and C from the top) are
displayed.
[0334]
A “recommendable educational program” soft key is displayed in the lower
right bottom part of the work analysis form screen 169, and when it is tapped, a web
application of the recommendable educational program is started based on the
determined 5 work level.
[0335]
In the present embodiment, anon-site work has been described as an example,
but the present invention can also be implemented in cooking, home appliance /
furniture attachment, assembly support, and the like. An example of cooking will be
described. In order to preserve the so-called “taste of mom’s home cooking” which is a
cooking method peculiar to each home and in order to teach to the next generation,
this invention can be used to hand it down to the next generation. The present
invention can be applied to cooking support using the techniques described herein
using the voice or screen via smartphone or tablet with respect to a support content
from which the subsequent cooking process, cooking precautions, injury prevention
points, and the like are extracted. Moreover, it is possible to point out the difference
by comparing the numerical value of the taste preserve saved in each process with the
numerical value of the taste preserve saved beforehand using a taste sensor. Further,
the difference may be pointed out as compared with the arrangement image stored in
advance.
[0336]
As described above, the present invention has an effect that it is possible to
prevent a work mistake by reliably performing a work check even for a worker having
a low work skill, and is useful in general in a work support system and a work support
method.
[0337]
[Appendix 1]
The work support system 1 of Appendix 1 is as follows.
[0338]

[Background Art]
In various on-site work including maintenance and inspection, various
information such as work procedure information, work target device information, and
work tool information to be used is required. On the other hand, the worker's work
skills vary widely. In view of this, a support system has been 5 proposed that provides
necessary information to workers who work on site to support the on-site work (see, for
example, Japanese Unexamined Patent Application Publication No. 2009-069954).
[0339]
Japanese Unexamined Patent Application Publication No. 2009-069954
discloses a work support apparatus that instructs the work to be performed in the
work site, the work support apparatus comprising: a measurement unit that grasps a
situation where a target to be worked is placed; an information processing unit that
controls the measurement unit, performs appropriate information processing on the
obtained information and constructs visual information to be presented to the worker;
and a display that gives the worker visual information superimposed on the target
object.
[0340]
[Summary of Invention]
[Technical Problem]
However, although the conventional work support apparatus described in
Japanese Unexamined Patent Application Publication No. 2009-069954 helps to
transfer the advanced work skills of the expert to the worker, there has been no
consideration in reliably preventing the work mistake made by the worker with the
low work skill. In actual work sites, there are an increasing number of cases where
foreign workers who have low work skills and are not accustomed to Japanese
language perform the work, so that there are increasing number of mistakes in the
work, occurrences of forgetting the work, and the accidents in the work. The social
impact of accidents is large, so that in addition to improving work quality as a
measure to prevent recurrence, confirmation by two-person work, refinement of work
checklists and work reports may be required.
[0341]
The present invention has been made to solve the above-described problems,
and it is an object according to the present invention to provide a work support system
that can securely prevent work mistakes even for workers with 5 low work skills.
[0342]
[Means to Solve the Problem]
In order to achieve the above object, a work support system (1) according to
the present invention comprises: a first video acquisition device (11) that acquires a
video of a work performed by a first worker (2);a voice acquisition device (12) that
acquires a voice of the first worker during the work; a work check information
generation device (30) that generates work check information (70) that serves as a
reference for a work check based on the work video acquired by the first video
acquisition device and the voice information acquired by the voice acquisition device; a
second video acquisition device (21) that acquires video of work performed by a second
worker (3);a support information output device (23) that outputs work support
information to the second worker; and a attention call control device (30) that has the
support information output device output attention information before a specific stage
during the work as the work support information based on the work check information
and elapsed temporal information of the work of the second worker
[0343]
By this configuration, the attention call control device outputs attention
information before a specific stage that requires attention based on work check
information and the temporal information (for example, work elapsed time or work
stage), thereby making it possible for a worker with low work skills to securely
prevent the work mistake.
[0344]
[Appendix 2]
A work support system 1A according to Appendix 2 is as described below.
[0345]
In the conventional work support apparatus described in Patent Document 2
and the like, no consideration is given to providing work support at a low cost without
impairing the workability of the worker. In actual work sites, there are an increasing
number of cases where foreign workers who have low work 5 skills and are not
accustomed to Japanese language perform the work, so that there is a demand to
provide work support at low cost with a petty device that is easy for workers to work
on.
[0346]
It is the object according to the present invention to provide a work support
system capable of providing work support even for workers with low work skills
without impairing the workability of the worker with a simple configuration.
[0347]
In order to achieve the above object, the work support system according to the
present is a work support system (1A), comprising: an information acquisition device
(120) that acquires information from a worker; a determination device (100) that
determines success or failure of the work; and an information output device (23) that
outputs work support information, wherein the determination device includes: a work
instruction unit (102) has instruction information to instruct the work outputted from
the information output device; an information detection unit (104) that detects
response information in response to an instruction information based on the
information acquired by the information acquisition device after output of the
instruction information by the information output device; and a determination unit
(112) that determines the success / failure of the work based on the response
information detected by the information detection unit.
[0348]
By this configuration, at the start of the work, the work instruction unit and
the information output device transmit the work instruction information to the worker
by voice or screen display, and at the end of the work, the information acquisition unit
acquires the information from the worker by voice, screen operation, or the like. At
that time, the information detection unit detects response information in response to
the instruction information, and the determination unit determines the success or
failure of the work based on the content of the response information. In this way, the
work support system and the worker support the worker's work 5 by, for example,
performing dialogue by voice or screen. The work support system can be constructed at
a low cost with a simple configuration, and since the worker can use both hands freely,
the work support system is easy to use, and since the success or failure of the work is
determined based on the information from the worker such as the utterance and
10 screen operation of the worker, it is possible to efficiently prevent work mistakes.
[EXPLANANTION OF REFERENCE NUMERALS]
1, 1A Work Support System
2. Expert (First Worker)
3. Worker (Second Worker)
4 Network
10 Expert Terminal (First Terminal)
11 Video Acquisition Device (First Video Acquisition Device)
12, 22 Voice Acquisition Device
20 Worker Terminal (Second Terminal)
21 Video Acquisition Device (Second Video Acquisition Device)
23 Support Information Output Device
30 Work Information Generation Server (Server)
30A Work Support Server
31 Database
51 Video File
52 Voice File
60 Element Work Information
61 Work Record of Element Work 1
62 Work Record of Element Work 2
63 Work Record of Element Work 3
70 Work Check Information
100 Control Device (Determination Device)
101 Dialogue Unit
102 Work 5 Instruction Unit
103 Attention Information Output Unit
104 Information Detection Unit
105 Utterance Detection Unit
106 Screen Operation Detection Unit
107 Inquiry Confirmation Unit
108 Information Input / Output Unit
109 Image / Video Information Acquisition Unit
110 Sensor Information Acquisition Unit
111 Support Information Output Unit
112 Determination Unit
113 Determination Switching Unit
114 First Determination Unit
115 Second Determination Unit
116 Third Determination Unit
117 Document Creation Unit
118 Data Acquisition Unit
119 Work Document Generation Unit
120 Information Acquisition Unit
121 Screen Operation Acquisition Device
122 Image Acquisition Device
123 Sensor Device
130 Work Check Information Storage Unit
131 Image / Video Storage Unit
132 Work Document Storage Unit
150 Display Screen of Worker Terminal
151 Work Guidance Screen
152 Work Instruction Area
153 Model Work Area
154 Attention 5 Calling Area
155 Complete Soft Key
156 Start Over Soft Key
157 Work Details Soft Key
158 Work Navigation Soft Key
160 Inquiry Confirmation Area
161 Response Soft Key
163 Photo Soft Key
164 Re-Request Area
165 Re-Take Photo Soft Key
166 Point Out Area
167 Response Soft Key
168 Work Document Screen
169 Work Analysis Form Screen
314 Voice Data Processing Unit (Language Converter)
323 Expert Work Operation Information Record Database
324 On-Site Worker Work Information Record Database
338 Work Record Analysis Unit (Determination Device, Attention Call Control
Device)
340 Work Check Information Generation Unit (Work Check Information Generation
Device)
341 Work Report Creation Unit(Work Report Creation Device)

We claim:
1. A work support system, comprising:
an information acquisition device that acquires information from a worker;
a determination device that determines success or failure of a work; and
an information output device that outputs work support information 5 based on a
determination result by the determination device, wherein
the determination device determines the success or failure of the work
according to a predetermined determination criterion based on information from the
worker acquired by the information acquisition device.
2. A work support system, comprising:
a first video acquisition device that acquires video of work performed by a first
worker; a voice acquisition device that acquires voice of the first worker during the
work; a work check information generation device that generates work check
information that serves as a reference for a work check based on the work video
acquired by the first video acquisition device and the voice information acquired by
the voice acquisition device;
a second video acquisition device that acquires video of work performed by a
second worker; a determination device that determines success or failure of the work of the
second worker; a support information output device that outputs the determination result by
the determination device as work support information, wherein
the determination device determines the success or failure of the work by
comparing the work video of the first worker and the work video of the second
worker based on the work check information.
3. The work support system according to claim 2, wherein the determination
device determines whether or not the work is successful depending on whether or not
a specific object included in the work video of the first worker is included in the work
video of the second worker.
4. The work support system according to claim 2, wherein the determination
device determines whether or not the work is successful based on whether or not an
object not included in the work video of the first worker is included in the work video
of the second worker.
5. The work support system according to claim 2, wherein the determination
device determines whether or not the work is successful or failure depending on
whether or not a specific object included in the work video of the first worker is
included in a predetermined sequence in the work video of the second worker.
6. The work support system according to any one of claims 2 to 5, wherein the
work check information generation device divides the work video of the first worker
into element works based on the voice information of the first worker, and the
determination device determines the success or failure of the work for each element
work.
7. The work support system according to any one of claims 2 to 6, wherein the
support information output device can selectively output the work support
information by at least one of voice and moving image.
8. The work support system according to any one of claims 2 to 7, further
comprising a language conversion device that converts a voice that transmits the
work support information output from the support information output device into a
specified language.
9. The work support system according to any one of claims 2 to 8, further
comprising a work report creation device that creates a work report including a
determination result by the determination device.
10. The work support system according to any one of claims 2 to 9, further
comprising an attention call control device that causes the support information output
device to output attention information as the work support information before a
specific stage during the work based on the work check information and the temporal
information of the work of the second worker.
11. The work support system according to claim 10, wherein the work check
information generation device determines the specific stage based on voice
information during work of the first worker as the work check information.
12. The work support system according to claim 10 or claim 11, wherein the
temporal information is information on an elapsed time from the start of work of the
second worker.
20 13. The work support system according to claim 10 or claim 11, wherein the
temporal information is work stage information that is recognized from a work video
of the second worker.
14. The work support system according to any one of claims 2 to 13, further
comprising:
a server including at least the work check information generation device;
a first terminal having the first video acquisition device and the voice
acquisition device and connected to the server; and
a second terminal having the second video acquisition device and the support
information output device and connected to the server.
15. A work support method, comprising:
a first video acquisition step to acquire a video of a work performed by a first
worker; a voice acquisition step to acquire the voice of the first worker during the
work; a work check information generation step to generate work check information
serving as a reference for the work check based on the work video acquired in the
first video acquisition step and the voice information acquired in the voice
acquisition step;
a second video acquisition step to acquire a video of the work performed by a
second worker;
a determination step to determine success or failure of the work of the second
worker; and a support information output step to output the determination result in the
determination step as work support information,
wherein the determination step determines the success or failure of the work by
comparing the video of the work of the first worker and the video of the work of the
second worker based on the work check information.
16. The work support system according to claim 1, wherein the determination
device includes:a work instruction unit to have instruction information for instructing the work
to be outputted from the information output device;
an information detection unit to detect response information in response to the
instruction information from the information acquired by the information acquisition
device, after the instruction information is outputted by the information output
device; and a determination unit that determines success or failure of the work based on the
response information detected by the information detection unit .
17. The work support system according to claim 16, wherein
the determination device further includes an inquiry confirmation 5 unit that
outputs inquiry information for confirming completion of work from the information
output device after the determination unit determines that the work is successful, and
the determination unit determines the success or failure of the work based on
the response information detected by the information detection unit, after the inquiry
10 information is outputted by the information output device.
18. The work support system according to claim 16 or claim 17, wherein the
determination device further includes an attention information output unit that causes
a work attention information to be outputted from the information output device.
19. The work support system according to any one of claims 16 to 18, wherein
an image acquisition device to acquire an image of a work target is further
provided,
the determination device further includes an image information acquisition
unit that acquires image information of the work target acquired by the image
acquisition device,
the determination unit determines that the work is incomplete, when the image
information acquisition unit has not acquired the image information of the work
target, and
the work instruction unit causes instruction information for requiring the
image information to be outputted from the information output device, when the
determination unit determines that the work is incomplete, on an account that the
image information has not been acquired.
20. The work support system according to claim 19, wherein the determination
unit includes:
a first determination unit that determines the success or failure of the work
based on the response information detected by the information detection unit, after
the instruction information is outputted by the information 5 output device;
a second determination unit that determines the success or failure of the work
based on the response information detected by the information detection unit , after
the inquiry information is outputted by the information output device; and
a third determination unit that determines that the work is incomplete when the
image information acquisition unit has not acquired the image information of the
work target, and determines the success or failure of the work based on the response
information detected by the information detection unit after the instruction
information is outputted by the information output device, when the image
information of the work object is acquired,
15 and the determination unit selects and uses one of the first determination unit,
the second determination unit and the third determination unit, in accordance with
importance of the work.
21. The work support system according to claim 19 or claim 20, wherein
20 the determination unit determines whether the image information acquired by
the image information acquisition unit is appropriate, and
the work instruction unit causes instruction information for requiring image
information to be outputted from the information output device, when the
determination unit determines that the image information is inappropriate.
22. The work support system according to any one of claims 19 to 21, wherein the
determination unit determines completion of the work based on the image
information acquired by the image information acquisition unit.
23. The work support system according to any one of claims 16 to 22, wherein
a video acquisition device that acquires a video of the work performed by the
worker is further provided,
the determination device further includes a video information acquisition unit
that acquires video information acquired by the video acquisition 5 device, and
the determination unit determines the completion of the work based on the
video of the work by the worker acquired by the video acquisition unit.
24. The work support system according to any one of claims 16 to 23, wherein
a sensor device that detects information related to the work is further provided,
the determination device further includes a sensor information acquisition unit
that acquires the information related to the work detected by the sensor device, and
the determination unit determines the completion of the work based on the
information related to the work acquired by the sensor information acquisition unit.
25. The work support system according to claim 23 or claim 24, wherein the work
instruction unit causes information indicating that the work is incomplete to be
outputted in real time from the information output device, when the determination
unit determines that the work is incomplete.
26. The work support system according to any one of claims 16 to 25, further
comprising:
a data acquisition unit that acquires data related to the work, when the
determination unit determines that the work has been normally completed; and
a work document generation unit that generates a work document based on the
data related to the work acquired by the data acquisition unit;
27. The work support system according to claim 26, wherein the work document
generation unit generates the work document, when the determination unit
determines that the work is completed based on the response information detected by
the information detection unit.
28. The work support system according to claim 26 or claim 27, wherein the data
acquisition unit acquires information on the result of the work success 5 / failure by the
determination unit and information on work time required for the work, and the work
document generation unit generates the work document inclusive of the
determination result and the information on the work time.
29. The work support system according to any one of claims 16 to 28, wherein the
determination device changes a scope of the work to be determined by the
determination unit in accordance with a skill level of the worker.
30. The work support system according to any one of claims 16 to 29, wherein the
information acquisition device acquires voice information during the work, and the
information output device outputs information to be outputted by voice.
31. The work support system according to any one of claims 16 to 29, wherein the
information acquisition device acquires information on a screen operation of the
worker, and the information output device displays information to be outputted on a
screen.
32. The work support system according to any one of claims 16 to 29, wherein the
information acquisition device acquires the voice information during the work and
the information on the screen operation of the worker, and the information output
device outputs information to be outputted by voice and displays the information on
the screen.