Title of invention: Surgical support system, information processing device, and program
Technical field
[0001]
　The present disclosure relates to surgical support systems, information processing devices, and programs.
Background technology
[0002]
　In the operating room, various cameras such as endoscopic cameras, surgical field cameras, and surgical field cameras are used, and the surgical images obtained by imaging with these cameras are displayed and recorded during the surgery. Sometimes. The recorded surgical images are used, for example, for verification and confirmation of postoperative procedures for the purpose of improving the procedures, and are used when doctors give presentations at academic conferences and lectures.
[0003]
　Further, Patent Document 1 below also includes a technique for recording a surgical image by automatically adding metadata at the time of imaging during the operation in order to improve the efficiency when the recorded surgical image is edited later. It has been disclosed.
Prior art literature
Patent documents
[0004]
Patent Document 1: Japanese Unexamined Patent Publication No. 2016-42882
Outline of the invention
Problems to be solved by the invention
[0005]
　However, such surgical images cannot be said to be sufficiently effectively utilized at present, and further utilization of surgical images has been desired.
Means to solve problems
[0006]
　According to the present disclosure, a storage unit that stores a judgment device obtained by learning a surgical image group using label information indicating a dangerous state during surgery as teacher data, and a storage unit that stores the judgment device obtained by learning the surgery image group, and the surgery image as input, said the judgment device. A surgical support system is provided that includes a predictive unit that predicts the occurrence of a dangerous condition.
[0007]
　Further, according to the present disclosure, the label information indicating the dangerous state during the operation is used as the teacher data, and the storage unit for storing the determination device obtained by learning the operation image group and the operation image are input. An information processing device including a prediction unit that predicts the occurrence of a dangerous state using a determination device is provided.
[0008]
　Further, according to the present disclosure, a function of learning a surgical image group and storing a judgment device obtained by using label information indicating a dangerous state during surgery as teacher data and a surgical image are input to a computer. , A program for realizing a function of predicting the occurrence of a dangerous state by using the determination device is provided.
Effect of the invention
[0009]
　As described above, according to the present disclosure, it is possible to predict the occurrence of a dangerous state by utilizing surgical images.
[0010]
　It should be noted that the above effects are not necessarily limited, and together with or in place of the above effects, any of the effects shown herein, or any other effect that can be grasped from this specification. May be played.
A brief description of the drawing
[0011]
FIG. 1 is a block diagram showing a schematic configuration of a surgical support system 1000 according to an embodiment of the present disclosure.
FIG. 2 shows an example of an image containing an alert that is displayed when an outbreak of a hazard is predicted.
[Fig. 3] Fig. 3 is a diagram showing a display example of an operation screen.
FIG. 4 is a block diagram showing a configuration example of a server 10 according to the same embodiment.
[Fig. 5] Fig. 5 is a flowchart showing an example of an operation related to learning.
[Fig. 6] Fig. 6 is a flowchart showing an example of an operation related to prediction of a dangerous state.
[Fig. 7] Fig. 7 is an explanatory diagram showing an example of a hardware configuration.
Mode for carrying out the invention
[0012]
　Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.
[0013]
　Further, in the present specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of the plurality of components having substantially the same functional configuration, only the same reference numerals are given.
[0014]
　The explanations will be given in the following order.
　<< 1. Background >>
　<< 2. Configuration >>
　　<2-1. Overall configuration of surgery support system>
　　<2-2. Server configuration>
　<< 3. Operation >>
　<< 4. Modification example >>
　　<4-1. Modification 1>
　　<4-2. Modification 2>
　　<4-3. Modification 3>
　<< 5. Hardware configuration example >>
　<< 6. Conclusion >>
[0015]
　<< 1. Background >>
　Before explaining one embodiment of the present disclosure, first, the background leading to the creation of one embodiment of the present disclosure will be described. In the operating room, various cameras such as an endoscopic camera, a surgical field camera, and a surgical field camera are used. Surgical images obtained during surgery by imaging with such a camera may be displayed and recorded during surgery. In the present specification, when it is not necessary to distinguish between a still image and a moving image, it may be simply referred to as an image. Further, in the present specification, the expression "surgical image" is used as an expression including a still image obtained during surgery or a moving image obtained during surgery.
[0016]
　At present, such surgical images cannot be said to be fully utilized. Therefore, in the present disclosure, in order to make effective use of surgical images, a plurality of surgical images (also referred to as surgical image groups) are learned, and a dangerous state that may occur during surgery is generated using a judgment device obtained by such learning. We propose a mechanism to automatically predict. Dangerous conditions that can occur during surgery may include, for example, accidents and events that cause accidents. In addition, in this specification, various symptoms caused by bleeding, perforation, medical accident, a state where a large change in vital information occurs before and after a medical action (treatment), a state where a change in surgical procedure is required, and other medical actions. The inconvenient symptoms that accompany the disease are collectively called accidents.
[0017]
　By the way, in the machine learning technique for performing such learning, in order to obtain a more accurate determination device, it is desirable to perform learning using appropriate teacher data. As such teacher data, for example, it is conceivable to use label information obtained by labeling each frame in a still image or each frame in a moving image. It is desirable that such label information is prepared according to the desired performance of the determination device.
[0018]
　However, manual labeling has a high human cost, and it is very difficult especially when the surgical images included in the surgical image group are moving images or when there are a large number of surgical images. In some cases, surgical images are classified and meta information is added manually or by an automatic method, but it is appropriate that it can be used as teacher data for obtaining a judgment device as described above by learning. Label information was not prepared.
[0019]
　Therefore, the present embodiment was created with the above circumstances as the first point of view. The surgery support system according to the present embodiment automatically generates label information indicating a dangerous state during surgery, which can be used as teacher data. Further, the surgical support system according to the present embodiment obtains a determination device by learning the surgical image group by using the generated label information as teacher data. Then, the operation support system according to the present embodiment can predict the occurrence of a dangerous state in real time from the operation image input during the operation by using the determination device obtained in this way. Hereinafter, in the present embodiment, configurations and operations for realizing the above effects will be sequentially described.
[0020]
　<< 2. Configuration >>
　　<2-1. Configuration of the entire surgery support system>
　FIG. 1 is a block diagram showing a schematic configuration of the surgery support system 1000 according to one embodiment of the present disclosure. The surgery support system 1000 according to the present embodiment includes a communication network 5, a server 10, and a surgical device that exists in operating rooms 20A to 20C and can be used during surgery. In the present specification, devices that can be used during surgery are collectively referred to as surgical devices, and not only devices for medical use but also devices (for example, general-purpose) that are not limited to medical use are also called surgical devices.
[0021]
　The communication network 5 is a wired or wireless transmission path for information transmitted from a device connected to the communication network 5. For example, the communication network 5 may include a public network such as the Internet, a telephone line network, or a satellite communication network, and various LANs (Local Area Network) including Ethernet (registered trademark), WAN (Wide Area Network), and the like. .. Further, the communication network 5 may include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).
[0022]
　The server 10 is connected to each surgical device existing in the operating rooms 20A to 20C via the communication network 5. The server 10 may exist in the operating rooms 20A to 20C, in the hospital where the operating rooms 20A to 20C exist, or may exist outside the hospital.
[0023]
　The server 10 receives surgical images (still images or moving images) from surgical devices existing in operating rooms 20A to 20C, and stores (records) them. Further, the server 10 obtains a determination device by learning the accumulated surgical image group. Further, the server 10 uses the surgical image received in real time as an input and predicts the occurrence of a dangerous state by using a determination device obtained in advance. Further, when the occurrence of a dangerous state is predicted, the server 10 outputs an alert to a surgical device existing in the operating room from which the surgical image is acquired and functioning as an output unit among the operating rooms 20A to 20C. .. The detailed configuration of the server 10 will be described later with reference to FIG.
[0024]
　The surgical instruments present in the operating rooms 20A to 20C are, for example, a camera 201, a vital monitor 202, an encoder 203, a monitor 204, a speaker 205, a decoder 206, a lighting device 207, and an electric knife (energy device), as shown in FIG. Includes 208 and the like. Among these surgical devices, the monitor 204, the speaker 205, the lighting device 207, and the electric knife 208 output an alert warning that an occurrence of a dangerous state is predicted by using an image display, sound, light, or vibration. Can function as an output unit. This makes it possible to issue visual, auditory or tactile alerts to healthcare professionals such as surgeons and staff members in the operating room. The surgical device shown in FIG. 1 is an example, and other surgical devices may be included in the surgical support system 1000. For example, surgical equipment such as a projector (an example of an output unit), a bipolar, and a surgical robot may be included in the surgical support system 1000. Further, although only the surgical equipment existing in the operating room 20A is shown in FIG. 1, the operating equipment is also present in the operating room 20B and the operating room 20C.
[0025]
　The camera 201 outputs the surgical image obtained by imaging to the encoder 203. The camera 201 may include, for example, an endoscopic camera, a surgical field camera, and a surgical field camera. The endoscopic camera is inserted into the body cavity of the patient, for example, and acquires an image of the surgical site. In addition, the surgical field camera acquires an image of the surgical site from the outside of the patient. In addition, the operating room camera is installed on the ceiling of the operating room, for example, and acquires an image of the entire operating room. The camera 201 may include another camera, for example, an electron microscope or the like.
[0026]
　The vital monitor 202 uses an image (an example of a surgical image) that visualizes patient vital information (for example, heart rate, respiratory rate (number), blood pressure, body temperature) measured during surgery by a vital information measuring device (not shown). , Output to encoder 203.
[0027]
　The encoder 203 (live encoder) transmits the surgical image output during the surgery from the camera 201 and the vital monitor 202 to the server 10 in real time.
[0028]
　The monitor 204 functions as an output unit, and the decoder 206 displays (outputs) an image received from the server 10. The image displayed by the monitor 204 may include a surgical image acquired by the camera 201 in the same operating room. Further, when the occurrence of a dangerous state is predicted by the server 10, the image displayed by the monitor 204 may include an alert, for example, an image in which the alert is superimposed on the surgical image.
[0029]
　FIG. 2 shows an example of an image including an alert displayed on the monitor 204 when the server 10 predicts the occurrence of a dangerous state. The image V10 shown in FIG. 2 is an image displayed on the monitor 204, and includes an alert A10 that warns of the predicted occurrence of bleeding (an example of a dangerous state) and the position where the occurrence is predicted. For example, it may be possible to avoid the occurrence of bleeding by having the doctor confirm the alert A10 shown in FIG. 2 and perform the operation after recognizing the portion where bleeding is likely to occur.
[0030]
　In addition, the monitor 204 may display an operation screen for giving an instruction for displaying an image or giving an instruction for operating a surgical device. In such a case, a touch panel may be provided on the display surface of the monitor 204 and can be operated by the user.
[0031]
　FIG. 3 is a diagram showing an example of an operation screen displayed on the monitor 204. FIG. 3 shows, as an example, an operation screen displayed on the monitor 204 when the operating room 20A is provided with at least two monitors 204 as output destination devices. Referring to FIG. 3, the operation screen 5193 is provided with a source selection area 5195, a preview area 5197, and a control area 5201.
[0032]
　In the source selection area 5195, the source device provided in the surgery support system 1000 and the thumbnail screen showing the display information possessed by the source device are linked and displayed. The user can select the display information to be displayed on the monitor 204 from any of the source devices displayed in the source selection area 5195.
[0033]
　In the preview area 5197, a preview of the screen displayed on the two monitors 204 (Monitor1 and Monitor2), which are the output destination devices, is displayed. In the illustrated example, four images are displayed in PinP on one monitor 204. The four images correspond to the display information transmitted from the source device selected in the source selection area 5195. Of the four images, one is displayed relatively large as the main image and the remaining three are displayed relatively small as the sub-image. The user can switch the main image and the sub image by appropriately selecting the area in which the four images are displayed. Further, a status display area 5199 is provided below the area where the four images are displayed, and the status related to the surgery (for example, the elapsed time of the surgery, the physical information of the patient, etc.) is appropriately displayed in the area. obtain.
[0034]
　The control area 5201 includes a source operation area 5203 in which GUI (Graphical User Interface) components for operating the source device are displayed, and GUI components for performing operations on the output destination device. Is provided with an output destination operation area 5205 and. In the illustrated example, the source operation area 5203 is provided with GUI components for performing various operations (pan, tilt, zoom) on the camera in the source device having an imaging function. The user can operate the operation of the camera in the source device by appropriately selecting these GUI components. Although not shown, when the source device selected in the source selection area 5195 is a recorder (that is, in the preview area 5197, an image recorded in the past is displayed on the recorder. In the case), the source operation area 5203 may be provided with a GUI component for performing operations such as playing, stopping, rewinding, and fast-forwarding the image.
[0035]
　Further, in the output destination operation area 5205, GUI parts for performing various operations (swap, flip, color adjustment, contrast adjustment, switching between 2D display and 3D display) for the display on the monitor 204 which is the output destination device are provided. It is provided. The user can operate the display on the monitor 204 by appropriately selecting these GUI components.
[0036]
　The operation screen displayed on the monitor 204 is not limited to the illustrated example, and the user may be able to input operations to each device provided in the surgery support system 1000 via the monitor 204.
[0037]
　The speaker 205 functions as an output unit, and the decoder 206 outputs the sound received from the server 10. For example, when the server 10 predicts the occurrence of a dangerous state, the speaker 205 outputs a voice (an example of an alert) warning that the occurrence of a dangerous state is predicted.
[0038]
　The decoder 206 receives images and sounds from the server 10 and outputs them to the monitor 204 and the speaker 205, respectively.
[0039]
　The lighting device 207 is a lighting device used in an operating room such as a surgical light. The lighting device 207 according to the present embodiment is connected to the server 10 via the communication network 5 as shown in FIG. Further, the lighting device 207 according to the present embodiment functions as an output unit, and outputs an alert warning that the occurrence of a dangerous state is predicted according to the control signal received from the server 10. For example, the lighting device 207 may output an alert by outputting light of a predetermined color or by making the lighting pattern different from the usual one.
[0040]
　The electric knife 208 is a surgical tool used for surgery, and it is possible to stop bleeding at the same time as dissection by passing a high frequency current through the human body, for example. Further, the electric knife 208 according to the present embodiment is connected to the server 10 via the communication network 5 as shown in FIG. Then, the electric knife 208 according to the present embodiment functions as an output unit, and outputs an alert warning that the occurrence of a dangerous state is predicted according to the control signal received from the server 10. For example, the electric knife 208 may output an alert by vibrating the handle portion.
[0041]
　As described above, when the occurrence of a dangerous state is predicted by the server 10, an alert warning that the occurrence of a dangerous state is predicted is output. As a result, the occurrence of a dangerous state can be avoided, for example, by the surgeon canceling an action leading to a medical accident or performing an operation after recognizing a portion likely to bleed.
[0042]
　　<2-2. Server configuration> The configuration
　of the surgery support system 1000 according to this embodiment has been described above. Subsequently, a more detailed configuration of the server 10 shown in FIG. 1 will be described with reference to FIG. FIG. 4 is a block diagram showing a configuration example of the server 10 according to the present embodiment. As shown in FIG. 4, the server 10 is an information processing device including a control unit 110, a communication unit 130, and a storage unit 150.
[0043]
　The control unit 110 functions as an arithmetic processing unit and a control device, and controls the overall operation in the server 10 according to various programs. Further, as shown in FIG. 4, the control unit 110 functions as a communication control unit 111, an information acquisition unit 112, a classification unit 113, a teacher data generation unit 114, a learning unit 115, a prediction unit 116, and an alert control unit 117.
[0044]
　The communication control unit 111 controls communication with other devices by the communication unit 130. For example, the communication control unit 111 controls the communication unit 130 to receive a surgical image from the encoder 203 shown in FIG. Further, the communication control unit 111 controls the communication unit 130 to receive the surgical attribute information in accordance with the instruction of the information acquisition unit 112 described later. Further, the communication control unit 111 controls the communication unit 130 in accordance with the instruction of the alert control unit 117 described later, and the operating room 20A shown in FIG. It is transmitted to the surgical equipment existing at ~ 20C.
[0045]
　The information acquisition unit 112 acquires (collects) surgical attribute information (meta information) corresponding to the surgical image received from the encoder 203. The information acquisition unit 112 outputs an instruction for acquiring the surgical attribute information to the communication control unit 111, and the communication control unit 111 controls the communication unit 130 according to the instruction to receive the surgical attribute information in the communication control unit 111. Can be obtained from.
[0046]
　The information acquisition unit 112 acquires surgical attribute information from not only the surgical equipment included in the operating rooms 20A to 20C shown in FIG. 1 but also databases and systems inside and outside the hospital (not shown), and includes each surgical image. Is associated with. Hereinafter, an example of the surgical attribute information acquired by the information acquisition unit 112 will be described.
[0047]
　The surgical attribute information may include patient information such as age, gender, race, and condition of the patient. Patient information can be obtained from, for example, HIS (Hospital Information System), EMR (Electronic Medical Record, also referred to as electronic medical record), or the like.
[0048]
　Further, the surgical attribute information may include doctor information such as a doctor's identification information, a doctor's name, a doctor's medical office, and a doctor's university of origin. Doctor information can be obtained from, for example, RIS (Radiology Information System, also referred to as ordering system), surgery planning system, anesthesia machine system, doctor information site on the Internet, or the like.
[0049]
　In addition, the surgical attribute information includes the surgical procedure name (for example, esophagectomy, total gastrectomy, small intestinal malignant tumor surgery, partial liver resection, pancreatic tail resection, lobectomy, TAPVR surgery, craniotomy hematoma removal, etc.) , Surgical information regarding the surgical procedure such as time allocation of the procedure procedure may be included. The surgical procedure information can be obtained from, for example, RIS, a surgical procedure database in a hospital, a surgical procedure information site on the Internet, or the like.
[0050]
　Further, the surgical attribute information may include surgical device information indicating the state (for example, usage status, status, etc.) of the surgical device such as the electric knife 208 or the surgical robot. For example, in the case of a surgical robot, the state of the joints of the arms constituting the robot, the posture of the arms, and the like may be included in the surgical equipment information. In the case of an electric knife, the status of ON / OFF operation may be included in the surgical equipment information. Surgical equipment information can be obtained from each surgical equipment present in operating rooms 20A-20C.
[0051]
　The surgical attribute information acquired by the information acquisition unit 112 is associated with the surgical image and output to the classification unit 113 and the teacher data generation unit 114.
[0052]
　The classification unit 113 classifies the surgical images based on the surgical attribute information. The classification unit 113 may classify surgical images for each surgical procedure based on, for example, surgical procedure information included in surgical attribute information. However, the method of classifying surgical images by the classification unit 113 is not limited to such an example, and more diverse classification can be performed based on various information included in the surgical attribute information.
[0053]
　The classification unit 113 outputs the classified surgical images and information regarding the classification of the surgical images to the teacher data generation unit 114, which will be described later. With such a configuration, the teacher data generation unit 114 can more efficiently generate label information as teacher data for each surgical image classified by the classification unit 113.
[0054]
　Further, the classification unit 113 provides information regarding the classification of the plurality of surgical image groups obtained by classifying the plurality of surgical images (still images or moving images) and the plurality of surgical image groups, which will be described later. Output to. With such a configuration, the learning unit 115 can perform learning for each surgical image group classified by the classification unit 113, and the performance of the determination device obtained by improving the learning efficiency is improved. The plurality of surgical image groups classified by the classification unit 113 may be stored in the storage unit 150.
[0055]
　In addition, the classification unit 113 outputs the classified surgical images and information regarding the classification of the surgical images to the prediction unit 116, which will be described later. With such a configuration, the prediction unit 116 can select a determination device based on the classification of surgical images to perform prediction, and the prediction accuracy is improved.
[0056]
　The teacher data generation unit 114 generates label information indicating a dangerous state during surgery based on the surgical images classified by the classification unit 113 and the surgical attribute information acquired by the information acquisition unit 112. The label information generated by the teacher data generation unit 114 is used as teacher data by the learning unit 115 described later.
[0057]
　For example, the teacher data generation unit 114 may generate label information by performing bleeding detection for detecting bleeding, rework detection for detecting rework due to a medical accident, hemostasis detection for detecting the implementation of hemostasis, and the like. For example, when bleeding is detected, label information indicating bleeding (an example of a dangerous state) may be generated and added to the frame in which bleeding is detected. In addition, when rework is detected, label information indicating a medical accident (an example of a dangerous state) is generated, and for a frame in which rework is detected or a frame corresponding to a medical accident that causes rework. It may be added. In addition, when hemostasis is detected, label information indicating bleeding (an example of a dangerous state) is generated, and a frame in which hemostasis is detected (or, if detectable, a frame during bleeding). ) May be added.
[0058]
　For example, the teacher data generation unit 114 may detect bleeding by detecting a feature amount such as red color or liquid from a surgical image by image recognition.
[0059]
　Further, the teacher data generation unit 114 may perform rework detection by detecting, for example, a scene change. Scene change can be detected by a method of detecting a change in a pattern from a surgical image, a method of detecting insertion / removal of an endoscope from a surgical image, or a method of detecting a change of a surgical instrument recognized in the surgical image.
[0060]
　Further, the surgical equipment information acquired by the information acquisition unit 112 may be used for detecting a scene change. For example, as the surgical equipment information, the usage status and status change of the electric knife 208 and the bipolar, the status of the surgical robot, the change of the forceps in use, and the like can be used.
[0061]
　Further, the surgical technique information acquired by the information acquisition unit 112 may be used for detecting a scene change. For example, as the surgical procedure information, information on the time allocation of the procedure procedure for each surgical procedure can be used. Since the information may differ depending on whether the patient is an adult or a child, the degree of obesity, etc., the time allocation of the surgical procedure may be properly used by using the patient information acquired by the information acquisition unit 112. It may be corrected.
[0062]
　For example, the teacher data generation unit 114 determines that a rework has occurred when the difference between the time allocation of the procedure procedure included in the surgical procedure information and the time allocation of the procedure procedure estimated from the surgical device information is large. You may. In addition, the teacher data generation unit 114 reworkes the frame in which a difference begins to occur between the time allocation of the procedure procedure included in the surgical procedure information and the time allocation of the procedure procedure estimated from the surgical device information. It may be detected as a frame corresponding to the medical accident that causes it.
[0063]
　In addition, the teacher data generation unit 114 detects that hemostasis is performed by detecting the characteristics of a surgical instrument for hemostasis (for example, a needle and thread for ligation, an electric knife 208, etc.) that has been learned in advance. You may. For example, when the electrosurgical knife 208 is in the coagulation mode, it is possible to detect that hemostasis is being performed.
[0064]
　The method by which the teacher data generation unit 114 generates label information indicating a dangerous state during surgery is not limited to the above-mentioned example. For example, the teacher data generation unit 114 determines that a medical accident has occurred when it is detected that the surgical image contains (collects) more doctors than usual, and indicates a medical accident. Label information may be generated.
[0065]
　The learning unit 115 uses the label information generated by the teacher data generation unit 114 as teacher data to learn the surgical image group classified by the classification unit 113, and generates (obtains) a determination device (learned model). .. The learning method by the learning unit 115 is not particularly limited, but for example, learning data in which label information and a surgical image group are linked is prepared, and the learning data is input to a calculation model based on a multi-layer neural network for learning. You may. Further, for example, a method based on DNN (Deep Neural Network) such as CNN (Convolutional Neural Network), 3D-CNN, RNN (Recurrent Neural Network) may be used.
[0066]
　The determination device generated by the learning unit 115 is used by the prediction unit 116, which will be described later, in order to predict the occurrence of a dangerous state. Therefore, the learning unit 115 learns the surgical image of the frame before the frame to which the label information indicating the dangerous state is added in the surgical image group as the surgical image leading to the occurrence of the dangerous state. With such a configuration, the determination device generated by the learning unit 115 can be used to predict the occurrence of the dangerous state before the occurrence of the dangerous state.
[0067]
　The learning unit 115 may generate a plurality of determination devices. As described above, since the classification unit 113 can classify a plurality of surgical images into a plurality of surgical image groups, the learning unit 115 may generate a determination device for each of the classified surgical image groups. That is, the same number of determination devices as the number of surgical image groups classified by the classification unit 113 may be generated.
[0068]
　The plurality of determination devices generated by the learning unit 115 are stored in the storage unit 150 in association with information regarding the classification of the surgical image group used for generating each determination device.
[0069]
　The prediction unit 116 receives the surgical image (still image or moving image) classified by the classification unit 113 as input, and predicts the occurrence of a dangerous state by using the determination device stored in the storage unit 150.
[0070]
　As described above, the storage unit 150 stores a plurality of determination devices. Therefore, the prediction unit 116 may select a determination device to be used for prediction from a plurality of determination devices stored in the storage unit 150 based on the classification of the surgical image by the classification unit 113.
[0071]
　With such a configuration, a determination device more suitable for the current surgery can be selected, and the accuracy of predicting the dangerous state can be improved. It should be noted that the selection of such a determination device may be performed for each frame, or may be performed only at the start of the operation and the same determination device may be used during the operation.
[0072]
　In addition, when the occurrence of a dangerous state is predicted, the prediction unit 116 includes the type of the dangerous state (bleeding, perforation, medical accident, etc.), the degree of danger of the dangerous state, the position where the occurrence of the dangerous state is predicted, and the like. Generates information related to prediction (hereinafter referred to as prediction information). In addition, the prediction unit 116 provides the generated prediction information to the alert control unit 117 when the occurrence of a dangerous state is predicted.
[0073]
　When the prediction unit 116 predicts the occurrence of a dangerous state, the alert control unit 117 outputs an alert based on the prediction information provided by the prediction unit 116. As mentioned above, the alert resides in the operating room where a dangerous condition is predicted to occur and functions as an output unit (in the example shown in FIG. 1, monitor 204, speaker 205, lighting equipment 207, and electricity). It is output by the female 20). The alert control unit 117 may output an alert by generating an image, a sound, or a control signal for these output units to output an alert and providing the alert control unit 111 to the communication control unit 111.
[0074]
　The alert control unit 117 may output different alerts according to the prediction information. Further, the alert control unit 117 may output an alert to an output unit (surgical device) according to the prediction information.
[0075]
　For example, when the prediction information includes information on the type of dangerous state, the alert control unit 117 may output an alert including the information on the type of dangerous state. For example, the alert control unit 117 may generate an image in which an alert indicating the type of danger state is combined with the surgical image and display it on the monitor 204. Further, the alert control unit 117 may output a voice including information on the type of the dangerous state from the speaker 205. Further, the alert control unit 117 may change the color of the light output to the lighting device 207 according to the type of the dangerous state. Further, the alert control unit 117 may make the vibration pattern of the electric knife 208 different depending on the type of dangerous state.
[0076]
　With such a configuration, the surgeon can grasp the type of the dangerous state in which the occurrence is predicted, and it becomes easier to avoid the dangerous state.
[0077]
　Further, when the prediction information includes information on the degree of danger of the dangerous state, the alert control unit 117 may output an alert according to the degree of danger of the dangerous state. For example, the alert control unit 117 may generate an image in which a more conspicuous alert is combined with the surgical image and display it on the monitor 204 when the risk level is high as compared with the case where the risk level is low. Further, the alert control unit 117 may change the display size and color of the alert. Further, the alert control unit 117 may make the volume of the alert output from the speaker 205 louder when the risk level is high than when the risk level is low. Further, the alert control unit 117 may increase the light intensity output to the lighting device 207 when the risk of the dangerous state is high, as compared with the case where the risk is low. Further, the alert control unit 117 may increase the vibration intensity of the electric knife 208 when the risk of the dangerous state is high as compared with the case where the risk is low.
[0078]
　With such a configuration, it is possible to call the surgeon more strongly, for example, when the risk of the predicted dangerous state is higher.
[0079]
　Further, when the prediction information includes information on the position where the occurrence of the dangerous state is predicted, the alert control unit 117 generates an image including an alert indicating the position where the occurrence of the dangerous state is predicted, and monitors 204. It may be displayed in. Further, the alert control unit 117 may control a projector (not shown) so that the alert is projected to a position where the occurrence of a dangerous state is predicted.
[0080]
　With such a configuration, the surgeon can grasp the position where the occurrence of the dangerous state is predicted, and it becomes easier to avoid the dangerous state.
[0081]
　Although an example of an alert output by the alert control unit 117 has been described above, the present technology is not limited to such an example, and an alert other than the above may be output.
[0082]
　The communication unit 130 is a communication module for transmitting / receiving data to / from another device by wire / wireless according to the control of the communication control unit 111. The communication unit 130 is an external device using, for example, a wired LAN (Local Area Network), a wireless LAN, Wi-Fi (Wireless Fidelity, registered trademark), infrared communication, Bluetooth (registered trademark), short-range / non-contact communication, or the like. Wirelessly communicate with or via a network access point.
[0083]
　The storage unit 150 stores programs and parameters for each configuration of the server 10 to function. For example, the storage unit 150 stores a plurality of surgical image groups classified by the classification unit 113, and information regarding the classification of the plurality of surgical image groups. Further, the storage unit 150 stores a plurality of determination devices generated by the learning unit 115. As described above, since the determination device is generated for each classified surgical image group, the storage unit 150 is associated with the determination device and the information regarding the classification of the surgical image group of the determination device. Will be remembered.
[0084]
　<< 3. Operation >> The
　configuration of the surgery support system 1000 and the server 10 according to the present embodiment has been described above. Subsequently, an operation example of the operation support system 1000 according to the present embodiment will be described. In the following, the operation related to learning will be described with reference to FIG. 5, and then the operation related to the prediction of a dangerous state performed during surgery will be described with reference to FIG.
[0085]
　FIG. 5 is a flowchart showing an example of the operation of the surgery support system 1000 for learning. The process shown in FIG. 5 may be performed in advance of, for example, the process related to the prediction of the dangerous state described later with reference to FIG.
[0086]
　First, the information acquisition unit 112 acquires surgical attribute information (S101). Further, the communication unit 130 receives (acquires) the surgical image from the encoder 203 (S103). The processes of step S101 and step S103 may be performed in parallel.
[0087]
　Subsequently, the classification unit 113 classifies the surgical images acquired in step S103 based on the surgical attribute information acquired in step S101 (S105). Subsequently, the teacher data generation unit 114 generates label information to be teacher data based on the surgical attribute information acquired in step S101 and the surgical image classified in step S105 (S107).
[0088]
　Subsequently, the learning unit 115 uses the label information generated in step S107 as teacher data, performs learning for each surgical image group classified in step S105, generates a determination device (S109), and stores the storage unit 150. Is stored in (S110).
[0089]
　The operation related to learning has been described above. Next, the operation related to the prediction of the dangerous state performed during the operation will be described. FIG. 6 is a flowchart showing an example of the operation of the operation support system 1000 for predicting a dangerous state. The process shown in FIG. 6 is performed after the process described with reference to FIG. 5, for example, is performed and the determination device is stored in the storage unit 150.
[0090]
　First, the information acquisition unit 112 acquires surgical attribute information (S201). Further, the communication unit 130 receives (acquires) a surgical image from the encoder 203 (S203). The processes of step S201 and step S203 may be performed in parallel.
[0091]
　Subsequently, the classification unit 113 classifies the surgical images acquired in step S203 based on the surgical attribute information acquired in step S201 (S205). Subsequently, the prediction unit 116 selects a determination device to be used for prediction from a plurality of determination devices stored in the storage unit 150 based on the classification performed in step S207 (S207).
[0092]
　Further, the prediction unit 116 uses the determination device selected in step S207 to predict the occurrence of a dangerous state by inputting the surgical image acquired in step S203 (S209).
[0093]
　When the occurrence of the dangerous state is not predicted as a result of the prediction in step S209 (NO in S211), the communication unit 130 receives (acquires) the surgical image from the encoder 203 again (S213). Then, the process returns to step S209, and the occurrence of the dangerous state is predicted by inputting the surgical image acquired in step S213.
[0094]
　On the other hand, when the occurrence of a dangerous state is predicted as a result of the prediction in step S209 (YES in S211), the output unit such as the monitor 204, the speaker 205, the lighting device 207, or the electric knife 208 is controlled by the alert control unit 117. Outputs an alert (S211).
[0095]
　<< 4. Modifications >> The
　configuration examples and operation examples according to the present embodiment have been described above. Hereinafter, a modified example of the present embodiment will be described. The modifications described below may be applied to the present embodiment alone or in combination with the present embodiment. Further, the present modification may be applied in place of the configuration described in the present embodiment, or may be additionally applied to the configuration described in the present embodiment.
　　<4-1. Modification 1>
　The configuration of FIGS. 1 and 4 described in the above embodiment is an example, and the present technology is not limited to such an example. For example, some or all of the functions of the server 10 described in the above embodiment may be provided in another device. For example, the functions related to learning such as the teacher data generation unit 114 and the learning unit 115 and the functions related to the prediction of the occurrence of a dangerous state such as the prediction unit 116 and the alert control unit 117 are provided in different devices. May be good. Then, the determination device obtained by learning may be provided from the device that performs learning to the device that performs prediction.
[0096]
　Further, the above-mentioned function related to the prediction is the camera 201, the vital monitor 202, the encoder 203, the monitor 204, the speaker 205, the decoder 206, the lighting device 207, or the electric knife 208 existing in the operating rooms 20A to 20C shown in FIG. It may be provided in surgical equipment such as.
[0097]
　Further, the generation of the image displayed on the monitor 204 as an alert does not have to be performed by the server 10. For example, the monitor 204 directly receives and displays the surgical image acquired by the camera 201 in the same surgery, and further synthesizes the alert and the surgical image when a control signal or the like related to the alert is received from the server 10. The image may be generated and displayed.
[0098]
　　<4-2. Modification 2> Further
　, in the above embodiment, an example in which a surgical image group composed of surgical images provided by a surgical device existing in an operating room is used for learning has been described, but the present technique is not limited to such an example. For example, a group of surgical images recorded in an external database or the like and corresponding surgical attribute information may be provided to the server 10 and used for learning.
[0099]
　　<4-3. Modification 3> Further
　, in the above embodiment, an example in which a determination device is generated in advance by learning and then prediction is performed has been described, but the present technology is not limited to such an example. For example, the surgical attribute information and the surgical image acquired in the prediction process described with reference to FIG. 6 may be used for learning, and the determination device may be updated at any time.
[0100]
　<< 5. Hardware Configuration Example >>
　The embodiments of the present disclosure have been described above. Finally, with reference to FIG. 7, the hardware configuration of the information processing apparatus according to the embodiment of the present disclosure will be described. FIG. 7 is a block diagram showing an example of the hardware configuration of the server 10 according to the embodiment of the present disclosure. The information processing by the server 10 according to the embodiment of the present disclosure is realized by the cooperation between the software and the hardware described below.
[0101]
　As shown in FIG. 7, the server 10 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, and a host bus 904a. Further, the server 10 includes a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 911, and a communication device 913. The server 10 may have a processing circuit such as a DSP or an ASIC in place of or in combination with the CPU 901.
[0102]
　The CPU 901 functions as an arithmetic processing unit and a control device, and controls the overall operation in the server 10 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs, calculation parameters, and the like used by the CPU 901. The RAM 903 temporarily stores a program used in the execution of the CPU 901, parameters that are appropriately changed in the execution, and the like. The CPU 901 may form, for example, the control unit 110.
[0103]
　The CPU 901, ROM 902, and RAM 903 are connected to each other by a host bus 904a including a CPU bus and the like. The host bus 904a is connected to an external bus 904b such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 904. It is not always necessary to separately configure the host bus 904a, the bridge 904, and the external bus 904b, and these functions may be implemented in one bus.
[0104]
　The input device 906 is realized by a device in which information is input by a user, such as a mouse, a keyboard, a touch panel, a button, a microphone, a switch, and a lever. Further, the input device 906 may be, for example, a remote control device using infrared rays or other radio waves, or an externally connected device such as a mobile phone or a PDA that supports the operation of the server 10. Further, the input device 906 may include, for example, an input control circuit that generates an input signal based on the information input by the user using the above input means and outputs the input signal to the CPU 901. By operating the input device 906, the user of the server 10 can input various data to the server 10 and instruct the processing operation.
[0105]
　The output device 907 is formed by a device capable of visually or audibly notifying the user of the acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices and lamps, audio output devices such as speakers and headphones, and printer devices. The output device 907 outputs, for example, the results obtained by various processes performed by the server 10. Specifically, the display device visually displays the results obtained by various processes performed by the server 10 in various formats such as texts, images, tables, and graphs. On the other hand, the audio output device converts an audio signal composed of reproduced audio data, acoustic data, etc. into an analog signal and outputs it audibly.
[0106]
　The storage device 908 is a data storage device formed as an example of the storage unit of the server 10. The storage device 908 is realized by, for example, a magnetic storage device such as an HDD, a semiconductor storage device, an optical storage device, an optical magnetic storage device, or the like. The storage device 908 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deleting device that deletes the data recorded on the storage medium, and the like. The storage device 908 stores programs executed by the CPU 901, various data, various data acquired from the outside, and the like. The storage device 908 may form, for example, a storage unit 150.
[0107]
　The drive 909 is a storage medium reader / writer, and is built in or externally attached to the server 10. The drive 909 reads out the information recorded in the removable storage medium such as the mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903. The drive 909 can also write information to the removable storage medium.
[0108]
　The connection port 911 is an interface connected to an external device, and is a connection port to an external device capable of transmitting data by, for example, USB (Universal Serial Bus).
[0109]
　The communication device 913 is, for example, a communication interface formed by a communication device or the like for connecting to the network 920. The communication device 913 is, for example, a communication card for a wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), WUSB (Wireless USB), or the like. Further, the communication device 913 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various communications, or the like. The communication device 913 can transmit and receive signals and the like to and from the Internet and other communication devices in accordance with a predetermined protocol such as TCP / IP. The communication device 913 may form, for example, the communication unit 130.
[0110]
　The network 920 is a wired or wireless transmission path for information transmitted from a device connected to the network 920. For example, the network 920 may include a public network such as the Internet, a telephone line network, a satellite communication network, various LANs (Local Area Network) including Ethernet (registered trademark), and a WAN (Wide Area Network). In addition, the network 920 may include a dedicated network such as IP-VPN (Internet Protocol-Virtual Private Network).
[0111]
　The above is an example of a hardware configuration capable of realizing the functions of the server 10 according to the embodiment of the present disclosure. Each of the above components may be realized by using a general-purpose member, or may be realized by hardware specialized for the function of each component. Therefore, it is possible to appropriately change the hardware configuration to be used according to the technical level at each time when the embodiment of the present disclosure is implemented.
[0112]
　It is possible to create a computer program for realizing each function of the server 10 according to the embodiment of the present disclosure as described above and implement it on a PC or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.
[0113]
　<< 6. Conclusion >> As
　described above, according to the embodiment of the present disclosure, it is possible to predict the occurrence of a dangerous state by utilizing the surgical image. Furthermore, when the occurrence of a dangerous condition is predicted, it is possible to output an alert before the occurrence of a dangerous condition. The occurrence of a dangerous condition can be avoided by performing an operation in.
[0114]
　As a result, the degree of invasiveness to the patient is reduced and the operation time is shortened. Furthermore, the QoL of patients will improve and the satisfaction level of patients will improve, leading to an increase in the number of customers as a hospital. In addition, it is expected that the profitability of hospitals will increase by improving the efficiency of operating room utilization. In addition, it is expected that the risk of accidents will be reduced, which will alleviate the tension of doctors, increase the satisfaction level of doctors at work, prevent turnover, and reduce labor costs.
[0115]
　Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that anyone with ordinary knowledge in the technical field of the present disclosure may come up with various modifications or modifications within the scope of the technical ideas set forth in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.
[0116]
　For example, each step in the above embodiment does not necessarily have to be processed in chronological order in the order described as a flowchart. For example, each step in the processing of the above embodiment may be processed in an order different from the order described in the flowchart, or may be processed in parallel.
[0117]
　In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.
[0118]
　The following configurations also belong to the technical scope of the present disclosure.
(1) Using the
　label information indicating the dangerous state during surgery as teacher data, a storage unit that stores the judgment device obtained by learning the surgery image group and the
　surgery image as input, using the judgment device. A
　surgical support system equipped with a prediction unit that predicts the occurrence of dangerous situations .
(2) The
　storage unit stores a plurality of the determination devices, and the
　prediction unit makes the prediction using the determination device corresponding to the surgical image among the plurality of the determination devices. The surgery support system according to (1).
(3)
　The surgical support system according to (2) above, wherein the determination device used for the prediction is selected based on the classification of the surgical images.
(4)
　The surgical support system according to (3) above, wherein the surgical images are classified based on surgical procedure information related to the surgical procedure.
(5)
　The surgical support system according to any one of (2) to (4) above, further comprising a learning unit that performs learning for each classified surgical image group and generates the determination device.
(6)
　The surgery support system according to (5) above, further comprising a teacher data generation unit that generates the label information.
(7)
　The teacher data generation unit generates the label information by performing at least one of bleeding detection for detecting bleeding, rework detection for detecting rework, and hemostasis detection for detecting the execution of hemostasis. , The surgical support system according to (6) above.
(8) The
　above (1) to (7) further include an output unit that outputs an alert warning that the occurrence of the dangerous state is predicted when the prediction unit predicts the occurrence of the dangerous state. The surgical support system according to any one of the above.
(9) The
　prediction unit generates prediction information related to the prediction when the occurrence of the danger state is predicted, and the
　output unit outputs the alert according to the prediction information. Surgical support system described in.
(10) The
　operation support system includes the plurality of output units, and the
　alert is output by the output unit according to the prediction information among the plurality of output units, according to the above (9). Surgical support system.
(11) The
　prediction information includes at least one of the type of the dangerous state predicted to occur, the degree of danger of the dangerous state, and the position where the occurrence of the dangerous state is predicted. 9) Or the surgical support system according to (10).
(12)
　The surgical support system according to any one of (1) to (11) above, wherein the dangerous state includes accidents or events that cause accidents.
(13)
　The surgical support system according to any one of (1) to (12) above, wherein the surgical image group includes a plurality of moving images, and the surgical image is a moving image.
(14) Using
　label information indicating a dangerous state during surgery as teacher data, a storage unit that stores a judgment device obtained by learning a surgical image group, and a storage unit
　that inputs a surgical image as input, using the judgment device. An
　information processing device including a prediction unit that predicts the occurrence of a dangerous state .
(15) 　A function of storing a judgment device obtained by learning a surgical image group by using label information indicating a dangerous state during surgery as teacher data in a
　computer and
a function
　of inputting a surgical image to input the judgment device.
　A program to realize the function of predicting the occurrence of dangerous situations by using it .
Description of the sign
[0119]
　10 Server
　20A to 20C Operating room
　110 Control unit
　111 Communication control unit
　112 Information acquisition unit
　113 Classification unit
　114 Teacher data generation unit
　115 Learning unit
　116 Prediction unit
　117 Alert control unit
　130 Communication unit
　150 Storage unit
　201 Camera
　202 Vital monitor
　203 Encoder
　204 Monitor
　205 Speaker
　206 Decoder
　207 Lighting equipment
　208 Electric knife
　1000 Surgical support system
The scope of the claims
[Claim 1]
　Using the label information indicating the dangerous state during surgery as teacher data, a storage unit that stores the judgment device obtained by learning the surgical image group and the
　surgical image as input, the dangerous state is used. A
　surgical support system equipped with a prediction unit that predicts the occurrence of .
[Claim 2]
　The storage unit stores a plurality of the determination devices, and the
　prediction unit makes the prediction by using the determination device corresponding to the surgical image among the plurality of the determination devices. The described surgical support system.
[Claim 3]
　The surgical support system according to claim 2, wherein the determination device used for the prediction is selected based on the classification of the surgical images.
[Claim 4]
　The surgical support system according to claim 3, wherein the surgical images are classified based on surgical procedure information related to the surgical procedure.
[Claim 5]
　The surgical support system according to claim 2, further comprising a learning unit that performs learning for each of the classified surgical image groups and generates the determination device.
[Claim 6]
　The surgery support system according to claim 5, further comprising a teacher data generation unit that generates the label information.
[Claim 7]
　The teacher data generation unit generates the label information by performing at least one of bleeding detection for detecting bleeding, rework detection for detecting rework, and hemostasis detection for detecting the execution of hemostasis. , The surgical support system according to claim 6.
[Claim 8]
　The operation according to claim 1, further comprising an alert control unit that outputs an alert warning that the occurrence of the dangerous state is predicted to the output unit when the occurrence of the dangerous state is predicted by the prediction unit. Support system.
[Claim 9]
　The
　eighth aspect of the present invention, wherein the prediction unit generates prediction information related to the prediction when the occurrence of the dangerous state is predicted, and the alert control unit outputs the alert according to the prediction information. Surgical support system.
[Claim 10]
　The surgery support system
　according to claim 9, wherein the surgery support system includes a plurality of the output units, and the alert is output by the output unit corresponding to the prediction information among the plurality of output units.
[Claim 11]
　The prediction information according to claim 9, which includes at least one of the type of the dangerous state predicted to occur, the degree of danger of the dangerous state, and the position where the occurrence of the dangerous state is predicted. Surgical support system.
[Claim 12]
　The surgical support system according to claim 1, wherein the risk condition includes a contingency or an event that causes a contingency.
[Claim 13]
　The surgical support system according to claim 1, wherein the surgical image group includes a plurality of moving images, and the surgical images are moving images.
[Claim 14]
　Using the label information indicating the dangerous state during surgery as teacher data, a storage unit that stores the judgment device obtained by learning the surgical image group and the
　surgical image as input, the dangerous state is used. An
　information processing device including a prediction unit that predicts the occurrence of .
[Claim 15]

　A function of storing a judgment device obtained by learning a group of surgical images by using label information indicating a dangerous state during surgery as teacher data in a 　computer and a function
　of inputting a surgical image and using the judgment device are used. A function to predict the occurrence of dangerous situations and
　a program to realize it.