Title of Invention : MEDICAL SYSTEM, MEDICAL APPARATUS, AND MEDICAL METHOD
Technical Field
[0001]
　The present disclosure relates to a medical system, a medical apparatus, and a medical method.
Background Art
[0002]
　Techniques relating to an annotation applied to a tumor area or the like in medical image data (including pathological image data and the like) have recently been actively developed. For example, NPL 1 below discloses a technique capable of improving the precision of diagnoses by applying accurate annotations to IHC (Immunohistochemistry) image data.
Citation List
Non Patent Literature
[0003]
nplcit 1 : Barnes M and other 13 authors, “Wholetumor section quantitative image analysis maximizes between-pathologists’ reproducibility for clinical immunohistochemistry-based biomarkers,” Laboratory Investigation, August 14th, 2017, Volume 97, pp. 1508-1515
Summary
Technical Problem
[0004]
　With the technique of NPL 1, and the like, properly applying an annotation may however be difficult. For example, in the case where a portion to be an annotation applying target is displayed on a display to be small relative to the display, the user switches the whole display to image data whose resolution is higher and thereby applies an annotation. The user however cannot see the portion to be the annotation applying target in a perspective view when the whole display is switched to the image data whose resolution is higher. Moreover, the operation of switching to the image data relating to the observation target object but being different (for example having more detailed visual information relating to the observation target object such as whose resolution is higher) may be troublesome for the user.
[0005]
　The present disclosure was therefore conceived in view of the above circumstances and provides a medical system, a medical apparatus, and a medical method that each are novel and improved and that each can more properly apply an annotation to pathological image data.
Solution to Problem
[0006]
　According to the present disclosure, medical system is provided that includes: a control part configured to receive pathological image data that is produced by imaging one observation target object and that includes first and second image data, the second image data relating to the observation target object but being different to the first image data, a display control part that causes a displaying part to display thereon the first image data and the second image data; and an annotation processing part that applies an annotation to each of the first image data and the second image data on a basis of inputting by a user.
[0007]
　Moreover, according to the present disclosure, a medical apparatus is provided that includes: a control part configured to receive pathological image data that is produced by imaging one observation target object and that includes first and second image data, the second image data relating to the observation target object but being different to the first image data, a display control part that causes a displaying part to display thereon the first image data and the second image data; and an annotation processing part that applies an annotation to each of the first image data and the second image data on a basis of inputting by a user.
[0008]
　Moreover, according to the present disclosure, a medical method executed by a computer is provided. The medical method including the steps of: receiving pathological image data that is produced by imaging one observation target object and that includes first and second image data, the second image data relating to the observation target object but being different to the first image data, causing a displaying part to display thereon the first image data and the second image data; and applying an annotation to each of the first image data and the second image data on a basis of inputting by a user.
[0009]
According to the present disclosure, a medical system is provided that includes: an imaging apparatus producing pathological image data including first and second image data, the second image data relating to the observation target object but being different to the first image data; and software used in processing for the pathological image data, wherein the software is executed by an information processing apparatus, and thereby realizes causing a displaying part to display thereon the first image data and the second image data, and applying an annotation to each of the first image data and the second image data on a basis of inputting by a user.
[0010]
　According to the present disclosure, the user can perform inputting relating to the annotation, watching both the first image data and the second image data. The present disclosure enables more proper application of the annotation to the pathological image data.
Advantageous Effects of Invention
[0011]
　As described above, according to the present disclosure, an annotation can more properly be applied to the pathological image data.
[0012]
　In addition, the above effect is not necessarily a limiting one, and any one of the effects described in the present specification or any other effects that are understandable from the present specification may be achieve together with the above effect or instead of the above effect.
Brief Description of Drawings
[0013]
[fig. 1] FIG. 1 is a diagram for explaining a specific example of an annotation 10 applied to medical image data.
[fig. 2] FIG. 2 is a block diagram depicting an example of the configuration of a medical system 100.
[fig. 3] FIG. 3 is a diagram depicting a pyramid structure for explaining pathological image data and the principle of displaying this data.
[fig. 4] FIG. 4 is a diagram for explaining the details of display control executed by a display control part 121.
[fig. 5] FIG. 5 is a diagram for explaining the details of the display control executed by the display control part 121.
[fig. 6] FIG. 6 depicts diagrams for explaining the details of the display control executed by the display control part 121.
[fig. 7] FIG. 7 is a diagram for explaining the details of the display control executed by the display control part 121.
[fig. 8] FIG. 8 is a diagram for explaining a specific example of the apparatus configuration.
[fig. 9] FIG. 9 is a diagram for explaining an input method conventionally executed by doctors when the doctors each apply an annotation 10.
[fig. 10] FIG. 10 is a diagram for explaining the input method conventionally executed by the doctors when the doctors each apply the annotation 10.
[fig. 11] FIG. 11 is a diagram for explaining a specific example of the apparatus configuration.
[fig. 12] FIG. 12 is a diagram for explaining a specific example of the apparatus configuration.
[fig. 13] FIG. 13 is a diagram depicting an example of the annotation 10 produced by an annotation processing part 122.
[fig. 14] FIG. 14 is a flowchart depicting an example of a process flow relating to the production of the annotation 10.
[fig. 15] FIG. 15 is a diagram for explaining correction of the annotation 10 executed by moving a control point 13.
[fig. 16] FIG. 16 is a diagram for explaining the correction of the annotation 10 executed by moving the control point 13.
[fig. 17] FIG. 17 is a flowchart depicting an example of a process flow relating to the correction of the annotation 10 executed by moving the control point 13.
[fig. 18] FIG. 18 is a diagram for explaining the correction of the first annotation 10 using a second annotation 14.
[fig. 19] FIG. 19 is a flowchart depicting an example of the process flow relating to the correction of the first annotation 10 using the second annotation 14.
[fig. 20] FIG. 20 is a diagram for explaining a change of the first control point 13 to be a target of connection with a second control point 15.
[fig. 21] FIG. 21 is a flowchart depicting an example of the process flow relating to the change of the first control point 13 to be the target of the connection with the second control point 15.
[fig. 22] FIG. 22 is a diagram for explaining a determination method for a correction range of the first annotation 10.
[fig. 23] FIG. 23 is a flowchart depicting an example of the process flow relating to the determination of the correction range of the first annotation 10.
[fig. 24] FIG. 24 is a block diagram depicting an example of the hardware configuration of an information processing apparatus 900 that realizes various types of function of a medical system 100.
Description of Embodiments
[0014]
　A preferred embodiment of the present disclosure will be described below in detail with reference to the accompanying drawings. In addition, in the present specification and the drawings, constituent elements having the substantially same function and configuration are given the same reference numeral and will thereby not again be described.
[0015]
　In addition, the description will be made in the following order.
　1. Overview
　2. Embodiment
　2.1. Configuration
　2.2. Details of Processing Relating to Annotation 10
　2.3. Hardware Configuration
　2.4. Variation of System Configuration
　3. Conclusion
[0016]
<1. Overview>
　The overview of the present disclosure will first be described.
[0017]
　To verify effects of a drug in the development of the drug and to give a pathological diagnosis, it is important to accurately calculate the size and the rate of a tumor area, an interstice, or the like, or the dimension of the nucleus in the tissue. Especially, associated with the digitalization of the pathological images and the boom of the cancer immunotherapy using a PD-L1 (Programmed cell Death 1-Ligand 1), and the like, it has become more important to quantify the area of a tumor area, and the distribution and the quantity of dyed cells in the tumor area, using immunostaining or the like using serial sections. Applying an annotation that indicates a tumor area or the like to medical image data is thereby actively performed by doctors. Moreover, as above, a technique of NPL 1 has also been developed according to which the precision of a diagnosis can be improved by applying an accurate annotation to IHC image data. It should be noted that applying an annotation may mean attaching an annotation to an image, embedding an annotation in an image, storing co-ordinates within the image defining an annotation, associating the annotation with a position and/or co-ordinate in the image or the like.
[0018]
　It may however be difficult to properly apply an annotation with the technique of NPL 1 or the like. For example, in the case where a portion to be an annotation applying target is displayed on a display to be small relative to the display, the user applies an annotation by switching the whole display to image data whose resolution is higher, superimposing the image data whose resolution is higher on the original image data, to be displayed, or the like. It should be noted that having image data relating to the observation target object but being different to the first image data is only one example. Other examples include providing more detailed visual information relating to the observation target object, providing different visual characteristics relating to the observation target, applying image data of observation target coloured with a different dye, or using a different imaging technique to that of the original image data. The image data that relates to the observation target object but being different to the first image data may sometimes be hereinafter referred to as an “annotation auxiliary image”. The user however cannot see the portion to be the annotation applying target in a perspective view when the whole display is switched to the image data whose, for example, resolution is high. Moreover, the operation of stitching to the image data whose resolution, for example, is high may also be troublesome for the user. Furthermore, even when the annotation auxiliary image is displayed as the superimposition on the display, in the case where the annotation is displayed only in the original image data and is not displayed in the annotation auxiliary image, the user is required to perform the inputting watching the portion displayed in the annotation auxiliary image and the portion to have the annotation actually applied thereto comparing these portions therebetween. It may therefore be difficult to apply an accurate annotation.
[0019]
　The person who discloses this invention finally created the technique according to the present disclosure in view of the above circumstances. A medical system according to the present disclosure, for pathological image data produced by imaging one observation target object and including pieces of image data having plural different resolutions, can cause a displaying part to display thereon first image data having a first resolution of the plural different resolutions and second image data having a second resolution equal to the first resolution, or equal to or higher than the first resolution, and can apply an annotation to each of the first image data and the second image data on the basis of inputting by a user. The user can thereby execute inputting that relates to the annotation, watching both the first image data and the second image data relating to the observation target object but being different to the first image data. The present disclosure therefore enables more proper application of an annotation to the pathological image data.
[0020]
　Moreover, in the medical system according to the present disclosure, control points to be plural points on the annotation are set and the annotation can more properly be produced and corrected by using the control points.
[0021]
　Moreover, a technique of applying an annotation using a machine learning technique has recently been developed while attaching meticulous annotations to a huge amount of medical image data is demanded to produce the learning data for the machine learning, and much time and many workers are therefore necessary.
[0022]
　For this point, in the medical system according to the present disclosure, inputting relating to an annotation and inputting relating to the display of the first image data or the second image data are realized each using a method different from that of each other. More specifically, in the medical system according to the present disclosure, the inputting relating to the annotation is performed by one hand of the user and the inputting relating to the display of the first image data or the second image data is performed by the other hand thereof. This input method resembles the input method conventionally performed by doctors when the doctors apply annotations and the medical system according to the present disclosure can therefore more intuitively and more efficiently realize the work by the user such as a doctor. Moreover, the medical system according to the present disclosure has a configuration for the user to more easily perform the inputting especially for annotations drawn free-hand.
[0023]
　On the basis of the above, the medical system according to the present disclosure enables more proper application of an annotation to the pathological image data and furthermore enables improvement of the quality and the efficiency of drug development and a pathological diagnosis.
[0024]
<2. Embodiment>
　The overview of the present disclosure has been described in the above. An embodiment of the present disclosure will next be described.
[0025]
　A medical system 100 according to the present embodiment is an information processing system that is used when a user applies an annotation 10 to medical image data, and includes, for example, a medical microscope (such as, for example, an optical microscope), an apparatus connected for communication to the medical microscope and the like. In addition, the case where the medical image data to which the annotation 10 is applied by the medical system 100 is pathological image data will hereinafter be described as an example while the medical image data to which the annotation 10 is applied only has to be certain image data relating to medical care and is not necessarily limited to the pathological image data. For example, the medical image data to which the annotation 10 is applied may be radiographic image data or ultrasonic image data. In other words, the medical image data to which the annotation may be applied is captured using different imaging techniques.
[0026]
　Concerning the above, a specific example of the annotation 10 applied by the medical system 100 to the medical image data will be described with reference to FIG. 1. As depicted in FIG. 1, the annotation 10 according to the present embodiment is applied to indicate an observation target object included in the pathological image data and is applied to surround a tumor area that is the observation target object. In other words, the annotation 10 according to the present embodiment is applied to indicate a closed region included in the pathological image data (such as, for example, a region having the tumor area that is the observation target object, and the like imaged therein). The annotation 10 is used for the output of the area of the region to which the annotation 10 is applied, the probability for the observation target object to be a tumor, the positivity of the tumor, various scores, and the like, is used in the pharmaceutical field for counting the number of immune cells in the tumor area and the number of cells in the tumor margin, and the like, and the selection of patients for a medicine on the basis of the number of immune cells infiltrating into the tumor, and the like are thereby performed. Moreover, it is known that the precision of a cancer genetic test is improved by determining and cutting out a tumor area (desirably, a tumor area including a specific number or more of idioblasts) and the like from a pathological section in the cancer generic test and, in this case, the annotation 10 is usable for the cutting out of the tumor area from the pathological section, the counting of the number of cells inside the tumor, and the like.
[0027]
(2.1. Configuration)
　An example of the configuration of the medical system 100 will be described with reference to FIG. 2. FIG. 2 is a block diagram depicting an example of the configuration of the medical system 100 according to the present embodiment. As depicted in FIG. 2, the medical system 100 includes an image obtaining part 110, a control part 120, an input part 130, a displaying part 140, and a storing part 150.
[0028]
(Image Obtaining Part 110)
　The image obtaining part 110 is configured to obtain pathological image data. The image obtaining part 110 may obtain the pathological image data by including an image sensor (such as, for example, a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device)) and functioning as a medical microscope, or may obtain the pathological image data by communication with an external medical microscope.
[0029]
　Concerning the above, the details of the pathological image data obtained by the image obtaining part 110 will be described with reference to FIG. 3. FIG. 3 is a diagram depicting a pyramid structure for explaining the pathological image data and the principle of displaying this data. The pathological image data having the pyramid structure in FIG. 3 is data that is produced by imaging one observation target object and that includes pieces of image data having plural different resolutions. In the case of FIG. 3, this explains the example embodiment where the relation to the observation target object but being different to the first image data is visual information having a higher image resolution. Furthermore, the pieces of image data included in the pathological image data are represented by the pyramid structure in FIG. 3 in accordance with the resolutions. At the lowermost position of the pyramid structure, the image data having the highest resolution (having the largest data size) is arranged and, at the uppermost position of the pyramid structure, the image data having the lowest resolution (having the smallest data size) is arranged. The highest resolution is, for example, 50×50 (Kpixel: kilo-pixel) or 40×60 (Kpixel), and the lowest resolution is, for example, 256×256 (pixel) or 256×512 (pixel). In addition, the resolutions of the pieces of the image data are not limited to these.
[0030]
　When the displaying part 140 described in a section below displays thereon these pieces of image data each at, for example, 100% (displays thereon each with the number of physical dots equal to the number of pixels of the image data), the image data having the highest resolution (having the largest data size) is displayed in the largest size and the image data having the lowest resolution (having the smallest data size) is displayed in the smallest size. Concerning this, in FIG. 3, the display range of the displaying part 140 is indicated by “D.” Moreover, the pieces of image data included in the pathological image data are managed using “tile” as the unit, that is the unit of a predetermined size. The size of one tile may be, for example, 256×256 (pixel) while the size is not limited to this.
[0031]
　The pathological image data obtained by the image obtaining part 110 is stored by the storing part 150 therein described in a section below. In addition, it should be noted that the pyramid structure described above is a concept that is used to absolutely describe the pathological image data. Practically, in the pathological image data, the pieces of image data having the plural different resolutions and pieces of information relating to these resolutions only have to be respectively correlated with each other.
[0032]
(Control Part 120)
　The control part 120 is configured to generally control the overall processes executed by the medical system 100. For example, the control part 120 produces a control signal, provides the control signal to each of the configurations, and can thereby control starting up and stoppage of each of the configurations. In addition, the function of the control part 120 is not especially limited. For example, the control part 120 may control the processes generally executed by each of various types of server, a general-purpose computer, a PC (Personal Computer), a tablet PC, and the like (such as, for example, the processes relating to an OS (Operating System)).
[0033]
　As depicted in FIG. 2, the control part 120 includes a display control part 121, an annotation processing part 122, and an image analyzing part 123.
[0034]
(Display Control Part 121)
　The display control part 121 is configured to cause the displaying part 140 to display thereon first image data and second image data. It is noted that in the disclosure, the second image data relates to the observation target object but is different to the first image data. The second image data is the data used as the annotation auxiliary image described above (in the present embodiment, described as “annotation auxiliary image 20”).
[0035]
　Concerning the above, the details of the display control executed by the display control part 121 will be described with reference to FIG. 4 and FIG. 5. As depicted in FIG. 4 and FIG. 5, the display control part 121 superimposes the second image data (the annotation auxiliary image 20) on the first image data and causes these pieces of data to be displayed. The user can perform meticulous inputting relating to the annotation 10 using the second image data, watching the observation target object in a perspective view using the first image data, by the fact that the second image data relates to the observation target object but being different to the first image data are displayed as above. Moreover, compared to the case where image data produced simply by electronically zooming the first image data is provided as one example of providing the second image, the display control part 121 can provide higher-definition image data to the user. Moreover, as depicted in FIG. 5, the second image data is displayed at a position distant from the input position (the cursor position) and thereby the first image data in the vicinity of the input position does not need to be hidden by the second image data. In addition, the display position of the second image data is not especially limited and, for example, the second image data may be displayed in the vicinity of the center of the first image data, or may be displayed in a display area or on a display that is different from that of the first image data without being superimposed on the first image data.
[0036]
　Moreover, as depicted in FIG. 4, the display control part 121 can cause a position in the first image data for which the inputting is performed by the user (an input point 11 in FIG. 4 and an input point 11a in FIG. 5) and a position in the second image data for which the inputting is performed by the user (the input point 11 in FIG. 4 and an input point 11b in FIG. 5) to be superimposed on each other and to be displayed (in other words, cause these positions to be displayed at a substantially same position). The second image data is thereby displayed at the position that the user sees to apply the annotation 10 to and the user can therefore more easily perform the inputting without changing the visual line thereof. In addition, the position in the first image data for which the inputting is performed by the user (the input point 11 in FIG. 4 and the input point 11a in FIG. 5) and the position in the second image data for which the inputting is performed by the user (the input point 11 in FIG. 4 and the input point 11b in FIG. 5) may not necessarily be displayed at the substantially same position. For example, the second image data may be displayed in the vicinity of the position in the first image data for which the inputting is performed by the user to the extent that the position in the first image data is not hidden thereby.
[0037]
　Moreover, the color (such as, for example, the transparency degree and the like), the size, or the shape of the display of the second image data superimposed on the first image data are not especially limited while the first image data displays the region of the observation target object that is larger than that of the second image data, and the user can apply the annotation watching the observation target object in a perspective view using the first image data by the fact that the observation target object is displayed in a display region that is larger than that of the second image data. In the case where the first image data and the second image data have the equal resolution and are displayed at display positions that are matched with each other, the display control part 121 may cause the second image data that is superimposed on the first image data, to be displayed to be see-through thereon. Moreover, the size of the second image data may freely be set in the display field angle (or in the first image data). Furthermore, the shape of the second image data may be a rectangular shape, a circular shape, a shape designated free-hand by user inputting, or the like. In the case where the shape of the second image data is a rectangular shape, it is desirable that the aspect ratio of the second image data is substantially equal to the aspect ratio of the displaying device (the aspect ratio of the first image data in the case where the first image data is displayed on the overall screen of the displaying device). In the case where the second image data has, for example, a substantially square shape, the user may feel that the information amount obtained from the second image data is small while this problem can be avoided by setting the aspect ratio of the second image data to be substantially equal to the aspect ratio of the displaying device. The color (such as, for example, the transparency and the like), the size, or the shape of the display of the second image data may be switched on the basis of the selection by the user.
[0038]
　In addition, the input position in the first image data is hidden by the second image data in the scheme in FIG. 4 while the user can immediately check the annotation 10 also on the first image data (or the user can check the annotation 10 applied to the region not having the second image data superimposed thereon) simultaneously when the inputting relating to the annotation 10 comes to an end, by the fact that the annotation 10 is also applied in real time to the first image data. It is therefore desirable that the annotation 10 is also applied in real time to the first image data, while the attachment is not limited to this and the annotation 10 may be applied to the first image data simultaneously when the inputting relating to the annotation 10 comes to an end.
[0039]
　Moreover, the display control part 121 can cause the displaying part 140 to display thereon the second image data on the basis of the inputting by the user (such as, for example, inputting to apply the annotation 10 by the user). For example, during pressing down of a button or the like by the user (or during clicking a mouse and dragging it by the user), the display control part 121 may cause the displaying part 140 to display thereon the second image data. Moreover, the display control part 121 may switch between a “display mode” in which the second image data is always kept displayed and a “non-display mode” in which the second image data is not displayed on the basis of an instruction by the user. The second image data is not displayed by these processes when any inputting to apply the annotation 10 is not performed and the user can therefore easily watch the observation target object in a perspective view using the first image data.
[0040]
　In addition, the position in each of the pieces of image data for which the user performs the inputting is displayed in accordance with the position in the image data relating to the observation target object but being different to the first image data, such as having the more detailed visual information, for example, higher resolution of the displayed pieces of image data. More specifically, the position in the first image data for which the inputting is performed by the user (the input point 11 in FIG. 4 and the input point 11a in FIG. 5) is displayed in accordance with the position in the second image data for which the inputting is performed by the user (the input point 11 in FIG. 4 and the input point 11b in FIG. 5). The user can thereby perform the inputting with the more detailed visual information, such as highest resolution of the resolutions that the displayed pieces of image data have (in other words, the user can apply the annotation 10 with the more detailed visual information, such as highest resolution). In addition, even in the case where the image data having the more detailed visual information, such as highest resolution is not displayed, the positions in the pieces of image data for which the inputting is performed by the user may be displayed in accordance with the position in the image data having the more detailed visual information, such as highest resolution.
[0041]
　In the event that the relation to the observation target object but being different to the first image data is more detailed visual information such as a higher resolution image, the first image data and the second image data may each have a resolution different from that of each other for one observation target object. The ratios of the first resolution that the first image data has and the second resolution that the second image data has are not especially limited while, in the case where the second resolution higher than the first resolution is set, when the second resolution is excessively higher than the first resolution, the positional relation becomes difficult to understand between the first image data and the second image data. The second resolution is therefore desirably approximately a two-fold to approximately an eight-fold value of the first resolution, and is more desirably approximately a two-fold to approximately a four-fold value thereof.
[0042]
　Moreover, the second resolution that the second image data has may be equal to the first resolution. This is because, in the case where, for example, the first image data has an extremely high resolution (such as, for example, a 4K resolution), the second image data having the extremely high resolution equal to that of the first image data is applied with predetermined image processing and is displayed and, the effect like the above is thereby achieved. The “predetermine image processing” may include, for example, the electronic zooming, contrast adjustment, and the like. Moreover, in the case where the first resolution is significantly low, or the like, the second image data may be image data produced by applying a super-resolution process to the first image data.
[0043]
　Moreover, the first image data and the second image data may be pieces of image data that focus positions different from each other on the one observation target object. In other words, the first image data and the second image data may be pieces of image data having visual characteristics different from each other.
For example, in the case where the first image data includes a portion that is not focused such as the case where a section to be the observation target object is bent or the case where the section is uplifted, the image data having the portion set to be focused may be displayed as the second image data (in other words, the second image data may be displayed whose focal position in the depth direction is different from that of the first image data).
[0044]
　Moreover, the first image data and the second image data may each be image data having therein an object different from that of each other, dyed by a dyeing reagent. In other words, the first image data and the second image data may be pieces of image data having visual characteristics different from each other. This second image data relates to the observation target object but is different to the first image data. For example, in the case where the effects of a drug is determined in drug development, or the like, the molecular characteristics of the cells are often observed first using an IHC stained image. The local existence, the distribution, and the like of the desired cells are observed in a perspective view using the IHC image and, for a specific region, the morphological characteristics are thereafter observed using the HE stained image. In this case: an immuno-stained image data indicating the immune state of the tissue such as the IHC stained image data, or specially stained image data visualizing specific tissue elements for each of the purposes may be displayed as the first image data; and general stained image data indicating the form of the tissue such as HE (Hematoxylin-Eosin) stained image data may be displayed as the second image data. For example, in the case where an observation target object is dyed by a dyeing reagent capable of dyeing also any object in addition to the tumor cells (such as, for example, Ki67), the user cannot accurately recognize the form of the tumor cells in the observation target object using only the IHC stained image data. The user can accurately recognize the form of the tumor cells using the HE stained image data by the fact that the HE stained image data is displayed as the second image data.
[0045]
　For example, in the case where an affected area is identified such as a diagnosis, the morphological characteristics are often observed first using the HE stained image. The positional relation between the affected area and the organs is identified using the HE stained image and the molecular characteristics are thereafter observed for a specific region using the IHC stained image and the specially stained image. Different from the above, in this case, the general stained image data (such as, for example, the HE stained image data) may be displayed as the first image data and the immuno-stained image data (such as, for example, the IHC stained image data) or the special stained image data (such as, for example, PAS stained image data) may be displayed as the second image data. Concerning this, a specific example of the case where the IHC stained image data to be the second image data is superimposed on the HE stained image data to be the first image data will be described with reference to FIG. 6. In A of FIG. 6, it is assumed first that only the HE stained image data to be the first image data is displayed and the user applies an annotation 10a by moving a cursor 12a using only the HE stained image data. Concerning this, in the case where the user considers, for example, that the user desires to apply the annotation 10a to a point whose stain state of HER2 is strong, the user cannot accurately recognize the point using only the HE stained image data. As depicted in B of FIG. 6, the display control part 121 causes the IHC stained image data that is the second image data (the annotation auxiliary image 20) to be superimposed thereon and displayed. As depicted in C of FIG. 6, the user can thereby accurately recognize the point whose stain state of HER2 is strong using the IHC stained image data and can thereby apply an annotation 10b and, when the IHC stained image data is caused not to be displayed, as depicted in D of FIG. 6, the annotation 10a is also applied to the HE stained image data. In addition, in the example of FIG. 6, because a cursor 12b in the IHC stained image data is set to be able to continuously be arranged at the center of the IHC stained image data, the display position of the IHC stained image data therefore moves in the HE stained image data as the annotation 10b is gradually drawn from B to C of FIG. 6 (in addition, the display form of the IHC stained image data is not limited to this).
[0046]
　Moreover, in the case where observation is desired for different molecular characteristics of a cell, different types of IHC stained image data may be displayed as the first image data and the second image data. For example, the IHC stained image data having CD4 dyed therein may be superimposed on the IHC stained image data having CD8 dyed therein. The user can thereby accurately recognize any presence or absence, and the position of both of the positive cells of CD8 and CD4. For example, it is known that the probability of survival is high when the labeling index of the growth factor Ki-67 of the CD8-positive T-cell in a tumor is high, and the user can mark the cells, can apply the annotation 10, and the like confirming whether or not both CD8 and CD4 are labelled. Furthermore, the user can also perform a quantification analysis for the positional relation between, the area of each of, and the like of CD8 and CD4 as post-processes. In addition, the target of the dyeing is not limited to the combination of CD8 and CD4. For example, the target of the dyeing may be a combination of CD8 and CK (Cytokeratin), CD8 and CD3, HER2 and ER・PgR, or the like (and surely is not limited to the above combinations). These combinations facilitate the determination relating to the probability of survival and the treatment method. Moreover, for each of these combinations, the target, the reagent, the method of the dyeing are not limited to the above and may be those only when the desired object to be imaged can be identified using those. The section for the imaging may be the same one or different sections may be used (such as, for example, serial sections).
[0047]
　Moreover, at least either one of the first image data or the second image data may be the image data to display therein the analysis result (such as, for example, the result of the marking made on a positive nucleus, a heat map, and the like). For example, the user may correct the annotation 10 applied to the membrane on the basis of the analysis result for the nucleus, may correct the annotation 10 applied to the membrane on the basis of the analysis result for the nucleus, and the like by the fact that either one of the first image data or the second image data is the IHC stained image data enabling recognition of the nucleus of one observation target object and the other is the IHC stained image data enabling recognition of the membrane of the same one observation target object. In addition, the “analysis result” may be the result of an analysis automatically executed on the basis of the machine learning technique, the AI technique, and the like, or may be the result of an analysis conducted by the user.
[0048]
　Concerning the above, the second image data may be two or more pieces of image data whose display contents are different from each other. More specifically, the display control part 121 may switch to the second image data displayed in one annotation auxiliary image 20 on the basis of a predetermined input (such as, for example, pressing down of a button by the user) after superimposing the one annotation auxiliary image 20 on the first image data. Moreover, as depicted in FIG. 7, the display control part 121 may superimpose plural annotation auxiliary images 20 displaying therein the pieces of second image data whose display contents are different from each other (that are an annotation auxiliary image 20a to an annotation auxiliary image 20c displayed in first image data 30 in the example of FIG. 7) on the first image data. Moreover, the display control part 121 may cause the plural annotation auxiliary images 20 displaying pieces of second image data whose display contents are different from each other, to be displayed outside the first image data after superimposing one annotation auxiliary image 20 on the first image data. In this case, when the user selects any one of the pieces of second image data displayed outside the first image data, the second image data displayed in the annotation auxiliary image 20 superimposed on the first image data may be switched to the selected image data. As above, for example, in the case where the IHC stained image data is displayed as the first image data, the user can cause the HE stained image data, the image data displaying the analysis result, and the like to be displayed as the second image data, by using the two or more pieces of image data whose display contents are different from each other as the second image data to be superimposed. Moreover, the user can also cause plural pieces of second image data having plural resolutions different from the resolution of the first image data, to be displayed. To perform comparing various pieces of data is required to the work of applying the annotation 10 to the pathological image data and it can therefore be stated that it is especially useful that the two or more pieces of image data whose display contents are different from each other are used as the second image data. In addition, the plural annotation auxiliary images 20 are placed side by side in the longitudinal direction on the right side in the first image data in the example of FIG. 7 while the positions for the plural annotation auxiliary images 20 to be displayed and the direction therefor to be placed side by side are not especially limited.
[0049]
　The second image data is image data obtained under the imaging conditions (the setting of the imaging apparatus) different from those of the first image data for the same one observation target, image data produced by applying image processing to the first image, or image data produced by imaging a tissue obtained from the same one tissue block as that of the observation target of the first image data. As described above, the accurate application of the annotation can be enabled by using image data having high recognizability for the observation target as the second image data, such as: high quality, high definition, or high image quality image data such as image data having a higher resolution than that of the first image data or image data having a specific imaging object more sharply focused therein; or high-contrast image data, electronically zoomed image data, or image data including differently stained portions. In other words, the second image data relates to the observation target object but is different to the first image data.
[0050]
(Annotation Processing Part 122)
　The annotation processing part 122 is configured to apply the annotation 10 to each of the first image data and the second image data on the basis of the inputting by the user. More specifically, in the case where the user performs free-hand inputting using the input part 130, the annotation processing part 122 applies the annotation 10 that is freehand-drawn on the basis of the inputting, to each of the first image data and the second image data. The user can thereby perform the inputting relating to the annotation 10 watching both the first image data and the second image data.
[0051]
　In addition, the approach used by the annotation processing part 122 to apply the annotation 10 to each of the first image data and the second image data is not especially limited. As an example, the annotation processing part 122 may apply the annotation 10 to either one of the first image data or the second image data on the basis of the inputting by the user and to the other thereof using SAL (Stain Alignment). For example, in the case where the annotation processing part 122 applies the annotation 10 to the second image data on the basis of the inputting by the user, the annotation processing part 122 may also apply the annotation 10 to the first image data by executing the process of SAL on the basis of the position information of the point at which the annotation 10 is applied. Concerning this, in the case where the first image data and the second image data are pieces of image data produced by imaging serial sections, pieces of image data produced by imaging the same one section and differing the focal positions from each other in the depth direction, or the like (in other words, in the case where objects having different forms are imaged in the pieces of image data), partial alignment is repeated for plural times for the pieces of image data in the process of SAL. In addition, in the case where the first image data and the second image data are pieces of image data that are produced by imaging the same one section and that have the equal focal distance in the depth direction (in other words, in the case where the objects having the same form are imaged in the pieces of image data), the annotation 10 is applied to the same positions in the first image data and the second image data by matching the position coordinates of these pieces of image data with each other.
[0052]
　Moreover, the annotation processing part 122 determines the timing at which the user causes a pen to touch a tablet in the case where the input part 130 described in a section below includes the pen-table, or the timing at which the user starts dragging using a mouse in the case where the input part 130 includes the mouse, to be the starting timing of the inputting relating to the annotation 10, and starts the production of the annotation 10. The annotation processing part 122 determines the timing at which the user moves the pen away from the tablet or the timing at which the user finishes the dragging, to be the ending timing of the inputting relating to the annotation 10, and ends the production of the annotation 10. Within a predetermined time period from the timing at which the user moves the pen away from the tablet or the timing at which the user finishes the dragging, the annotation processing part 122 may determine that the inputting relating to the annotation 10 still continues. In this case, the annotation processing part 122 may produce the annotation 10 to connect the position at which the pen is caused by the user to move away from the tablet and the position at which the pen is caused by the user to again touch the tablet to each other, or to connect the position at which the dragging is caused by the user to come to an end and the position at which the dragging is caused by the user to again start to each other.
[0053]
　Moreover, as depicted in FIG. 4 and FIG. 5, the annotation processing part 122 may apply not only the annotation 10 but also may concurrently attach the cursor 12 (the cursor 12a and the cursor 12b in FIG. 5) that indicates the input point 11 (the input point 11a and the input point 11b in FIG. 5) at which the inputting is performed by the user. The user can thereby easily recognize the input point 11 and can therefore smoothly perform the inputting relating to the annotation 10. In addition, the cursor 12 is desirably an icon such as a small arrow or a small pencil indicating the position or co-ordinate at which the inputting is performed by the user (such as, for example, an icon of 60 px or smaller and desirably 32 px or smaller) or an icon such as a circular icon which is a small occlusion (an image is hidden) while the cursor 12 is not limited to this. The annotation is then associated with the position/co-ordinate so that the annotation can be later easily viewed by the user.
[0054]
　Concerning to the above, the annotation processing part 122 sets the control points 13 that are plural points on the annotation 10 (that may each be referred to as “CP 13”) and executes the production and the correction of the annotation 10 using the control points 13. The details of the processing executed by the annotation processing part 122 will separately be described in a section below.
[0055]
(Image Analyzing Part 123)
　The image analyzing part 123 is configured to analyze image data included in the pathological image data. More specifically, the image analyzing part 123 realizes various types of functions by analyzing the first image data or the second image data (or pieces of image data other than the first image data and the second image data, that are included in the pathological image data).
[0056]
　The contents of the functions realized by the image analyzing part 123 is not especially limited. For example, the image analyzing part 123 can realize identification of the area in the annotation 10 and the observation target object that is present inside or outside the annotation 10, counting of the number of observation target object present inside or outside the annotation 10, evaluation of the gap between the annotation 10 and the observation target object, the segmentation (region dividing based on the observation target object), or the like, by analyzing the image data. In addition, the image analyzing part 123 and the annotation processing part 122 can execute the processing in proper cooperation with each other. For example, after the image analyzing part 123 evaluates the gap between the annotation 10 and the observation target object, the annotation processing part 122 may execute correction of the annotation 10 and the like when necessary (in the case such as where the gap is larger than a predetermined value, or the like).
[0057]
(Input Part 130)
　The input part 130 is configured to receive the inputting by the user. As depicted in FIG. 2, the input part 130 includes a first input part 131 and a second input part 132. In addition, the input part 130 may include various types of input mechanism in addition to the first input part 131 and the second input part 132 described below. For example, the input part 130 may include a sound input mechanism and the like.
[0058]
(First Input Part 131)
　The first input part 131 is configured to receive the inputting relating to the annotation 10 by the user. Concerning this, a specific example of the first input part 131 will be described with reference to FIG. 8. As depicted in FIG. 8, into the first input part 131, inputting can be performed by touching of a pen 131a or the like, and the first input part 131 may be an apparatus including a touch panel 131b capable of displaying the first image data and the like (such as, for example, a pen-tablet or the like). The inputting by the pen 131a or the like (the inputting by free-hand especially) enables much intuitive inputting compared to the inputting by a mouse or the like, and can reduce the physical load. In addition, the type and the shape of the first input part 131 are not limited to those depicted in FIG. 8. For example, the first input part 131 may include a keyboard, a lever, and the like.
[0059]
(Second Input Part 132)
　The second input part 132 is configured to receive the inputting relating to the displaying of the first image data or the second image data, by the user. Concerning this, a specific example of the second input part 132 will be described with reference to FIG. 8. As depicted in FIG. 8, the second input part 132 may be an apparatus including buttons 132a, sticks 132b, and the like (such as, for example, a controller). In addition, similar to the first input part 131, the type and the shape of the second input part 132 are not limited to those depicted in FIG. 8.
[0060]
　The user may select, switch, or the like the mode of the medical system 100 by operating the buttons 132a and the sticks 132b. For example, the user may switch between a “display mode” in which the second image data is always kept displayed and a “non-display mode” in which the second image data is not displayed by pressing down the button 132a or inclining the stick 132b in a predetermined direction (in addition, it is assumed that these modes can be switched therebetween regardless of whether or not the annotation 10 is currently produced, and the switching is surely not limited to this). The user can thereby more easily perform the inputting relating to the annotation 10. More specifically, the user can cause the second image data to be displayed at a necessary timing without performing any operation to switch the overall display to image data having a higher resolution or having a more detailed visual information relating to the observation target object for example.
[0061]
　Either one of the first input part 131 and the second input part 132 described above receives the inputting by the right hand of the user and the other one receives the inputting by the left hand of the user. This input method is similar to the input method conventionally executed by doctors when the doctors apply annotations 10, and the user such as a doctor can more intuitively and more efficiently perform the inputting. FIG. 9 and FIG. 10 depict an aspect of the inputting conventionally performed by doctors when the doctors each apply the annotation 10. For example, as depicted in FIG. 9, doctors each conventionally perform free-hand inputting and the like using a dedicated pen (or a mouse), or the like as the inputting relating to the annotation 10 using one hand (the right hand in FIG. 9) and rotates a nob (a dial) as the inputting relating to the display of the image data (such as, for example, change of the display range (change in the upward, downward, rightward, or leftward direction), a change of the resolution (enlarging or shrinking), or focusing) using the other hand (the left hand in FIG. 9). Moreover, for the inputting relating to the display of the image data, for example, as depicted in FIG. 10, doctors each conventionally move a slide glass (a change of the display range) or the like using one hand (the left hand in FIG. 10). In the present embodiment, as is conventionally performed by doctors, the inputting relating to the annotation 10 and the inputting relating to the display of the first image data or the second image data are each performed by an individual hand and the user such as a doctor can thereby more intuitively and more efficiently perform the inputting.
[0062]
　In addition, the first input part 131 and the second input part 132 are not limited to those in the example in FIG. 8 above. For example, as depicted in FIG. 11, a display 133 may separately be included in addition to the first input part 131 and the second input part 132. In this case, the first image data and the like are displayed on the display 133 and the user can perform the inputting using the first input part 131 (such as, for example, the pen 131a, and the touch pad 131b executing no display) and the second input part 132 (such as, for example, a controller) watching the display. The displaying apparatus is disposed separately from the input apparatus and the user can thereby select a desired displaying apparatus (such as, for example, a displaying apparatus capable of displaying in a desired size and desired color tone, or the like), can use the existing displaying apparatus included in the working environment (such as, for example, a display disposed in a conference room, or the like), and the like. Moreover, for example, the user does not need to perform the work of drawing the annotation 10 lifting up the arm thereof toward the touch panel (that is an example of the case where the functions of a displaying apparatus and an input apparatus are realized by one apparatus) leaning on something, does not need to incline the head thereof to watch the touch panel placed flat on a desk, and the like (that is, the fatigue of the user is reduced). The work of applying the annotation 10 often takes a significantly long time and it can therefore be stated that it is especially useful that the displaying apparatus and the input apparatus are disposed separately from each other. Moreover, for example, as depicted in FIG. 12, the second input part 132 is properly not disposed and only the first input part 131 (such as, for example, the pen 131a, and the touch panel 131b available for inputting and capable of displaying) may be disposed. The user can thereby easily take out the device to be used in the work (that is the first input part 131) with the user and can therefore perform the work in various places.
[0063]
　Moreover, in the present embodiment, because it is assumed that the inputting relating to the annotation 10 is performed by free-hand, the first input part 131 receiving the inputting relating to the annotation 10 is more desirably operated by the dominant hand of the user (the operation is surely not limited to this). Moreover, the inputting relating to the annotation 10 may be performed using an optional method other than the free-hand (such as, for example, visual line inputting). Moreover, the setting relating to the assignment of the processes realized by the first input part 131 and the second input part 132 (such as, for example, enlarging, shrinking, a move of the visual field, or switching of the mode) may be able to be changed by the user. For example, the setting relating to the assignment of the processes realized by the buttons 132a and the sticks 132b included in the second input part 132 may be able to be changed by the user. The user can thereby perform the inputting in a desired form in accordance with the user’s characteristics (such as, for example, the dominant hand, the habits, or the physical handicap).
[0064]
(Displaying Part 140)
　The displaying part 140 is configured to display thereon the first image data and the second image data on the basis of the control by the display control part 121. In the present embodiment, the displaying part 140 may be embodied by the touch panel 131b of the first input part 131 while the displaying part 140 is not limited to this.
[0065]
(Storing Part 150)
　The storing part 150 is configured to store therein various types of information. For example, the storing part 150 stores therein the pathological image data obtained by the image obtaining part 110, the information relating to the annotation 10 applied by the annotation processing part 122, and the like. Moreover, the storing part 150 stores therein the various types of program, the various types of parameter, and the like, that are used in the processes in the medical system 100. In addition, the pieces of information stored in the storing part 150 are not limited to the above.
[0066]
　The example of the configuration of the medical system 100 has been described as above. In addition, the above configuration described with reference to FIG. 2 is absolutely an example and the configuration of the medical system 100 is not limited to this example. For example, the medical system 100 may include components other than the components depicted in FIG. 2. Moreover, the configuration depicted in FIG. 2 may be embodied by any apparatus configuration. For example, the image obtaining part 110 may be embodied by a medical microscope, the first input part 131 and the displaying part 140 may be embodied by a pen-tablet, the second input part 132 may be embodied by a controller, and the control part 120 and the storing part 150 may be embodied by a server or the like, or the overall configuration may be embodied by one apparatus. Moreover, the configuration of the medical system 100 may flexibly be changed in accordance with the specification and the operation thereof.
[0067]
　Moreover, the function of the annotation processing part 122 or the image analyzing part 123 described above may be realized by artificial intelligence (AI) and especially by machine learning or the like. For example, the function of the annotation processing part 122 or the image analyzing part 123 can be realized on the basis of a machine learning approach such as a neural network or a regression model, or a statistical approach. More specifically, in the case of the machine learning approach, learning data tying up the image data and the annotation 10 with each other is input into a predetermined calculation model that uses a neural network or a regression model, the learning is thereby executed, and the function of the annotation processing part 122 is realized by a processing circuit that has a processing model including the produced parameters, implemented therein.
[0068]
(2.2. Details of Processing Relating to Annotation 10)
　The example of the configuration of the medical system 100 has been described above. The details of the processing relating to the annotation 10 will be described next.
[0069]
(Production Process for Annotation 10)
　A production process for the annotation 10 executed by the annotation processing part 122 will be described with reference to FIG. 13 and FIG. 14. FIG. 13 depicts an example of the annotation 10 produced by the annotation processing part 122.
[0070]
　As depicted in FIG. 13, the annotation processing part 122 sets the control points 13 that are the plural points on the annotation 10, and produces the annotation 10 using the control points 13. More specifically, the annotation processing part 122 sets the plural control points 13 on the annotation 10 and interpolates the control points 13 using a predetermined method.

Claims
[Claim 1]
　　A medical system comprising:
　　a control part configured to receive pathological image data that is produced by imaging one observation target object and that includes first and second image data, the second image data relating to the observation target object but being different to the first image data,
　　a display control part that causes a displaying part to display thereon the first image data and the second image data; and
　　an annotation processing part that applies an annotation to each of the first image data and the second image data on a basis of input by a user.
[Claim 2]
　　The medical system according to claim 1, wherein
　　the display control part superimposes the second image data on the first image data, to be displayed.
[Claim 3]
　　The medical system according to claim 2, wherein
　　the display control part superimposes a position in the first image data for which the input is performed by the user and a position in the second image data for which the input is performed by the user on each other, to be displayed.
[Claim 4]
　　The medical system according to claim 1, wherein the second image data includes more detailed visual information than the first image data.
[Claim 5]
　　The medical system according to claim 4, wherein the more detailed visual information is a higher resolution image compared to the first image data.
[Claim 6]
　　The medical system according to claim 1, wherein
　　the position in the first image data for which the input is performed by the user is in accordance with the position in the second image data for which the input is performed by the user.
[Claim 7]
　　The medical system according to claim 1, wherein
　　the first image data and the second image data include pieces of image data having visual characteristics different from each other.
[Claim 8]
　　The medical system according to claim 7, wherein
　　the first image data and the second image data include pieces of image data that each focus a position different from that of each other on the one observation target object.
[Claim 9]
　　The medical system according to claim 7, wherein
　　the first image data and the second image data include pieces of image data having targets that are dyed by a dyeing reagent and that are different from each other.
[Claim 10]
　　The medical system according to claim 1, wherein
　　at least either one of the first image data or the second image data includes image data that displays a result of an analysis.
[Claim 11]
　　The medical system according to claim 1, wherein
　　the second image data includes two or more pieces of image data whose display contents are different from each other.
[Claim 12]
　　The medical system according to claim 1, wherein
　　the annotation processing part applies an annotation to either one of the first image data or the second image data on a basis of the input by the user, to an other using SAL (Stain Alignment).
[Claim 13]
　　The medical system according to claim 1, wherein
　　the input by the user includes input by free-hand, and
　　the annotation processing part applies the annotation drawn free-hand.
[Claim 14]
　　The medical system according to claim 13, wherein
　　the annotation is applied to indicate a closed region included in each of the first image data and the second image data.
[Claim 15]
　　The medical system according to claim 14, wherein
　　the annotation processing part sets control points that are a plurality of points on the annotation, and interpolates the control points using a predetermined method.
[Claim 16]
　　The medical system according to claim 15, wherein the difference between the first image and the second image is the resolution of the first image data and the second image data; and
　　intervals among the control points are each determined in accordance with the resolution of the second image data.
[Claim 17]
　　The medical system according to claim 15, wherein
　　the annotation processing part executes correction of the annotation using the control points on a basis of the input by the user.
[Claim 18]
　　The medical system according to claim 17, wherein
　　the annotation processing part executes the correction for a first annotation applied first, using the control points included in the second annotation applied later.
[Claim 19]
　　The medical system according to claim 18, wherein
　　the annotation processing part determines a correction range of the first annotation on a basis of a relative positional relation between first control points included in the first annotation and second control points included in the second annotation.
[Claim 20]
　　The medical system according to claim 19, wherein
　　the second annotation includes a free line that is not closed, and
　　the annotation processing part executes the correction by connecting the second control points included in the second annotation on its both ends to any one of the first control points.
[Claim 21]
　　The medical system according to claim 20, wherein
　　the annotation processing part executes the correction by connecting the second control points included in the second annotation on its both ends to the first control point for which an offset distance to each of the second control points is shortest.
[Claim 22]
　　The medical system according to claim 21, wherein
　　the annotation processing part changes the first control point to be a connection target on a basis of an angle having a vertex that is either one of the first control point and the second control point that are connected to each other.
[Claim 23]
　　The medical system according to claim 1, further comprising:
　　a first input part that receives input relating to the annotation; and
　　a second input part that receives input relating to the display of the first image data or the second image data, the second input part being different from the first input part.
[Claim 24]
　　The medical system according to claim 23, wherein
　　either one of the first input part or the second input part receives input by a right hand of the user, and
　　an other thereof receives input by a left hand of the user.
[Claim 25]
　　The medical system according to claim 1, wherein the annotation is associated with a co-ordinate in the image.
[Claim 26]
　　A medical apparatus comprising:
　　a control part configured to receive pathological image data that is produced by imaging one observation target object and that includes first and second image data, the second image data relating to the observation target object but being different to the first image data,
　　a display control part that causes a displaying part to display thereon the first image data and the second image data; and
　　an annotation processing part that applies an annotation to each of the first image data and the second image data on a basis of input by a user.
[Claim 27]
　　The medical apparatus according to claim 26, further comprising:
　　the displaying part that displays thereon the first image data and the second image data on a basis of the control by the display control part.
[Claim 28]
　　The medical apparatus according to claim 26, further comprising:
　　an image obtaining part configured to obtain the pathological image data.
[Claim 29]
　　A medical method executed by a computer, the medical method comprising the steps of:
　　receiving pathological image data that is produced by imaging one observation target object and that includes first and second image data, the second image data relating to the observation target object but being different to the first image data,
　　causing a displaying part to display thereon the first image data and the second image data; and
　　applying an annotation to each of the first image data and the second image data on a basis of input by a user.
[Claim 30]
　　A medical system comprising:
　　an imaging apparatus producing pathological image data including first and second image data, the second image data relating to the observation target object but being different to the first image data; and
　　software used in processing for the pathological image data, wherein
　　the software is executed by an information processing apparatus, and thereby realizes
　　　　causing a displaying part to display thereon the first image data and the second image data, and
　　　　applying an annotation to each of the first image data and the second image data on a basis of input by a user.