Title of invention: Image coloring device, image coloring method, image learning device, image learning method, program, and image coloring system
Technical field
[0001]
　The present disclosure relates to an image coloring device, an image coloring method, an image learning device, an image learning method, a program, and an image coloring system.
Background technology
[0002]
　Conventionally, for example, Patent Document 1 below describes that the dye amount distribution of a stained sample image is approximated to the dye amount distribution of a standard sample image, and the dye amount at each image position of the stained sample image is appropriately corrected. There is. Further, the following Patent Document 2 describes a method of performing stain separation processing on a pathological image by vector analysis.
Prior art documents
Patent literature
[0003]
Patent Document 1: Japanese Patent Laid-Open No. 2009-14355
Patent Document 2: US Pat. No. 9036889
Summary of the invention
Problems to be Solved by the Invention
[0004]
　However, according to the method described in Patent Document 1, the dye amount is corrected by comparing the distribution shape feature of the dye amount distribution of the target image with the distribution shape feature of the dye amount distribution of the standard sample class. It is difficult to color a desired color according to the form and shape of the.
[0005]
　Therefore, it has been required to accurately color the target image based on information such as the form and shape of the target image.
Means for solving the problem
[0006]
　According to the present disclosure, reference is made to an acquisition unit that acquires a decolorized image in which colors are decolorized, and learning result information in which the correspondence between the morphological information and the color information is learned in advance based on the morphological information obtained from the decolorized image. Then, an image coloring device is provided, which includes a coloring unit that colors the decolorized image with a color corresponding to the morphological information.
[0007]
　According to the present disclosure, by obtaining a decolorized image in which a color is decolorized, and based on morphological information obtained from the decolorized image, reference is made to learning result information in which correspondence between the morphological information and color information is learned in advance. And coloring the decolorized image with a color corresponding to the form information.
[0008]
　According to the present disclosure, a means for acquiring a decolorized image in which a color has been decolorized, based on morphological information obtained from the decolorized image, with reference to learning result information in which correspondence between the morphological information and color information is learned in advance. A program that causes a computer to function as means for coloring the decolorized image with a color corresponding to the morphological information is provided.
[0009]
　According to the present disclosure, a decolorization processing unit that performs a decolorization process on a colored learning image, morphological information obtained from the learning image subjected to the decolorization process, and the learning image not subjected to the decolorization process An image learning apparatus is provided, which includes a learning unit that learns by associating color information obtained from.
[0010]
　According to the present disclosure, decolorization processing is performed on a colored learning image, morphological information obtained from the learning image subjected to the decolorization processing, and learning information obtained from the learning image not subjected to the decolorization processing. And learning the associated color information.
[0011]
　According to the present disclosure, means for performing decolorization processing on a colored learning image, morphological information obtained from the learning image subjected to the decolorization processing, and learning information obtained from the learning image not subjected to the decolorization processing A program that causes a computer to function as means for learning by associating color information is provided.
[0012]
　Further, according to the present disclosure, a decolorization processing unit that performs decolorization processing on a colored learning image, morphological information obtained from the learning image subjected to the decolorization processing, and the non-decolorization processing performed Based on the morphological information obtained from the decolorized image, an image learning device having a learning unit for learning the color information obtained from the learning image in association with each other, an acquisition unit that acquires the decolorized image in which the color has been decolorized, An image coloring system is provided, comprising: an image coloring device having a coloring unit that colors the decolorized image with a color corresponding to the morphological information with reference to the learning result information by the learning unit.
Effect of the invention
[0013]
　As described above, according to the present disclosure, it is possible to accurately color a target image based on information such as the shape and shape of the target image.
　Note that the above effects are not necessarily limited, and in addition to or in place of the above effects, any of the effects shown in this specification, or other effects that can be grasped from this specification. May be played.
Brief description of the drawings
[0014]
FIG. 1 is a schematic diagram showing a configuration of a system according to an embodiment of the present disclosure.
FIG. 2 is a schematic diagram showing an example of data of a learning image.
FIG. 3 is a diagram schematically showing a separation result by a stain separation process when an input image is an HE image by hematoxylin-eosin (HE) staining.
FIG. 4 is a diagram showing an actual example of a separation result by a stain separation process when a HE image is used as an input image.
FIG. 5 is a flowchart showing a process of color conversion of a bright field image.
FIG. 6 is a diagram showing an example of converting an IHC image into an HE image.
FIG. 7 is a flowchart showing a process of converting a fluorescence image into a bright field image.
FIG. 8 is a diagram showing an example of converting a fluorescence image into an HE image.
FIG. 9 is a flowchart showing a process of coloring a sample including foreign matter when the sample has foreign matter such as artifacts.
FIG. 10 is a schematic diagram showing an example in which an IHC image that is an input image and an HE image that is an output image are displayed side by side.
FIG. 11 is a schematic diagram showing an example in which an observation target is displayed on the entire display screen, in which the left side of the screen is displayed as an IHC image and the right side of the screen is displayed as an HE image.
FIG. 12 is a schematic view showing an example in which an image having a plurality of cross sections in the depth direction is displayed, and an IHC image which is an input image is displayed on the left side and an HE image which is an output image is displayed on the right side. ..
MODE FOR CARRYING OUT THE INVENTION
[0015]
　Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, constituent elements having substantially the same functional configuration are designated by the same reference numerals, and duplicate description will be omitted.
[0016]
　The description will be given in the following order.
　1. System configuration example
　2. Specific Example of Coloring Process by System of Present Embodiment
　　2.1. Color conversion of bright field image
　　2.2. Color conversion of fluorescence image
　　2.3. If there are artifacts in the sample in the bright-field image
　3. Display application example
[0017]
　1. Configuration example of a system
　the present disclosure is directed to a method for color conversion of an image using machine learning. Examples of the target image include a pathological image such as a pathological stained image. In the present embodiment, in particular, when converting the color of a pathologically stained image, a method of automatically and accurately coloring the cell nucleus and other cell regions and tissue regions by using the dye position and staining information of the pathological image Will be described.
[0018]
　FIG. 1 is a schematic diagram showing a configuration of a system 1000 according to an embodiment of the present disclosure. As shown in FIG. 1, the system 1000 includes a learning device 100, a color converter 200, and a display device 250. The learning device 100 includes a learning image input unit 102, a learning unit 104, and a decolorizing filter 106.
[0019]
　RGB (red, green, blue) image data of a learning image (reference image) is input to the learning image input unit 102. The learning image is, for example, the pathological staining image described above. The learning image input to the learning image input unit 102 is input to the learning unit 104. The learning image input to the learning image input unit 102 is decolorized by the decolorizing filter 106. Grayscale image data is obtained by the decolorization process. The grayscale image data is input to the learning unit 104. The learning unit 104 receives a large number of learning images and learns information about the form and shape of the grayscale image, such as texture and shade (hereinafter referred to as form information), and the corresponding colors of the RGB data.
[0020]
　Note that an edge image may be used instead of the grayscale image. In this case, processing for detecting an edge from RGB data is performed, and the obtained edge image is input to the learning unit 104. The learning unit 104 learns the information of the edge image and the corresponding color of the RGB data. That is, the decolorization filter 106 functions as a decolorization processing unit that removes a color from the learning image.
[0021]
　The learning by the learning unit 104 can use deep learning. Deep learning is a
machine learning method using a multilayer neural network (Deep Neural Network ( DNN)). The learning image may include regions such as cell nuclei, fibers, blood cells, fats, and bacteria. Desirably, by limiting the learning image input in the learning process to the pathological staining image, it is possible to configure the learning device 100 in which color errors are unlikely to occur.
[0022]
　FIG. 2 is a schematic diagram showing an example of data of a learning image. Grayscale image data 300 obtained by the decoloring process of the decolorizing filter 106 and RGB image data 400 which is the learning image itself are pixel by pixel. Showing. In FIG. 2, grayscale image data 300 having morphological information is shown on the left side. In addition, image data 400 corresponding to the grayscale image data 300 and having ideal correct color information is shown on the right side. In this way, the learning unit 104 learns the color of each pixel from arbitrary shape and shape information for each pixel. In the present embodiment, the learning unit 104 basically only learns a characteristic color due to staining such as a pathological image. Characteristic colors for staining a pathological image are about 2 to 3 colors in a bright field. When increasing the types of dyeing for which color conversion is desired, it is desirable to provide a plurality of learning devices 100.
[0023]
　As a result of learning by the learning unit 104, learning result information in which the morphological information of the learning image and the color information are associated with each other is obtained. The learning result information is sent to the coloring unit 206 of the coloring device 200.
[0024]
　The color converter 200 includes an image input unit 202, a decolorization filter 204, a coloring unit 206, a stain separation unit 208, a display processing unit 210, and an operation input unit 212. Each component of the learning device 100 and the color converter 200 shown in FIG. 1 may be configured by a circuit (hardware) or a central processing unit such as a CPU and a program (software) for causing the same to function. it can.
[0025]
　An input image to be colored is input to the image input unit 202. The input image is sent to the decolorizing filter 204. The decolorization filter 204 performs a decolorization process on the input image to create a grayscale image. An edge image may be used instead of the grayscale image, and in this case, the edge image is detected from the input image. The decolorization filter 204 functions as a decolorization processing unit that removes a color from the learning image, and may have a function of edge detection. When the input image is a fluorescent image, the fluorescent image is a grayscale image, and therefore the processing by the decolorizing filter 202 need not be performed. The image input unit 202 or the decolorization filter 204 functions as an acquisition unit that acquires the decolorized image. As the decoloring filter 204 of the color converter 200, it is preferable to use the same one as the decoloring filter 106 of the learning device 100. An emphasis filter or the like may be applied.
[0026]
　By passing the input image through the decolorizing filter 204, a grayscale image of the input image is obtained. The obtained grayscale image is input to the coloring unit 206. Further, in parallel with the decoloring process by the decolorizing filter 204, the stain separating unit 208 performs the stain separating process on the input image. The stain separating unit 208 separates the color of the input image based on the stain vector representing the color feature of the image, using the method described in Patent Document 2 described above. The stain separation process makes it possible to distinguish between the cell nucleus and other regions other than the nucleus.
[0027]
　The coloring unit 206 acquires the color information corresponding to the morphological information by applying the morphological information of the input grayscale image to the learning result information obtained from the learning unit 104. Then, the grayscale image is colored using the color information corresponding to the morphological information.
[0028]
　At the time of coloring, coloring by the coloring unit 206 is possible only with a grayscale image. For example, when an object having the same size and shape as the cell nucleus but not the cell nucleus exists in the input image, coloring is performed as the cell nucleus. There is a possibility that it will end up. Therefore, the information obtained by the stain separation process is used as a coloring hint (auxiliary information) to improve the coloring accuracy.
[0029]
　FIG. 3 is a diagram
schematically showing the separation result by the stain separation process when the input image is an image by Hematoxylin Eosin ( HE) staining (hereinafter referred to as HE image). Further, FIG. 4 is a diagram showing an actual example of the separation result by the stain separation process when the HE image is used as the input image. In HE images, hematoxylin and eosin staining stains cell nuclei purple and tissue components other than cell nuclei stain red. As shown in FIGS. 3 and 4, when the HE image is subjected to the stain separation processing, the cell nuclei are separated into purple and the other parts than the cell nuclei are separated into red.
[0030]
　The stain separation unit 202 extracts coordinates that are hints for coloring by the coloring unit 206 from the image that has been color-separated into two or more. In the coordinate extraction, a high-luminance representative point is found by threshold processing, a high-luminance region having a predetermined threshold value or higher is extracted, and the center of gravity or the center coordinate of the region is calculated. An algorithm such as NMS (Non-Maximum Suppression) may be used when narrowing down the detection position. By providing the coloring unit 206 with the position information of the color and coordinates separated by the stain separating unit 202 as a hint, highly accurate color conversion by the coloring unit 206 can be realized. The stain vector used in the stain separation may be the one obtained from the learning image. Note that the stain vector used in the stain separation may be the one obtained from the learning image.
[0031]
　According to the stain separation processing by the stain separating unit 202, the region of the cell nucleus is recognized as a purple region in FIGS. 3 and 4. Therefore, when coloring by the coloring unit 206, even if the area other than the cell nucleus is recognized as the cell nucleus from the morphological information of the grayscale image, the coordinates of the area are determined by the auxiliary information obtained from the stain separation process. It will not match the coordinates of the cell nucleus. Therefore, the region can be determined to be a region other than the cell nucleus. Therefore, by using the auxiliary information obtained by the stain separation processing when coloring by the coloring unit 206, it is possible to perform coloring accurately.
[0032]
　The coloring unit 206 extracts the color of the region obtained by the stain separation process from the original image input to the image input unit 202 and colors the region. For example, in the area determined not to be the cell nucleus area based on the morphological information, if the area is colored with the cell nucleus color by the stain separation processing, the color obtained from the original image is colored in the area. On the other hand, the coloring unit 206 can also refer to the color of the learning image based on the area obtained by the stain separation processing. When referring to the color of the learning image, a color of the learning image in which the stain vector of the original image and the stain vector (reference vector) of the learning image are close to each other is selected and used as auxiliary information at the time of coloring. In this case, the learning unit 104 extracts the stain vector of the learning image in advance. Machine learning may be used when extracting the stain vector of the learning image.
[0033]
　As described above, in the coloring process by the color converter 200, the process of discriminating and storing the color of the cell nucleus of the input image and other cell regions and tissue regions for each region or point, and applying the digital filter to the image Decolorization by using the color information stored for each area or point, and the learned color is derived based on the shape or shape information for each area, and the ideal color is digitalized. The process of dyeing is included. The "region" in the coloring process can be detected automatically by using a stain separation process. The "point" may be detected in the coloring step by the center of the area obtained by the stain separation or the barycentric coordinates, or by the random coordinates in the image. Note that, as described above, the learning unit 104 basically only learns a characteristic color due to staining such as a pathological image. Therefore, when coloring is performed by the coloring unit 206, for example, a region of a color that does not originally appear in HE staining in the input image is limited to the range of HE staining. Therefore, it is possible to prevent the color from being colored unexpectedly.
[0034]
　The image colored by the coloring unit 206 is input to the display processing unit 210 as an output image. The display processing unit 210 performs processing for displaying the output image on the display device 250. The operation input unit 212 receives user operation information from a mouse, a keyboard, or the like. The operation input unit 212 sends the operation information to the display processing unit 210. The display processing unit 210 controls the display content displayed on the display device 250 based on the operation information. The processing performed by the display processing unit 210 will be described later.
[0035]
　2. Specific Examples of Coloring Processing by System of Present Embodiment
　Next, some specific examples of coloring processing (color conversion processing) by the system of the present embodiment will be described.
[0036]
　　2.1. Color conversion
　of bright field image First, color conversion of a bright field image will be described. Examples of the bright field image include the HE image and the IHC image described above. Here, an image used for specifically detecting a specific substance in a tissue or a cell by utilizing an antigen-antibody reaction by immunohistochemistry (IHC) is referred to as an IHC image. According to this embodiment, it is possible to mutually perform color conversion processing between these bright-field images.
[0037]
　First, when the input image is the HE image and the HE image is also output as the output image, the color information generated by the stain separation process is held as the reference color for coloring. As described above, the color of the area obtained by the stain separation process may be extracted from the original image, or the color of the learning image may be referred to.
[0038]
　When the input image is an IHC image and an HE image is to be output as an output image, or when the image is to be converted into an image with a different stain, a reference color is assigned to the color obtained by the stain separation process and used as a reference color. Is desirable. For example, when the input image is the IHC image and the output image is the HE image, the cell nucleus is dyed blue in the IHC image. Therefore, the reference color of the region of the cell nucleus after coloring the purple color of the HE image against the blue color of the IHC image. Give as. Thereby, when the IHC image is colored into the HE image, the region of the cell nucleus can be colored purple. The coloring unit 206 assigns a reference color to the color after stain separation by the stain separation unit 208, and performs coloring with this as a reference color.
[0039]
　As described above, the dyeing position is sampled in the region obtained by the stain separation processing, and coloring is performed based on the grayscale image after the decolorization conversion and the reference color information. Note that color conversion from an HE image to an IHC image is also possible.
[0040]
　FIG. 5 is a flowchart showing the process of color conversion of the bright field image. First, in step S10, the stain separation unit 208 performs stain separation processing on the input image. In the next step S12, the staining position (nuclear information) is sampled based on the image obtained by the stain separation. In the next step S14, the decolorization filter 204 performs a decolorization process (decolorization conversion) on the input image to obtain a grayscale image. In the next step S16, the coloring unit 206 applies the form information of the grayscale image to the learning result information obtained from the learning unit 104 to color the grayscale image.
[0041]
　FIG. 6 is a diagram showing an example of converting an IHC image into an HE image (Simulated HE). The IHC image is a stain that utilizes an antigen-antibody reaction and is suitable for identifying, for example, cancer cells, but may not be suitable for accurately observing the shape of cells. In such a case, the shape of a cell or the like can be easily observed by converting an IHC image into an HE image by the method of the present embodiment. As described above, the cell nuclei are stained blue in the IHC image, and thus the cell nuclei are stained blue in the input image of the IHC image shown on the left side of FIG. 6 (the area indicated by the gray dot in the left diagram of FIG. 6). On the other hand, in the HE image after coloring shown on the right side of FIG. 6, the cell nuclei are colored purple (regions indicated by black dots in the right diagram of FIG. 6), and the other parts are colored red. Therefore, it can be understood that the HE image after color conversion and the original IHC image have corresponding cell nucleus positions, and the conversion is accurately performed.
[0042]
　　2.2. Color Conversion
　of Fluorescent Image Next, bright field color conversion of the fluorescent image will be described. For example, there is a case where color conversion is performed from a fluorescence image to an HE image or an IHC image. For example, using DAPI as a fluorescent dye, the observation target is dyed to obtain a fluorescence image of the observation target. Then, the fluorescence image is sampled as nuclear information. Then, the reference color of the reference is assigned to the core information to perform coloring. The reference reference color is purple for HE images and blue for IHC images.
[0043]
　The fluorescent image requires a gain adjustment because the signal is weak. Further, for example, in the case of a grayscale image, since the lightness and darkness are opposite, it is desirable to add a negative-positive inversion process.
[0044]
　When the coloring is completed, the same process is performed for the number of fluorescent images and the images are overlaid. For example, in the case of multiple staining of two or more colors such as an IHC image, it is necessary to learn each color, and therefore, the learning device 100 side preliminarily learns the multiple stained bright field image.
[0045]
　When handling an image after multiplexing (superposition), it is necessary to adjust the gain according to the color assignment of each fluorescence image.
[0046]
　FIG. 7 is a flowchart showing a process of converting a fluorescence image into a bright field image. First, in step S20, a nuclear stain image showing a region in which the nucleus is stained by fluorescence is sampled. Here, a nuclear-stained image is sampled from a fluorescent image obtained by irradiating a living tissue stained with a fluorescent substance with predetermined light. In the next step S22, the gain of the sampled image is adjusted. In the next step S24, negative-positive reversal processing is performed.
[0047]
　In the next step S26, the sampled cell nuclei are colored. In the next step S28, image multiplexing is performed. In fluorescence observation in which a plurality of stained images are mixed, a pseudo bright-field stained image can be obtained by determining a color for each region of the fluorescent image and advancing the coloring process on the mixed image.
[0048]
　FIG. 8: is a figure which shows the example which converted the fluorescence image into the HE image (Simulated HE). As for the fluorescence image, the input image of the fluorescence image shown on the left side of FIG. 8 is basically a grayscale image, and the white-looking portion basically corresponds to the region of the cell nucleus. In the fluorescence image, the nuclei are brighter than the surroundings, and are therefore sampled as the above-mentioned nuclear staining image. As a result, the coloring unit 206 can color the nucleus portion based on the sampled shape of the cell nucleus. Referring to FIG. 8, in the HE image (Simulated HE) obtained by converting the fluorescence image, it can be seen that the region of the cell nucleus that appears white in the fluorescence image is colored with the color of the region 450 of the cell nucleus in the HE image. .. The learning device 100 does not need to newly learn the fluorescence image, and can convert the fluorescence image into the HE image based on the learning data of the HE image.
[0049]
　　2.3. Case
　where artifact exists in sample in bright field image Here, a case where artifact (artifact, foreign substance) exists in sample in bright field image will be described. In the case of a pathologically stained image, an air bubble is an example of the artifact. The artifact is a very deep black color that is easy to distinguish from other colors. A mask image is generated by the threshold value process for detecting the black color, and the red (pink) reference color of the HE image is assigned to the region, so that the color becomes the same as that of normal tissue and the visibility is improved. By linking this processing with the processing of "2.1. Color conversion of bright field image" described above, it is possible to reproduce a color close to that of the reference image even with respect to a sample having an artifact.
[0050]
　FIG. 9 is a flowchart showing a process of coloring an input image sample including artifacts (foreign matter). First, in step S30, threshold value processing of the pixel value of the input image is performed to detect a black portion. In the next step S32, masking is performed to remove the black portion detected in step S30. In the next step S34, stain separation processing is performed on the image from which the black portion has been removed.
[0051]
　In the next step S36, the staining position is sampled from the image obtained by the stain separation process. In the next step S38, a grayscale image is obtained by subjecting the input image to decolorization processing. In the next step S39, the grayscale image is colored by applying the morphological information of the grayscale image to the learning result information obtained from the learning unit 104.
[0052]
　3. Example of Display Application
　Next, an example of an application for displaying an output image on the display device 250 will be described. In the present embodiment, both the input image and the output image can be displayed on the display device 250 by the processing of the display processing unit 210. For example, when the IHC image is the input image and the HE image is the output image, it is conceivable to observe the IHC image to discriminate cancer cells and the like and to observe the HE image to discriminate the shape of cells. ..
[0053]
　FIG. 10 is a schematic diagram showing an example in which an IHC image that is an input image and an HE image that is an output image are displayed side by side. In the example shown in FIG. 10, the input image IHC image is displayed on the left side and the output image HE image is displayed on the right side on the display screen 252 of the display device 250. Thereby, the observer can recognize the position of the cancer cell in the IHC image on the left side and the shape of the cell in the HE image on the right side. When the user operates the mouse or the like to input an instruction to move the image without the screen from the operation input unit 212, the display processing unit 210 displays the IHC image as the input image and the HE image as the output image on the display screen 252. Move in the same direction as above. As a result, the convenience at the time of observation by the user can be enhanced.
[0054]
　FIG. 11 is a schematic diagram showing an example in which the observation target is displayed on the entire display screen, in which the left side of the screen is displayed as an IHC image and the right side of the screen is displayed as an HE image. The boundary 254 between the IHC image and the HE image can be moved left and right when the user operates the mouse or the like to input information from the operation input unit 212. By performing the display as shown in FIG. 11, it is possible to visually recognize the cancer cells on the left side of the screen and recognize the shape of the one observation target on the right side of the screen.
[0055]
　FIG. 12 shows an example in which an image having a plurality of cross sections in the depth direction is displayed, and an IHC image which is an input image is displayed on the left side and an HE image which is an output image is displayed on the right side. .. The IHC image and the HE image are associated with the images of the dark surface at the same depth position. The user can display both the IHC image and the HE image in the cross section at the desired depth position by inputting information from the operation input unit 212.
[0056]
　As described above, according to the present embodiment, by performing the coloring by the color converter 200, it is possible to clearly reproduce the color of the sample whose staining is deteriorated. In addition, the pathological stain image may have a different shade depending on the scanner, but by performing coloring by the color converter 200, the tint of the stain can be corrected to a desired reference color. In addition, by performing the coloring by the color converter 200, it is possible to make the stain of the sample invisible and improve the visibility. It is also possible to observe the fluorescent image by coloring it like a bright field image. Further, unlike the tone curve correction, it is possible to correct each color in the slide of each image.
[0057]
　The preferred embodiments of the present disclosure have been described above in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive various changes or modifications within the scope of the technical idea described in the claims. It is understood that the above also naturally belongs to the technical scope of the present disclosure.
[0058]
　Further, the effects described in the present specification are merely illustrative or exemplary, and are not limiting. That is, the technique according to the present disclosure may have other effects that are apparent to those skilled in the art from the description of the present specification, in addition to or instead of the above effects.
[0059]
　The following configurations also belong to the technical scope of the present disclosure.
(1) With
　reference to learning result information obtained by learning in advance the correspondence between the morphological information and the color information, based on an acquisition unit that acquires a bleached image with decolorized colors and morphological information obtained from the bleached image, An
　image coloring device, comprising: a coloring unit that colors the decolorized image with a color corresponding to the morphological information .
(2) The image coloring device according to (1), wherein the acquisition unit includes a decolorization processing unit that performs a decolorization process on a colored input image.
(3) The image coloring device according to (1) or (2), wherein the learning result information is obtained by learning correspondence between morphological information of a plurality of learning images and color information corresponding to the morphological information. ..
(4) The image
　coloring according to (2), including a stain separation unit that performs a stain separation process on the input image, and the coloring unit performs the coloring by using auxiliary information obtained by the stain separation process. apparatus.
(5) The auxiliary information is position information of a predetermined part in the input image separated by the stain separation process, and the
　coloring part, for a part of the input image corresponding to the position information, The image coloring device according to (4), which colors the color corresponding to the predetermined portion.
(6) The image coloring device according to (5), wherein the predetermined part is a part corresponding to a cell nucleus.
(7) The image coloring device according to (5) or (6), wherein the coloring unit colors the color obtained from the input image at the predetermined portion separated by the stain separation processing.
(8) In the above (5) or (6), the coloring unit colors the reference color obtained from the learning result information based on the stain separation processing at the predetermined portion obtained by the stain separation processing. The image coloring device described.
(9) The input image is an HE image obtained by HE staining, and the coloring unit colors the input image that has been subjected to the decolorization processing with a color corresponding to the HE image. Image coloring device.
(10) The input image is an IHC image obtained by IHC staining, and the coloring section colors the input image subjected to the decolorization processing with a color corresponding to the HE image obtained by HE staining. ) The image coloring device described in.
(11) The acquisition unit acquires a fluorescent image obtained by applying a phosphor to an observation target object as the decolorized image, and the
　colored unit uses an HE image obtained by HE staining of the fluorescent image or an IHC stain. The image coloring device according to (1), which colors the color corresponding to the obtained IHC image.
(12) The image coloring device according to (11), wherein the coloring unit colors the plurality of images forming the fluorescent image and superimposes the plurality of colored images.
(13) The image coloring device according to (11) or (12), which performs gain adjustment or negative/positive inversion processing of the fluorescent image before coloring by the coloring unit.
(14) The coloring section obtains a black region from the decolorized image, and colors the black region with a predetermined color which is set in advance, according to any one of (1) to (13) above. Image coloring device.
(15) The image coloring device according to any one of (1) to (14), further including a display processing unit that performs processing for displaying an image before coloring and an image after coloring by the coloring unit on a display device. ..
(16) With respect to the same observation object, the display processing unit displays an image before coloring in the first area of ​​the observation object, and displays an image after coloring in the second area of ​​the observation object. The image coloring device according to (15) above.
(17) The display processing unit displays an image before coloring and an image after coloring on the same screen, and moves the image before coloring and the image after coloring in the same direction based on operation information. ) The image coloring device described in.
(18) The display processing unit displays, for each of the image before coloring and the image after coloring, an image having the same cross-sectional position in the depth direction, and changes the cross-sectional position based on operation information. The image coloring device described.
(19) The image coloring device according to claim 1, which colors the pathologically stained image.
(20) Obtaining a decolorized image in which a color is decolorized,
　and referring to learning result information in which the correspondence between the morphological information and the color information is preliminarily learned based on the morphological information obtained from the decolorized image, Coloring the decolorized image with a color corresponding to morphological information
　.
(21) means for acquiring a decolorized image in which the color has been decolorized,
　Based on the morphological information obtained from the bleached image, with reference to learning result information that learned the correspondence between the morphological information and the color information in advance,
　as means for coloring the bleached image with a color corresponding to the morphological information, A program that causes a computer to function.
(22) Decolorization processing unit that performs decolorization processing on a colored learning image,
　morphological information obtained from the learning image subjected to the decolorization processing, and demorphization processing obtained from the learning image not subjected to the decolorization processing An
　image learning device , comprising: a learning unit configured to learn color information in association with each other .
(23) Decolorization processing is performed on a colored learning image,
　morphological information obtained from the learning image subjected to the decolorization processing, and color information obtained from the learning image not subjected to the decolorization processing And learning by associating with each other
　.
(24) means
　for performing a decolorization process on a colored learning image, morphological information obtained from the learning image subjected to the decolorization process, and color information obtained from the learning image not subjected to the decolorization process.
　A program that causes a computer to function as means for learning in association with each other .
(25) Decolorization processing unit that performs decolorization processing on the colored learning image, morphological information obtained from the learning image subjected to the decolorization processing, and the learning image not subjected to the decolorization processing An image learning device, comprising: a learning unit configured to learn color information in association with each other;
　An acquisition unit that acquires a decolorized image in which a color has been decolorized, and based on morphological information obtained from the decolorized image, referring to learning result information by the learning unit, a color corresponding to the morphological information is set to the decolorized image. An image coloring system, comprising: an image coloring device having a coloring section
　for coloring.
Explanation of symbols
[0060]
　100 Learning device
　104 Learning part
　106 Decolorization filter
　200 Coloring device
　202 Image input part
　204 Decolorization filter
　206 Coloring part
　208 Stain separating part
　210 Display processing part
The scope of the claims
[Claim 1]
　An acquisition unit that acquires a decolorized image in which a color is decolorized
　, based on morphological information obtained from the decolorized image, with reference to learning result information in which the correspondence between the morphological information and the color information is learned in advance, the morphological information And a coloring section for coloring the decolorized image with a color corresponding to
　.
[Claim 2]
　The image coloring device according to claim 1, wherein the acquisition unit includes a decolorization processing unit that performs a decolorization process on a colored input image.
[Claim 3]
　The image coloring device according to claim 1, wherein the learning result information is obtained by learning correspondence between morphological information of a plurality of learning images and color information corresponding to the morphological information.
[Claim 4]

　The image coloring device according to claim 2 　, further comprising a stain separating unit that performs a stain separating process on the input image, wherein the coloring unit performs the coloring by using auxiliary information obtained by the stain separating process.
[Claim 5]
　The auxiliary information is position information of a predetermined part in the input image separated by the stain separation process, and the
　coloring part is the predetermined part for the part of the input image corresponding to the position information. The image coloring device according to claim 4, wherein a color corresponding to the part is colored.
[Claim 6]
　The image coloring device according to claim 5, wherein the predetermined portion is a portion corresponding to a cell nucleus.
[Claim 7]
　The image coloring device according to claim 5, wherein the coloring section colors the color obtained from the input image at the predetermined portion separated by the stain separation processing.
[Claim 8]
　The image coloring device according to claim 5, wherein the coloring section colors a reference color obtained from the learning result information based on the stain separation processing at the predetermined portion obtained by the stain separation processing.
[Claim 9]
　The image coloring device according to claim 2, wherein the input image is an HE image obtained by HE staining, and the coloring unit colors the input image subjected to the decolorization processing with a color corresponding to the HE image.
[Claim 10]
　The said input image is an IHC image obtained by IHC dyeing, The said coloring part colors the said input image by which the said decolorization process was carried out with the color corresponding to the HE image obtained by HE dyeing. Image coloring device.
[Claim 11]
　The acquisition unit acquires a fluorescence image obtained by applying a phosphor to an observation object as the decolorized image, and the
　coloring unit obtains an HE image obtained by HE staining of the fluorescence image or an IHC obtained by IHC staining. The image coloring device according to claim 1, which colors a color corresponding to the image.
[Claim 12]
　The image coloring device according to claim 11, wherein the coloring unit colors the plurality of images forming the fluorescent image and superimposes the plurality of colored images.
[Claim 13]
　The image coloring device according to claim 11, wherein gain adjustment or negative/positive reversal processing of the fluorescence image is performed before coloring by the coloring unit.
[Claim 14]
　The image coloring device according to claim 1, wherein the coloring unit acquires a black region from the decolorized image and colors the black region with a predetermined color.
[Claim 15]
　The image coloring apparatus according to claim 1, further comprising a display processing unit that performs processing for displaying an image before coloring and an image after coloring by the coloring unit on a display device.
[Claim 16]
　The display processing unit causes the first area of ​​the observation object to display an image before coloring and the second area of ​​the observation object to display an image after coloring for the same observation object. Item 15. The image coloring device according to item 15.
[Claim 17]
　The display processing unit displays the image before coloring and the image after coloring on the same screen, and moves the image before coloring and the image after coloring in the same direction based on operation information. Image coloring device.
[Claim 18]
　The image coloring according to claim 15, wherein the display processing unit displays an image having the same cross-sectional position in the depth direction for each of the image before coloring and the image after coloring, and changes the cross-sectional position based on operation information. apparatus.
[Claim 19]
　The image coloring device according to claim 1, which colors a pathologically stained image.
[Claim 20]
　Obtaining a decolorized image in which the color is decolorized
　, based on the morphological information obtained from the decolorized image, with reference to the learning result information previously learned the correspondence between the morphological information and the color information, the morphological information Coloring the decolorized image with a corresponding color
　.
[Claim 21]
　Means for obtaining a bleaching image in which the color is decolorized,
　based on the form information obtained from said decolorized image, by referring to the learning result information previously learned the correspondence between the configuration information and the color information, corresponding to the form information
　A program that causes a computer to function as means for coloring the decolorized image with a desired color .
[Claim 22]
　A decolorization processing unit that performs decolorization processing on a colored learning image,
　morphological information obtained from the learning image subjected to the decolorization processing, and color information obtained from the learning image not subjected to the decolorization processing. An
　image learning device , comprising: a learning unit that performs learning in association with each other .
[Claim 23]
　Decolorization processing is performed on a colored learning image,
　morphological information obtained from the learning image subjected to the decolorization processing and color information obtained from the learning image not subjected to the decolorization processing are associated with each other. And
　an image learning method comprising:
[Claim 24]
　Means
　for performing decolorization processing on a colored learning image, form information obtained from the learning image subjected to the decolorization processing, and color information obtained from the learning image not subjected to the decolorization processing in association with each other.
　A program that causes a computer to function as a means of learning .
[Claim 25]
　A decolorization processing unit that performs decolorization processing on a colored learning image, morphological information obtained from the learning image subjected to the decolorization processing, and color information obtained from the learning image not subjected to the decolorization processing. An image learning device having a learning unit that learns in association with each other,
　an acquisition unit that acquires a decolorized image in which colors are decolorized, and learning result information by the learning unit based on morphological information obtained from the decolorized image. And an image coloring device having a coloring unit for coloring the decolorized image with a color corresponding to the morphological information
　.