1. A system (102) for performing segmentation of one or more anomaly regions in a plurality of multi-modal three-dimensional (3D) images of an organ, the system (102) comprising: a preprocessing module (210) configured to: receive the plurality of multi-modal three-dimensional (3D) images corresponding to the organ, and apply one or more preprocessing operations on the plurality of multi-modal 3D images; a segmenting module (212) comprising: a generator sub-module (218) configured to generate a plurality of predicted segmentation masks corresponding to one or more anomaly sub-regions, and a discriminator sub-module (220) configured to classify one or more anomaly sub-regions in each of the plurality of predicted segmentation masks into one of a real-class or a synthetic-class based on the plurality of the multi-modal 3D images and the plurality of predicted segmentation masks; a validating module (214) configured to: generate a similarity score based on an overlap between the plurality of predicted segmentation masks and a plurality of predefined ground truth masks, and validate an accuracy of each of the plurality of predicted segmentation masks based on the similarity score; and a generating module (216) configured to generate one or more segmented anomaly regions by aggregating the plurality of predicted segmentation masks based on the validation.

2. The system (102) as claimed in claim 1, wherein the segmenting module (212) corresponds to a transformer based generative adversarial network (GAN), the generator sub-module (218) is a Shifted Windows-Unet Transformer (Swin-UNETR) generator, and the discriminator sub-module (220) is a patch-based 3D Convolutional Neural Network (CNN) discriminator.

3. The system (102) as claimed in claim 1, wherein the generator sub-module (218) and the discriminator sub-module (220) are trained based on an unequal-weighted combination of at least two loss functions, wherein the loss functions represent a spatial-overlap accuracy and a voxel-level accuracy between the plurality of predicted segmentation masks and the plurality of predefined ground truth masks.

4. The system (102) as claimed in claim 3, wherein the at least two loss functions comprises a Dice loss function and a binary-cross entropy (BCE) loss.

5. The system (102) as claimed in claim 3, wherein one or more unequal weights utilized in the unequal-weighted combination of the at least two loss functions are determined experimentally.

6. The system (102) as claimed in claim 1, wherein one or more first parameters of the generator sub-module (218) are updated based on the classification by the discriminator sub-module (220) and the overlap between the plurality of predicted segmentation masks and the plurality of predefined ground truth masks, and wherein one or more second parameters of the discriminator sub-module (220) are updated based on the plurality of predicted segmentation masks and the plurality of predefined ground truth masks.

7. The system (102) as claimed in claim 1, wherein the one or more preprocessing operations comprise a foreground cropping operation, an intensity normalization operation, a data augmentation operation, and a Region of Interest (ROI) based random spatial cropping operation.

8. The system (102) as claimed in claim 1, wherein the plurality of the multi-modal 3D images comprises at least one of T1-weighted (T1) image, a contrast enhanced T1-weighted (T1-Gd) image, a T2-weighted (T2) image, and a Fluid-Attenuated Inversion Recovery (FLAIR) image.

9. The system (102) as claimed in claim 1, wherein the one or more anomaly sub-regions comprise an enhancing anomaly sub-region, an anomaly core sub-region, and a whole anomaly sub-region.

10. The system (102) as claimed in claim 1, wherein the organ is the brain, and the one or more anomaly regions comprise a tumor.

Description:FIELD OF INVENTION
The present disclosure relates to artificial intelligence (AI)-based medical image analysis, and more particularly, to a system and method for performing segmentation of one or more anomaly regions in a plurality of multi-modal three-dimensional (3D) images of an organ.
BACKGROUND
Segmentation of anomalies in radiological images constitutes a core task in neuro‑oncology, supporting both diagnostic assessment and therapeutic planning. Magnetic resonance imaging (MRI) systems generate high‑resolution, multi‑sequence three‑dimensional (3D) volumes within short acquisition times, enabling detailed visualization of anatomical structures and pathological regions within the brain and other organs.
However, manually isolating and analyzing anomaly components from such volumetric datasets is labor‑intensive and inherently subjective. Expert neuroradiologists typically delineate anomalous areas on MRI scans manually or with semi‑automated tools, which is time‑consuming and susceptible to inter‑observer variability.
Conventional deep learning techniques for medical image segmentation predominantly utilize discriminative architectures designed to map input images directly to pixel‑level class predictions. Numerous architectures based on encoder-decoder frameworks have been introduced, employing contracting and expanding paths to capture contextual information and generate segmentation masks. Many of these solutions rely on convolutional neural networks for local feature extraction. However, the integration of transformer modules and attention mechanisms has improved segmentation accuracy by capturing broader contextual cues.
Transformer‑based architectures facilitate effective modeling of global context for segmentation tasks but often require substantial computational resources. Developments such as shifted‑window transformers mitigate computational inefficiencies while enhancing representational capacity. Likewise, self‑adapting framework architectures provide improved robustness in segmentation. Contemporary state‑of‑the‑art solutions frequently utilize ensemble methods that integrate predictions from multiple architectures, coupled with extensive data augmentation, to achieve superior performance.
Despite these advances, discriminative models exhibit limitations in learning fine‑grained structural characteristics and generating coherent segmentation outputs. Generative modeling approaches, which aim to learn the underlying data distribution rather than fixed pixel mappings, offer increased representational richness. The application of generative methodologies within transformer‑based architectures for medical image segmentation remains relatively underexplored, presenting an opportunity for further innovation.
Accordingly, there is a need for improved systems and methods that address one or more of the foregoing limitations of existing techniques.
SUMMARY
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
According to an embodiment of the present disclosure, a system for performing segmentation of one or more anomaly regions in a plurality of multi-modal three-dimensional (3D) images of an organ is disclosed. The system includes a preprocessing module configured to receive the plurality of multi-modal three-dimensional (3D) images corresponding to the organ. The preprocessing module is also configured to apply one or more preprocessing operations on the plurality of multi-modal 3D images. The system also includes a segmenting module including a generator sub-module configured to generate a plurality of predicted segmentation masks corresponding to one or more anomaly sub-regions. The segmenting module also includes a discriminator sub-module configured to classify one or more anomaly sub-regions in each of the plurality of predicted segmentation masks into one of a real-class or a synthetic-class based on the plurality of the multi-modal 3D images and the plurality of predicted segmentation masks. The system further includes a validating module configured to generate a similarity score based on an overlap between the plurality of predicted segmentation masks and a plurality of predefined ground-truth masks. The validating module is also configured to validate the accuracy of each of the plurality of predicted segmentation masks based on the similarity score. Moreover, the system includes a generating module configured to generate one or more segmented anomaly regions by aggregating the plurality of predicted segmentation masks based on the validation.
The foregoing general description of the illustrative embodiments and the following detailed description thereof are merely exemplary aspects of the teachings of this disclosure and are not restrictive.
BRIEF DESCRIPTION OF FIGURES
Non-limiting and non-exhaustive examples are described with reference to the following figures.
Figure 1A illustrates an example environment for implementation of a system for performing segmentation of one or more anomaly regions in a plurality of multi-modal three-dimensional (3D) images of an organ, in accordance with an embodiment of the present disclosure;
Figure 1B illustrates various types of multi-modal 3D images, in accordance with an embodiment of the present disclosure;
Figure 2 illustrates a block diagram of the system for performing the segmentation of the one or more anomaly regions in the plurality of multi-modal 3D images of the organ, in accordance with an embodiment of the present disclosure;
Figure 3A illustrates an image before foreground cropping and an image after foreground cropping, in accordance with an embodiment of the present disclosure;
Figure 3B illustrates an intensity histogram before normalization and an intensity histogram after normalization, in accordance with an embodiment of the present disclosure;
Figure 3C illustrates an image before intensity normalization and an image after intensity normalization, in accordance with an embodiment of the present disclosure;
Figure 3D illustrates an image before data augmentation and an image after data augmentation, in accordance with an embodiment of the present disclosure;
Figure 3E illustrates an image before a random spatial crop based on a Region of Interest (ROI) and an image after the random spatial crop based on the ROI, in accordance with an embodiment of the present disclosure;
Figure 4A illustrates a process flow for training, validation, and testing of the system, in accordance with an embodiment of the present disclosure;
Figure 4B illustrates a dataset split configuration, in accordance with an embodiment of the present disclosure;
Figure 5 illustrates an example network architecture of a generator sub-module, in accordance with an embodiment of the present disclosure;
Figure 6 illustrates an example transformer block architecture, in accordance with an embodiment of the present disclosure;
Figure 7 illustrates an example discriminator network, in accordance with an embodiment of the present disclosure;
Figure 8 illustrates a pixel-based image representation versus a patch-based image representation, in accordance with an embodiment of the present disclosure;
Figure 9 illustrates segmentation results, in accordance with an embodiment of the present disclosure;
Figure 10 illustrates a bar chart depicting a Dice score comparison for multiple test samples, in accordance with an embodiment of the present disclosure;
Figure 11 illustrates a box‑whisker plot depicting the overall performance of the system, in accordance with an embodiment of the present disclosure; and
Figure 12 illustrates a performance comparison table depicting Dice Score values for multiple test samples, in accordance with an embodiment of the present disclosure.
DETAILED DESCRIPTION
The following description sets forth exemplary aspects of the present disclosure. It should be recognized, however, that such description is not intended as a limitation on the scope of the present disclosure. Rather, the description also encompasses combinations and modifications to those exemplary aspects described herein.
The present disclosure relates to a system and method for performing segmentation of one or more anomaly regions in a plurality of multi-modal three-dimensional (3D) images of an organ. For this, the present disclosure may utilize a transformer-based generative adversarial network (GAN) architecture including a Shifted Windows-Unet Transformer (Swin-UNETR) generator and a patch-based 3D Convolutional Neural Network (CNN) discriminator in a manner as explained in detail in the forthcoming paragraphs.
Figure 1A illustrates an example environment 100 for implementation of a system 102 for performing segmentation of the one or more anomaly regions in the plurality of multi-modal 3D images of the organ, in accordance with an embodiment of the present disclosure. The environment 100 may correspond to a clinical or research setting where the system 102 may be deployed for automated anomaly segmentation from medical imaging data.
In an embodiment, the system 102 may be configured to receive the plurality of multi-modal 3D images of the organ and generate one or more segmented anomaly regions for clinical diagnosis and treatment planning. In an embodiment, the organ may be the brain, and the one or more anomaly regions may include a tumor. The one or more anomaly sub-regions may include an enhancing anomaly sub-region, an anomaly core sub-region, and a whole anomaly sub-region.
Figure 1B illustrates various types of multi-modal 3D images, in accordance with an embodiment of the present disclosure. The multi-modal 3D images may include a T1-weighted image 104a, a contrast-enhanced T1-weighted (T1-Gd) image 104b, a T2-weighted image 104c, and a Fluid-Attenuated Inversion Recovery (FLAIR) image 104d. Each type of the multi-modal 3D image may provide a distinct information related to organ tissue and potential anomalies. The T1-weighted image 104a may depict brain anatomy, the T1-Gd image 104b may represent regions where blood-brain barrier is disrupted, the T2-weighted image 104c may be sensitive to water content, such as edema, and the FLAIR image 104d may suppress cerebrospinal fluid to better reveal abnormalities near fluid-filled spaces. Using all four types of multi-modal 3D images collectively may enable the system 102 to detect abnormalities with greater accuracy and reliability.
Figure 2 illustrates a block diagram of the system 102 for performing the segmentation of the one or more anomaly regions in the plurality of multi-modal 3D images of the organ, in accordance with an embodiment of the present disclosure. The system 102 may include one or more processors 202, a memory 204, one or more modules 206, and data 208. The system may be explained in the forthcoming paragraphs using the brain as an example of the organ, however, it should be noted that similar explanations are equally applicable to other organs.
In an embodiment, the one or more processor 202 may be operatively coupled to each of the memory 204, and the one or more modules 206. The processor 202 may include specialized processing units such as integrated system (bus) controllers, memory management control units, floating point units, graphics processing units, digital signal processing units, etc. In one embodiment, the processor 202 may include a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or both. The processor 202 may be one or more general processors, Digital Signal Processors (DSPs), application-specific integrated circuits, Field-Programmable Gate Arrays (FPGAs), servers, networks, digital circuits, analog circuits, combinations thereof, or other now known or later developed devices for analyzing and processing data. The processor 202 may execute a software program, such as code generated manually (i.e., programmed) to perform the desired operation. The processor 202 may implement various techniques such as, but not limited to, data extraction, Artificial Intelligence (AI), Machine Learning (ML), Deep Learning (DL), and so forth to achieve the desired objective.
In some embodiments, the memory 204 may be communicatively coupled to the processor 202. The memory 204 may be configured to store data, instructions executable by the processor 202. In one embodiment, the memory 204 may communicate via a bus within the system 102. The memory 204 may include, but is not limited to, a non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media including, but not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like. In one example, the memory 204 may include a cache or random-access memory for the processor 202. In alternative examples, the memory 204 is separate from the processor 202, such as a cache memory of a processor, the system memory, or other memory. The memory 204 may be an external storage device or database for storing data. The memory 204 may be operable to store instructions executable by the processor 202. The functions, acts or tasks illustrated in the figures or described may be performed by the programmed processor 202 for executing the instructions stored in the memory 204. The functions, acts or tasks are independent of the particular type of instructions set, storage media, processor or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro-code and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, and the like. The memory 204 may further include a database to store the data. Further, the memory 204 may include an operating system for performing one or more tasks of the system 102.
In an embodiment, the one or more modules 206, amongst other things, include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement data types. The module(s) 206 may also be implemented as, signal processor(s), state machine(s), logic circuitries, and/or any other device or component that manipulates signals based on operational instructions. Further, the module(s) 206 may be implemented in hardware, instructions executed by a processing unit, or by a combination thereof. The processing unit may comprise a computer, the processor 202, a state machine, a logic array, or any other suitable devices capable of processing instructions. The processing unit can be a general-purpose processor which executes instructions to cause the general-purpose processor to perform the required tasks, or the processing unit can be dedicated to performing the required functions. In another embodiment of the present disclosure, the modules 206 may be machine-readable instructions (software) which, when executed by a processor/processing unit, perform any of the described functionalities.
In an embodiment, the data 208 may serve, amongst other things, as a repository for storing data processed, received, and generated by the one or more modules 206.
In an embodiment, the one or more modules 206 may include a preprocessing module 210, a segmenting module 212, a validating module 214, and a generating module 216.
In an embodiment, the preprocessing module 210 may be configured to receive the plurality of multi-modal 3D images corresponding to the organ. The plurality of the multi-modal 3D images may include at least one of the T1 image, the T1-Gd image, the T2 image, and the FLAIR image. In an embodiment, raw Magnetic Resonance Imaging (MRI) data may be reviewed by radiologists, after which the T1, T1-Gd, T2, and FLAIR images may be extracted and named according to a standard naming convention. The raw data may be converted from a first format (for example, Digital Imaging and Communications in Medicine (DICOM) format) to a second format (for example, Neuroimaging Informatics Technology Initiative (NIfTI) format), and non-brain tissues may be removed using techniques such as skull-stripping. Each image may be resampled to a fixed spatial resolution with dimensions, for example, of 182x218×182 and registered to a linear symmetric atlas to ensure spatial consistency across subjects.
The preprocessing module 210 may also be configured to apply one or more preprocessing operations on the plurality of multi-modal 3D images. The one or more preprocessing operations may include a foreground cropping operation, an intensity normalization operation, a data augmentation operation, and a Region of Interest (ROI)-based random spatial cropping operation. The one or more preprocessing operations are explained in detail in the forthcoming paragraphs in conjunction with Figures 3A-3E.
In an embodiment, the foreground cropping operation may be performed to ensure that image dimensions are compatible with network architecture and to reduce the large amount of background (non-anomaly) regions present in the multi-modal 3D image, which may otherwise lead to class imbalance. As an initial step, the preprocessing module 210 may be configured to compute a tight bounding cube enclosing the brain. Further, the preprocessing module 210 may be configured to compare dimensions of the computed tight bounding cube with a predefined region of interest (ROI). To ensure compatibility with the network architecture, the preprocessing module 210 may be configured to apply padding to adjust the dimensions to the nearest multiple of ROI size.
Figure 3A illustrates an image 302 before foreground cropping and an image 304 after foreground cropping, in accordance with an embodiment of the present disclosure. The foreground cropping operation may remove non-brain tissues and reduce the large amount of background regions present in the images. The image 302 before foreground cropping may show an original MRI scan with surrounding background, while the image 304 after foreground cropping may show cropped brain region with reduced background.
In an embodiment, the intensity normalization operation may be performed to rescale input data (i.e., the plurality of multi-modal 3D images) to a common range, which helps stabilize and accelerate training process. Without the intensity normalization operation, large variations in input values may lead to unstable gradients and slow or unreliable convergence. The intensity normalization operation may reduce the intensity variability across scanners and patients in the MRI scan, allowing the system 102 to focus on meaningful patterns rather than absolute intensity differences, thereby improving training stability and generalization. In some embodiments, the preprocessing module 210 may apply z-score normalization to the input data. The preprocessing module 210 may compute mean and standard deviation of voxel intensities after excluding background voxels. The preprocessing module 210 may normalize each voxel intensity by subtracting the mean and dividing by the standard deviation, according to Equation (1) provided below, where Z denotes a normalized image, X represents an original voxel intensity, μ is the mean of non-background voxel intensities, and σ is the standard deviation of the non-background voxel intensities.
Z=(X-μ)/σ … (1)
Figure 3B illustrates an intensity histogram 306 before normalization and an intensity histogram 308 after normalization, in accordance with an embodiment of the present disclosure. The intensity histogram 306 before normalization shows a highly skewed distribution with voxels concentrated at low intensity values. The intensity histogram 308 after normalization shows a more symmetric, bell-shaped distribution centered around a normalized intensity value, demonstrating the effect of z-score normalization on the input data. Figure 3C illustrates an image 310 before intensity normalization and an image 312 after intensity normalization, in accordance with an embodiment of the present disclosure.
In an embodiment, the data augmentation operation may be applied to increase diversity of training data and improve ability of the system 102 to generalize. Intensity scaling and shifting may be used to simulate variations in image contrast and brightness that arise from differences in scanners and acquisition settings, thereby making the system 102 more robust to intensity variations. In some embodiments, the voxel intensities may be scaled within the range [0.9, 1.1] and shifted within the range [-0.1, 0.1]. Additionally, random mirror flips may be applied along the three spatial axes with a probability of 0.5 to introduce geometric variability and encourage the system 102 to learn orientation-invariant features. The data augmentation may reduce overfitting, promote robust feature learning, and improve performance on previously unseen data.
Figure 3D illustrates an image 314 before data augmentation and an image 316 after data augmentation, in accordance with an embodiment of the present disclosure. The image 314 before data augmentation shows the normalized MRI scan, and the image 316 after data augmentation shows the normalized MRI scan after intensity scaling, shifting, and spatial flipping operations have been applied.
In an embodiment, the ROI-based random spatial cropping operation may be employed to reduce computational cost and enable efficient training on large 3D MRI volumes. The system 102 may be trained on randomly extracted 3D patches biased toward regions containing relevant anatomical or pathological information instead of processing the plurality of multi-modal 3D images simultaneously, which is memory-intensive and computationally inefficient. Thus, the present disclosure improves training efficiency while allowing the system 102 to learn meaningful local spatial context. In some embodiments, a fixed ROI (patch) size of, for example, 128×128×128 may be used for all four types of the multi-modal 3D images. During real-time deployment, the system 102 may process the full image volume in a sliding or sequential patch-wise manner, and patch-level predictions may be subsequently stitched together to generate the one or more segmented anomaly regions (i.e., a final whole-volume segmentation output).
Figure 3E illustrates an image 318 before a random spatial crop based on a Region of Interest (ROI) and an image 320 after the random spatial crop based on the ROI, in accordance with an embodiment of the present disclosure, in accordance with an embodiment of the present disclosure. The images 318 and 320 show the ROI-based random spatial cropping operation, where smaller fixed-size patches are extracted from the larger MRI volume for efficient training.
Referring to Figure 2, the segmenting module 212 may include a generator sub-module 218 and a discriminator sub-module 220. In an embodiment, the segmenting module 212 may correspond to a transformer-based generative adversarial network (GAN). The generator sub-module 218 may be configured to generate a plurality of predicted segmentation masks corresponding to one or more anomaly sub-regions. The one or more anomaly sub-regions may comprise an enhancing anomaly sub-region, an anomaly core sub-region, and a whole anomaly sub-region. In an embodiment, the generator sub-module 218 may correspond to a Shifted Windows-Unet Transformer (Swin-UNETR) generator configured to generate the plurality of predicted segmentation masks corresponding to the one or more anomaly sub-regions based on the plurality of multi-modal 3D images. The discriminator sub-module 220 may be configured to classify the one or more anomaly sub-regions in each of the plurality of predicted segmentation masks into one of a real-class or a synthetic-class based on the plurality of the multi-modal 3D images and the plurality of predicted segmentation masks. In an embodiment, the discriminator sub-module 220 may be a patch-based 3D Convolutional Neural Network (CNN) discriminator. In an adversarial setup, the discriminator sub-module 220 may be used to distinguish between a plurality of predefined ground-truth masks and the plurality of predicted segmentation masks given the plurality of multi-modal 3D images. The generator sub-module 218 may be configured to generate the plurality of predicted segmentation masks such that the discriminator sub-module 220 is unable to distinguish between a predicted segmentation mask and a predefined ground-truth mask.
In an embodiment, the generator sub-module 218 and the discriminator sub-module 220 may be trained based on an unequal-weighted combination of at least two loss functions. The loss functions may represent a spatial-overlap accuracy and a voxel-level accuracy between the plurality of predicted segmentation masks and the plurality of predefined ground-truth masks. The at least two loss functions may include a Dice loss function and a binary-cross entropy (BCE) loss function. The BCE loss function may be used as an objective function to train the discriminator sub-module 220. For training the generator sub-module 218, a weighted combination of BCE Loss and Dice Loss functions may stabilize the training.
In an embodiment, the BCE loss function may be defined based on Equation (2) provided below, where Yi is a ground truth voxel label, Ŷi is a predicted voxel label, and n is the total number of voxels in the predicted segmentation mask. The BCE loss function may measure an agreement between the plurality of predicted segmentation masks and the plurality of predefined ground-truth masks.
L_BCE=-1/n ∑_(i=1)^n▒(Y_iⓜ┤·ⓜlogⓜY┴^_iⓜ+ⓜ(ⓜ1ⓜ-ⓜY_iⓜ)ⓜ·ⓜlogⓜ(ⓜ1ⓜ-ⓜY┴^_iⓜ)) … (2)
In an embodiment, the Dice loss function may be defined based on Equation (3) provided below, where J denotes the number of classes, I denotes the number of voxels, yi,jpred represents the probability of voxel i belonging to class j, yi,jtrue denotes the corresponding predefined ground-truth mask, and ε is a small constant value (set to 10-5) added for numerical stability. The Dice loss function may evaluate a spatial overlap between the plurality of predicted segmentation masks and the plurality of predefined ground-truth masks.
L_Dice=1-2/J×∑_(j=1)^J (∑_(i=1)^I▒〖(y_pred^(i,j)×y_true^(i,j))〗+ε)/(∑_(i=1)^I 〖(y_pred^(i,j))〗^2+∑_(i=1)^I 〖(y_true^(i,j))〗^2+ε) … (3)
In an embodiment, the one or more unequal weights utilized in the unequal-weighted combination of the at least two loss functions may be determined experimentally. The total loss may be formulated using Equation (4) provided below. In some embodiments, α may be empirically found to be 0.8 to give the best results, where α≠0.5. Using equal weights for both losses may not always lead to optimal performance. Therefore, different weight combinations may be used, and an unequally weighted sum of the two losses may yield better optimization and segmentation performance.
Loss_total=α·Loss_Dice+(1-α)·Loss_BCE … (4)
In an embodiment, one or more first parameters of the generator sub-module 218 may be updated based on the classification by the discriminator sub-module 220 and the overlap between the plurality of predicted segmentation masks and the plurality of predefined ground-truth masks. One or more second parameters of the discriminator sub-module 220 may be updated based on the plurality of predicted segmentation masks and the plurality of predefined ground-truth masks. In some embodiments, the generator sub-module 218 may be trained on every batch and the discriminator sub-module 220 may be trained once every three batches. Hence, during each training epoch, the generator sub-module 218 and discriminator sub-module 220 may be trained in a 3:1 ratio. The discriminator sub module 220 may be intentionally configured to be simpler and trained less frequently, since performing image classification is generally less complex than generating segmentation masks. If the discriminator sub module 220 becomes overly accurate, gradient information may fail to propagate to the generator sub module 218, thereby reducing the efficiency of training the generator sub module 218. The generator sub-module 218 and the discriminator sub-module 220 are explained in detail in the forthcoming paragraphs in conjunction with Figures 5-7.
In an embodiment, the validating module 214 may be configured to generate a similarity score based on an overlap between the plurality of predicted segmentation masks and a plurality of predefined ground-truth masks. The validating module 214 may also be configured to validate an accuracy of each of the plurality of predicted segmentation masks based on the similarity score. The similarity score may be determined using a Dice Similarity Coefficient (DSC). The DSC may refer to a metric used to measure the similarity between two binary masks. The DSC may indicate an extent to which the two binary masks overlap, with values ranging from 0 to 1, where 0 indicates no overlap and 1 indicates perfect agreement. The DSC may be defined based on Equation (5) provided below, where ypred is a binary mask of predicted class, and ytrue is the binary mask of ground-truth class.
DSC=(2×|y_pred∩y_true |)/(|y_pred |+|y_true |) … (5)
In an embodiment, the generating module 216 may be configured to generate one or more segmented anomaly regions by aggregating the plurality of predicted segmentation masks based on the validation. In some embodiments, the one or more segmented anomaly regions may be used to support clinical diagnosis and assist in treatment planning by providing precise information about the location of an anomaly, a spatial extent of the anomaly, and sub-regions of the anomaly within the organ. The segmentation of the one or more anomaly regions may separate key anomaly components, such as anomaly core and surrounding regions, in a consistent and stable manner, thereby enabling reliable quantitative analysis across cases.
Figure 4A illustrates a process flow 400 for training, validation, and testing of the system 102, in accordance with an embodiment of the present disclosure. Figure 4B illustrates a dataset split configuration 410, in accordance with an embodiment of the present disclosure. For clarity, Figures 4A and 4B have been explained together in the forthcoming paragraphs. The process flow 400 may include a data preprocessing block 402, a training block 404, and an inference block 406.
Referring to Figure 4A, dataset 401 may be received as an input at the data preprocessing block 402. The dataset 401 may include multi-modal MRI brain scan data from multiple patients. At the data preprocessing block 402, the preprocessing module 210 may apply the one or more preprocessing operations to prepare the dataset 401 for model training. The one or more preprocessing operations have already been explained above in conjunction with Figure 2 and, therefore, description thereof have been omitted in respect of Figure 4A for the sake of brevity.
Referring to Figure 4B, the dataset 401 may be partitioned for training, validation, and testing. In an example embodiment, the dataset 401 may include approximately 1251 data samples, with each data sample corresponding to a multi-modality MRI brain scan image of a patient. The dataset 401 may be divided into six parts after random shuffling of multi-modality MRI brain scan images. Five parts (train and validation dataset 403A) may be used for training five different segmenting modules using a 5-fold cross-validation strategy. The sixth part (testing dataset 403B) may be kept separate and used only for testing the accuracy of the trained segmenting modules. In some aspects, approximately 208 samples may be randomly separated as the test set to evaluate the segmentation performance of the system 102.
Referring to Figure 4A, the training block 404 may include the generator sub-module 218 and the discriminator sub-module 220 configured in an adversarial training setup. During training, the discriminator sub-module 220 may be frozen while the generator sub-module 218 may be trained using the BCE Loss, the Dice Loss, and a weighted loss combination. Further, the generator sub-module 218 may be frozen while the discriminator sub-module 220 may be trained using the BCE Loss. Such an alternating training scheme between the generator sub-module 218 and discriminator sub-module 220 may represent the adversarial training methodology where the generator sub-module 218 and discriminator sub-module 220 are trained in opposition to each other. The inference block 306 may receive test data and generate the plurality of predicted segmentation masks as output using the trained generator sub-module 218.
Figure 5 illustrates an example network architecture 500 of the generator sub-module 218, in accordance with an embodiment of the present disclosure. The generator sub-module 218 may be implemented based on a Swin-UNETR architecture, which operates as an image encoder-decoder network, utilizes visual transformers to capture global contextual information, and requires comparatively lower memory and computational complexity.
The network architecture 500 may include partitioning each of the plurality of multi-modal 3D images into one or more patches and projecting the one or more patches into a higher-dimensional embedding space, followed by four hierarchical layers. Each layer may include an encoder block 502 and a decoder block 504. The encoder block 502 may process each multi-modal 3D image using two different Swin transformer blocks. At each layer, the encoder block 502 may reduce spatial dimensions by a factor of two and increase channel dimensions by a factor of two. An input may have dimensions D×H×W×4, representing the plurality of multi-modal 3D images. After patch partitioning and linear embedding, one or more features may be projected to dimensions D/2×H/2×W/2×48. The encoder block 502 may progressively reduce the spatial dimensions while increasing channel depth, with feature maps transitioning through dimensions such as D/4×H/4×W/4×96, D/8×H/8×W/8×192, D/16×H/16×W/16×384, and D/32×H/32×W/32×768 at the bottleneck.
The decoder block 504 may include one or more residual connections from the corresponding encoder block 502, a convolutional layer, and an upward deconvolution block configured to upsample the spatial dimensions by a factor of two while reducing the channel dimensions by a factor of two. The decoder block 504 may progressively restore spatial resolution while decreasing the channel depth. A final output may generate the plurality of predicted segmentation masks with dimensions D×H×W×3, corresponding to the three anomaly sub-regions, namely the enhancing anomaly sub-region, the anomaly core sub-region, and the whole anomaly sub-region.
The residual connections may be defined using Equation (6) provided below, where ‘Norm’ denotes Instance Normalization, Conv3×3 denotes 3D Convolution, LeReLU denotes the Leaky Rectified Linear Unit non-linearity, x represents the input, y1 denotes an output after a first layer, and y2 denotes an output after a second layer.
y_1=LeReLU(Norm(Conv_(3×3) (x))),y_2=LeReLU(y_1+Norm(Conv_(3×3) (y_1))) … (6)
The deconvolutional block may be defined using Equation (7) provided below, where x1 denotes an input to the deconvolutional block, x2 denotes a residual input from a layer above the current layer, and y1 denotes an output.
y_1=LeReLU(Norm(Conv_(3×3) (Deconv(x_1)+x_2))) … (7)
Figure 6 illustrates an example transformer block architecture 600, in accordance with an embodiment of the present disclosure. The transformer block architecture 600 may include a first Swin transformer block 602-1 and a second Swin transformer block 602-2. The first Swin transformer block 602-1 may include a first layer normalization layer 604-1, a windowed multi-head self-attention (W-MSA) module 606-1, a second layer normalization layer 608-1, and a multi-layer perceptron (MLP) block 610-1. The second Swin transformer block 602-2 may include a third layer normalization layer 604-2, a shifted window multi-head self-attention (SW-MSA) module 606-2, a fourth layer normalization layer 608-2, and a multi-layer perceptron (MLP) block 610-2.
The first Swin transformer block 602-1 may implement the W-MSA module 606-1, followed by the MLP block 610-1 and the one or more residual connections. The second Swin transformer block 602-2 may implement the SW-MSA 606-2. The Swin Transformer blocks 602-1 and 602-2 may be defined using Equation (8) provided below, where WMSA and SWMSA may denote the regular and shifted window-based MSA modules 606-1 and 606-2, respectively. LN and MLP represent Layer Normalization and the Multi-Layer Perceptron, respectively.
y┴^^l=WMSA(LN(y^(l-1)))+y^(l-1),y^l=MLP(LN(y┴^^l))+y┴^^l,y┴^^(l+1)=SWMSA(LN(y^l))+y^l,y^(l+1)=MLP(LN(y┴^^(l+1)))+y┴^^(l+1) … (8)
At layer l, input features may be first normalized and processed through the WMSA module 606-1, and a resulting output may be added back to the input features through a residual connection. This may be followed by the MLP block 610-1 with another residual connection. To enable interaction flow across neighboring windows, a subsequent layer l+1 may apply SW-MSA, which introduces shifted window partitions. A 3D cyclic shifting mechanism may be employed to efficiently implement the shifted window operation.
Self-attention operation may be defined using Equation (9) provided below, where Q, K, and V denote query, key, and value matrices, respectively, and dh denotes the dimensionality of query and key vectors.
Attention(Q,K,V)=Softmax((QK^T)/√(d_h ))V … (9)
Figure 7 illustrates an example discriminator network 700, in accordance with an embodiment of the present disclosure. The discriminator network 700 may include a first convolutional layer 702-1, a second convolutional layer 702-2, and a third convolutional layer 702-3. The discriminator network 700 may be implemented as a 3D convolutional neural network (CNN) configured to downsample the multi-modal 3D image into reduced spatial dimensions and classify each region as belonging to the real-class or the fake-class.
In an embodiment, the first convolutional layer 702-1 may receive the plurality of multi-modal 3D images along with, for example, three predicted segmentation masks (D×H×W×7). The first convolutional layer 702 1 may reduce the spatial dimensions by a factor of two and increase the number of channels to 32. The second convolutional layer 702 2 may further reduce the spatial dimensions by a factor of two and expand the number of channels to 64. The third convolutional layer 702 3 may map the 64 channels to a single output channel, thereby classifying each pixel in the downsampled representation (corresponding to a patch in the original image) as real or fake.
The discriminator network 700 may be defined using Equation (10) provided below, where x denotes the input image, y1 is an output after the first convolutional layer 702-1, y2 is an output after the second convolutional layer 702-2, and y3 is an output after the third convolutional layer 702-3. Conv4×4 denotes the 3D convolution operation with kernel size of 4, stride of 2, and padding of 1. LReLU denotes the Leaky Rectified Linear Unit non-linearity after convolution operation.
y_1=LReLU(Conv_(4×4) (x)),y_2=LReLU(Conv_(4×4) (y_1)),y_3=Conv_(4×4) (y_2) … (10)
Figure 8 illustrates a pixel-based image representation versus a patch-based image representation, in accordance with an embodiment of the present disclosure. The system 102 may employ a patch-based discriminator to achieve improved computational efficiency and reduced redundancy as neighbouring pixels typically exhibit similar characteristics. The pixel level image representation 802 may depict classification on an individual pixel basis, and the patch level image representation 804 may illustrate pixels grouped into patches. The processed patch based image 806 may further demonstrate the patch based classification approach implemented by the discriminator sub module 220.
Figure 9 illustrates segmentation results, in accordance with an embodiment of the present disclosure. Figure 9 may include a first input multi-modal 3D image 902, a predefined ground-truth mask 904, a first segmented anomaly region 906, a second input multi-modal 3D image 908, a second ground-truth mask 910, and a second segmented anomaly region 912. The segmented anomaly regions may demonstrate the performance of the system 102 in accurately identifying and classifying distinct anomaly components from the multi-modal 3D images. The segmented anomaly regions 906, 912 show close visual correspondence with the corresponding ground-truth masks 904, 910, with the system 102 successfully identifying and delineating the different anomaly sub-regions including the enhancing anomaly sub-region, the anomaly core sub-region, and the whole anomaly sub-region.
Figure 10 illustrates a bar chart 1000 depicting a Dice score comparison for multiple test samples, in accordance with an embodiment of the present disclosure. The bar chart 1000 represents the DSC performance of the system 102 across a plurality of test cases. For each test case, three bars may be presented, corresponding to the three anomaly sub‑regions, namely the enhancing anomaly sub-region, the anomaly core sub-region, and the whole anomaly sub-region. The DSC values may consistently fall within a high range, thereby demonstrating strong segmentation performance across all evaluated test samples.
Figure 11 illustrates a box‑whisker plot 1100 depicting the overall performance of the system 102, in accordance with an embodiment of the present disclosure. The box‑whisker plot 1100 may depict the distribution of Dice Score values across test samples for the three anomaly sub‑regions. The whole anomaly sub-region may exhibit the highest and most consistent performance among the three sub-regions, characterized by a higher median DSC and minimal variance. The enhancing anomaly sub-region may demonstrate the lowest median Dice Score and a higher degree of variability. The anomaly core sub-region may exhibit greater variability than the whole anomaly sub-region however fewer outliers relative to the enhancing anomaly sub-region.
Figure 12 illustrates a performance comparison table 1200 depicting Dice Score values for multiple test samples, in accordance with an embodiment of the present disclosure. In case the anomaly is the tumor, the performance comparison table 1200 may present the DSC values for the Tumor Core (TC), Whole Tumor (WT), and Enhancing Tumor (ET) across the plurality of test cases. The system 102 may achieve a mean DSC of 0.91489 for overall anomaly segmentation. Specifically, the system 102 may achieve a DSC of 0.92289 for the tumor core (TC), 0.93232 for the whole tumor (WT), and 0.88946 for the enhancing tumor (ET) component segmentation.
In some embodiments, the system 102 (or, the segmenting module 212) may be trained on a high‑performance computing platform utilizing a graphical processing unit (GPU). The training may be performed for approximately 500 epochs (100 epochs per fold) using a 5‑fold cross‑validation strategy. The system 102 may employ a hidden feature size of 48 dimensions. For a single forward pass, the system 102 may utilize approximately 3132 Giga Floating Point Operations per Second (GFLOPs) and may include a total of about 60 million parameters. The inference overlap ratio may be set to 0.25 for the validation dataset and 0.6 for the testing dataset. The system 102 may be validated every 10 epochs to evaluate performance and monitor potential overfitting.
During training, two optimizers may be used - one for the discriminator sub‑module 220 (initialized at 2e‑4) and one for the generator sub‑module 218 (initialized at 1e‑4). Additionally, two learning‑rate schedulers may be employed to reduce the learning rate when the accuracy or loss values begin to plateau. For the generator sub‑module 218, the learning rate may be reduced upon accuracy plateau, whereas for the discriminator sub‑module 220, the learning rate may be reduced upon loss plateau.
For testing, one or more overlapping patches may be sequentially sampled from an input multi-modal 3D image, as the full image may exceed the input size of the segmenting module 212. The degree of overlap may be controlled using an overlapping ratio. The model outputs for the patches may then be stitched together to generate a predicted segmentation mask with dimensions matching those of the input multi-modal 3D image. The DSC may be determined using the Equation (5) to evaluate the performance of the system 102 on the testing dataset.
The present disclosure provides various advantages. The present disclosure facilitates consistent and reproducible anomaly delineation across imaging centers and hospitals, thereby reducing manual intervention and inter‑observer variability in clinical diagnosis. The present disclosure further provides efficient and accurate performance across multiple MRI modalities, enabling a comprehensive representation of anomaly characteristics. The architectural design of the present disclosure incorporates minimal data augmentation and a simplified configuration, resulting in lower computational complexity and faster processing relative to conventional deep‑learning models. Additionally, the underlying framework of present disclosure is adapted for other medical imaging applications, including segmentation of different pathological regions or integration into multi‑modality diagnostic systems.
In some embodiments, the present disclosure offers several advantages over existing solutions. The present disclosure enables consistent and reproducible anomaly delineation at clinical facilities while reducing manual effort and inter‑observer variability. The present disclosure achieves high accuracy across multiple MRI modalities and delivers a comprehensive characterization of anomaly regions. The simplified architecture, combined with limited data augmentation, results in reduced computational overhead and faster inference relative to traditional deep‑learning methods. Moreover, the present disclosure achieves higher accuracy compared to existing approaches through its robust combination of a transformer‑based generator and a patch‑based discriminator. The present disclosure further exhibits efficient use of computational resources, maintaining near‑maximum GPU utilization and high memory occupancy throughout training.
A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.
, Claims:1. A system (102) for performing segmentation of one or more anomaly regions in a plurality of multi-modal three-dimensional (3D) images of an organ, the system (102) comprising:
a preprocessing module (210) configured to:
receive the plurality of multi-modal three-dimensional (3D) images corresponding to the organ, and
apply one or more preprocessing operations on the plurality of multi-modal 3D images;
a segmenting module (212) comprising:
a generator sub-module (218) configured to generate a plurality of predicted segmentation masks corresponding to one or more anomaly sub-regions, and
a discriminator sub-module (220) configured to classify one or more anomaly sub-regions in each of the plurality of predicted segmentation masks into one of a real-class or a synthetic-class based on the plurality of the multi-modal 3D images and the plurality of predicted segmentation masks;
a validating module (214) configured to:
generate a similarity score based on an overlap between the plurality of predicted segmentation masks and a plurality of predefined ground truth masks, and
validate an accuracy of each of the plurality of predicted segmentation masks based on the similarity score; and
a generating module (216) configured to generate one or more segmented anomaly regions by aggregating the plurality of predicted segmentation masks based on the validation.

2. The system (102) as claimed in claim 1, wherein the segmenting module (212) corresponds to a transformer based generative adversarial network (GAN), the generator sub-module (218) is a Shifted Windows-Unet Transformer (Swin-UNETR) generator, and the discriminator sub-module (220) is a patch-based 3D Convolutional Neural Network (CNN) discriminator.

3. The system (102) as claimed in claim 1, wherein the generator sub-module (218) and the discriminator sub-module (220) are trained based on an unequal-weighted combination of at least two loss functions, wherein the loss functions represent a spatial-overlap accuracy and a voxel-level accuracy between the plurality of predicted segmentation masks and the plurality of predefined ground truth masks.

4. The system (102) as claimed in claim 3, wherein the at least two loss functions comprises a Dice loss function and a binary-cross entropy (BCE) loss.

5. The system (102) as claimed in claim 3, wherein one or more unequal weights utilized in the unequal-weighted combination of the at least two loss functions are determined experimentally.

6. The system (102) as claimed in claim 1, wherein one or more first parameters of the generator sub-module (218) are updated based on the classification by the discriminator sub-module (220) and the overlap between the plurality of predicted segmentation masks and the plurality of predefined ground truth masks, and wherein one or more second parameters of the discriminator sub-module (220) are updated based on the plurality of predicted segmentation masks and the plurality of predefined ground truth masks.

7. The system (102) as claimed in claim 1, wherein the one or more preprocessing operations comprise a foreground cropping operation, an intensity normalization operation, a data augmentation operation, and a Region of Interest (ROI) based random spatial cropping operation.

8. The system (102) as claimed in claim 1, wherein the plurality of the multi-modal 3D images comprises at least one of T1-weighted (T1) image, a contrast enhanced T1-weighted (T1-Gd) image, a T2-weighted (T2) image, and a Fluid-Attenuated Inversion Recovery (FLAIR) image.

9. The system (102) as claimed in claim 1, wherein the one or more anomaly sub-regions comprise an enhancing anomaly sub-region, an anomaly core sub-region, and a whole anomaly sub-region.

10. The system (102) as claimed in claim 1, wherein the organ is the brain, and the one or more anomaly regions comprise a tumor.