Description:FIELD OF INVENTION
[0001] The present disclosure relates to ophthalmic analysis, and more particularly to a system and a method for ophthalmic analysis using a vision-language based model.

BACKGROUND
[0002] Current ophthalmic diagnostic methods utilize deep learning. However, they often lack interpretability and thereby limiting clinical acceptance. Traditional explainability methods such as Grad-CAM are coarse and lack textual grounding. Existing Vision-language-based models (VLMs) align images and textual inputs but have not been optimized specifically for ophthalmic diagnostics, leaving a critical gap in interactive and interpretable Artificial Intelligence (AI)-driven disease assessment.
[0003] Regulatory-cleared vision-only Convolutional Neural Networks (CNNs), such as IDx-DR and EyeArt, are single-task classifiers that excel at flagging referable diabetic retinopathy but output little more than a binary “refer/don't-refer” decision. These systems generally operate on fundus photographs, require separate workflows or models for Optical Coherence Tomography (OCT), and provide limited localization or textual rationale, leaving clinicians to trust a black-box scores without understanding the underlying reasoning.
[0004] Post-hoc explainability overlays, including Grad-CAM, Guided Back-propagation, and Integrated Gradients, are widely bolted onto retinal CNNs to visualize “important” pixels. However, these methods yield low-resolution, class-level heat-maps that often spill into irrelevant regions. They are not tied to specific clinical terms, may contradict textual reports, and rarely satisfy audit requirements for biomarker-level visual evidence.
[0005] Foundation Vision-Language Models adapted to medicine, such as EyeCLIP, MedCLIP, and LLaVA-Med, align images and text for retrieval or visual question answering tasks. However, they are not optimized for ophthalmic disease grading. These models may return a freeform caption or a coarse attention map, often re-encode the image for every follow-up question, and lack token-grounded region masks, clinical coding, quantitative biomarker metrics, or Electronic Medical Record (EMR) integration.
[0006] Uncertainty-aware retinal AI systems, including open-set detectors and ensemble confidence scoring approaches, often focus on a single scalar confidence or anomaly score for the whole image. Users cannot reliably see where uncertainty concentrates spatially, nor is the information consistently propagated into downstream clinical reports, limiting its practical safety value.
[0007] Prompt-guidance and visual-prompt engineering frameworks use prompts to steer models toward pathology regions, yet often still rely on coarse Grad-CAM-like outputs, omit token-grounded biomarker localization, and stop at research metrics without mapping findings to International Classification of Diseases, Tenth Revision (ICD-10) or Systematized Nomenclature of Medicine (SNOMED) terminology or exporting Fast Healthcare Interoperability Resources (FHIR) Diagnostic Reports with Health Level Seven (HL7) Observation entries.
[0008] Therefore, in view of the above-mentioned problems, it is advantageous to provide an improved system and method that may overcome or at least mitigate the above-mentioned problems and limitations.

SUMMARY
[0009] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0010] According to an aspect of the present disclosure, a method for ophthalmic analysis using a vision-language-based model is disclosed. The method includes serializing a set of multimodal ophthalmic inputs into a multimodal representation comprising a set of modality tokens using a Unified Multimodal Gateway. The set of multimodal ophthalmic inputs comprises ophthalmic image data and associated metadata. Further, the method includes generating fused embeddings comprising one or more global classification tokens and lesion-level visual patch tokens based on encoding the multimodal representation, using a multimodal encoder. Furthermore, the method includes decoding, in a single forward pass from the fused embeddings, a suggested output, a user-readable biomarker caption output, and a saliency–uncertainty output comprising a saliency map co-registered with per-pixel uncertainty. Hereinafter, the term output is used in this disclosure to mean suggested output, for the sake of brevity. Moreover, the method includes generating token-grounded region masks for each token in a set of biomarker tokens appearing in the user-readable biomarker caption output with corresponding image regions using a Token-to-Region Alignment (T2R-A) mechanism. Further, the method includes enforcing Tri-Output Consistency by penalizing, resolving, or flagging disagreement among the output, the user-readable biomarker caption output, and the saliency-uncertainty output. Furthermore, the method includes deriving instance-level biomarker masks and quantitative biomarker metrics based on performing uncertainty-gated biomarker analytics by fusing the saliency-uncertainty output with the token-grounded region masks for each biomarker token. The instance-level biomarker masks correspond to biomarker tokens for which the token-grounded region masks are generated for each token. Further, the method includes generating a standards-aware clinical report comprising coded clinical terminology and provenance information based on the output, the user-readable biomarker caption output, the instance-level biomarker masks for each biomarker token, the saliency-uncertainty output, and the quantitative biomarker metrics.
[0011] According to another aspect of the present disclosure, a system for ophthalmic analysis using a vision-language-based model is disclosed. The system includes a Unified Multimodal Gateway, a multimodal encoder, a multi-headed decoder, a Token-to-Region Alignment (T2R-A) module, a loss computation module, a biomarker engine, and a standards-aware output layer. The Unified Multimodal Gateway is configured to ingest multimodal ophthalmic inputs and serialize ophthalmic image data and associated metadata into a common multimodal representation including a set of modality tokens. The multimodal encoder is configured to encode the multimodal representation and generate fused embeddings comprising one or more global classification tokens and lesion-level visual patch tokens. The multi-headed decoder is configured to decode, in a single forward pass from the fused embeddings, an output, a user-readable biomarker caption output, and a saliency–uncertainty output comprising a saliency map co-registered with per-pixel uncertainty. The Token-to-Region Alignment (T2R-A) module is configured to generate token-grounded region masks for each token that align a set of biomarker tokens in the biomarker caption output with corresponding image regions. The loss computation module is configured to enforce Tri-Output Consistency among the output, the biomarker caption output, and the saliency-uncertainty output. The biomarker engine is configured to perform uncertainty-gated biomarker analytics by fusing the saliency-uncertainty output with the token-grounded region masks for each biomarker token to derive instance-level biomarker masks and quantitative biomarker metrics, wherein the instance-level biomarker masks correspond to biomarker tokens for which the token-grounded region masks are generated for each token. The standards-aware output layer is configured to generate a clinical report comprising coded clinical terminology and provenance information based on the decoded outputs.
[0012] To further clarify the advantages and features of the disclosed system and method, a more particular description of the disclosed system and method will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawing. It is to be appreciated that these drawings depict only typical embodiments of the disclosed system and method and are therefore not to be considered limiting its scope. The disclosed system and method will be described and explained with additional specificity and detail with the accompanying drawings.

BRIEF DESCRIPTION OF FIGURES
[0013] The foregoing and other features of embodiments will become more apparent from the following detailed description of embodiments when read in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements.
[0014] Figure 1 illustrates a block diagram depicting an environment for implementation of a system for ophthalmic analysis using a vision-language-based model, in accordance with an embodiment of the present disclosure;
[0015] Figure 2 illustrates a block diagram of the system of Figure 1, in accordance with an embodiment of the present disclosure;
[0016] Figure 3 illustrates a block diagram of a system for ophthalmic analysis using the vision-language-based model, in accordance with an embodiment of the present disclosure;
[0017] Figure 4 illustrates an enhanced operational view of the system for ophthalmic analysis, in accordance with an embodiment of the present disclosure;
[0018] Figure 5 illustrates a detailed block diagram of a biomarker engine, in accordance with an embodiment of the present disclosure;
[0019] Figure 6 illustrates a block diagram of a Follow-up Chat Interface for servicing follow-up clinician prompts, in accordance with an embodiment of the present disclosure;
[0020] Figure 7 illustrates a user interface flow diagram for Image Only mode, in accordance with an embodiment of the present disclosure;
[0021] Figure 8 illustrates a user interface flow diagram for Image and Text mode showing interaction between a user and an AI Studio system, in accordance with an embodiment of the present disclosure; and
[0022] Figure 9 illustrates a flowchart depicting a method 900 for ophthalmic analysis using the vision-language-based model, in accordance with an embodiment of the present disclosure.
[0023] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the present disclosure. Furthermore, in terms of the construction of the circuit, one or more components of the circuit may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

DETAILED DESCRIPTION
[0024] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the various embodiments and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the present disclosure is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the present disclosure as illustrated therein being contemplated as would normally occur to one skilled in the art to which the present disclosure relates.
[0025] It will be understood by those skilled in the art that the foregoing general description and the following detailed description are explanatory of the present disclosure and are not intended to be restrictive thereof.
[0026] Whether or not a certain feature or element was limited to being used only once, it may still be referred to as “one or more features” or “one or more elements” or “at least one feature” or “at least one element.” Furthermore, the use of the terms “one or more” or “at least one” feature or element do not preclude there being none of that feature or element, unless otherwise specified by limiting language including, but not limited to, “there needs to be one or more…” or “one or more elements is required.”
[0027] Reference is made herein to some “embodiments.” It should be understood that an embodiment is an example of a possible implementation of any features and/or elements of the present disclosure. Some embodiments have been described for the purpose of explaining one or more of the potential ways in which the specific features and/or elements of the proposed disclosure fulfil the requirements of uniqueness, utility, and non-obviousness.
[0028] Use of the phrases and/or terms including, but not limited to, “a first embodiment,” “a further embodiment,” “an alternate embodiment,” “one embodiment,” “an embodiment,” “multiple embodiments,” “some embodiments,” “other embodiments,” “further embodiment”, “furthermore embodiment”, “additional embodiment” or other variants thereof do not necessarily refer to the same embodiments. Unless otherwise specified, one or more particular features and/or elements described in connection with one or more embodiments may be found in one embodiment or may be found in more than one embodiment, or may be found in all embodiments, or may be found in no embodiments. Although one or more features and/or elements may be described herein in the context of only a single embodiment, or in the context of more than one embodiment, or in the context of all embodiments, the features and/or elements may instead be provided separately or in any appropriate combination or not at all. Conversely, any features and/or elements described in the context of separate embodiments may alternatively be realized as existing together in the context of a single embodiment.
[0029] Any particular and all details set forth herein are used in the context of some embodiments and therefore should not necessarily be taken as limiting factors to the proposed disclosure.
[0030] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises a list of steps does not include only those steps but may include other steps not expressly listed or inherent to such process or method. Similarly, one or more devices or sub-systems or elements or structures or components proceeded by “comprises... a” does not, without more constraints, preclude the existence of other devices or other sub-systems or other elements or other structures or other components or additional devices or additional sub-systems or additional elements or additional structures or additional components.
[0031] Embodiments of the present disclosure will be described below in detail with reference to the accompanying drawings. For enhancing clarity and understanding, certain terms used throughout the present disclosure are defined herein.
[0032] It is an object of the present disclosure to provide a method and system that overcomes the problems and limitations of ophthalmic analysis.
[0033] It is another object of the present disclosure to provide a system and method for ophthalmic analysis using a vision-language-based model that generates token-grounded region masks corresponding to biomarker tokens appearing in a user-readable biomarker caption, thereby providing audit-ready visual evidence that may be stored and reviewed.
[0034] It is another object of the present disclosure to provide a system and method that enables real-time interactive dialogue by reusing cached fused embeddings, cached lesion-level patch tokens, and cached token-grounded region masks, allowing follow-up prompts to be answered rapidly without complete re-encoding of the ophthalmic input.
[0035] It is another object of the present disclosure to provide a Unified Multimodal Gateway that harmonizes fundus images, optional Optical Coherence Tomography (OCT) B-scans or volumes, free-text clinician prompts, and metadata or context into a shared vision-language-based pipeline.
[0036] It is another object of the present disclosure to propagate aleatoric and epistemic uncertainty into saliency-uncertainty outputs and evidence overlays, allowing ambiguous regions to be visualized and helping clinicians judge reliability region by region.
[0037] It is another object of the present disclosure to derive instance-level biomarker masks and quantitative biomarker metrics by combining token-grounded region masks with saliency and uncertainty information, and to translate these metrics into disease-specific severity grades using ontology-driven rules.
[0038] Figure 1 illustrates a block diagram depicting an environment 100 for implementation of a system 106 for ophthalmic analysis using a vision-language-based model, in accordance with an embodiment of the present disclosure. The environment 100 includes a set of multimodal inputs 102 and a set of outputs 104. The set of multimodal inputs 102 and the set of outputs are communicably coupled to the system 106.
[0039] In an embodiment, the set of multimodal inputs 102 may include a Fundus Image 108, an OCT B-scan or Volume 110, a Clinician Prompt 112, and Metadata or Context 114. The Fundus Image 108 may be a color image of the back of the eye, including retina, optic disc, vessels, macula, and foveal region. The OCT B-scan or Volume 110 may be a cross-sectional retinal imaging data comprising a single OCT slice or a stack of OCT slices representing a three-dimensional retinal structure. The Clinician Prompt 112 may be a free-text request from a clinician, such as “Highlight exudates and measure area”. The Metadata or Context 114 may include additional contextual information associated with the ophthalmic examination. In a non-limiting example, the additional contextual information may include age, sex, gender, medical history, examination date, imaging-device information, acquisition parameters, laterality, clinical indication, prior examination findings, or other subject-specific or examination-specific information relevant to ophthalmic analysis.
[0040] In an embodiment, the set of outputs 104 may include a Standards-Aware Clinical Report 116 and a Follow-up Chat Interface 118. The Standards-Aware Clinical Report 116 may be a FHIR Diagnostic Report comprising coded clinical terminology, HL7 Observation entries, ICD-10 codes, SNOMED terminology, calibrated probabilities, quantitative biomarker metrics, clinician-readable captions, token-grounded region masks, evidence overlays, and provenance information. The Follow-up Chat Interface 118 may have a bidirectional connection with the system 106, in which an arrow points from the system 106 to the Follow-up Chat Interface 118, and a return arrow labeled “follow-up prompt/query” points back from the Follow-up Chat Interface 118 to the system 106, indicating an interactive dialogue capability where clinicians may submit follow-up queries and receive responses without requiring complete reprocessing of the original inputs.
[0041] In an embodiment, the system 106 may be implemented as a cloud-based server, an on-premise server, an edge computing device, or a combination thereof. The system 106 may include one or more hardware accelerators, such as high-memory Graphics Processing Units (GPUs) including NVIDIA H100, NVIDIA A100, AMD MI300, Tensor Processing Unit (TPU) v5e, or equivalent accelerators supporting mixed-precision training and inference.
[0042] Figure 2 illustrates a block diagram of the system 106 of Figure 1, in accordance with an embodiment of the present disclosure.
[0043] In an embodiment, the system 106 may include, but is not limited to, at least one processor 202 (alternatively referred hereinafter as a processing unit 202 or the processor 202), a memory 204, one or more modules 206, and a data unit 208. The one or more modules 206 and the memory 204 may be coupled to the processor 202.
[0044] The system 106, in accordance with alternative embodiments, may be configured to operate with one or more alternative sensor configurations, platform configurations, deployment architectures, and communication technologies, depending on an operational environment and application requirements.
[0045] In an embodiment, the system 106 may comprise heterogeneous configurations of components of the vision–language-based ophthalmic analysis system, including one or more input acquisition modules configured for capturing multimodal ophthalmic data and one or more processing modules configured for performing multimodal encoding and inference. The system 106 may comprise variants of imaging modalities including fundus imaging systems, OCT systems, and slit-lamp imaging devices, configured for generating ophthalmic image data under different clinical conditions. Further, the system 106 may comprise lightweight deployment configurations configured for point-of-care or portable diagnostic settings, and high-performance configurations configured for hospital or cloud-assisted environments. Additionally, the system 106 may include specialized processing modules configured for biomarker localization, saliency-uncertainty estimation, and token-to-region alignment across diverse ophthalmic imaging scenarios.
[0046] In an embodiment, the system 106 may employ one or more vision-language-based model for processing ophthalmic image data. Further, the one or more vision-language-based model may operate independently, sequentially, or in an ensemble configuration.
[0047] In an embodiment, the system 106 may comprise an edge-enabled architecture configured to perform inference locally at clinical devices or imaging systems for latency-sensitive ophthalmic assessment. Alternatively, the system 106 may comprise a cloud-based processing framework configured to offload computationally intensive multimodal encoding and reporting tasks to remote servers. Further, the system 106 may comprise a hierarchical architecture including one or more intermediate processing nodes configured to coordinate data flow, perform partial inference, and synchronize outputs with a central system for generating standardized clinical reports. Such an architecture may facilitate scalable deployment across distributed clinical environments while maintaining consistency in outputs, biomarker captions, and saliency-uncertainty estimations.
[0048] In an embodiment, the functionalities of the system 106 described in connection with Figure 1 may be executed by the at least one processor 202 in conjunction with the memory 204 and the one or more modules 206.
[0049] The processor 202 may be a single processing unit or several units, all of which could include multiple computing units. The processor 202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 202 is adapted to fetch and execute computer-readable instructions and data stored in the memory 204. At this time, one or a plurality of processors may be a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or an AI-dedicated processor such as a neural processing unit (NPU). One or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (AI) model stored in the non-volatile memory and the volatile memory.
[0050] The memory 204 may include any non-transitory computer-readable medium known in the art including, for example, volatile memory, such as static random-access memory (SRAM) and dynamic random-access memory (DRAM), and/or non-volatile memory, such as read-only memory (ROM), erasable programmable ROM, electrically erasable PROM (EEPROM), flash memories, hard disks, optical disks, and magnetic tapes. The predefined configuration may be stored in the memory 204.
[0051] The one or more modules 206, among other things, include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement data types. The one or more modules 206 may also be implemented as signal processor(s), state machine(s), logic circuitries, or any other device or component that manipulates signals based on operational instructions or a combination thereof.
[0052] Further, the one or more modules 206 may be implemented in hardware, instructions executed by a processing unit, or by a combination thereof. The processing unit may comprise a computer, a processor, a state machine, a logic array, or any other suitable device capable of processing instructions. The processing unit may be a general-purpose processor that executes instructions to cause the general-purpose processor to perform the required tasks, or the processing unit may be dedicated to performing the required functions. In another embodiment of the present disclosure, the processor 202, via the one or more modules 206, is configured to execute machine-readable instructions (software) that perform the working of the system 106 within the scope of the present disclosure as described in forthcoming paragraphs.
[0053] In an embodiment, the data unit 208 serves, among other things, as a repository for storing data processed, received, and generated by the one or more modules 206.
[0054] A detailed working and explanation of the system 106 will be provided in the forthcoming paragraphs of Figure 3-8. The reference numerals are kept the same in the disclosure wherever applicable for ease of explanation.
[0055] Figure 3 illustrates a block diagram of a system 106 for ophthalmic analysis using the vision-language-based model, in accordance with an embodiment of the present disclosure. The system 106 includes a Unified Multimodal Gateway 302, a multimodal encoder 304, a multi-headed decoder 306, a Token-to-Region Alignment (T2R-A) module 314, a biomarker engine 316, a latent cache 318, a Standards-Aware Clinical Report generator 116, and a Follow-up Chat Interface 118.
[0056] In an embodiment, the system 106 may be configured to ingest multimodal ophthalmic inputs and serialize ophthalmic image data and associated metadata into a common multimodal representation including a set of modality tokens using the Unified Multimodal Gateway 302. In a non-limiting example, the Unified Multimodal Gateway 302 may receive fundus images, OCT B-scans or volumes, free-text clinician prompts, and metadata or context. The Unified Multimodal Gateway 302 may apply modality-appropriate preprocessing, such as speckle suppression for OCT and color or illumination normalization for fundus images. The Unified Multimodal Gateway 302 may perform text tokenization, vision patchification or visual token generation, and modality-token creation. For example, the modality tokens may act as source identity tags, such as FUNDUS, OCT_3D, OCT_BSCAN, TEXT, or other ophthalmic input modalities, enabling the multimodal encoder 304 to distinguish the origin of each token during early fusion.
[0057] In an embodiment, the system 106 may be configured to encode the multimodal representation and generate fused embeddings comprising one or more global classification tokens and lesion-level visual patch tokens using the multimodal encoder 304. In a non-limiting example, the multimodal encoder 304 may ingest vision-patch tokens, text tokens, and modality tokens from the Unified Multimodal Gateway 302. Each encoder block may blend modalities immediately, wherein self-attention heads carry learned query/key biases that favor vision-text links, and every second block applies a lightweight Feature-wise Linear Modulation (FiLM) gate so statistics from one stream condition the other. For example, this early, in-encoder fusion lets language cues reshape retinal patch representations, and vice versa, at every depth, yielding denser joint features, sharper saliency maps, and interactive latency without an additional vision pass.
[0058] In an embodiment, the system 106 may be configured to decode, in a single forward pass from the fused embeddings, an output, a user-readable biomarker caption output, and a saliency-uncertainty output comprising a saliency map co-registered with per-pixel uncertainty using the multi-headed decoder 306. In a non-limiting example, the multi-headed decoder 306 may include an Output head 308, a Caption Head 310, and a Saliency-Uncertainty Head 312. The Output head 308 may convert the one or more global classification tokens into an output and output probabilities. The Caption Head 310 may generate a user-readable biomarker caption comprising biomarker tokens or phrases such as “microaneurysm,” “hemorrhage,” or “hard exudate.” The Saliency-Uncertainty Head 312 may produce a saliency map alongside per-pixel epistemic and aleatoric uncertainty estimates, ensuring spatial co-registration of attention and confidence. For example, the three outputs may be jointly regularized by a Tri-Output Consistency Loss that penalizes semantic or spatial mismatch among the output, the user-readable biomarker caption output, and the saliency-uncertainty output.
[0059] In an embodiment, the system 106 may be configured to generate token-grounded region masks for each token that align a set of biomarker tokens in the biomarker caption output with corresponding image regions using the Token-to-Region Alignment (T2R-A) module 314. In a non-limiting example, given lesion-level visual patch tokens from the multimodal encoder 304 and biomarker tokens or phrases from the user-readable biomarker caption, the T2R-A module 314 may predict a soft spatial distribution over the relevant visual patch tokens for each biomarker token or phrase. For example, the T2R-A module 314 may output one token-grounded region mask for “exudate” and another for “hemorrhage”. The token-grounded region masks are produced during the forward pass rather than via Grad-CAM and align with the biomarker vocabulary emitted by the Caption Head 310.
[0060] In an embodiment, the system 106 may be configured to enforce Tri-Output Consistency among the output, the biomarker caption output, and the saliency-uncertainty output using a loss computation module. The Tri-Output Consistency may establish a consistency relationship among the classification confidence represented by the output, the semantic evidence represented by the biomarker caption output, and the spatial reliability represented by the saliency-uncertainty output. In a non-limiting example, the loss computation module may penalize missing visual evidence if a biomarker mentioned in the user-readable biomarker caption lacks corresponding saliency evidence or a corresponding token-grounded region mask. The loss computation module may penalize severity discrepancy if a severity grade derived from quantitative biomarker metrics conflicts with the output or diagnosis probabilities. For example, the loss computation module may flag output disagreement if the output, user-readable biomarker caption, and saliency-uncertainty output provide inconsistent evidence for the same ophthalmic input. Further, the such computed consistency loss, disagreement score, or consistency-weighted probability may further be provided to derive a Confidence Estimation module 402, such that the Confidence Estimation module 402 derives or calibrates confidence based at least in part on whether the output, the biomarker caption output, and the saliency-uncertainty output are mutually consistent.
[0061] In an embodiment, the system 106 may be configured to perform uncertainty-gated biomarker analytics by fusing the saliency-uncertainty output with the token-grounded region masks for each biomarker token to derive instance-level biomarker masks and quantitative biomarker metrics using the biomarker engine 316. The instance-level biomarker masks correspond to biomarker tokens for which the token-grounded region masks are generated for each token. In a non-limiting example, the biomarker engine 316 may ingest four streams from the multi-headed decoder 306 and the T2R-A module 314: diagnosis probabilities, token-grounded region masks, the saliency map, and the corresponding uncertainty map. These layers may be first upsampled to native image resolution and fused, wherein pixels whose uncertainty exceeds a dynamic threshold, such as a dynamic percentile threshold, may be suppressed so that higher-confidence regions preferentially propagate downstream. For example, a lesion-separation module may derive instance-level biomarker masks from the fused saliency, uncertainty, and token-grounded region-mask information using saliency-guided seed detection followed by seeded watershed segmentation and morphological cleanup.
[0062] In an embodiment, the system 106 may be configured to generate a clinical report comprising coded clinical terminology and provenance information based on the decoded outputs using a standards-aware output layer. In a non-limiting example, the standards-aware output layer may package the output, diagnosis probabilities, calibrated probabilities, user-readable biomarker caption, saliency-uncertainty output, token-grounded region masks, instance-level biomarker masks, quantitative biomarker metrics, evidence overlays, ICD-10 and SNOMED terminology, and provenance information into a FHIR Diagnostic Report. For example, the quantitative biomarker metrics may be represented as HL7 Observation entries, while visual evidence outputs, including saliency maps, uncertainty maps, token-grounded region masks, and instance-level biomarker overlays, may be stored or referenced in PNG or Digital Imaging and Communications in Medicine Segmentation (DICOM-SEG) format for EMR or Picture Archiving and Communication System (PACS) integration.
[0063] In an embodiment, the multimodal encoder 304 may include feature-wise conditioning and learned cross-modal attention biases between vision and text streams. In a non-limiting example, the multimodal encoder 304 may apply feature-wise conditioning using lightweight FiLM gates so statistics from one stream condition the other. For example, the learned cross-modal attention biases may favor vision-text interactions to refine lesion-level representations.
[0064] In an embodiment, the biomarker engine 316 may derive instance-level biomarker masks using saliency-guided seed detection and seeded watershed segmentation without a separate segmentation network. In a non-limiting example, the biomarker engine 316 may use saliency-guided seed detection to locate likely lesion centers in a heat-map or mask, followed by seeded watershed segmentation to separate touching lesion regions into distinct biomarker instances. For example, equivalent segmentation, clustering, connected-component, contour-based, or learned refinement techniques may be used to obtain discrete biomarker instances without requiring a separate heavy segmentation network.
[0065] In an embodiment, outputs generated by the multi-head decoder 306, including outputs from the output head 308, the caption head 310, and the saliency-uncertainty head 312, are provided to the T2R-A 314. The T2R-A 314 is configured to aggregate and transform the outputs of the plurality of heads and, optionally, the fused embeddings comprising global classification token(s) and lesion-level visual patch tokens, into structured intermediate representations suitable for further processing.
[0066] In an embodiment, the biomarker engine 316 may perform morphological cleanup and biomarker instance separation prior to quantitative metric computation. In a non-limiting example, morphological cleanup may include image-processing steps that tidy lesion shapes, remove noise, and fill small gaps. For example, the morphological cleanup may be performed after seeded watershed segmentation to obtain discrete biomarker instances.
[0067] In an embodiment, the system 106 may further comprise a lightweight classifier configured to classify instance-level biomarker masks into biomarker categories with associated probabilities. In a non-limiting example, each instance-level biomarker mask may be extracted as a compact region and classified using a lightweight classifier, such as a lightweight CNN, to assign a granular biomarker label, such as microaneurysm or hard exudate, together with an associated probability score. For example, the lightweight classifier may allow new biomarker categories to be supported by updating the classifier rather than retraining the entire system.
[0068] In an embodiment, the standards-aware output layer may generate a FHIR Diagnostic Report comprising at least one of: HL7 Observation entries, ICD-10 codes, SNOMED terminology, or RFC-3881 provenance. In a non-limiting example, each report may be stamped with provenance information, including model version, model hash, parameter version, input references, and processing metadata, thereby supporting traceability and audit requirements for clinical and regulatory review.
[0069] In an embodiment, the system 106 may further comprise the latent cache 318 configured to store fused embeddings and the token-grounded region masks for each biomarker token and to support follow-up clinician prompts. In a non-limiting example, the latent cache 318 may store cached fused embeddings, cached lesion-level patch tokens, and cached token-grounded region masks after the initial inference. For example, for subsequent clinician prompts, such as "highlight exudates and measure area," the system 106 may reuse the cached representations and route the prompt through a lightweight follow-up reasoning path.
[0070] In an embodiment, the loss computation module may be configured to penalize missing saliency or severity discrepancy and to flag inconsistency when conflicts persist. In a non-limiting example, the loss computation module may penalize missing saliency if a biomarker mentioned in the user-readable biomarker caption lacks corresponding saliency evidence. For example, the loss computation module may flag inconsistency when the output indicates “no diabetic retinopathy” while the user-readable biomarker caption mentions “exudates”.
[0071] Figure 4 illustrates an enhanced operational view of the system 106 for ophthalmic analysis, in accordance with an embodiment of the present disclosure. In particular, Figure 4 further elaborates downstream processing and clinical translation pipeline by explicitly depicting a Confidence Estimation module 402 and a Severity Rules module 404 in association with the biomarker engine 316.
[0072] In an embodiment, the Confidence Estimation module 402 is configured to calibrate diagnosis probabilities and align the output with the saliency-uncertainty output by utilizing the Tri-output Consistency, thereby improving reliability of uncertainty gated biomarker analytics prior to derivation of instance level biomarker masks and quantitative biomarker metrics.
[0073] In an embodiment, the Severity Rules module 404, operatively coupled to the biomarker engine 316, is configured to translate the quantitative biomarker metrics into a disease specific severity grade using ontology driven rules compliant with established clinical grading schemes, thereby enabling clinically interpretable outputs and decision support.
[0074] Figure 5 illustrates a detailed block diagram 500 of the biomarker engine 316, in accordance with an embodiment of the present disclosure.
[0075] As shown, the biomarker engine 316 includes Diagnosis Probabilities 502, Token-Grounded Region Masks 504, Saliency Map 506, Uncertainty Map 508, an Upsample and Fuse module 510, a Seed Detection module 512, a Lesion Separation module 514, Instance-Level Biomarker Masks 516, a Lightweight Classifier 518, a Metric Calculator 520, an Evidence Overlay generator 522, a Structured Biomarker Record generator 524, and the Severity Rules 404.
Output head
[0076]
[0077] In an embodiment, the diagnosis probabilities 502, the token-grounded region masks 504, the saliency map 506, and the uncertainty map 508 are provided to an upsample and the fuse module 510. The upsample and fuse module 510 is configured to perform uncertainty-gated fusion to generate the one or more refined representations. The one or more refined representations are provided to the seed detection module 512.
[0078] In an embodiment, the seed detection module 512 is configured to detect one or more lesion seeds. The one or more lesion seeds are provided to a lesion separation module 514.
[0079] In an embodiment, the lesion separation module 514 is configured to generate instance-level biomarker masks 516. The instance-level biomarker masks 516 are provided to the lightweight classifier 518 and to the metric calculator 520.
[0080] In an embodiment, the lightweight classifier 518 is configured to generate the one or more classification outputs based on the instance-level biomarker masks 516, and the one or more classification outputs are provided to the metric calculator 520. The metric calculator 520 is configured to compute one or more metrics including count, area, and foveal offset based on the instance-level biomarker masks 516 and the one or more classification outputs.
[0081] In an embodiment, the one or more metrics are provided to severity rules 404, wherein the severity rules 404 are configured to determine a severity level. The instance-level biomarker masks 516 are further provided to an evidence overlay module 522. The evidence overlay module 522 is configured to generate the evidence overlay 522. The severity level, the one or more metrics, and the evidence overlay 522 are provided to generate a structured biomarker record 524.
[0082] Figure 6 illustrates a block diagram 600 of the Follow-up Chat Interface 118 for servicing follow-up clinician prompts, in accordance with an embodiment of the present disclosure. As shown, the Follow-up Chat Interface 118 includes a Dialogue/Session State component 604, a Prompt Interpreter 606, a Latent Cache 318, a Quick Biomarker Slice 608, a Heat-Map Side-Car 610, a Rapid Inference Hub 612, a Metric Formatter 614, an Overlay Builder 616, a Response Assembler 618, and a Web/App Front-End 620.
[0083] In an embodiment, when a clinician sends a follow-up clinician prompt such as “Highlight exudates and measure”, the utterance may be preserved in the Dialogue/Session State component 604 and routed to the Prompt Interpreter 606 that performs intent extraction and slot-filling.
[0084] In an embodiment, the Prompt Interpreter 606 may be operatively connected to the Latent Cache 318, the Quick Biomarker Slice block 608, and the Heat-Map Side-Car block 610. The Latent Cache 318 may be configured to store fused embeddings and visual patch tokens and may further be operatively coupled to the Rapid Inference Hub 612. The Quick Biomarker Slice block 608, configured to generate instance masks and metrics, may also be operatively coupled to the Rapid Inference Hub 612. The Heat-Map Side-Car block 610, configured to generate masks including saliency and uncertainty, may be operatively connected to both the Overlay Builder 616 and the Rapid Inference Hub 612.
[0085] In an embodiment, the Overlay Builder 616 may be communicatively coupled to the Response Assembler 618 and may further be connected to the Metric Formatter 614. The Prompt Interpreter 606 may be directly coupled to the Metric Formatter 614, thereby enabling formatted metric generation based on interpreted intent. The Metric Formatter 614 may further be operatively connected to the Response Assembler 618.
[0086] In an embodiment, the Rapid Inference Hub 612, configured to perform micro-pass inference on relevant tokens, may aggregate inputs from the Latent Cache 318, the Quick Biomarker Slice block 608, and the Heat-Map Side-Car block 610, and may further be operatively coupled to the Response Assembler 618. The Response Assembler 618, configured to combine metrics, overlays, and textual responses, may receive inputs from the Overlay Builder 616, the Metric Formatter 614, and the Rapid Inference Hub 612.
[0087] Finally, the Response Assembler 618 may be communicatively coupled to the Web/Desktop based Application Front-End 620 for presentation of outputs, and a feedback loop may be established from the Web/Desktop based Application Front-End 620 to the Follow-up Clinician Prompt 602, thereby enabling iterative clinician interaction within the system 600.
[0088] Figure 7 illustrates a user interface flow diagram 700 for Image Only mode, in accordance with an embodiment of the present disclosure. The workflow depicts interaction between a user and an AI studio component for ophthalmic image analysis. Initially, the user provides an Image 702 as input, which flows into the AI Studio component.
[0089] In an embodiment, within the AI Studio, the system 106 may process the image and generate an Original Image along with a Summary output 704. The output 704 may be presented back to the user, who evaluates the information and reaches a decision point labeled “Doubt?” 706 represented by a diamond-shaped decision node. If the user has doubts or questions about the analysis (indicated by the “Yes” path), the system 106 may direct them to the Follow-up Chat interface 118 within the AI Studio, allowing the user to engage in interactive dialogue to clarify findings or request additional information about the ophthalmic image. The Follow-up Chat interface 118 may loop back to the “Doubt?” 706 decision point, enabling iterative questioning until the user is satisfied. When the user has no further doubts (indicated by the “No” path), the system 106 may proceed to generate the Standards-Aware Clinical Report 116 as the output.
[0090] Figure 8 illustrates a user interface flow diagram for Image and Text mode showing the interaction between a user and an AI Studio system, in accordance with an embodiment of the present disclosure. The workflow begins with the user providing an “Image + Text: Generated Heatmap” 802 as input, which represents the initial multimodal ophthalmic input comprising an image along with text and a generated heatmap visualization.
[0091] In an embodiment, within the AI Studio, a processing element labeled “Original Image + Heatmap + Summary” 804 may receive the relevant cached information including the original ophthalmic image, the generated heatmap visualization, and the summary information. The element 804 may connect to a Follow-up Chat component, which represents the interactive dialogue capability that allows clinicians to ask additional questions about the analysis. The Follow-up Chat component may have a feedback loop returning to the “Doubt?” decision point 806, indicating that after receiving a response from the follow-up chat, the user may evaluate whether their concerns have been addressed and either proceed to report generation or continue the interactive dialogue with additional queries.
[0092] Figure 9 illustrates a flowchart depicting a method 900 for ophthalmic analysis using the vision-language-based model, in accordance with an embodiment of the present disclosure. The method 900 may be a computer-implemented method executed, for example, by the system 106. For the sake of brevity, constructional and operational features of the system 106 that are already explained in the description of Figures 1-8 are not explained in detail in the description of Figure 9.
[0093] At step 902, the method 900 comprises serializing the set of multimodal ophthalmic inputs into a multimodal representation comprising the set of modality tokens using the Unified Multimodal Gateway. The set of multimodal ophthalmic inputs comprises ophthalmic image data and associated metadata. In a non-limiting example, the Unified Multimodal Gateway 302 may receive the fundus images, the OCT B-scans or volumes, the free-text clinician prompts, and the metadata or context, and may apply modality-appropriate preprocessing, text tokenization, vision patchification, and modality-token creation.
[0094] At step 904, the method 900 comprises generating fused embeddings comprising the one or more global classification tokens and the lesion-level visual patch tokens based on encoding the multimodal representation, using a multimodal encoder. In a non-limiting example, the multimodal encoder 304 may ingest the vision-patch tokens, the text tokens, and the modality tokens from the Unified Multimodal Gateway 302 and perform early fusion with cross-modal attention and feature-wise conditioning.
[0095] At step 906, the method 900 comprises decoding, in a single forward pass from the fused embeddings, the output, the useruser-readable biomarker caption output, and the saliency-uncertainty output comprising the saliency map co-registered with per-pixel uncertainty. In a non-limiting example, the multi-headed decoder 306 may include the OutputOutput head 308, the Caption Head 310, and the Saliency-Uncertainty Head 312 that generate the three coordinated outputs.
[0096] At step 908, the method 900 comprises generating token-grounded region masks for each token in the set of biomarker tokens appearing in the user-readable biomarker caption output with corresponding image regions using the T2R-A mechanism. In a non-limiting example, the T2R-A module 314 may predict a soft spatial distribution over the relevant visual patch tokens for each biomarker token or phrase.
[0097] At step 910, the method 900 comprises enforcing Tri-Output Consistency by penalizing, resolving, or flagging disagreement among the output, the user-readable biomarker caption output, and the saliency-uncertainty output. In a non-limiting example, the loss computation module may penalize missing visual evidence, severity discrepancy, or output disagreement.
[0098] At step 912, the method 900 comprises deriving instance-level biomarker masks and quantitative biomarker metrics based on performing uncertainty-gated biomarker analytics by fusing the saliency-uncertainty output with the token-grounded region masks for each biomarker token. The instance-level biomarker masks correspond to biomarker tokens for which the token-grounded region masks are generated for each token. In a non-limiting example, the biomarker engine 316 may upsample and fuse the inputs, perform seed detection and lesion separation, and compute quantitative metrics including lesion count, aggregate lesion area, centroid location, and minimum foveal distance.
[0099] At step 914, the method 900 comprises generating the standards-aware clinical report comprising coded clinical terminology and provenance information based on the output, the user-readable biomarker caption output, the instance-level biomarker masks for each biomarker token, the saliency-uncertainty output, and the quantitative biomarker metrics. In a non-limiting example, the standards-aware output layer may package all artifacts into a FHIR Diagnostic Report with HL7 Observation entries, ICD-10 and SNOMED terminology, and provenance information.
[0100] In an embodiment, the method 900 comprises assigning a modality tag, based on the set of multimodal ophthalmic inputs, prior to encoding using the Unified Multimodal Gateway. In a non-limiting example, the modality tokens may act as source identity tags, such as FUNDUS, OCT_3D, OCT_BSCAN, TEXT, or other ophthalmic input modalities.
[0101] In an embodiment, the method 900 comprises applying feature-wise conditioning and learned cross-modal attention biases that favor vision-text interactions to refine lesion-level representations using the multimodal encoder. In a non-limiting example, the multimodal encoder 304 may apply lightweight FiLM gates so statistics from one stream condition the other.
[0102] In an embodiment, the method 900 comprises generating the saliency-uncertainty output comprising epistemic uncertainty and aleatoric uncertainty whose composite uncertainty modulates an alpha channel of a rendered evidence overlay and is produced during the single forward pass. In a non-limiting example, epistemic uncertainty may be estimated using Monte Carlo dropout over the decoder heads, whereas aleatoric uncertainty may be learned as a heteroscedastic log-variance channel.
[0103] In an embodiment, the method 900 comprises training the T2R-A mechanism using a contrastive loss that aligns biomarker tokens with visual patch tokens using dense or weak supervision. In a non-limiting example, a contrastive mask loss may maximize the probability mass assigned to ground-truth patch sets when mask annotations are available, or to weakly supervised regions inferred from bounding boxes, caption supervision, or biomarker-level labels when dense labels are absent.
[0104] In an embodiment, the method 900 comprises enforcing inconsistency under the Tri-Output Consistency comprising at least one of: misalignment, absence, or disagreement between output, the user-readable biomarker caption output, and the saliency-uncertainty output. In a non-limiting example, the loss computation module may penalize a user-readable biomarker caption mentioning “exudates” while the output indicates “no diabetic retinopathy”.
[0105] In an embodiment, the method 900 comprises performing the uncertainty-gated biomarker analytics comprising deriving instance-level biomarker masks using saliency-guided seed detection followed by the seeded watershed segmentation. In a non-limiting example, the Seed Detection module 512 may use local maxima identification to locate likely lesion centers, followed by the Lesion Separation module 514 employing seeded watershed segmentation.
[0106] In an embodiment, the method 900 comprises performing the seeded watershed segmentation followed by morphological cleanup to obtain discrete biomarker instances without invoking a separate segmentation network. In a non-limiting example, morphological cleanup may include image-processing steps that tidy lesion shapes, remove noise, and fill small gaps.
[0107] In an embodiment, the method 900 comprises classifying each instance-level biomarker mask using a lightweight classifier to assign a biomarker category and an associated probability. In a non-limiting example, the Lightweight Classifier 518, such as a CNN, may assign a granular biomarker label, such as microaneurysm or hard exudate, together with an associated probability score.
[0108] In an embodiment, the method 900 comprises translating the quantitative biomarker metrics into a disease-specific severity grade using ontology-driven rules compliant with established clinical grading schemes. In a non-limiting example, the Severity Rules 404 may follow ETDRS or ICDR for diabetic retinopathy or AREDS for age-related macular degeneration.
[0109] In an embodiment, the method 900 comprises suppressing pixels whose uncertainty exceeds a dynamic percentile threshold computed per image using the uncertainty-gated biomarker analytics. In a non-limiting example, the Upsample and Fuse module 510 may suppress pixels whose uncertainty exceeds a dynamic threshold so that higher-confidence regions preferentially propagate downstream.
[0110] In an embodiment, the method 900 comprises calibrating output probabilities using Monte-Carlo dropout or ensemble inference, and aligning calibrated probabilities with pixel-level uncertainty. In a non-limiting example, the Confidence Estimation module 402 may use ensemble inference and Monte Carlo dropout to calibrate diagnosis probabilities and merge calibrated probabilities with pixel-level uncertainty information.
[0111] In an embodiment, the method 900 comprises generating the standards-aware clinical report as a FHIR Diagnostic Report comprising at least one of: HL7 Observation entries, ICD-10 codes, SNOMED terminology, or RFC-3881 provenance. In a non-limiting example, each report may be stamped with provenance information, including model version, model hash, parameter version, input references, and processing metadata.
[0112] In an embodiment, the method 900 comprises generating the standards-aware clinical report comprising render-ready overlays in compatible image formats or DICOM-SEG format for visualization in PACS or EMR systems. In a non-limiting example, visual evidence outputs, including saliency maps, uncertainty maps, token-grounded region masks, and instance-level biomarker overlays, may be stored or referenced in PNG or DICOM-SEG format.
[0113] In an embodiment, the method 900 comprises caching the fused embeddings and the token-grounded region masks for each token from an initial inference and servicing follow-up clinician prompts. In a non-limiting example, the latent cache 318 may store cached fused embeddings, cached lesion-level patch tokens, and cached token-grounded region masks after the initial inference, enabling rapid follow-up interaction while preserving consistency with the original outputs.
[0114] In an embodiment, the system and method of the present disclosure may include a training methodology comprising dataset preparation and augmentation. The corpus composition may include colour fundus images from public and in-house sources, OCT B-scans or OCT volumes covering common retinal diseases, and paired natural-language captions, clinician prompts, diagnosis labels, and metadata or context where available. Each training example may be stored in a unified sample wrapper using a structured record format, such as protobuf or Parquet, with fields including image_pixels, modality_tag, optional_text, and metadata_dict. Fundus inputs may be represented as RGB image arrays, while OCT inputs may be represented as a B-scan slice, slice stack, or voxel cube with standardized spacing where applicable.
[0115] In an embodiment, the system and method of the present disclosure may include online transforms for data augmentation. Geometric flips or rotations may be applied for fundus images where clinically appropriate. Elastic or depth-axis warps may be applied for OCT inputs where clinically appropriate. Prompt augmentation may be performed using synonym substitution within the SNOMED hierarchy or other controlled clinical terminology.
[0116] In an embodiment, the system and method of the present disclosure may include a Stage I foundation pre-training phase for contrastive alignment. The goal of Stage I may be to align visual and textual embeddings in a shared metric space. The visual backbone may include a ViT-family model or another suitable visual encoder. The language backbone may include a BERT-family model or another suitable text encoder trained or fine-tuned on ophthalmic literature. An AdamW optimizer or an equivalent optimizer may be used with a cosine-decay scheduler and warm-up. An InfoNCE or CLIP-style contrastive objective with a configurable temperature parameter may be used as the loss function. Pre-training may continue until image-text retrieval metrics plateau on an internal validation set.
[0117] In an embodiment, the system and method of the present disclosure may include a Stage II joint multimodal fine-tuning phase. Training batches may be constructed containing fundus images, optional OCT inputs, prompts or captions, diagnosis labels, and available biomarker supervision. The loss functions for each head may include cross-entropy on caption token IDs for the Caption Head 310, focal loss or class-balanced loss on diagnosis labels for the Output Head 308, and KL divergence to T2R-A outputs and variance regression for the Saliency-Uncertainty Head 312. The Tri-Output Consistency Loss may penalize mismatch among the output, the user-readable biomarker caption, and the saliency-uncertainty output. The total loss may be computed as a weighted sum of caption loss, diagnosis loss, saliency-uncertainty loss, T2R-A alignment loss where applicable, and Tri-Output Consistency Loss, with adjustable lambda weights.
[0118] In an embodiment, the system and method of the present disclosure may include uncertainty propagation during training by combining aleatoric uncertainty with epistemic uncertainty, such as MC-dropout-based uncertainty, to form pixel-level uncertainty maps.
[0119] In an embodiment, the system and method of the present disclosure may include latent-cache integration during training. After early epochs, encoder latents may be serialized using image UID or ophthalmic input UID, allowing cache hits to accelerate later epochs where applicable.
[0120] In an embodiment, the system and method of the present disclosure may include curriculum learning and hard-example mining. Every few epochs, the training process may intersect saliency maps or token-grounded region masks with ontology keywords, such as microaneurysm, drusen, exudate, or hemorrhage. Misclassified or under-represented biomarker cases may be up-sampled. Class sampling may be re-balanced within a tolerated skew.
[0121] In an embodiment, the system and method of the present disclosure may include distributed and mixed-precision training. High-depth OCT inputs may be split across accelerators, such as by spatial or channel parallelism, when memory is limited. For large clusters, suitable distributed communication strategies may be used, such as NCCL hierarchical trees or ring-plus-tree topologies. Checkpoints may be stored in a precision format suitable for reliable save and loading, and may be converted dynamically on-the-fly as required.
[0122] In an embodiment, the system and method of the present disclosure may include a lightweight clinical-edge variant for inference nodes with constrained VRAM. Selected vision blocks may be frozen and only the decoder heads may be fine-tuned where appropriate. A distilled, quantized, or otherwise lightweight language encoder may be used where suitable. MC-dropout may be replaced with entropy-based uncertainty or another lightweight uncertainty estimate if runtime budgets are tight. Target latency may be interactive, such as sub-second per ophthalmic input and prompt on suitable embedded GPUs.
[0123] In an embodiment, the system and method of the present disclosure provide industrial applicability supporting both cloud-scale training for maximal accuracy and incremental on-device or edge fine-tuning for chair-side ophthalmic systems. Built-in mask logging and provenance hashing facilitate regulatory audits, while the latent cache 318 enables rapid iteration. New prompts, biomarker categories, or ontologies may be accommodated without retraining the full backbone where cached representations, modular heads, Retrieval-Augmented Generation (RAG), and a vector database storing ophthalmic knowledge embeddings, biomarker ontology embeddings, or prior-report embeddings are sufficient.
[0124] In an embodiment, the system and method of the present disclosure may include a knowledge and safety layer. A disease-to-biomarker ontology may be stored in JSON-LD, providing a mapping between diseases and relevant biomarkers used for biomarker selection and severity rules. RAG may be used to pull trusted information into generation or reasoning. Governance and compliance processes may support safety, traceability, and regulatory readiness. A human-in-the-loop workflow may be provided where clinicians may review, accept, reject, or correct AI outputs.
[0125] In an embodiment, the system and method of the present disclosure facilitate DICOM compliance checks. Each imaging stream may be subjected to DICOM compliance checks, glare removal, color-constancy normalization, and modality-appropriate preprocessing before being processed by the Unified Multimodal Gateway 302.
[0126] In an embodiment, the system and method of the present disclosure enable mixed batches and cross-modal curriculum learning. Mixed batches of fundus and OCT images may be co-trained, promoting implicit cross-modal curriculum learning and eliminating the operational overhead of maintaining distinct model endpoints.
[0127] In an embodiment, the system and method of the present disclosure may identify common retinal findings including: microaneurysm, which is a tiny red dot caused by localized dilation of retinal capillaries and is often an early sign of diabetic retinopathy; hard exudate, which comprises yellow-white deposits caused by leakage of lipids and proteins from blood vessels; hemorrhage, which is a red or dark retinal region caused by leakage of blood from retinal vessels, including dot-blot hemorrhage seen in deeper retinal layers and flame-shaped hemorrhage seen in the nerve fiber layer; soft exudate or cotton-wool spot, which is a white fluffy retinal lesion associated with nerve fiber layer ischemia; and drusen, which comprises yellow deposits beneath the retina often associated with age-related macular degeneration.
[0128] In an embodiment, the system and method of the present disclosure may analyze ophthalmic diseases including: Diabetic Retinopathy (DR), which is retinal damage caused by diabetes commonly assessed using microaneurysms, hemorrhages, exudates, and other lesions; Non-Proliferative Diabetic Retinopathy (NPDR), which is a stage of diabetic retinopathy without abnormal new vessel growth; Proliferative Diabetic Retinopathy (PDR), which is advanced diabetic retinopathy involving abnormal new vessel growth; Age-Related Macular Degeneration (AMD), which is a retinal disease affecting the macular region often associated with drusen and other changes; Glaucoma, which is optic nerve damage often associated with structural and functional visual changes; and Hypertensive Retinopathy, which comprises retinal changes caused by high blood pressure.
[0129] In an embodiment, the Follow-up Chat Interface 118 may support interactive dialogues with clinicians. For example, a clinician may ask "presence of diabetic retinopathy?" and the system 106 may respond by analyzing the presence of multiple intraretinal hemorrhages and other findings to indicate the presence of diabetic retinopathy. The clinician may further ask "comment on tortuosity and its significance on the diabetic retinopathy" and the system 106 may explain that increased retinal vessel tortuosity, particularly venous tortuosity, may indicate impaired blood flow and retinal ischemia, vascular remodeling, and disease severity. The clinician may ask "now comment on? hemorrhages" and the system 106 may describe the types and distribution of hemorrhages, including dot-and-blot hemorrhages in deeper retinal layers characteristic of NPDR, and their significance in indicating microvascular damage. The clinician may ask "comment on presence of any other ophthalmic disease" and the system 106 may analyze the image for signs of other conditions such as glaucoma, hypertensive retinopathy, or age-related macular degeneration.
[0130] In an embodiment, the system and method of the present disclosure provides portability across different hardware accelerators. Global batch size, micro-batch size, activation precision, and optimizer settings may be scaled to match available accelerators. For example, FP8 or BF16 precision on NVIDIA H100 may be adapted to BF16 or FP16 on NVIDIA A100, bfloat16 on TPU, or INT8 plus FP16 on edge devices.
[0131] The present disclosure provides the following advantages:
a) The present disclosure provides a single Unified Multimodal Gateway for fundus images, optional OCT B-scans or volumes, free-text clinician prompts, and metadata or context that simplifies clinical integration by collapsing traditionally siloed pipelines into one gateway.
b) The present disclosure enables low-latency interactive reasoning through early fusion with cached fused embeddings, cached lesion-level patch tokens, and cached token-grounded region masks.
c) The present disclosure provides co-registered saliency and uncertainty maps that enable spatial risk visualization in a single model pass, allowing clinicians to distinguish high-confidence pathology from ambiguous or noisy regions.
d) The present disclosure facilitates self-auditing predictions through Tri-Output Consistency Loss that encourages coherence across diagnostic prediction, textual explanation, and visual evidence, thereby reducing hallucinated or contradictory outputs.
e) The present disclosure provides ontology-driven biomarker analytics that supply actionable quantitative biomarker metrics and biomarker-derived severity grades compliant with established clinical grading schemes such as ETDRS and AREDS.
f) The present disclosure enables standards-aware clinical reports with calibrated probabilities, uncertainty information, biomarker metrics, evidence overlays, and provenance information that support transparency and regulatory review.
g) The present disclosure provides token-grounded region masks that supply audit-ready biomarker-specific visual evidence exceeding the granularity of conventional ophthalmic explainability methods.
h) The present disclosure enables rapid follow-up interaction through a latent-cache dialogue handler that reuses cached representations, allowing follow-up prompts to be answered without complete re-encoding of the ophthalmic input. Further, one or more guardrails may be employed to assess domain relevance of a user query and to filter, reject, or redirect queries unrelated to ophthalmic analysis.
i) The present disclosure facilitates reduced reliance on extensively labeled datasets through weakly supervised biomarker grounding using ontology-driven biomarker highlighting, T2R-A, and prompt-guided reasoning.
[0132] While specific language has been used to describe the present subject matter, any limitations arising on account thereof are not intended. As would be apparent to a person in the art, various working modifications are made to the method in order to implement the inventive concept as taught herein. The drawings and the foregoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements are split into multiple functional elements. Elements from one embodiment are added to another embodiment. For example, orders of processes described herein are changed and are not limited to the manner described herein. , Claims:WE CLAIM:

1. A method (900) for ophthalmic analysis using a vision–language-based model, the method comprising:
serializing (902) a set of multimodal ophthalmic inputs into a multimodal representation comprising a set of modality tokens using a Unified Multimodal Gateway (302), wherein the set of multimodal ophthalmic inputs comprises ophthalmic image data and associated metadata;
generating (904) fused embeddings comprising one or more global classification tokens and lesion-level visual patch tokens based on encoding the multimodal representation, using a multimodal encoder (304);
decoding (906), in a single forward pass from the fused embeddings, an output, a user-readable biomarker caption output, and a saliency-uncertainty output comprising a saliency map co-registered with per-pixel uncertainty, using a multi-headed decoder (306);
generating (908) token-grounded region masks for each token in a set of biomarker tokens appearing in the user-readable biomarker caption output with corresponding image regions using a Token-to-Region Alignment (T2R-A) mechanism (314);
enforcing (910) Tri-Output Consistency by penalizing, resolving, or flagging disagreement among the output, the user-readable biomarker caption output, and the saliency-uncertainty output;
deriving (912) instance-level biomarker masks and quantitative biomarker metrics based on performing uncertainty-gated biomarker analytics by fusing the saliency-uncertainty output with the token-grounded region masks for each biomarker token using a biomarker engine (316), wherein the instance-level biomarker masks correspond to biomarker tokens for which the token-grounded region masks are generated for each token; and
generating (914) a standards-aware clinical report comprising coded clinical terminology and provenance information based on the output, the user-readable biomarker caption output, the instance-level biomarker masks for each biomarker token, the saliency-uncertainty output, and the quantitative biomarker metrics.

2. The method as claimed in claim 1, wherein the Unified Multimodal Gateway assigns a modality tag, based on the set of multimodal ophthalmic inputs, prior to encoding.

3. The method as claimed in claim 1, wherein the multimodal encoder applies feature-wise conditioning and learned cross-modal attention biases that favor vision–text interactions to refine lesion-level representations

4. The method as claimed in claim 1, wherein the saliency–uncertainty output comprises epistemic uncertainty and aleatoric uncertainty whose composite uncertainty modulates an alpha channel of a rendered evidence overlay and is produced during the single forward pass.

5. The method as claimed in claim 1, wherein the Token-to-Region Alignment (T2R-A) mechanism is trained using a contrastive loss that aligns biomarker tokens with visual patch tokens using dense or weak supervision.

6. The method as claimed in claim 1, wherein the inconsistency enforced under the Tri-Output Consistency comprises at least one of: misalignment, absence, or disagreement between output, the user-readable biomarker caption output, and the saliency–uncertainty output.

7. The method as claimed in claim 1, wherein performing the uncertainty-gated biomarker analytics comprises deriving instance-level biomarker masks using saliency-guided seed detection followed by a seeded watershed segmentation.

8. The method as claimed in claim 7, wherein the seeded watershed segmentation is followed by morphological cleanup to obtain discrete biomarker instances without invoking a separate segmentation network.

9. The method as claimed in claim 1, further comprising:
classifying each instance-level biomarker mask using a lightweight classifier to assign a biomarker category and an associated probability.

10. The method as claimed in claim 1, wherein the quantitative biomarker metrics are translated into a disease-specific severity grade using ontology-driven rules compliant with established clinical grading schemes.

11. The method as claimed in claim 1, wherein the uncertainty-gated biomarker analytics suppress pixels whose uncertainty exceeds a dynamic percentile threshold computed per image.

12. The method as claimed in claim 1, further comprising:
calibrating output probabilities using Monte-Carlo dropout or ensemble inference, and aligning calibrated probabilities with pixel-level uncertainty.

13. The method of claim 1, wherein the standards-aware clinical report is generated as a FHIR Diagnostic Report comprising at least one of: HL7 Observation entries, ICD-10 codes, SNOMED terminology, and RFC-3881 provenance.

14. The method as claimed in claim 1, wherein the standards-aware clinical report comprises render-ready overlays in compatible image formats or DICOM-SEG format for visualization in PACS or EMR systems.

15. The method as claimed in claim 1, further comprising:
caching the fused embeddings and the token-grounded region masks for each token from an initial inference and servicing follow-up clinician prompts.

16. A system (106) for ophthalmic analysis using a vision-language-based model, comprising:
a Unified Multimodal Gateway (302) configured to ingest multimodal ophthalmic inputs and serialize ophthalmic image data and associated metadata into a common multimodal representation including a set of modality tokens;
a multimodal encoder (304) configured to encode the multimodal representation and generate fused embeddings comprising one or more global classification tokens and lesion-level visual patch tokens;
a multi-headed decoder (306) configured to decode, in a single forward pass from the fused embeddings, an output, a user-readable biomarker caption output and a saliency–uncertainty output comprising a saliency map co-registered with per-pixel uncertainty;
a Token-to-Region Alignment (T2R-A) module (314) configured to generate token-grounded region masks for each token that align a set of biomarker tokens in the biomarker caption output with corresponding image regions;
a loss computation module configured to enforce Tri-Output Consistency among the output, the biomarker caption output, and the saliency–uncertainty output;
a biomarker engine (316) configured to perform uncertainty-gated biomarker analytics by fusing the saliency-uncertainty output with the token-grounded region masks for each biomarker token to derive instance-level biomarker masks and quantitative biomarker metrics, wherein the instance-level biomarker masks correspond to biomarker tokens for which the token-grounded region masks are generated for each token; and
a standards-aware output layer configured to generate a clinical report comprising coded clinical terminology and provenance information based on the decoded outputs.

17. The system as claimed in claim 16, wherein the multimodal encoder includes feature-wise conditioning and learned cross-modal attention biases between vision and text streams.

18. The system as claimed in claim 16, wherein the biomarker engine derives instance-level biomarker masks using saliency-guided seed detection and seeded watershed segmentation without a separate segmentation network.

19. The system as claimed in claim 16, wherein the biomarker engine performs morphological cleanup and biomarker instance separation prior to quantitative metric computation.

20. The system as claimed in claim 16, further comprising a lightweight classifier configured to classify instance-level biomarker masks into biomarker categories with associated probabilities.

21. The system as claimed in claim 16, wherein the standards-aware output layer generates a FHIR Diagnostic Report comprising at least one of: HL7 Observation entries, ICD-10 codes, SNOMED terminology, and RFC-3881 provenance.

22. The system as claimed in claim 16, further comprising a latent cache configured to store fused embeddings and the token-grounded region masks for each biomarker token and to support follow-up clinician prompts.

23. The system as claimed in claim 16, wherein the loss computation module is configured to penalize missing saliency or severity discrepancy and to flag inconsistency when conflicts persist.