Description:FIELD OF THE INVENTION
The present invention relates to artificial intelligence and security auditing. More specifically, it pertains to a system and method for orchestrating security audits in agentic workflows using auxiliary large language models.
BACKGROUND OF THE INVENTION
Large Language Models (LLMs) have become a core part of modern artificial intelligence systems, particularly in agentic workflows where autonomous agents carry out complex tasks such as code generation, data interpretation, decision-making, and system orchestration. These agentic systems, powered by LLMs, are capable of reasoning through tasks and producing meaningful outputs with little or no human intervention. As their adoption grows across industries, these workflows increasingly involve sensitive inputs and produce phenomenal outputs, making their secure and responsible operation a matter of significant concern. Agentic workflows using LLMs are increasingly used for autonomous execution of complex tasks such as code generation, data analysis, decision-making, and system orchestration. These workflows often require sensitive data inputs and produce mission-critical outputs. However, there is a growing concern over security vulnerabilities introduced during LLM inferencing, such as data leakage, code injection, unauthorized access, or hallucinated commands.
Despite their capabilities, the existing systems that attempt to ensure the security within such workflows remain limited. Traditional security audit mechanisms are mostly static; they mostly rely on predefined rules and cannot adapt to the dynamic nature of agentic interactions. Moreover, they lack contextual awareness and the ability to interpret intermediate reasoning steps or infer intent behind generated actions. As a result, they often fail to detect and prevent advanced threats such as prompt injections, inadvertent data leaks, hallucinated or unauthorized commands, and violations of data security or compliance policies. These limitations pose a real challenge to maintaining trust and reliability in AI-driven automation.
Prior Art:
For instance, US11916767B1 discloses systems and methods for security analysis agents using large language models. It introduces a multi-agent architecture where LLMs generate prompts based on system events and assess the relevance or risks of those events. While this disclosure presents a dynamic approach to analyzing LLM behavior, it focuses largely on event classification and prompt fragment reuse, and does not propose a structured framework for intercepting and auditing the full input-output lifecycle of autonomous agents in a real-time agentic workflow. It also lacks the integration of a dedicated orchestration layer that actively governs LLM outputs based on contextual security scoring.
US12034616B2, a continuation of the above, expands on prompt evaluation and risk-based response mechanisms. However, the system remains centered on event classification and response generation without disclosing a modular orchestration engine that monitors, validates, and dynamically alters the behavior of agentic LLMs in production workflows. It does not address layered input-output interception, structured packaging of LLM reasoning steps, or integration of a learning feedback loop to refine security policies over time.
US20250016183A1 discloses a system using large language models to respond to information security incidents. It introduces a dual-LLM setup where one model handles initial processing while another reviews the result against security rules. Though conceptually close, this disclosure is limited to incident response scenarios and does not provide a generalized method for embedding auxiliary LLMs directly within agentic workflows to validate all stages of inferencing. It further lacks a universal scoring mechanism that guides output remediation or escalation, and does not disclose a structured feedback mechanism that enables adaptive refinement of security protocols based on evolving threats.
Although existing systems aim to monitor the behavior of large language models and improve Artificial Intelligence safety, they do not offer a complete framework that can continuously intercept agentic workflow activity, semantically analyze the input, reasoning, and output of LLM agents, and carry out contextual security validation using an auxiliary model. These approaches also miss key components such as dynamic risk scoring, real-time remediation through allow, block, or modify actions, structured packaging of reasoning steps for audit purposes, and a feedback loop that helps refine threat models over time.
DEFINITIONS
The expression “system” used hereinafter in this specification refers to an ecosystem comprising, but is not limited to a system with a user, input and output devices, processing unit, plurality of mobile devices, a mobile device-based application to identify dependencies and relationships between diverse businesses, a visualization platform, and output; and is extended to computing systems like mobile, laptops, computers, PCs, etc.
The expression “input unit” used hereinafter in this specification refers to, but is not limited to, mobile, laptops, computers, PCs, keyboards, mouse, pen drives or drives.
The expression “output unit” used hereinafter in this specification refers to, but is not limited to, an onboard output device, a user interface (UI), a display kit, a local display, a screen, a dashboard, or a visualization platform enabling the user to visualize, observe or analyse any data or scores provided by the system.
The expression “processing unit” refers to, but is not limited to, a processor of at least one computing device that optimizes the system.
The expression “large language model (LLM)” used hereinafter in this specification refers to a type of machine learning model designed for natural language processing tasks such as language generation. LLMs are language models with many parameters, and are trained with self-supervised learning on a vast amount of text.
The expression “Agentic Workflow” used hereinafter in this specification refers to an autonomous execution process involving one or more LLM agents used for tasks such as code generation, data analysis, decision-making, and system orchestration.
The expression “Tuple” (as implied from “structured tuple”) used hereinafter in this specification refers to the structured packaging of the input, reasoning, and output of the LLM agent for validation by the Audit LLM.
The expression “Risk Score” used hereinafter in this specification refers to the contextual score generated by the Audit LLM indicating the security integrity or risk level of an LLM’s inference output.
The expression “Action Decision” used hereinafter in this specification refers to the system’s response based on the risk score, which may include allowing, blocking, or replacing the output, or escalating it to a human operator.
OBJECTS OF THE INVENTION
The primary object of the present invention is to provide a system and method for orchestrating security audits in agentic workflows using an auxiliary large language model.
Another object of the invention is to continuously intercept and monitor the input, intermediate reasoning, and output generated by primary LLM agents during inferencing in real-time workflows.
Yet another object is to enable contextual security validation by leveraging a specialized Audit LLM trained to detect prompt injection, sensitive data leakage, hallucinations, unsafe commands, and policy violations.
A further object of the invention is to assign dynamic risk scores and apply decisions such as allowing, modifying, or blocking the output, or escalating it to a human operator, based on the level of risk.
An additional object is to incorporate a feedback loop that learns from past audit decisions and continuously updates the system’s security policies and detection capabilities.
SUMMARY
Before the present invention is described, it is to be understood that the present invention is not limited to specific methodologies and materials described, as these may vary as per the person skilled in the art. It is also to be understood that the terminology used in the description is for the purpose of describing the particular embodiments only and is not intended to limit the scope of the present invention.
The present invention describes a system and method for orchestrating security audits in agentic workflows using auxiliary large language models. The present invention leverages one or more Large Language Models (LLMs) to orchestrate and perform security audits on inputs and outputs during an agentic workflow involving autonomous agents powered by LLMs.
According to an aspect of the present invention, the system orchestrating security audits in agentic workflows injects a security audit mechanism within LLM-driven workflows. The system uses a secondary LLM (Audit LLM) to contextually analyze, validate, and score the security integrity of each step. It performs both static and dynamic validation on inputs and outputs. The system automatically learns and updates risk profiles across workflows. It also flags or corrects potentially harmful outputs and raise alerts to human supervisors.
According to an aspect of the present invention, the process flow begins when a user provides an input to the agentic system. This system forwards the task to the primary LLM agent module for processing. Upon processing the task, the primary LLM agent module transmits a tuple comprising the input (I), the reasoning (R), and the output (O) to the Audit Orchestrator module. The Audit Orchestrator then engages a specialized Audit LLM module to validate the received tuple for correctness, integrity, or potential risk. The Audit LLM module responds with a risk assessment report, which the Audit Orchestrator module uses to determine the appropriate course of action. Based on this decision, whether to allow or block the output, the Audit Orchestrator module instructs the primary LLM agent module accordingly. Finally, the primary LLM agent module communicates the approved final output back to the agentic system for delivery to the user. This modular and auditable design enhances trust, reliability, and governance in the deployment of AI agents.
BRIEF DESCRIPTION OF DRAWINGS
A complete understanding of the present invention may be made by reference to the following detailed description which is to be taken in conjugation with the accompanying drawing. The accompanying drawing, which is incorporated into and constitutes a part of the specification, illustrates one or more embodiments of the present invention and, together with the detailed description, it serves to explain the principles and implementations of the invention.
FIG. 1 illustrates a flowchart of the workflow of the present invention .
DETAILED DESCRIPTION OF INVENTION:
Before the present invention is described, it is to be understood that this invention is not limited to methodologies described, as these may vary as per the person skilled in the art. It is also to be understood that the terminology used in the description is for the purpose of describing the particular embodiments only and is not intended to limit the scope of the present invention. Throughout this specification, the word “comprise”, or variations such as “comprises” or “comprising”, will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps. The use of the expression “at least” or “at least one” suggests the use of one or more elements or ingredients or quantities, as the use may be in the embodiment of the invention to achieve one or more of the desired objects or results. Various embodiments of the present invention are described below. It is, however, noted that the present invention is not limited to these embodiments, but rather the intention is that modifications that are apparent are also included.
The present invention describes a system and method for orchestrating security audits in agentic workflows using auxiliary large language models. The present invention leverages one or more Large Language Models (LLMs) to orchestrate and perform security audits on inputs and outputs during an agentic workflow involving autonomous agents powered by LLMs. To address these limitations, the present invention focuses on the need for a more intelligent and adaptable security audit mechanism within agentic workflows supported by large language models. The theme of the present invention is to go beyond static rule-based systems by introducing a way to monitor not just the inputs and outputs, but also the reasoning steps taken by LLM agents. By leveraging an auxiliary LLM specifically trained to assess potential risks such as prompt injections, policy violations, hallucinations, or unsafe outputs the system can evaluate each step in real time. It can then decide whether to allow, block, or modify the output, or escalate it to a human operator. This entire process is supported by a feedback loop that learns from past outcomes and helps improve the system’s ability to handle future risks, making it more reliable and responsive over time.
According to an aspect of the present invention, the system comprises of an input unit , a processing unit and output unit , wherein the processing unit further comprises of primary LLM agent(s) module, secondary audit LLM agent module, audit orchestrator module, security policy database module and feedback loop module. The primary LLM Agent(s) module executes the main task such as coding, release planning, test cases, test plan, business plan, etc. in an agentic workflow. The Audit LLM agent module is a secondary agent which is a specialized language model trained or fine-tuned to identify security flaws, policy violations, or risk vectors. The secondary audit LLM uses prompts, context and tools that are designed to review. Example of Prompts are -“you are a reviewer who will review this task”, example of context - use this guide or governance or objectives and review against them, and example of tools- generate diff to find what changed etc. The Audit Orchestrator module is a control unit that intercepts the input and output during inferencing and dispatches them for validation. The Security Policy database module stores the predefined validation rules, compliance requirements, and dynamic learning logs at the reviewer stage. The Feedback Loop module is used for adaptive fine-tuning and updating the threat models.
According to an embodiment of the invention, the process flow begins when a user provides an input to the agentic system. This system forwards the task to the primary LLM agent module for processing. Upon processing the task, the primary LLM agent module transmits a tuple comprising the input (I), the reasoning (R), and the output (O) to the Audit Orchestrator module. The Audit Orchestrator then engages a specialized Audit LLM module to validate the received tuple for correctness, integrity, or potential risk. The Audit LLM module responds with a risk assessment report, which the Audit Orchestrator module uses to determine the appropriate course of action. Based on this decision, whether to allow or block the output, the Audit Orchestrator module instructs the primary LLM agent module accordingly. Finally, the primary LLM agent module communicates the approved final output back to the agentic system for delivery to the user. This modular and auditable design enhances trust, reliability, and governance in the deployment of AI agents.
According to the aspect of the present invention, the method for method for orchestrating security audits in agentic workflows using auxiliary large language models as described in FIG. 1 comprises the steps of :
• Receiving inputs from user to the Primary LLM Agent module: The workflow begins when the user provides an input to an agent powered by the primary large language model (LLM). This input could involve a query, task instruction, or command depending on the application context (e.g., coding, financial advice, orchestration).
• Intercepting data by Audit Orchestrator module: Before the agent proceeds to deliver an output, the Audit Orchestrator intercepts the full context, this includes the original user input, any intermediate reasoning or decision-making steps, and the agent’s proposed output. These elements are structured into a standardized data tuple for further analysis.
• Evaluating risks by the secondary audit LLM module: The structured tuple is transmitted to the secondary Audit LLM module, which is specially trained to evaluate security, compliance, and factual reliability. It checks for risks such as prompt injection attacks, data leakage, hallucinated content, unsafe commands, and policy violations.
• Generating Risk Score and Suggested Fix: The Audit LLM module returns a detailed report to the Audit Orchestrator module. This report includes a quantitative risk score as well as recommendations for mitigation, which may include suggested modifications to the output or warnings regarding identified threats.
• deciding whether the output is allowed or replaced: Based on the returned risk score and mitigation advice, the Audit Orchestrator makes a policy-informed decision. It either approves the original output, blocks it, or replaces it with a safer, corrected version generated with input from the Audit LLM.
• Logging and Feedback for Continuous Learning: Each transaction in the workflow, including input, reasoning, risk analysis, decision outcomes, and final output, is logged in a secure audit trail. This log is used to refine and update security policies and the Audit LLM’s learning over time, enabling the system to become more adaptive and resilient to emerging risks.
According to the embodiment of the invention, the present invention discloses a method for orchestrating security audits within agentic workflows involving large language model (LLM) agents. The method includes intercepting and capturing the input provided to the primary LLM agent, the intermediate reasoning generated during task processing, and the final output produced. These elements are organized into a structured tuple and transmitted to the secondary Audit LLM module which is specifically trained to identify a range of security risks, including but not limited to prompt injection attacks, data leakage, hallucinations, policy violations, and unsafe command generation. The Audit LLM module analyzes the tuple, assigns a risk score, and suggests appropriate mitigations. Based on this evaluation, the system dynamically determines one or more actions such as permitting the original output, substituting it with a secure alternative, or triggering an alert. Additionally, the method incorporates a continuous feedback loop that enables adaptive refinement of the security audit policies over time, thereby enhancing the robustness and trustworthiness of agentic AI systems.
According to the embodiment of the present invention, in the given system and method, every step taken by an autonomous agent powered by a large language model (LLM) is monitored by a dedicated security orchestration layer. For each agent action, the system captures the input, intermediate reasoning, and the generated output and sends this package to a secondary LLM—called the Audit LLM—designed to assess security risks. This Audit LLM checks for issues like data leaks, code injections, hallucinations, and policy violations and returns a detailed risk report with a severity score and recommended actions. If the risk score crosses a defined threshold, the system can block the unsafe output, substitute it with a secure version, or escalate the issue to a human operator. All events are logged into a learning database, creating a feedback loop that helps the system improve its understanding of emerging threats, ensuring continuous protection in dynamic, multi-agent AI environments.
According to the embodiment of the present invention, the Audit LLM module performs a comprehensive suite of validations on the agentic workflow to ensure that the generated responses and underlying processes are secure, compliant, and trustworthy. These validations include:
1. Prompt Injection Detection: The Audit LLM analyses user inputs and agent responses for signs of prompt injection attack, where maliciously crafted inputs attempt to manipulate the behaviour or logic of the language model. This validation helps prevent unauthorized override of system instructions or unintended execution paths.
2. Sensitive Data Leakage Prevention: The system checks for unintentional exposure of sensitive or protected information, including personally identifiable information (PII), access credentials, private keys, or internal metadata. This validation is critical for maintaining data privacy and preventing regulatory violations.
3. Command or Shell Injection Identification: If the LLM is involved in generating code, automation scripts, or command-line instructions, the Audit LLM scans for unsafe or malicious commands that could result in command injection vulnerabilities or unauthorized system access.
4. Policy Violation Detection: The Audit LLM validates outputs against predefined organizational, legal, and regulatory policies or internal ethical guidelines. It flags content that could lead to compliance breaches or reputational risk.
5. Hallucination Risk Assessment: The Audit LLM evaluates the factual correctness of the output, particularly in high-stakes domains like healthcare, finance, or scientific research. It flags outputs that are inconsistent with trusted knowledge sources or lack verifiable basis, thereby mitigating risks of hallucination.
6. External API Call Scrutiny: For workflows involving external API calls, the Audit LLM inspects the structure, intent, and parameters of such calls to detect misuse, leakage, or unsafe interactions.
Advantages:
The disclosed method offers several key advantages, including proactive security enforcement that operates without compromising the autonomy of agentic systems. It features a real-time adaptive feedback loop, allowing the system to continuously improve its risk detection and mitigation capabilities. By identifying and addressing hallucinations and unsafe behaviors, the method ensures safer and more reliable performance in production environments. Furthermore, the architecture is designed to be modular, scalable, and model-agnostic, making it compatible with a wide range of deployment scenarios.
Potential applications include autonomous coding agents, financial decision-making bots, multi-agent scientific workflows, and systems that rely on LLM-based orchestration across distributed components.
Example:
For each agent step:
Capture Input (I), Intermediate Reasoning (R), and Out
Package (I, R, 0) as Tuple T
Send T to Audit LLM
Audit LLM returns Risk Report R(T): [Score, Tags, Sugg
If Score > Threshold:
Perform Mitigation:
-Block Output
-Replace Output with Safe Alternative
-Raise Alert
-Log Incident
Else:
Allow execution
, Claims:We claim,
1. A system and method for orchestrating security audits in agentic workflows
characterized in that
the system leverages Large Language Models (LLMs) to orchestrate and perform security audits on inputs and outputs during an agentic workflow involving autonomous agents powered by LLMs;
the system comprises of an input unit , a processing unit and output unit , wherein the processing unit further comprises of primary LLM agent(s) module, secondary audit LLM agent module, audit orchestrator module, security policy database module and feedback loop module;
the method for method for orchestrating security audits in agentic workflows comprises the steps of :
• receiving inputs from user to the primary LLM agent module;
• intercepting data by audit orchestrator module;
• evaluating risks by the secondary audit LLM module;
• generating risk score and suggested fix;
• deciding whether the output is allowed or replaced;
• logging and feedback for continuous learning.

2. The system and method as claimed in claim 1, wherein the secondary audit LLM agent module is a specialized language model trained and fine-tuned to identify security flaws, policy violations, or risk vectors.

3. The system and method as claimed in claim 1, wherein the Security Policy database module stores the predefined validation rules, compliance requirements, and dynamic learning logs.

4. The system and method as claimed in claim 1, wherein the Audit Orchestrator module is a control unit that intercepts the input and output during inferencing and dispatches them for validation.

5. The system and method as claimed in claim 1, wherein the Feedback Loop module is used for adaptive fine-tuning and updating the threat models.

6. The system and method as claimed in claim 1, wherein the process flow begins when a user provides an input to the agentic system, the system forwards the task to the primary LLM agent module for processing, then upon processing the task, the primary LLM agent module transmits a tuple comprising the input, the reasoning, and the output to the Audit Orchestrator module, which then engages a specialized Audit LLM module to validate the received tuple for correctness, integrity, or potential risk and it responds with a risk assessment report, which the Audit Orchestrator module uses to determine the appropriate course of action and based on this decision, whether to allow or block the output, the Audit Orchestrator module instructs the primary LLM agent module accordingly and finally, the primary LLM agent module communicates the approved final output back to the agentic system for delivery to the user.

7. The system and method as claimed in claim 1, wherein the Audit LLM module analyzes the tuple, assigns a risk score, and suggests appropriate mitigations.

8. The system and method as claimed in claim 1, wherein the types of Validations by Audit LLM module are Prompt Injection Detection, Sensitive Data Leakage , Command or Shell Injection, Policy Violations , Hallucination Risk and External API call scrutiny.