Description:The present invention relates generally to the fields of software engineering, artificial intelligence, and development tooling. More specifically, it relates to dynamic orchestrator system and method for converting code editor plugins into LLM-compatible tools for agentic systems that enables 5 large language models (LLMs) to autonomously discover and interpret the code editor plugins as callable tools within agentic AI frameworks.
BACKGROUND OF THE INVENTION Code editor plugins have become an integral part of modern software development workflows. 10 Platforms such as Visual Studio Code, IntelliJ, and other popular editors support a wide range of plugins that offer utilities such as linters, formatters, runners, code generators, and more. These plugins are typically installed from online marketplaces and contribute valuable enhancements by integrating deeply into the editor environment. However, these capabilities are primarily tailored for human developers, relying on graphical interfaces, manual configuration, and context-based 15 interactions. While plugin ecosystems have matured significantly, their interfaces are not readily accessible or compatible with large language model (LLM)-based systems that now play an increasing role in code generation, refactoring, and task automation. Despite the growing interest in enabling LLMs to act as autonomous agents in software engineering workflows, there exists no effective method to convert these human-centric plugins 20 into tools that can be programmatically invoked during LLM inference. Manually wrapping each plugin is impractical given the scale and diversity of available tools, and many plugin behaviours are only documented through sparse text or unstructured video demonstrations. The existing systems do not support automatic ingestion of plugin metadata, nor do they provide mechanisms to analyse plugin behaviour using static analysis or multimodal understanding. This highlights the 25 need for a dynamic, intelligent system that can bridge the gap between traditional editor plugin ecosystems and agentic LLM frameworks. The present invention addresses this need by introducing a dynamic orchestrator that discovers, interprets, and transforms code editor plugins
3
into standardized, callable tools that LLMs can seamlessly leverage during real-time inference workflows. Prior arts- US11908476B1 presents a method for generating interface models based on video demonstrations. While it effectively uses audio and visual inputs to infer user interactions, it is centered around 5 learning user flows rather than transforming editor plugins into AI-compatible tools. It lacks the ability to discover and understand plugins, analyze their static and runtime behaviour, or convert them into callable interfaces suitable for real-time LLM use. US20240143296A1 explores how code-based LLMs can work with compilers to improve output accuracy. Although it demonstrates orchestration between LLMs and development tools, its scope 10 is limited to compiler integrations. It does not address the wider plugin ecosystem, nor does it propose ways to automatically extract plugin behavior, wrap functions into APIs, or expose them for agentic workflows using multimodal analysis. US12266107B1 discloses a system for agent orchestrators that invoke tools in response to user queries. However, it does not address the challenge of interfacing with existing editor plugins. It 15 lacks plugin-specific behavior extraction, static code parsing, video and document understanding, or a user-facing wizard for customizing tool parameters. It does not offer real-time integration with live editor sessions or mechanisms for dynamic tool registration and chaining. While these prior inventions contribute to the broader field of artificial technology tool orchestration, none of them offer a comprehensive system that dynamically converts diverse code 20 editor plugins into structured, LLM-compatible tools. The present invention goes further by introducing a multimodal, context-aware, and extensible orchestrator that not only analyzes and wraps plugins but also integrates them directly into live inference workflows. This enables a new level of collaboration between LLM agents and traditional development environments.
25
4
DEFINITIONS
The expression “system” used hereinafter in this specification refers to an ecosystem comprising, but is not limited to a system with a user, input and output devices, processing unit, plurality of mobile devices, a display unit and output; and is extended to computing systems like mobile, laptops, computers, PCs, etc. 5
The expression “input unit” used hereinafter in this specification refers to, but is not limited to, mobile, laptops, computers, PCs, keyboards, mouse, pen drives or drives.
The expression “output unit” used hereinafter in this specification refers to, but is not limited to, an onboard output device, a user interface (UI), a display unit, a local display, a screen, a dashboard, or a visualization platform enabling the user to visualize the results provided as output 10 by the system.
The expression “processing unit” refers to, but is not limited to, a processor of at least one computing device that optimizes the system, and acts as the functional unit of the system.
The expression “large language model (LLM)” used hereinafter in this specification refers to a type of machine learning model designed for natural language processing tasks such as language 15 generation. LLMs are language models with many parameters, and are trained with self-supervised learning on a vast amount of text.
The expression “plugin” used hereinafter in this specification refers to a functional component that adds specific features or functionality to a code editor or IDE (integrated development environment) activated through predefined APIs (application programming interface) or user 20 actions.
The expression “code editor plugin” used hereinafter in this specification refers to a plugin designed specifically for code editors, enabling features like linting, formatting, debugging, or code generation for human users
The expression “dynamic orchestrator” used hereinafter in this specification refers to the core 25 system that automatically discovers, analyses, and converts code editor plugins into tools usable by LLMs, managing their lifecycle and integration.
5
The expression “multimodal analysis” used hereinafter in this specification refers to a method that combines data from code, text, audio, and visual inputs to understand the behaviour and functionality of plugins.
The expression “static analysis” used hereinafter in this specification refers to the examination of plugin source code and configuration files without execution to extract metadata and structural 5 information.
The expression “dynamic instrumentation” used hereinafter in this specification refers to the execution of plugins in an isolated environment to monitor runtime behaviour through tracing of inputs, outputs, and UI changes.
The expression “API synthesizer” used hereinafter in this specification refers to a component that 10 transforms plugin functions into callable APIs with structured schemas suitable for LLM tools.
The expression “wizard UI” used hereinafter in this specification refers to an interactive, step-by-step interface within the editor that guides users in configuring and exporting plugin tools.
The expression “sandboxed environment” used hereinafter in this specification refers to a secure, isolated setup used to safely execute plugins and observe their behaviour during dynamic analysis. 15
The expression “sandboxed environment” used hereinafter in this specification refers to a formal representation of a plugin’s function, detailing its name, parameters and usage constraints. The expression “DOM hooks” used hereinafter in this specification refers to programmatic interception points attached to elements of the Document Object Model (DOM) of an application (like a code editor built on web technologies, e.g., VS Code’s electron-based frontend); thereby 20 allowing the orchestrator to monitor changes to UI elements, events, or attributes triggered during plugin execution, and optionally alter or extend those behaviors.
The expression “JSON schema” used hereinafter in this specification refers to a declarative specification that describes the structure, constraints, and validation rules of data represented in JavaScript Object Notation (JSON); that defines what a valid JSON object should look like in 25 terms of properties, data types, required fields, formats, and default values.
6
The expression “test invocation” used hereinafter in this specification means executing the newly generated plugin-tool interface (the wrapped API) with sample or mock inputs in order to verify that the tool runs correctly when invoked outside its native UI, the inputs and outputs conform to the generated JSON schema, and the tool’s behavior matches the original plugin functionality as observed through instrumentation or documentation. 5
OBJECTS OF THE INVENTION The primary object of the present invention is to provide a dynamic orchestrator system and method for converting code editor plugins into LLM-compatible tools for agentic systems. Another object of the present invention is to provide a dynamic orchestrator system configured to 10 enable automatic discovery and conversion of code editor plugins into (LLM)-compatible tools for invocation within agentic systems. Another object of the invention is to facilitate LLM interaction with such plugins through standardized APIs, eliminating the need for manual integration and thereby improving scalability and automation. 15 A further object of the invention is to utilize multimodal analysis, including Natural Language Processing (NLP), Automatic Speech Recognition (ASR), and Computer Vision techniques, for inferring plugin behavior from unstructured documentation. Yet a further object is to support dynamic instrumentation of plugins executed within sandboxed to capture runtime behavior, input parameters, and output responses. 20 An additional object of the invention is to provide an interactive wizard-based user interface integrated within the code editor to assist users in the step-by-step configuration, validation, and packaging of plugin tools.
SUMMARY 25
7
The present invention discloses a dynamic orchestrator for converting code editor plugins into LLM-compatible tools for agentic systems, comprising an input unit, a processing unit further comprising of a plugin discovery engine, a behavior extraction layer, a dynamic instrumentation module, an API synthesizer and tool wrapper, an interactive wizard UI, an editor runtime integration, an LLM inference orchestrator, and an output unit. The plugin discovery engine 5 initiates the process by scanning local installations and online marketplaces such as the VSCode Marketplace or JetBrains Plugin Repository. It leverages plugin manifest formats like package.json or plugin.xml to extract detailed metadata including plugin type, version, dependencies, extension points, and contributed commands. A trained classification model then organizes these plugins into categories, such as linters, formatters, runners, and generators, allowing downstream 10 components to process them intelligently.
The behavior extraction layer follows with a hybrid analysis approach. It uses static analysis techniques such as AST parsing to identify plugin functions, input forms, configuration schemas, and exposed commands. Additionally, it applies NLP to documentation assets like READMEs and help files to extract usage patterns, parameters, and API or CLI command mappings. For more 15 complex plugins with video content, a multimodal pipeline is employed: video URLs are retrieved and processed through Whisper for transcription, then analyzed with CLIP for visual interaction classification. These insights are aligned to generate an inferred action graph capturing expected inputs and outputs.
The dynamic instrumentation module activates a sandboxed or headless instance of the editor, such 20 as VSCode with development flags. It injects tracing mechanisms via monkey-patching editor APIs (e.g., vscode.commands.registerCommand) and uses developer tools and DOM hooks to monitor runtime behavior. This includes capturing input parameters, return values, visual feedback, and generating dynamic call graphs from execution traces—automatically grouping similar calls for streamlined processing. 25
The API synthesizer and tool wrapper transforms plugin behavior into callable interfaces like REST, JSON-RPC, or gRPC by normalizing the extracted functions. It generates JSON schemas for each tool, validates parameter types and defaults, and wraps them into interoperable formats for systems like OpenAI Tools, LangChain, or AutoGen. A UI wizard embedded within the editor
8
assists users in customizing, testing, and packaging the plugin-based tools. Runtime integration ensures LLM agents can invoke these tools directly within the editor, update the code contextually, and orchestrate complex workflows via chaining or multi-agent planning and bridging the gap between raw plugin capabilities and intelligent automation through LLMs.
5
BRIEF DESCRIPTION OF DRAWINGS
A complete understanding of the present invention may be made by reference to the following detailed description which is to be taken in conjugation with the accompanying drawing. The accompanying drawing, which is incorporated into and constitutes a part of the specification, illustrates one or more embodiments of the present invention and, together with the detailed 10 description, it serves to explain the principles and implementations of the invention.
FIG.1. illustrates a schematic representation of the structural and functional components of the system.
FIG.2. illustrates a sequence diagram.
FIG.3. illustrates a multimodal extraction flow. 15
FIG.4. illustrates an agentic tool invocation pipeline.
FIG. 5. illustrates an overall system architectural flowchart.
DETAILED DESCRIPTION OF INVENTION
Before the present invention is described, it is to be understood that this invention is not limited to 20 methodologies described, as these may vary as per the person skilled in the art. It is also to be understood that the terminology used in the description is for the purpose of describing the particular embodiments only and is not intended to limit the scope of the present invention. Throughout this specification, the word “comprise”, or variations such as “comprises” or “comprising”, will be understood to imply the inclusion of a stated element, integer or step, or 25
9
group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps. The use of the expression “at least” or “at least one” suggests the use of one or more elements or ingredients or quantities, as the use may be in the embodiment of the invention to achieve one or more of the desired objects or results. Various embodiments of the present invention are described below. It is, however, noted that the present invention is not 5 limited to these embodiments, but rather the intention is that modifications that are apparent are also included.
The present invention describes a dynamic orchestrator system and method for converting code editor plugins into LLM-compatible tools for agentic systems. The said dynamic orchestrator system [10] as illustrated in Fig. 1. comprises of an input unit [1], a processing unit [2] further 10 comprising of a plugin discovery engine [21], a behavior extraction layer [22], a dynamic instrumentation module [23], an API synthesizer and tool wrapper [24], an interactive wizard UI [25], an editor runtime integration [26], an LLM inference orchestrator [27], and an output unit [3].
In an embodiment of the invention, input is provided using an input unit [1] to the processing unit 15 [2], wherein the plugin discovery engine [21] scans local installations and online marketplaces such as the VSCode Marketplace and JetBrains Plugin Repository. It uses plugin manifest formats like package.json and plugin.xml to extract metadata, including plugin type, version, dependencies, extension points, and contributed commands. A trained classification model is then used to categorize the plugins into types such as linters, formatters, runners, and generators. 20
In yet a next embodiment of the invention, the behavior extraction layer [22] wherein plugin analysis via multimodal knowledge extraction in which, static analysis using AST parsing is performed to extract functions, commands, input forms, and configuration schemas from plugin source code. NLP-based analysis of certain files including, but not limited to README files, help files, and documentation is carried out to extract sample usage patterns, expected parameters, and 25 CLI/API mappings. A multimodal analysis pipeline is implemented, wherein video URLs are retrieved from plugin metadata or through external search, transcribed using Whisper (ASR), and frame-sampled for classification with CLIP to detect interactions. An alignment model is then used
10
to correlate the transcript with visual actions, enabling the inference of action graphs and input/output expectations.
In yet a next embodiment of the invention, the dynamic instrumentation modular [23] launches a headless or sandboxed instance of the editor (e.g., VSCode using --extensionDevelopmentPath), and tracing code is injected via monkey-patching editor APIs (e.g., 5 vscode.commands.registerCommand). Input parameters, return values, and visual changes are captured using DOM hooks and developer tools automation; wherein the DOM hooks are crucial for dynamic instrumentation that enable capturing runtime behavior of plugins in a sandboxed editor; for multimodal alignment that enable matching UI changes (visual cues) with audio/video/documentation analysis; and for tool wrapping that enable inferring input-output 10 mappings so the plugin can be exposed as an API for LLMs. A dynamic call graph is then built with execution traces, automatically grouping similar invocations.
In yet a next embodiment of the invention, the API synthesizer and tool wrapper [24] or tool wrapping and API exposure, it normalizes into callable endpoints using language-agnostic representations, and plugin operations are converted into REST, JSON-RPC, or gRPC interfaces 15 based on configuration. A JSON schema is created for each plugin tool or the tool interface with validated parameter types and default values, ensures predictable interaction between the LLM agent and the wrapped plugin, and is then provided as a universal representation so tools can be registered across different frameworks such as OpenAI Tool, LangChain Tool, or AutoGen Tool formats. The tool behavior is validated through test invocation and result comparison. 20
In yet a next embodiment of the invention, the interactive wizard UI [25] is a step-by-step UI within the code editor enabled to selecting a discovered plugin; previewing the extracted behavior and flow; customizing input/output parameter names, descriptions, and data types; mapping contextual elements (file path, selection, cursor) into input fields, testing the plugin operation with mock values, and finalizing tool packaging for export to the registry. Further, the editor runtime 25 integration [26] uses internal APIs (e.g., vscode.workspace.applyEdit, editor.insertSnippet) to apply plugin results to code, hooks into the editor event bus to reflect changes triggered by LLM tool invocations, and enables bi-directional communication between the LLM agent and the live code editor session.
11
In yet a next embodiment of the invention, LLM inference integration [27] or real time inference orchestration, it registers tools dynamically during the chat session with available metadata, uses tool descriptions to inform tool selection in multi-agent planners, injects tool results into the LLM context either via prompt expansion or context window streaming, and supports chaining multiple plugin-tools in workflow graphs for complex task decomposition. 5
According to an embodiment of the invention, FIG. 2. illustrates a sequence diagram of the components of the system and its workflow; comprising the steps as follows:
-
installing the plugin into the editor by the user,
-
registering the plugin with the orchestrator using the editor,
-
analyzing the plugin manifest and source using the orchestrator, 10
-
fetching the documentation and videos from the Internet using the orchestrator,
-
extracting the behavior by the orchestrator using LLM, CLIP, and Whisper,
-
opening the wizard UI by the orchestrator for user interaction,
-
mapping the inputs and outputs in the wizard,
-
creating and returning a final tool specification to the orchestrator, 15
-
invoking the tool with the mapped inputs using the orchestrator,
-
executing the requested function by the plugin and returning the result,
-
injecting the result into the LLM context using the orchestrator,
-
enabling the result to be surfaced back in the editor or wizard UI for the user.
According to an embodiment of the invention, FIG. 3. illustrates a multimodal extraction flow 20 wherein the system enables a workflow comprising the steps of:
-
providing plugin ID or URL as the entry point for discovery,
-
searching YouTube Tutorial for relevant video tutorials related to the plugin,
-
enabling Whisper transcription after searching tutorial; thereby transcribing the audio of the selected videos using Whisper ASR, 25
-
producing a transcript i.e. the textual output the Whisper transcription,
-
analyzing the CLIP Visual Frame Classifier alternatively, the sampled video frames with CLIP to identify visual intents,
-
enabling frame intents, the visual actions or UI elements inferred from frame classification,
12
-
aligning audio & visual signals such as the transcript with detected frame intents to correlate spoken instructions with on screen actions.
-
inferring an action graph from the aligned audio and visual signals to model sequential actions and dependencies,
-
alternatively fetching documentation after providing plugin ID or URL, thereby retrieving 5 the plugin README and other written resources,
-
parsing LLM based documentation using an LLM to extract functions, usage patterns, and schemas from the fetched documentation,
-
constructing function schema by combining insights from the action graph and parsed documentation into a validated tool interface. 10
According to an embodiment of the invention, FIG. 4. illustrates an agentic tool invocation pipeline; wherein the LLM Agent Plan formulates a high-level sequence of actions and goals for the agent comprising the steps of;
-
choosing the most appropriate plugin or tool to fulfill each step of the plan using a tool selector, 15
-
determining and filling the required input parameters using contextual data, defaults, or user prompts by parameter resolver,
-
invoking the chosen plugin through the orchestrator using the resolved parameters by the plugin tool invocation,
-
executing the plugin runs using the editor wherein the plugin within the editor environment 20 is configured to perform the requested operation,
-
returning the output to orchestrator where the system collects the plugin’s result and sends it back to the orchestrator for processing,
-
injecting the returned results into LLM context so the agent can update its plan or respond.
According to yet another embodiment, the FIG. 5. illustrates an overall system architectural 25 flowchart; wherein a first stepwise workflow comprises the steps including:
-
the plugin discovery scans the local and marketplace where it discovers and indexes plugins from local sources and online marketplaces;
-
the behavior extraction docs, video, clip and whisper analyzes documentation and tutorial videos, using clip for visual frames and whisper for audio transcription to extract behavior; 30
13
-
the sandbox & tracing editor hooks and API monitoring launches sandboxed editor instances and instruments editor APIs to capture runtime traces and monitor activity;
-
or parallelly the API and tool wrapper REST/JSON schema generation normalizes extracted behaviors into rest/json schemas and wraps them as callable tools;
-
followed by tool registration store in orchestrator registry enables registering finalized tools 5 and stores their metadata and schemas in the orchestrator registry; and
-
finally, the LLM result injection output is passed back to the model wherein the system injects the plugin’s output into the LLM context so the model can update its plan or present results;
and wherein a simultaneous alternate stepwise workflow comprises the steps including: 10
-
the plugin discovery scans the local and marketplace where it discovers and indexes plugins from local sources and online marketplaces;
-
the wizard UI user maps inputs and outputs thereby allowing the users to map, customize, and validate input and output parameters for the generated tool;
-
the LLM tool selection agent chooses best plugin tool based on the agent’s plan and tool 15 descriptions;
-
or parallelly the API and tool wrapper REST/JSON schema generation normalizes extracted behaviors into rest/json schemas and wraps them as callable tools;
-
the plugin execution parameters are sent and the function runs invokes the chosen plugin with resolved parameters and executes the requested function within the editor; and 20
-
finally, the LLM result injection output is passed back to the model wherein the system injects the plugin’s output into the LLM context so the model can update its plan or present results.
Advantages:
According to an embodiment, the system and method of the present invention offers several 25 groundbreaking advantages such as being the first system to integrate multimodal AI—combining Whisper, CLIP, and LLMs—to reverse-engineer plugin functionality directly from user-generated content. The system enables automatic behavior graph generation by aligning code with corresponding video segments, streamlining understanding and documentation. A reusable API
14
abstraction layer ensures compatibility with any plugin, regardless of language or platform. The context-aware wizard intelligently auto-suggests parameter names and documentation fields based on code snippets and user context, reducing manual effort. Runtime dynamic tool registration eliminates the need to restart the LLM inference session, while the editor-LLM co-execution loop delivers real-time feedback and file updates for faster iterations. Its agent-based architecture 5 efficiently orchestrates multi-step tasks by chaining tools within the plugin ecosystem, enhancing scalability and automation.
While considerable emphasis has been placed herein on the specific elements of the preferred embodiment, it will be appreciated that many alterations can be made and that many modifications can be made in preferred embodiment without departing from the principles of the invention. These 10 and other changes in the preferred embodiments of the invention will be apparent to those skilled in the art from the disclosure herein, whereby it is to be distinctly understood that the foregoing descriptive matter is to be interpreted merely as illustrative of the invention and not as a limitation. , C , Claims:CLAIMS:
We claim,
1.
A dynamic orchestrator system and method for converting code editor plugins into LLM-compatible tools for agentic systems; wherein the said dynamic orchestrator system [10] comprises of an input unit [1], a processing unit [2] with a plugin discovery engine [21], a 5 behavior extraction layer [22], a dynamic instrumentation module [23], an API synthesizer and tool wrapper [24], an interactive wizard UI [25], an editor runtime integration [26], an LLM inference orchestrator [27], and an output unit [3];
characterized in that: 10
-
the plugin discovery engine [21] scans local installations and online marketplaces, uses plugin manifest formats to extract metadata, and categorizes plugins via a trained classification model;
-
the behavior extraction layer [22] performs multimodal knowledge extraction via AST parsing, NLP-based analysis of files (README, help, docs) to get usage patterns, 15 parameters, and mappings; retrieves and processes video URLs using whisper and clip; aligns transcript with visual actions to infer action graphs and input or output expectations;
-
the dynamic instrumentation modular [23] launches headless or sandboxed editor instance, injects tracing code, captures parameters, return values, and visual changes, 20 and builds a dynamic call graph grouping similar invocations;
-
the API synthesizer and tool wrapper [24] normalizes plugin operations into callable REST/JSON-RPC/GRPC endpoints, creates JSON schema with validated parameters, wraps into OpenAI/LangChain/Autogen tool formats, and validates behavior through test invocation; 25
-
the interactive wizard UI [25] acts as a step-by-step editor UI for selecting plugins, previewing behavior, customizing parameters, mapping contextual elements, testing with mock values, and finalizing tool packaging for export;
16
-
editor runtime integration [26] applies plugin results to code via internal APIs, hooks into event bus for LLM-triggered changes, and enables bi-directional communication between LLM agent and live code editor session;
-
LLM inference integration or real time inference orchestration [27] registers tools during chat, uses descriptions for multi-agent planning, injects results into LLM 5 context, and chains plugin-tools in workflow graphs for complex task decomposition.
2.
The orchestrator system as claimed in claim 1, wherein the sequential arrangement of the components and its workflow comprises the steps of;
-
installing the plugin into the editor by the user, 10
-
registering the plugin with the orchestrator using the editor,
-
analyzing the plugin manifest and source using the orchestrator,
-
fetching the documentation and videos from the Internet using the orchestrator,
-
extracting the behavior by the orchestrator using LLM, CLIP, and Whisper,
-
opening the wizard UI by the orchestrator for user interaction, 15
-
mapping the inputs and outputs in the wizard,
-
creating and returning a final tool specification to the orchestrator,
-
invoking the tool with the mapped inputs using the orchestrator,
-
executing the requested function by the plugin and returning the result,
-
injecting the result into the LLM context using the orchestrator, 20
-
enabling the result to be surfaced back in the editor or wizard UI for the user.
3.
A method for converting code editor plugins into LLM-compatible tools; wherein a first stepwise workflow comprises the steps including:
-
scanning the local sources and marketplaces by the plugin discovery where it discovers 25 and indexes plugins from these local sources and online marketplaces;
-
analyzing documentation and tutorial videos using the behavior extraction from the docs, video, CLIP and whisper such that the system uses CLIP for visual frames and Whisper for audio transcription to extract behavior;
17
-
launching the sandbox and tracing editor hooks and API monitoring enabling the sandboxed editor instances and instruments editor APIs to capture runtime traces and monitor activity;
-
normalizing extracted behaviors into REST/JSON schemas and wrapping them as callable tools; parallelly using the API and tool wrapper REST/JSON schema 5 generation;
-
registering finalized tools by tool registration store in orchestrator registry and storing their metadata and schemas in the orchestrator registry; and
-
passing the LLM result injection output finally back to the model wherein the system injects the plugin’s output into the LLM context so the model can update its plan or 10 present results;
and wherein a simultaneous alternate stepwise workflow comprises the steps including:
-
scanning the local sources and marketplaces by the plugin discovery where it discovers and indexes plugins from these local sources and online marketplaces;
-
mapping inputs and outputs using the wizard UI thereby allowing the users to map, 15 customize, and validate input and output parameters for the generated tool;
-
choosing the best plugin tool by the LLM tool selection agent, based on the agent’s plan and tool descriptions;
-
normalizing extracted behaviors into REST/JSON schemas and wrapping them as callable tools; parallelly using the API and tool wrapper REST/JSON schema 20 generation;
-
sending the plugin execution parameters and the function runs invokes the chosen plugin with resolved parameters, thereby executing the requested function within the editor; and
-
passing the LLM result injection output finally back to the model wherein the system 25 injects the plugin’s output into the LLM context so the model can update its plan or present results.
4.
The method as claimed in claim 3, wherein a multimodal extraction flow enabled by the system comprises the steps of: 30
-
providing plugin ID or URL as the entry point for discovery,
18
-
searching YouTube Tutorial for relevant video tutorials related to the plugin,
-
enabling Whisper transcription after searching tutorial; thereby transcribing the audio of the selected videos using Whisper ASR,
-
producing a transcript i.e. the textual output the Whisper transcription,
-
analyzing the CLIP Visual Frame Classifier alternatively, the sampled video frames 5 with CLIP to identify visual intents,
-
enabling frame intents, the visual actions or UI elements inferred from frame classification,
-
aligning audio & visual signals such as the transcript with detected frame intents to correlate spoken instructions with on screen actions. 10
-
inferring an action graph from the aligned audio and visual signals to model sequential actions and dependencies,
-
alternatively fetching documentation after providing plugin ID or URL, thereby retrieving the plugin README and other written resources,
-
parsing LLM based documentation using an LLM to extract functions, usage patterns, 15 and schemas from the fetched documentation,
-
constructing function schema by combining insights from the action graph and parsed documentation into a validated tool interface.
5.
The method as claimed in claim 3, wherein the agentic tool invocation pipeline enabled by 20 the LLM agent plan formulates a high-level sequence of actions and goals for the agent comprising the steps of;
-
choosing the most appropriate plugin or tool to fulfill each step of the plan using a tool selector,
-
determining and filling the required input parameters using contextual data, defaults, 25 or user prompts by parameter resolver,
-
invoking the chosen plugin through the orchestrator using the resolved parameters by the plugin tool invocation,
-
executing the plugin runs using the editor wherein the plugin within the editor environment is configured to perform the requested operation, 30
19
-
returning the output to orchestrator where the system collects the plugin’s result and sends it back to the orchestrator for processing,
-
injecting the returned results into LLM context so the agent can update its plan or respond.
5
6.
The system and method as claimed in claim 1, wherein the components of the system integrates multimodal AI, combining Whisper, CLIP, and LLMs, to reverse-engineer plugin functionality directly from user-generated content; enables automatic behavior graph generation by aligning code with corresponding video segments, streamlining understanding and documentation; ensures compatibility with any plugin, regardless of 10 language or platform; auto-suggests parameter names and documentation fields based on code snippets and user context, reducing manual effort; eliminates the need to restart the LLM inference session; delivers real-time feedback and file updates for faster iterations; efficiently orchestrates multi-step tasks by chaining tools within the plugin ecosystem, enhancing scalability and automation.