Claims:We claim:
1. A system (100) for creating extended reality, wherein the system (100) provides a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface, the system (100) comprising:
- one or more user input terminal (101) connected to a network (103), the user input terminal (101) allowing one or more users to provide input and configured to output AR content, VR content, 3D content, 360 degree content using an HTML5 web environment-based VR and AR interaction coding service related web page, program, or simulation;
- a server (104) configured to provide one or more interactive content services to the one or more users; and
- one or more output request terminal (102) configured to access the server (104) through the network (103) and upload the produced AR content or VR content by running a script in a web browser (114) to the server (104),
wherein said at least one user input terminal (101) is a terminal interworking with one or more output peripheral devices including smart phone, smart pad, laptop, desktop, HMD and controller, and
wherein the server (104) comprises:
- one or more databases,
- at least one content development module,
- at least one sensing and detecting module configured to detect access to the user input terminal (101),
- a control module (105) with code input information acquisition unit configured to toggle output content in AR, VR, custom from web page accessed from the user input terminal (101) selection, and input to at least one user interface of the user input terminal (101),
- a content converter module configured to receive the produced AR content and VR content and convert into corresponding HTML5 platform,
- an output node, and
- at least one output manager request terminal configured to manage the user input terminal (101) to output AR content and VR content from the accessed web page.

2. The system (100) for creating extended reality as claimed in claim 1, wherein the user input terminal (101) is one or more of laptop, desktop, mobile phone and tablet supporting a plurality of web browser (114).

3. The system (100) for creating extended reality as claimed in claim 1, wherein the one or more interactive content services provides an HTML5 web graphical coding environment-based VR and AR interactive service web page, simulator, program, or application, uploading and transmitting an AR content or VR content request from at least one output request terminal, interaction corresponding to the providing input selection to the user input terminal (101) as 3D, AR or VR.

4. The system (100) for creating extended reality as claimed in claim 1, wherein the network (103) comprises one or more of LTE (Long Term Evolution) network (103), 5GPP (5th Generation Partnership Project) network (103), LAN (Local Area Network) (103), Wireless LAN (Wireless Local Area Network) (103), WAN (Wide Area Network) (103), and PAN (Personal Area Network) (103).
5. The system (100) for creating extended reality as claimed in claim 1, wherein the web page is an HTML5-based web browser (114) page comprising one or more of a function to play pictures, videos, and music.

6. A method (200) of creating extended reality, wherein the method (200) provides a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface, the method (200) comprising steps of:
- accessing (201) a web page, by a user input terminal (101), the web page providing an interactive content based on user selection as 3D/AR/VR, with geographical coding environment;
- outputting (202) a list of coding blocks and control to perform function based on coding instruction mapped with 3D assets and free play scenes with animations; and
- detecting, sensing, transforming, and rendering (203) 3D to AR/VR content to be output from the web page accessed from the output request terminal;
- interactive simulations allowing server (204) comprising an output unit, mobile devices, computer web systems (100), smart TV browser (114), AR glasses, when a user selection is present in any of the modes, AR, VR or 3D at the interface of the user input terminal (101).

7. The method (200) of creating extended reality as claimed in claim 6, comprising managing, by an output manager request terminal, the AR content to be output from the user input terminal (101) by integrating image recognition to enable AR rendering within the web browser (114).

8. The method (200) of creating extended reality as claimed in claim 6, comprising:
- providing a tool bar area defined within an enhanced graphical coding environment, the tool bar area having defined graphical blocks and 3D assets blocks;
- providing a free play workspace area defined within the enhanced graphical coding environment, the free play area permitting the connection of one more graphical blocks and 3D assets models/environment together to form the simulation; and
- dragging from the toolbar area one or more graphical blocks and 3D assets to add the one or more graphical blocks and 3D assets to an existing program simulation or content.

9. The method (200) of creating extended reality as claimed in claim 6, comprising:
- automatically detecting, by the enhanced graphical coding environment, freespace work in 3D models, environment, textures, particle effect, text, audio; and
- automatically changing the toolbar area to a block drag and dropdown toolbar.

10. The method (200) of creating extended reality as claimed in claim 6, comprising:
- playing VR simulation to the user input terminal (101), when a Head Mounted Display (HMD) linked with the user input terminal (101) is detected; and
- allowing, toggling, and transforming the VR content from 3D simulation, when a user selection exists as a VR through the user interface while initializing content type of the user input terminal (101).

11. The method (200) of creating extended reality as claimed in claim 6, comprising providing the interactive simulations based on sensor-based user’s eye data sets by the HMD or input data of at least a control module (105) in the server (104) and linked with the user input terminal (101).

, Description:FIELD OF THE INVENTION
The present invention relates to a system and a method for creating extended reality, and particularly, a WebXR environment based 3D, AR, and VR interactive coding platform for extended reality.

BACKGROUND OF THE INVENTION

Background description includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art.
Web Technology refers to the way computers/devices communicate via the internet using the markup language. A web browser is used to decode the markup language and render information on the display devices either laptop/desktop/mobile/wearable/head mounted/workstation. Web browsers can be defined as programs that display text, data, pictures, animation, and video on the Internet. Uniform Resource Locator(URL) on the World Wide Web(WWW) can be accessed using software interfaces provided by Web browsers.
Web Technology can be classified into the following sections:
a) World Wide Web (WWW): The World Wide Web is based on several different technologies : Web browsers, Hypertext Markup Language (HTML) and Hypertext Transfer Protocol (HTTP).
b) Web Browser: The web browser is an application software to explore www (World Wide Web). It provides an interface between the server and the client and requests to the server for web documents and services.
Web server is a program which processes the network requests of the users and serves them with files that create web pages. This exchange takes place using Hypertext Transfer Protocol (HTTP). Further, a web page is a digital document that is linked to the World Wide Web and viewable by anyone connected to the internet who has a web browser
Augmented reality “augments” virtual objects on top of the real world while virtual reality provides a computer generated virtual environment that gives the user an experience as if they are in a real situation in the virtual world. As a technology, virtual reality puts the user in a completely computerized environment, whereas augmented reality aims to directly register information in the real environment. The detailed technologies that make up virtual/augmented reality include Display Technology, Interaction Technology, Tracking Technology and Image Processing Technology. In addition, augmented reality majorly includes hand handled mobile devices AR Glasses. The market size of the industry is growing as a variety of virtual and augmented reality products are released.
Visual-based interface includes programming within an environment where instructions are mainly represented as blocks. Visual programming language is also referred to as block-based coding, block modality, graphical programming. Examples of block-based programming languages include Scratch, Modkit, Robotc Graphical, and many others. Blocks consist of one or more instructions that specify the declarations and statements belonging to logic of the program. In most block-based programming languages, blocks can be embedded within other blocks to complete the intent of the program. Blocks are fundamental to structured programming, in which control structures are created, variables are initiated, animations can be given, multi scenes can also be created by blocks. The aim is to use a visual interface so that young learners can boost their programming logic by using the functionality of dragging and dropping of the blocks into an editor workplace to create the program. The embodiment of the present invention removes the present barrier to entry in understanding the computer science concept without remembering the complex syntax or semantics with easy logical coding blocks.
A certain available art discloses methods and systems for a scripting framework and implementations thereof for mixed reality software applications of heterogeneous systems. These methods or systems create a mixed-reality software application that executes across heterogeneous platforms on a server-side instance of a scripting framework and manage a change in the mixed-reality software application using the server-side instance of the scripting framework. Moreover, the change in the mixed-reality software application using a client-side instance of the scripting framework; and the mixed-reality software application may be interactively executed on a mixed-reality device.
Another available art discloses systems, methods, and non-transitory computer readable media for generating trained neural network with increased robustness against adversarial attacks by utilizing a dynamic dropout routine and/or a cyclic learning rate routine.
At present, the conventional art has a technology that enables interaction when providing augmented reality content. It recognizes the event that occurs according to the interaction between the user and the virtual object, recognizes and transmits the information of the virtual object to the smart device. The smart device applies the user input for the virtual object to the augmented reality device. A configuration that implements augmented reality by transmitting and rendering a pre-stored virtual object based on the information of the deformed virtual object, and a configuration that supports marker-based user interaction so that learners can manipulate virtual content using markers/markerless as disclosed. However, the conventional technology does not offer child friendly 3D designing & coding interface.
There are a lot of 2D creator tools available, but there are no 3D tools which provide a one-stop solution for the kid’s education in 3D rather than 2D.
Also, to experience the interactive & immersive content in AR/VR users need to download the content from the respective application stores. This, not only burdens the user device storage, but also needs to have a good internet connection for downloading the resources as requested. If the capacity of the memory or storage device is continuously reduced, the user simply deletes the interest-oriented, non-essential applications.
Therefore, there is a need for technology with AR and VR interactive contents that do not depend on the user’s computing resources are required. Further, there is also a need to change the perspective of consumption in the education domain. Learning by doing is motivated among the student’s in the early phase of education so that they are empowered to create & innovate something new & different. With this education learning platform student are not only empowered with the skill of programming but also designing scenes in 3D/AR/VR.
Creating interactive AR/VR/3D content is expensive and time-consuming, especially in making them work on multiple devices (e.g., smartphones, laptops, smart TV) and within a single user hardware interface (e.g., Head Mounted Displays or HMDs).
Existing work and research are focused on the use of 3D professional software, such as CAD drawing, Unity 3D and Unreal, that require prior knowledge and mastery of programming and design. Because of this, the bulk of educational user programming is with advanced engineering students. Yet, research suggests younger learners benefit from experiences with software development and programming in enhancing their cognitive understandings of complex concepts, contents and creative skills.
In this regard, Unity3D approach is novel in significantly reducing the amount of time needed to create 3D models, but has a drawback that it requires a heavy demand for user’s programming skills.

OBJECT OF THE INVENTION

The present disclosure is aimed at providing a system and method for reducing time to create AR and VR content.
Further, the present disclosure is aimed at providing a system and method to create AR and VR content that does not require skilled programmers, and is economically viable.
In other words, the present disclosure is aimed at providing a system and method that emphasizes more on the logical & analytical skill rather than programming syntax & semantics for generating the interactive simulation.
A further object of the present disclosure is to allow the users to create their own AR and VR interactive content along with enabling them to experience and share their creative work through embedded links and QR codes with other learners across the world, thereby, enabling the system, to meet users’ needs of utility and ease of use, to facilitate their acceptance towards technology.
Yet further object of the present invention is to provide a platform where students collaboratively via group-work, create and design an AR and VR application.

SUMMARY OF THE INVENTION

This summary is provided to introduce a selection of concepts in a simplified format that are further described in the detailed description of the present disclosure. This summary is not intended to identify key or essential inventive concepts of the present disclosure, nor is it intended for determining the scope of the present disclosure.
According to an embodiment of the present disclosure, a system for creating extended reality is disclosed. The system provides a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface, and comprises one or more user input terminal connected to a network, the user input terminal allowing one or more users to provide input and configured to output AR content, VR content, 3D content, 360 degree content using an HTML5 web environment-based VR and AR interaction coding service related web page, program, or simulation; a server configured to provide one or more interactive content services to the one or more users; and one or more output request terminal configured to access the server through the network and upload the produced AR content or VR content by running a script in a web browser to the server, wherein at least one user input terminal is a terminal interworking with one or more output peripheral devices including smart phone, smart pad, laptop, desktop, HMD and controller.
In some embodiments, the server comprises one or more databases, at least one content development module, at least one sensing and detecting module configured to detect access to the user input terminal, a control module with code input information acquisition unit configured to toggle output content in AR, VR, custom from web page accessed from the user input terminal selection, and input to at least one user interface of the user input terminal, a content converter module configured to receive the produced AR content and VR content and convert into corresponding HTML5 platform, an output node, and at least one output manager request terminal configured to manage the user input terminal to output AR content and VR content from the accessed web page.
One embodiment of the present invention generates interactive 3D, AR and VR content, but does not depend on the user’s computing resources by performing HTML5 Canvas & WebXR support so that it can be driven only by accessing a web page without downloading an application or program. Visual block coding is introduced in the process of generating interactive content to support the events/action done on the user input terminal (101). It is possible to provide interactive & immersive content that is not dependent on the source, and stream VR interactive content to the user input terminal (101), based on eye tracking of HMD (Head Mounted Display) linked with user input terminal (101) and input data input from the controller to transform the VR interactive content, and provide a method (200) for VR and AR interaction content service based on HTML5 canvas web environment that outputs AR content through AR markers/markerless, but can transform AR interaction content based on even one data input from a user input terminal (101). However, the technical task to be achieved by the present embodiment is not limited to the technical problems as described above, and other technical tasks may exist.
According to an embodiment of the present disclosure, the user input terminal is one or more of laptop, desktop, mobile phone and tablet supporting a plurality of web browsers.
According to an embodiment, the one or more interactive content services provides an HTML5 web graphical coding environment-based VR and AR interactive service web page, simulator, program, or application, uploading and transmitting an AR content or VR content request from at least one output request terminal, interaction corresponding to the providing input selection to the user input terminal as 3D, AR or VR.
According to an embodiment, the network comprises one or more of LTE (Long Term Evolution) network, 5GPP (5th Generation Partnership Project) network, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), and PAN (Personal Area Network).
According to an embodiment, the web page is an HTML5-based web browser page comprising one or more of a function to play pictures, videos, and music.
According to an embodiment of the present disclosure, a method of creating extended reality is disclosed. The method provides a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface, the method (200) comprising steps of: accessing a web page, by a user input terminal, the web page providing an interactive content based on user selection as 3D/AR/VR, with geographical coding environment; outputting a list of coding blocks and control to perform function based on coding instruction mapped with 3D assets and free play scenes with animations; and detecting, sensing, transforming, and rendering 3D to AR/VR content to be output from the web page accessed from the output request terminal. This is followed by the step of interactive simulations allowing server comprising an output unit, mobile devices, computer web systems, smart TV browsers, AR glasses, when a user selection is present in any of the modes, AR, VR or 3D at the interface of the user input terminal.
According to an embodiment of the present disclosure, the method comprises managing, by an output manager request terminal, the AR content to be output from the user input terminal by integrating image recognition to enable AR rendering within the web browser.
According to an embodiment of the present disclosure, the method comprises providing a tool bar area defined within an enhanced graphical coding environment, the tool bar area having defined graphical blocks and 3D assets blocks; providing a free play workspace area defined within the enhanced graphical coding environment, the free play area permitting the connection of one more graphical blocks and 3D assets models/environment together to form the simulation; and dragging from the toolbar area one or more graphical blocks and 3D assets to add the one or more graphical blocks and 3D assets to an existing program simulation or content.
According to an embodiment of the present disclosure, the method comprises automatically detecting, by the enhanced graphical coding environment, freespace work in 3D models, environment, textures, particle effect, text, audio; and automatically changing the toolbar area to a block drag and dropdown toolbar.
According to an embodiment of the present disclosure, the method comprises playing VR simulation to the user input terminal, when a Head Mounted Display (HMD) linked with the user input terminal is detected; and allowing, toggling, and transforming the VR content from 3D simulation, when a user selection exists as a VR through the user interface while initializing content type of the user input terminal.
According to an embodiment of the present disclosure, the method comprises providing the interactive simulations based on sensor-based user’s eye data sets by the HMD or input data of at least a control module in the server and linked with the user input terminal.

BRIEF DESCRIPTION OF DRAWINGS

To further clarify the advantages and features of the present disclosure, a more particular description of the disclosure will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting of its scope. The disclosure will be described and explained with additional specificity and detail with the accompanying drawings.
The foregoing and other aspects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings.
Figure 1 illustrates a system for providing a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface, according to an embodiment of the present invention;
Figure 1a illustrates a graphical representation of the 3D editor window where the kids may have access to the Widgets Panel, Coding Editor, Property Panel and other gizmo tools like scale, position, rotation;
Figure 1b illustrates a way for application initialization in the front end;
Figure 2 is a block diagram illustrating an interactive content service providing server and client, according to an embodiment of the present invention;
Figure 3a illustrates an editor interface to design interactively in 3D/AR/VR, according to an embodiment of the present invention;
Figure 3b illustrates a coding interface to provide logic on user input terminal, according to an embodiment of the present invention;
Figure 3c illustrates a scene selection interface to create interactive in 3D/AR/VR based on user selection, according to an embodiment of the present invention;
Figure 3d illustrates a user authentication terminal interface, according to an embodiment of the present invention;
Figure 3e illustrates a public landing page gallery, according to an embodiment of the present invention;
Figure 3f illustrates generation of a Virtual Reality (VR) Scene developed by the innovation, according to an embodiment of the present invention;
Figure 3g illustrates generation of an Augmented Reality (AR) Scene developed by the user interface, according to an embodiment of the present invention;
Figure 4 illustrates real time user responses being recorded and admin rating the user content with AI data driven technique used CNN algorithms so as to give feedback based on the performance, according to an embodiment of the present invention; and
Figure 5 is a flowchart illustrating a method of providing a VR and AR interactive content service based on an HTML5 web environment, according to an embodiment of the present invention.
Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having benefit of the description herein.

DETAILED DESCRIPTION OF THE INVENTION

For the purpose of promoting an understanding of the principles of the invention, reference will now be made to the embodiment illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the invention is thereby intended. Such alterations and further modifications in the illustrated system, and such further applications of the principles of the invention as illustrated therein would be contemplated as would normally occur to one skilled in the art to which the invention relates. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skilled in the art. The system, methods, and examples provided herein are illustrative only and are not intended to be limiting.
The term “some” as used herein is to be understood as “none or one or more than one or all.” Accordingly, the terms “none,” “one,” “more than one,” “more than one, but not all” or “all” would all fall under the definition of “some.” The term “some embodiments” may refer to no embodiments or to one embodiment or to several embodiments or to all embodiments, without departing from the scope of the present disclosure.
The terminology and structure employed herein is for describing, teaching, and illuminating some embodiments and their specific features. It does not in any way limit, restrict or reduce the spirit and scope of the claims or their equivalents.
More specifically, any terms used herein such as but not limited to “includes,” “comprises,” “has,” “consists,” and grammatical variants thereof do not specify an exact limitation or restriction and certainly do not exclude the possible addition of one or more features or elements, unless otherwise stated, and furthermore must not be taken to exclude the possible removal of one or more of the listed features and elements, unless otherwise stated with the limiting language “must comprise” or “needs to include.”
Whether or not a certain feature or element was limited to being used only once, either way, it may still be referred to as “one or more features” or “one or more elements” or “at least one feature” or “at least one element.” Furthermore, the use of the terms “one or more” or “at least one” feature or element do not preclude there being none of that feature or element, unless otherwise specified by limiting language such as “there needs to be one or more . . . ” or “one or more element is required.”
Unless otherwise defined, all terms, and especially any technical and/or scientific terms, used herein may be taken to have the same meaning as commonly understood by one having ordinary skills in the art.
Reference is made herein to some “embodiments.” It should be understood that an embodiment is an example of a possible implementation of any features and/or elements presented in the attached claims. Some embodiments have been described for the purpose of illuminating one or more of the potential ways in which the specific features and/or elements of the attached claims fulfill the requirements of uniqueness, utility and non-obviousness.
Use of the phrases and/or terms including, but not limited to, “a first embodiment,” “a further embodiment,” “an alternate embodiment,” “one embodiment,” “an embodiment,” “multiple embodiments,” “some embodiments,” “other embodiments,” “further embodiment”, “furthermore embodiment”, “additional embodiment” or variants thereof do not necessarily refer to the same embodiments. Unless otherwise specified, one or more particular features and/or elements described in connection with one or more embodiments may be found in one embodiment, or may be found in more than one embodiment, or may be found in all embodiments, or may be found in no embodiments. Although one or more features and/or elements may be described herein in the context of only a single embodiment, or alternatively in the context of more than one embodiment, or further alternatively in the context of all embodiments, the features and/or elements may instead be provided separately or in any appropriate combination or not at all. Conversely, any features and/or elements described in the context of separate embodiments may alternatively be realized as existing together in the context of a single embodiment.
Further, throughout the specification, the term "part" means a unit realized by hardware. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Some of the operations or functions described as being performed by the terminal, apparatus, or device. It may instead be performed in the device or a server connected to the terminal apparatus or device.
Similarly, some of the operations or functions described as being performed by the server may also be performed in a terminal device or device connected to the server. Some of the operations or functions described as a mapping or matching with the terminal interpreted as mapping or matching the terminal's unique number or personal identification information, which is the identification data of the terminal.
10) In the specification, XR throughout to refers to the spectrum of hardware, software, and techniques used for Virtual Reality, Augmented Reality and 3D, examples including, but not limited to:
a) Head-mounted displays, whether they are opaque, transparent, or utilize
video pass-through,
b) Mobile devices with positional tracking.
c) Fixed displays with head tracking capabilities
Any particular and all details set forth herein are used in the context of some embodiments and therefore should not be necessarily taken as limiting factors to the attached claims. The attached claims and their legal equivalents can be realized in the context of embodiments other than the ones used as illustrative examples in the description below. Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.
The present disclosure provides a system (100) (100) for creating extended reality, wherein the system (100) (100) provides a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface.
Figure 1 illustrates a system (100) for providing a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface, according to an embodiment of the present invention. However, since the HTML5 web environment-based VR and AR interactive coding platform as illustrated in Fig.1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through Fig. 1.
The system (100) may comprise one or more user input terminal (101) connected to a network (103), the user input terminal (101) allowing one or more users to provide input and configured to output AR content, VR content, 3D content, 360 degree content using an HTML5 web environment-based VR and AR interaction coding service related web page, program, or simulation, and a server (104) configured to provide one or more interactive content services to the one or more users.
Further, referring to Figure 1, the system (100) may comprise one or more output request terminal (102) configured to access the server (104) through the network (103) and upload the produced AR content or VR content by running a script in a web browser (114) to the server (104), wherein at least one user input terminal (101) is a terminal interworking with one or more output peripheral devices including smart phone, smart pad, laptop, desktop, HMD and controller.
According to an embodiment, the server (104) may comprise one or more databases, at least one content development module, at least one sensing and detecting module configured to detect access to the user input terminal (101), a control module (105) with code input information acquisition unit configured to toggle output content in AR, VR, custom from web page accessed from the user input terminal (101) selection, and input to at least one user interface of the user input terminal (101), a content converter module configured to receive the produced AR content and VR content and convert into corresponding HTML5 platform, an output node, and at least one output manager request terminal configured to manage the user input terminal (101) to output AR content and VR content from the accessed web page. The interactive content service providing server (104) (interchangeably used as “server (104)”) may be capable of accessing a remote server (104) or terminal through a network (103). It may be implemented with a computer. The computer may include, for example, a laptop, a desktop equipped with a navigation system (100) and a web browser (114).
In an embodiment, the one or more user input terminal (101) may be a laptop or desktop supporting all major web browsers (114) (google chrome, safari, internet explorer, opera, firefox, etc.).
In an embodiment, each component as illustrated in Figure 1 may be connected through a network (103). For example, as shown in Fig.1, at least one user input terminal (101) and output request terminal may be connected to the interactive content service providing a server (104) through the network (103). Here, the network (103) may refer to a connection structure in which information exchange is possible between each node, such as a plurality of terminals and servers (104). In an embodiment, networks (103) may include LTE (Long Term Evolution) network (103), 5GPP (5th Generation Partnership Project) network, LAN (Local Area Network) (103), Wireless LAN (Wireless Local Area Network) (103), WAN (Wide Area Network) (103), PAN (Personal Area Network) (103), etc., but are not limited thereto.
According to an embodiment, the at least one user input terminal (101) may output AR content, VR content, 3D content, 360 degree content using an HTML5 web environment-based VR and AR interaction coding service related web page, program, or simulation. In this case, at least one user input terminal (101) may be a terminal interworking with output peripheral devices such as smart phone, smart pad, laptop, desktop, HMD or a controller, etc.
According to an embodiment, at least one user input terminal (101) may be implemented as a computer that can access a remote server (104) or terminal through a network (103). Here, the computer may include, for example, navigation, a laptop equipped with a web browser (114), a desktop, and a laptop. In this case, at least one user input terminal (101) may be implemented as a terminal capable of accessing a remote server (104) or terminal through a network (103).
Figure 1a is the graphical representation of the 3D editor window where the kids may have access to the Widgets Panel, Coding Editor, Property Panel and other gizmo tools like scale, position, rotation. Referring to Figure 1a, the illustration diagram is divided into user scene names, options like save and publish simulation with the feature like autosave on ten seconds interval. The users may have the action to redo or undo the action performed in creating the simulation design. A property panel may be marked with the custom defined property of each 3D/2D model/item.
Setting may provide the user with a custom canvas setting like Anti Aliasing/Rendering Type/Preserve Drawing option. Further, the users may also be provided with the hierarchical graph of the models/items loaded in the design which includes the systematic allocation of the resource as per the history they are loaded in the scene. Widget panel may provide the user with the option of custom model upload with the variety of the inbuilt option of Environment, Primitive 3D Models, Character Models, Audio, Video, Text and custom markers.
Figure 1b refers to the way application initialization in the front end. In an embodiment, Init.js may be the initializing file where the controlling action happens after that it goes for the selection of the Production or the Development server based on the instance where the solution is run. Also the Init.js may be the main file for selecting the criteria whether the required scene is open in the editor mode or play mode. A Play mode may be the scenario where users don’t have the freedom to edit the simulation, instead they can visualize only the scene created by authorized users. An Edit mode, instead, may give users freedom to add the 3D Models/Audio/Text/Particle Effect/Textures. In the edit mode, an application layer logic may be designed to create the HTML element using the vanilla js and jquery. Said HTML components may include the creation of the Property Panel, Widget Panel and other Gizmo Panel as discussed in Figure 1a, while said 3D canvas may be initialized with ‘WebGL’ context that is the reason for displaying the 3D related graphics on the web. During the initialization of the 3D engine, it may setup the camera matrix and model matrix to create MVP matrix and the same is sent to the rendering function. After complete initialization of the 3D engine and HTML elements 3D models are loaded from the scene API which asynchronously load the models either from the cloud (S3 Storage, localStorage, external cloud services) and bind the required events with the loaded model. Said loaded model in the memory are now sent to the renderer where the models are displayed to the user terminal based on the selection of scene creation by the user which could be either in Augmented Reality format or Virtual Reality Format, default here is 3D if none of the option is user selected in the previous mode.
Figure 2 is a block diagram illustrating an interactive content service providing server (104) and client, according to an embodiment of the present invention. A three-layered architecture may consist of presentation layer (111), logic layer (112), and data layer (113). The presentation layer (111) may include the Graphical User Interface (GUI) and the main canvas for 3D rendering according to an embodiment. The logic layer (112) may contain a number of web services written in Python and deployed in Django Web Framework. The data layer (113) may focus on the storage and retrieval of data.
Further referring to Figure 2, the server (104) may include a sensing unit, a control unit (105) (in the control module (105), and interchangeably used with control module (105) throughout the specification), an output VR/AR providing unit, a conversion unit, and a database unit. On the other hand, the interactive content service providing client web browser (114) may include a 3D game engine and code manager to execute and output the simulation in AR, VR and 3D.
In an embodiment, the code manager may include the graphical visual blocks that are used in the user interface input terminal, and may have two associated definition files. The first file may contain information that describes the graphical visual appearance of the block, including any input or output fields it may have. The second file may describe how a block is embedded to 3D assets on the toolbar side of the platform. The user may input the required function of the blocks and retrieve the parameters that have been supplied to the 3D models and web environments to create the AR/VR/Custom simulations. Javascript may be used as the language for the block definition and code generation files in an embodiment. Also, the entire collection of blocks with functions in creating the particular simulation may be converted to textual code in the background.
The sensing unit may detect the access of the user input terminal (101), accessing a web page providing an interactive content service from the user input terminal (101). In this case, the web page may be an HTML5-based web browser (114) page. HTML5 is the latest standard of HTML (Hypertext Markup Language), which is a web document standard and includes a function to play pictures, videos, music, etc. Additionally, HTML5 eliminates the need to separately install programs such as plug-in-based Active-X and Flash, and most of the problems caused by this can be solved. At this time, the biggest difference between the conventional HTML standard and HTML5 is that it supports semantic markup and provides an additional API. Semantic markup is a new element that may semantically structure web documents. For example, to separate the content of the document into header, body, and footer. Therefore, if one writes a document, he/she can clearly understand the contents of the document structurally. In addition, HTML5 provides various APIs. The API provided in HTML5 may the user to add various functions to web applications, such as supporting two-dimensional and three-dimensional graphics.
The postMessage mechanism in HTML5 is used to enable web content from different origins (third party content like sketchfab and others to upload 3D models) to communicate with each other, thus relaxing the same origin policy. Each message contains accurate information. The effective visualization of 3D meshes based on the build-in features of a web browser (114) supporting HTML5(Canvas, WebGL) standards is presented. JavaScript libraries integrate the physics and 3D processing algorithms. HTML5 based interactive content system (100) supports hyperlink, user interaction, audio and video, animation and 3D model view. In addition, HTML5 is a markup language used to display hyperlinks and texts, to create web pages, and to express and provide various applications such as multimedia. HTML5 can be implemented including HTML, a markup language for designing web document structure, CSS for design expression, and JavaScript for interactive action, and new tags can be added to the existing HTML tags, and a structure for containing contents can be improved to even play the role of a platform.
Cascading Style Sheets (CSS) describes the HTML5 elements displayed at the output requirement terminal (102), saves and controls the layout of multiple web pages all at once. On the other hand, front-end technology used here is JavaScript (JS), which is a high-level, interpreted programming language. JavaScript has curly-bracket syntax, dynamic typing, prototype-based object-orientation, and first-class functions. Alongside HTML and CSS, JavaScript is one of the core technologies of the World Wide Web. JavaScript enables interactive web pages and is an essential part of this web environment platform. The support includes but is not limited to event-driven, functional and imperative including object-oriented and prototype-based styles. It is important to validate that the form submitted by the user at the user interface can have inappropriate values. So, validation is a must to authenticate the user. JavaScript provides the facility to validate the form on the client-side so data processing will be faster than server (104)-side validation.
Bootstrap, is an open-source CSS framework directed at responsive user input terminal (101), mobile-first front-end web environment interface. It contains CSS and JavaScript-based design templates for typography, forms, buttons, navigations, and other interface components.
In Back-end side, Python is used in this invention, which is an interpreted, high-level, object-oriented approach, a general-purpose programming language. In the present disclosure, python is used as a backend language to code database parts and all functionalities that the platform can perform. The version of Python used in this development process is Python 3.7.
The version of Django used in this development process of this platform is Django 4.0.1. Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. The framework emphasizes reusability and "pluggability" of components, less code; low coupling, rapid development, and the principle of don't repeat yourself. Python is used throughout, even for settings files and data models of the web based interactive content system (100). Django also provides an administrative create, read, update and delete interface that is generated dynamically through introspection and configured via administration models.
Django used SQLite 3.372 as its default database. SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. Django provides a specific way to define this platform database using the python programming language.
SHA-256 is one of the hashing algorithms used for encryption of storing secure passwords in a database. All SHA-256 values have 256 bits, or 32 bytes. To verify a user’s password at the login interface, this keeps the user’s submitted password in memory, so it never needs to touch the disk and compute its hash. If the computed hash matches the stored hash, the user has fronted up with the right password, and the platform can let users login securely.
In addition, although HTML5 is not a standard for mobile web, it provides many functions necessary for the implementation of the mobile web among various functions described above. It provides many features like 2D graphics, audio, and various web forms that can enrich and diversify the UI (User Interface) of the mobile web. That is, it is possible to provide multimedia information or support a user's dynamic interaction even in a mobile web environment. In the mobile web environment, there is a limitation in transmitting large amounts of data due to the nature of the mobile environment, it is the web that can overcome these constraints. Some other functions are summarized below in Table 1 below,
Main Function Explanation
Web Form Provides extension of user input form (additional web form such as e-mail, URL, number, range, calendar, etc.)
Canvas (WebGL) API for drawing 3D graphics
SVG (Scalable Vector
Graphic) XML-based three-dimensional vector graphic expression language
Video/ Audio Device for playing audio and video without plug-ins
SQL Database SQL that uses standard MySQL for database entry
Local Storage Ability to store data in web client
Web Socket An API that allows web client to communicate directly with the server (104) side in both directions
WebXR API to call xr interface in the devices which includes the support of the controller, surface detection, camera optimization.
REST API Transmitting the user & scene information using REST API structure
Visual Coding A system (100), comprising: create a graphical programming coding environment, wherein the visual programming coding environment defines a plurality of graphical programming blocks, each graphical programming block, of the plurality of graphical programming blocks, is configured to represent a predefined programming element.

In one embodiment of the present invention in consideration of the above-described HTML5 function, the service of the present invention may be implemented with two models, however the present invention is not limited to those listed below. The first model may store data on the server (104) side like a web application, and the client may receive information in real time. The second model may use the client's cache and local database to run as a preview web application while developing, and if necessary, receive information from the server (104) side.
In addition to this, it may be possible to use SaaS (Software as a Service) using a cloud server (104) connected to a network (103). SaaS, is a service that provides users with application programs running in the cloud environment. SaaS can be easily used anywhere on the internet and can be accessed from anywhere with a web browser (114). The above-described application may be defined as an application provided through a cloud server (104), or an HTML5-based web browser (114), rather than an application to be downloaded or installed.
Referring to Figure 2, the control unit (105) may control to output AR and VR content from a web page having code input information, code executioner that may be accessed from the user input terminal (101). Accordingly, the user input terminal (101) may output AR content or VR content with respect to the function of the written code and available to access as a web page, and may not install an application or program.
Also, when an AR marker is recognised by the user input terminal (101), the control unit (105) may control to output AR content at the location of the AR marker. In this case, the AR marker may be the course content images, when the requesting terminal is an organisation or individual users, but is not limited thereto. In addition, it can be configured to enable interaction by a touch of the screen. The markerless method (200) detects and understands an unknown or outdoor real-world environment (e.g., the locations of walls), and no pre knowledge of the environment is required, which will efficiently promote large-scale Mobile AR.
For different devices, many interaction models are offered and automatically adapted, on mobile devices for instance (smartphones and tablets) and touch-screens, a multi-touch interaction is provided, while on desktop devices (laptops, PCs) a keyboard+mouse model is enabled. On immersive VR/AR devices, different interaction models are automatically enabled, depending on HMD degrees-of- freedom (3-DoF or 6-DoF). There are different navigation modes in QwikXR, that can be activated, available depending on the typology of the user device: (A) orbit mode (default); (B) first-person mode; (C) device-orientation mode; and (D) immersive VR navigation mode.
Orbit mode is a classic navigation model offered by the vast majority of Web3D presenters: in QwikXR it offers retargeting features (double-tap/double click on surfaces to adjust camera target) with smooth transitions for a good user experience. First-person mode allows the user to explore the environment through a common point-and-go model Device orientation mode (available on mobile devices) accesses user device built-in sensors and uses such information to control the virtual camera. The system (100) automatically adapts to 3-DoF and 6-Dof HMDs, switching pointing method (200) accordingly also depending on the presence of VR controllers. When no controllers are present (e.g., card- boards) a view-aligned/gaze pointing is activated, otherwise, one of the VR controllers is used. This allows the system (100) to seamlessly adapt to 3-DoF (e.g., cardboards) and 6-DoF interaction models offered by high-end HMDs (e.g., Oculus Quest, HTC Vive, etc.).
WebXR Device API supports both VR and AR on the web. Web extended Reality, WebXR is used to support the rendering of a 3D scene with appropriate hardware to present a virtual environment in a web environment or to add graphic images to the real environment. WebXR device API algorithm is an XR input source that includes, but is not limited to, handheld controllers, optically tracked hands, and gaze-based input methods (200) that operate on the viewer's pose. WebGL, JavaScript API is used for rendering high-performance interactive 3D and 2D graphics without the use of plugins.
WebRTC provides an API that gives the web environment the ability to communicate with users in real-time, to stream video, voice and generic data between peers. Web Speech API is also used for providing support for speech recognition and synthesis. Users can select the custom content creation for using this feature in real-time in AR and VR output requirement terminal (102). Users can also join the communication by just opening an URL/embedded link or scan QR code on their devices. Moreover, large downloads are not required and users can immediately experience virtual environments on the website.
The part of the embodiment in this invention, the XR renderer stores the desired user interaction environment option selected by the user and uses it to interpret the XR code properly according to the user’s interaction environ- ment. If the user’s interaction environment is AR, the XR renderer initially sets all virtual objects not-to-be-rendered and selectively renders only the augmentation object when the AR engine tracks the real-world object. On the other hand, all objects are always rendered regardless of the ar-target property in VR. The XR renderer specifies that the background and virtual objects are not rendered in AR so that the camera video frames are visible to the user. On the other hand, if the user’s environment is VR, the renderer provides a realistic immersion to the user by drawing the background and virtual objects.
The present disclosure used sensor-based method (200) is a relatively lightweight Mobile AR implementation approach, while in contrast, the vision-based approach places high demands on the computing and storage capabilities of the runtime platform, as well as network (103) capability. Mobile devices nowadays already support a variety of sensors, such as accelerometers, gyroscopes, compasses, magnetometers, GPS, and so on. The output unit, when an input is present in at least one user interface of the user input terminal (101), the AR content may be transformed to the corresponding input so that the interactive content may be output from the user input terminal (101). Accordingly, the user input terminal (101) may output the interactive content corresponding to the input of the user interface. At this time, the output unit, a touch event input to the touch display of the user input terminal (101), is recognized through the camera (the camera can be enabled to capture the surrounding environment, but only for the display of the environment as the background) of the user input terminal (101) any one of the gesture data and the sensor data sensed through at least one sensor built into the user input terminal (101). AR content may be modified by any one or a combination of at least one of them.
In addition to Mobile AR implementation mechanisms based on a single sensor, combining different sensors allows many applications to achieve more accurate tracking results. The increasing of sensor category, as well as the continuous enhancement of sensor functionality, provides the basis and opportunities for the diversification of Web XR applications. When the camera is looking at the real environment, if the camera is moved, point cloud information about the real environment is extracted using a Simultaneous Localization And Mapping (SLAM) algorithm.
The present invention offloads computation-intensive tasks to the network (103) cloud for efficient computation acceleration and powerful rendering of the AR content. Cloud servers (104) always have a more powerful computing capability and also provide a performance improvement for Web XR content. Current mobile network (103) 4G/5G, supports real-time operations like tracking, rendering and interaction.
There are two kinds of AR contents at user input terminal (101) output requirement terminal (102), first one is Self-contained method (200) that executes all tasks on the mobile device locally, offline approach. The advantage of this method (200) is that it is less dependent on mobile network (103), so the real-time tracking performance will not be degraded by additional communication delay. However, the inefficient computing capability of the mobile device becomes its fatal flaw; current mobile devices still cannot carry out these tasks very well.
The other method (200) which is used in this invention is the Computation outsourcing method (200) that leverages the computation and storage capabilities of the cloud server (104), and it can usually provide a better user experience than the aforementioned self-contained due to stronger computing capability of the server (104) that reduces the computing capability at the user output requirement terminal (102). However, this method (200) has a strong dependence on the mobile network (103), therefore 4G and 5G connection fits very well with this system (100). The emerging 5G network (103) can achieve even a 1-Gb/s data rate as well as millisecond end-to-end communication delay, improving the AR performance at the output but not limited to. The upcoming 5G network (103) provide an efficient and powerful platform for WebXR.
The platform mentioned in the invention (Fig. 2), is developed on Django Framework for optimized computation outsourcing; the backend server (104) used are Python, MySQL and Socket. Cloudinary, an online cloud service has been used for storing data. Augmented reality, face recognition, natural language processing, gaming, video processing, 3D modeling, real-time rendering, among others usually are resource-hungry, requiring intensive computation and high energy usage. On the other hand, a mobile device, due to its portable nature, is invention used Cloud-based offloading that may improve mobile application performance and reduce energy consumption by migrating mobile code execution to the cloud. Computation offloading is a technique that empowers the capabilities of mobile system (100), for e.g. wearables, mobile phones, sensors, among others tiny devices. Such improvement is done by migrating the entire program or just its intensive tasks to more powerful computers located in the cloud or cloudlet in nearby infrastructure.
Django web applications mentioned in Fig. 2, typically group the code that handles each of these steps into separate files: a) URLs: While it is possible to process requests from every single URL via a single function, it is much more maintainable to write a separate view function to handle each resource. A URL mapper is used to redirect HTTP requests to the appropriate view based on the request URL. The URL mapper can also match particular patterns of strings or digits that appear in a URL and pass these to a view function as data. b) View: A view is a request handler function, which receives HTTP requests and returns HTTP responses. Views access the data needed to satisfy requests via models, and delegate the formatting of the response to templates. c) Models: Models are Python objects that define the structure of an application's data, and provide mechanisms to manage (add, modify, delete) and query records in the database. d) Templates: A template is a text file defining the structure or layout of a file (such as an HTML page), with placeholders used to represent actual content. A view can dynamically create an HTML page using an HTML template, populating it with data from a model. A template can be used to define the structure of any type of file; it doesn't have to be HTML. Django further enhances the security of the web environment by preventing a variety of security attacks cross-site scripting (XSS), cross-site request forgery (CSRF), SQL injection, and click jacking. The security issues are resolved in order to prevent any unwanted attack on the database as well as the server (104). In the proposed framework, spatio-temporal data is stored in the data server (104) and is sent to user browser (114) with WebSocket.
In addition, the three-dimensional interaction method (200) to be used in augmented reality is not limited to any one because there are various, but according to an embodiment of the present invention, the core rendering system (100) uses Three.js, the rendering system (100) also supports dynamic shadow, dynamic pixel density, to control or fine-tune frame rate on devices with poor graphics performances. Foveated rendering lowering the rendering workload by reducing resolution in the peripheral vision.
This WebXR platform mentioned in this invention supports VR environments at the output requirement terminal (102) with the use of the HMD and a web browser (114), including smartphones, tablets, laptops, desktop computers, and the entire room displays. Also, the WebXR Device API supports both VR and AR on the web. This invention uses Three.js, that allows users to enjoy not only the available VR contents on the platform gallery but also enable users to create their own content with panorama pictures or 360 videos with a smartphone camera and graphical coding environment. Although these preset 3D models and functions in visual coding have greatly influenced the way users interact with the world.
The output unit may increase the performance of hybrid-based tracking by using a sensor. A feature point-based tracking technique, is one of the general tracking techniques, uses an image coming from the camera to track the camera's posture and position. The feature point-based tracking technique guarantees tracking performance to some extent when the camera movement is small, but tracking fails when the camera rotates or moves rapidly or when a moving object appears in front of the camera interferes with feature extraction that occurs. In this case, it causes a problem that the sense of reality is reduced without the accurate augmentation of the graphic, which is one of the major problems in augmented reality. In order to compensate for this problem, it is also possible to additionally use a sensor built into the smartphone.
When a Head Mounted Display (HMD), interlocked with the user input terminal (101), is detected, the VR providing unit streams VR content to the user input terminal (101) and input is received through the user interface of the user input terminal (101). It is possible to output the interactive content from the user input terminal (101) by transforming it. In this case, the VR providing unit may provide the interactive content based on the user’s pupil data tracked by the HMD or input data of at least one controller interworking with the user input terminal (101). Eye tracking is a method (200) of tracking eye movement by recognizing the pupil center and the corneal reflex of the eye by the eye tracker.
Eye tracking, which tracks the gaze, is a method (200) to find out where you are looking by measuring the movement of the gaze or pupil. It is a process of tracking the gaze from the position of the eye. At this time, the interaction may be made according to the gazed time, the number of gazes, the gaze distribution, the average pupil size, the eye blink frequency, etc. Depending on the user interaction, VR content can be provided so that it literally corresponds to the direction in which the eyes are moving. In this case, in addition to eye tracking, whenever the user moves by using a gyro sensor, an angular axis etc., it may also cause you to view the corresponding output.
The conversion unit may receive the produced AR content and VR content and convert it in to correspond to the HTML5 platform. HTML5- based content streaming service should be able to play web browser (114)-based video. However, since the video/audio codec information supported by HTML5-based web browser (114) so far is all different, it is necessary to convert to content supported by each web browser (114) until the codec standard is determined.
After this conversion, multimedia content can be played as if viewing an image in a web browser (114). For this, video and audio tags can be defined in HTML5. Prior to HTML5, in order to embed a video into a web page, an object tag that can put an external object and many parameters were duplicated. On the other hand, HTML5 provides a way to directly play and display content in the web browser (114) itself by defining a standardized method (200) to include video and audio as one element of the web without using plug-ins anymore. All content consists of tracks such as video, audio, and data, which are usually interrelated. In addition, each track stores metadata such as video aspect ratio or audio language information, and the container may also include metadata such as video title and cover design.
The database conversion unit may receive the request of AR content and VR content from at least one user input terminal (101), and may store the AR content and VR content by mapping the AR content and VR content to a requesting terminal (not shown), the interaction content service providing server (104) collects and organizes these events and periodically reports can be provided or data can be shared with the requesting terminal.
Hereinafter, an operation process according to the configuration of the interactive content service providing server (104) of Fig. 2 will be described in detail with reference to Fig. 3 & 4 as an example. However, it will be obvious that the embodiment is only any one of various embodiments of the present invention, and is not limited thereto
Figure 3a illustrates an editor interface to design interactively in 3D/AR/VR, according to an embodiment of the present invention. Fig. 3a, shows the 3D editor window workplace where users can select the appropriate 3D models, animations, low-poly models, 3D environment, images, videos, 3D texts, sounds, particle effects, lightings, and also you can upload the external 3D models from open sources or sketchfab, then drag and drop them into the workspace. By default, the workspace includes a trash can that enables users to remove unwanted blocks, models or any other unnecessary assets. The high-level functions may be performed by said freeplay window:
Function 1 (F1). It may enable users to program AR/VR content visually.
Function 2 (F2). It may include multiple content types, including 3D models, 3D text engine, sound, primitive models, animation, low-poly, environment, particle engine, images, videos, panorama, lego, and external upload 3D models (e.g., GLTF, OBJ).
Function 3 (F3). It may allow the learners to generate both AR/VR content and animations associated with it.
Function 4 (F4). It may enable users to experience the visual interface in AR and VR mode.
Function 5 (F5). It may allow users to share AR/VR content they have developed with others by shareable embedded links/QR codes without downloading the AR and a VR application.
Figure 3b illustrates a coding interface to provide logic on user input terminal (101), according to an embodiment of the present invention. In particular, Referring to Fig. 3b, Blockly.js is used for creating block-based visual programming languages (VPLs) and editors. As opposed to dragging and dropping blocks, text-based programming is the process of typing letters, numbers and symbols. Text-based programming, also referred to as text modality, used in programming languages such as C, Javascript and Python, requires programmers to utilize formal syntax to compile a program. In this invention, Block modality encourages users to quickly create, compile and then test their programs.
The main value of this window is the inclusion of customizable blocks along with some existing blocks for common operations. Due to this unique feature, it can generate code in any textual programming language. However, new block types each require a definition and a generator. The definition describes the block’s appearance and how it behaves, including the text, color, shape, and the connections to other blocks while the generator translates the block to executable code.
The present embodiment includes but is not limited to the following descriptions that embed the visual interface with 3D editor panel workspace,
Animations: This category may consist of blocks that define events in the 3D web interface, AR and VR content simulations. Said block may provide multiple actions on 3D models, environment and syncronize with camera, text and audio engine.
Display message: The display message block may define a 2D and 3D message to be displayed on the simulation or web interface. Said display message may take text block as the input and show output in the user logic format with different colors options.
Variables: Said category may declare and initialize the custom variable used in games logic scenes in the 3D, AR and VR simulations. Since QwikXR allows the addition of any arbitrary 3D object in the multiscene storytelling mode, each object needs to be saved on cloud server and have unique identity.
Figure 3c illustrates a scene selection interface to create interactive in 3D/AR/VR based on user selection, according to an embodiment of the present invention. Fig. 3c, explained the user input terminal (101) selection of the scenes that output at the request output requirement terminal (102) through control terminal and network (103) terminal. Users can select 3D, AR, VR scenes and initialise the interface of the editor window. Sensing unit, control unit (105) and converter outputs the content in respective mode of user selection.
Figure 3d illustrates a user authentication terminal interface, according to an embodiment of the present invention. Fig. 3d, shows the framework that provides a built-in authentication system (100) that allows content creators and users to access, manage and modify their own collections and scenes on the deployed instance. The framework offers a built-in lightweight, responsive back-end where users, editors, or content creators can authenticate to publish and manage 3D scenes or simulations with ease.
Figure 3e illustrates a public landing page gallery, according to an embodiment of the present invention. Fig. 3e, shows the public landing gallery web page for direct accessing of AR, VR, 3D content with functionalities of graphical code informations. Token based authentication, allowing a fine-grained control on requests involving content access, modification or other tasks. Also allows authenticated users to create online scenes with ease starting from the tool kits 3D assets in galleries. Authenticated users can then publish their scenes on the main landing page with public access, thus allowing remote users to consume the 3D scene on any devices based on AR and VR scenes selected by authenticated users respectively. The landing page also provides a search box to filter public scenes by term or keyword, very useful to create custom galleries (e.g., collection coding, history, geography, science, chemistry, music, STEM, etc.)
Figure 3f illustrates generation of a Virtual Reality (VR) Scene developed by the innovation, according to an embodiment of the present invention. In Fig. 3f, each frame is sequentially configured like a scene, but this is an embodiment of each scene for VR. In this way, the character/models and the background can be combined and output from the HMD, and the content as shown in Fig. 3e is output on the AR with marker or markerless as per the input of the user selection and output on the augmented reality camera screen. It uses a Three.js library for rendering virtual objects. Thus the executable code is attributed to Three.js JavaScript. The architecture of Three.js is built on the WebGL which is used for rendering interactive 3D graphics within any compatible web browser (114) without the use of plug-ins. WebGL is fully integrated with other web standards, allowing GPU-accelerated usage of physics and image processing and effects as part of the web page canvas. On the other hand, AR.js allows developers to detect images and patterns, estimate spatial positions, and superimpose virtual objects on the detected images/patterns.
Figure 4 illustrates real time user responses being recorded and admin rating the user content with AI data driven technique used CNN algorithms so as to give feedback based on the performance, according to an embodiment. In an embodiment, heat maps may be generated based on the user performance in turn based on the factors including but not limited to logical thinking, creativity, analytical thinking, computation skills, coding proficiency, storytelling, and problem solving skills.
Figure 5 is a flowchart illustrating a method (200) of providing a VR and AR interactive content service based on an HTML5 web environment, according to an embodiment of the present invention. In Fig. 5 is an operation flowchart illustrating a method (200) of providing a 3D, VR and AR interactive coding platform service based on an HTML5 WebXR environment according to an embodiment of the present invention. Below, an example of a process in which data is transmitted/received between respective components will be described with reference to Fig.5, but the present application is not limited to such an embodiment, and the data shown in Fig.5 according to various embodiments described above it is apparent to those skilled in the art that the illustrated data transmission/reception process may be changed, referring to Fig.5. In this case, when the interactive content service providing server (104) accesses a web page that provides an interactive content service from a user input terminal (101), the access of the user input terminal (101) is detected.
In particular, the method (200) may provide a WebXR based VR and AR interactive content based on the HTML5 web environment with graphical coding interface, the method (200) comprising steps of: accessing a web page, by a user input terminal (101), the web page providing an interactive content based on user selection as 3D/AR/VR, with geographical coding environment; outputting a list of coding blocks and control to perform function based on coding instruction mapped with 3D assets and free play scenes with animations; and detecting, sensing, transforming, and rendering 3D to AR/VR content to be output from the web page accessed from the output request terminal. This is followed by the step of interactive simulations allowing server (104) comprising an output unit, mobile devices, computer web system (100), smart TV browser (114), AR glasses, when a user selection is present in any of the modes, AR, VR or 3D at the interface of the user input terminal (101).
Further, the method (200) comprises managing, by an output manager request terminal, the AR content to be output from the user input terminal (101) by integrating image recognition to enable AR rendering within the web browser (114).
Next, the method (200) comprises providing a tool bar area defined within an enhanced graphical coding environment, the tool bar area having defined graphical blocks and 3D assets blocks; providing a free play workspace area defined within the enhanced graphical coding environment, the free play area permitting the connection of one more graphical blocks and 3D assets models/environment together to form the simulation; and dragging from the toolbar area one or more graphical blocks and 3D assets to add the one or more graphical blocks and 3D assets to an existing program simulation or content.
Furthermore, the method (200) comprises automatically detecting, by the enhanced graphical coding environment, freespace work in 3D models, environment, textures, particle effect, text, audio; and automatically changing the toolbar area to a block drag and dropdown toolbar. The method (200) comprises playing VR simulation to the user input terminal (101), when a Head Mounted Display (HMD) linked with the user input terminal (101) is detected; and allowing, toggling, and transforming the VR content from 3D simulation, when a user selection exists as a VR through the user interface while initializing content type of the user input terminal (101).
According to an embodiment of the present disclosure, the method (200) comprises providing the interactive simulations based on sensor-based user’s eye data sets by the HMD or input data of at least a control module (105) in the server (104) and linked with the user input terminal (101).
The figures and the forgoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, orders of processes described herein may be changed and are not limited to the manner described herein. Moreover, the actions of any flow diagram need not be implemented in the order shown; nor do all of the acts necessarily need to be performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of the embodiments is by no means limited by these specific examples. Numerous variations, whether explicitly given in the specification or not, such as differences in structure, dimension, and use of material, are possible.