Method And System For Augmented Reality Based Smart Classroom Environment

OF INVENTION

[001] The present invention relates to augmented reality environment, and more particularly to processing recognition information to provide interactive augmented reality based environment. BACKGROUND OF INVENTIO5 N

[002] Augmented Reality (AR) applications combine real world data and computer-generated data to create a user environment. The real world data may be collected using any data acquisition unit such as mobile phone, Personal Digital Assistant (PDA), smart phone, camera, 10 communicator, wireless electronic device, or any other data acquisition unit. The augmented reality can be used in video games, Industrial designs, mapping, navigation, advertising, medical, visualization, military, emergency services, or in any other application. One of the most common approaches to the AR is the use of live or recorded videos/images, captured 15 with a camera or mobile phone, which are processed and augmented with computer-generated data to provide an interactive augmented reality environment to the user. In many augmented reality applications, information about the surrounding real world of the user becomes interactive and digitally manipulated. In order to interact in an augmented 20 reality environment, a user may need location information of other users in the virtual environment.

[003] The present invention provides a smart and robust method and system for providing an interactive augmented reality based environment to a user. 25 3/20 OBJECT OF INVENTION

[004] The principal object of the embodiments herein is to provide a method and system for providing augmented reality environment using a portable electronic device.

[005] Another object of the invention is to provide a method an5 d system for processing recognition information to provide an augment reality environment to a user.

[006] Another object of the invention is to provide a mechanism for providing an interactive augmented reality platform that allows users to 10 interact with each other and digitally manipulate the information.

[007] Another object of the invention is to provide a method and system for deriving location coordinates of users to provide an interactive user environment. 15 SUMMARY

[008] Accordingly the invention provides a method for providing augmented reality based environment using a portable electronic device. The method includes capturing an image of users, recognizing the users in the image, and fetching information associated with the recognized users. 20 Further, the method includes determining location of the users in the image, mapping the fetched information associated with the users with the determined location of the users and communicating with the users based on the mapped information.

[009] Furthermore, the method includes adjusting position of the 25 portable electronic device according to position of the users. In an embodiment, the position of the portable electronic device is adjusted according to a predetermined region of the portable electronic device.

[0010] Furthermore, the method includes sending the image to a server for recognizing the users. In an embodiment, the server performs a 4/20 facial recognition function on the image to determine face portion of the users and authenticate the determined face portion in the image to recognize the users.

[0011] Furthermore, the method includes transferring digital information to the users using the information and the determined locatio5 n of the users. In an embodiment, the digital information is transferred by dragging and dropping the digital information in the determined location of the users.

[0012] Furthermore, the method includes performing an adaptive 10 communication with the users based on the fetched information. Furthermore, the method includes using the information and the determined location of the users to take attendance in the environment.

[0013] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the 15 following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating preferred embodiments and numerous specific details thereof, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing 20 from the spirit thereof, and the embodiments herein include all such modifications. BRIEF DESCRIPTION OF FIGURES

[0014] This invention is illustrated in the accompanying drawings, throughout which like reference letters indicate corresponding parts in the various figures. The embodiments herein will be better understood from 25 the following description with reference to the drawings, in which:

[0015] FIG. 1 illustrates generally, among other things, a top view of a classroom, according to embodiments as disclosed herein; 5/20

[0016] FIG. 2 illustrates generally, among other things, an example of classroom environment, according to embodiments as disclosed herein;

[0017] FIG. 3 depicts an exemplary image of the classroom environment of the FIG. 2, according to embodiments as disclosed herein;

[0018] FIG. 4 illustrates generally, among other things, exemplar5 y components of a system in which various embodiments of the present invention operates;

[0019] FIG. 5 illustrates generally, among other things, an exemplary predetermined region or field of view of the FIG. 2, according 10 to embodiments as disclosed herein;

[0020] FIG. 6 depicts a sequence diagram illustrating operations performed by the system of the FIG. 4 using a server, according to embodiments as disclosed herein;

[0021] FIG. 7 depicts a sequence diagram illustrating operations 15 performed by the system of the FIG. 4 without using the server, according to embodiments as disclosed herein;

[0022] FIG. 8 depicts a flowchart illustrating a method for providing augmented reality environment, according to embodiments as disclosed herein; 20 [0023] FIG. 9 depicts a flowchart illustrating operations performed by an instructor, according to embodiments as disclosed herein; and

[0024] FIG. 10 depicts a computing environment implementing the application, in accordance with various embodiments of the present invention. 25 6/20 DETAILED DESCRIPTION OF INVENTION

[0025] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well5 -known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein can be practiced and to further enable those skilled in 10 the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

[0026] The embodiments herein achieve a method and system for providing augmented reality based environment using a portable electronic device (hereinafter “PED”). The method allows an instructor to capture an 15 image of audience using the PED. The instructor adjusts the position of the PED according to position of the audience. The instructor sends the captured image to a server for performing a facial recognition function to recognize the audience. The server recognizes the audience face(s) and fetches information associated with the recognized audience. Further, the 20 server determines location coordinates of the audience and sends to the PED. The instructor maps the fetched information associated with the audience with the determined location of the audience. The instructor communicates with the audience based on the mapped information. Furthermore, the instructor performs an adaptive communication with the 25 audience based on the fetched information.

[0027] The method and system disclosed herein is simple, robust, and reliable to provide an intelligent and smart augmented reality based environment. The method and system can be used to take attendance, interact, or perform any other activity inside a classroom, meeting room, or 7/20 any other gathering. Further, the method and system provides an interactive platform to the instructor to easily interact and exchange digital information with the audience.

[0028] Referring now to the drawings, and more particularly to FIGS. 1 through 10, where similar reference characters denot5 e corresponding features consistently throughout the figures, there are shown preferred embodiments.

[0029] Throughout the description, the term audience and one or more users is used interchangeably. 10 [0030] FIG. 1 illustrates generally, among other things, a top view 100 of a classroom 102, according to embodiments as disclosed herein. The classroom 102 can include an instructor 104 conducting a session with audience 106. The instructor 104 is standing or sitting in front of the audience 106 such that the instructor 104 is able to make direct eye-to-eye 15 contact with the audience 106. In an embodiment, the instructor 104 described herein can be, for example, teacher, demonstrator, speaker, professor, presenter, guider, controller, or any other person. In an embodiment, the audience 106 described herein can be, for example, an individual or a group of students, users, participants, attendees, or any other 20 person. The audience 106 can see, hear, and interact with the instructor 104 and among each other easily.

[0031] The classroom 102 described in the FIG. 1 is only for illustrative purpose and does not limit the scope of the invention. Further, in real type, the classroom 102 can be circular, square, rectangular, or in 25 any other shape. Though the FIG. 1 is described with respect to a classroom but the present invention can be used in any type of gathering such as meeting, conferencing, social platform or environment, public or private event, company or organization workspace, or any other gathering. 8/20

[0032] FIG. 2 illustrates generally, among other things, an example of classroom environment 200, according to embodiments as disclosed herein. In an embodiment, the classroom 102 includes audience 106 facing towards the instructor 104. The classroom 102 provides a room space 202 for sitting arrangement of the audience 106 as shown in the FIG. 2. In a5 n example, a sequential setting arrangement of the audience 106 is made in which some of the audience appears to be close to the instructor 104 as shown at 204 and some of the audience 106 appears to be far from the instructor 104 as shown at 206. 10 [0033] In an embodiment, the instructor 104 can have a portable electronic device (PED) 208 to capture an image or video of the audience 106. The PED 208 described herein can include, for example, mobile phone, personal digital assistant, smart phone, tablet, or any other wireless customer electronic device. The PED 208 is capable of including imaging 15 sensor 210 to capture single or multiple images or videos of the audience 106. In an embodiment, the instructor 104 can adjust the position of the PED 208 according to position of audience 106 such that the room space 202 is visible within a predetermined region 212 (or field of view 212) of the imaging sensor 210. The instructor 104 can adjust the position of the 20 PED 208 according to the position of audience 106 such that face of every individual in the audience 106 can be clearly visible in the image. In an embodiment, the PED 208 can be a placed at any specific location of the classroom 102 in a way that the predetermined region 212 of the imaging sensor 210 covers the entire room space 202. The specific location 25 described herein can provide a clear facial view of the audience 106 present in the classroom 102.

[0034] FIG. 3 depicts an exemplary image 300 of the classroom environment 200 of the FIG. 2, according to embodiments as disclosed herein. The exemplary image 300 can be displayed on the display screen of 9/20 the PED 102. In an embodiment, the image 300 can be viewed on any other display device, for example, liquid crystal display device, cathode ray tube monitor, plasma display, light-emitting diode (LED) display device, image projection device, or any other type of display device capable of presenting the image 300. The image 300 includes a scene having a visua5 l representation of the audience 106, physical items, locality (for example, exact coordinates of where an individual user is currently located in the classroom 102), and other objects of the classroom 102.

[0035] FIG. 4 illustrates generally, among other things, exemplary 10 components of a system 400 in which various embodiment of the present invention operates. The system 400 can include a server 402 configured to be connected to the PED 208 through a wired or wireless communication network 404. In an embodiment, the instructor 104 can use the PED 208 to capture and send the image of the audience 106 to the server 402. 15 [0036] In an example, multiple images can be continuously sent to the server 402 as a video stream. Each image generally includes a scene at which the imaging sensor 210 of the PED 208 is pointed. Such scene can include visual representation of the audience, physical items, location coordinates of the audience, or any other object present in the classroom 20 102. In an embodiment, the instructor 104 sends the captured image to the server 402 for further processing. The operations performed by the system 400 to provide augmented reality environment using the server 402 are described in conjunction with FIG. 6.

[0037] In an embodiment, the PED 208 creates an image in memory 25 of the PED 208 and uses the image for further processing without sending it to the server 204. The operations performed by the system 400 to provide the augmented reality environment, without using the server 402, is described in conjunction with FIG. 7. 10/20

[0038] FIG. 5 illustrates generally, among other things, an exemplary predetermined region 212 or field of view of the FIG. 2, according to embodiments as disclosed herein. The use of a common coordinates can assist in presenting the captured image to the instructor 104 for providing an interactive augmented reality environment. Th5 e coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) can define the predetermined region 212 that is adjusted, by the instructor 104, according to the position of the audience 106. In an embodiment, the server 402 is configured to perform a facial recognition function on the image to 10 determine face portion 502 of the audience 106. In an example, the server 402 determines the location of the audience 106 by deriving the location coordinates of each individual audience in the image. The server 402 determines that the audience 106 is at point 504 facing towards the instructor 104. The server 402 is configured to determine the position of 15 audience 106 relative to axes of the coordinates, such as axes x, y, and z illustrated as a part of the system 400. In an embodiment, the PED 208 can also determine the location of the audience 106 by using local coordinates, Global Positioning System (GPS) coordinates of the PED 106, or any other technology known in the art. 20 [0039] FIG. 6 depicts a sequence diagram 600 illustrating operations performed by the system 400 of the FIG. 4 using the server 402, according to embodiments as disclosed herein. In an embodiment, at 602, the audience 106 can provide profile information (or recognition information) for requesting registration of the recognition information with 25 the server 402. Although such registration is not required in other embodiments. In an embodiment, the instructor 104 can also provide the recognition information associated with the audience 106 to the server 402. In an embodiment, the instructor 104 can also register the audience instantly based on the instructor knowledge. Further, the instructor 104 can 11/20 perform functions to correct/modify or delete the recognition information associated with the audience 106. The server 402 is configured to record the profile information in one or more databases.

[0040] At 604, the instructor 104 can use the PED 208 to capture an image of the audience 106. In an example, the instructor 104 can adjust th5 e position of the PED 208 in a way that the audience 106 is within the predetermined region 212 of the imaging sensor 210. In an example, the instructor 104 adjusts the position of the PED 208 according to the position of the audience 106 such that face of every individual in the audience 106 10 can be clearly visible in the image.

[0041] At 606, the instructor 104 can use the PED 208 to send the image to the server 402 through the communication network 404. In an embodiment, the PED 208 can process the image without sending to the server 402 as described in the FIG. 7. At 608, the server 402 is configured 15 to perform facial recognition functions on the image to recognize the audience 106 in the image. The facial recognition functions described herein are any facial recognition functions or techniques known in the art used to determine the facial portions of the audience 106. In an example, the server 402 is configured to recognize the audience 106 by 20 authenticating the determined face portions with the data stored in the database.

[0042] At 610, the server 402 is configured to fetch the information associated with the recognized audience 106. In an example, the information extracted by the server 402 can include the profile information, 25 previous records, field information, or any other information. At 612, the server 402 is configured to determine location of the audience 106 in the image. In an example, the server 402 derives the location coordinates of the audience 106 using the standard location coordinate systems known in the art. At 614, the server 402 is configured to provide the information 12/20 associated with the recognized audience 106 and determined location coordinates of the audience 106 to the PED 208 through the communication network 404.

[0043] In an embodiment, at 616, the instructor 104 can map the information with the determined location of the audience 106. In addition5 , the instructor 104 can use the information to take attendance, view previous records, manipulate information, or to perform any other action. At 618, the instructor 106 can communicate with the audience 106 based on the mapped information. In an example, an interactive user interface is 10 displayed on the PED 208 of the instructor 104 to transfer data to the audience 106. The instructor 104 can use the interactive user interface to transfer or manipulate digital information to the audience 106, through the communication network 404, by dragging and dropping the digital information in the location coordinates of the audience 106. In an example, 15 the instructor 104 can perform an adaptive communication with the audience 106 based on the information received from the server 402.

[0044] FIG. 7 depicts a sequence diagram 700 illustrating operations performed by the system 400 of the FIG. 4, without using the server 402, according to embodiments as disclosed herein. In an 20 embodiment, at 702, the instructor 104 can use the PED 208 to capture an image of the audience 106. In an example, the instructor 104 can adjust the position of the PED 208 in a way that the audience 106 is within the predetermined region 212 of the imaging sensor 210. In an example, the instructor 104 adjusts the position of the PED 208 according to the position 25 of the audience 106 such that face of every individual in the audience 106 can be clearly visible in the image.

[0045] At 704, the PED 208 is configured to create an image in internal memory and perform a facial recognition function on the image to recognize the audience 106. The PED 208 can determine the facial 13/20 portions and recognizes the audience 106 by authenticating the determined face portions with the data stored in the internal memory. At 706, the PED 208 is configured to fetch the information associated with the recognized audience 106. At 708, the server 402 is configured to determine location of the audience 106 in the image. In an example, the server 402 derives th5 e location coordinates of the audience 106 using the local coordinate system or GPS coordinate system of the PED 208.

[0046] At 710, the PED 208 is configured to display the information and determined location coordinates of the audience 106. In an example, 10 the PED 208 provides an interactive user interface to the instructor 104 to transfer digital information to the audience 106. At 712, the instructor 104 can map the information with the determined location coordinates of the audience 106. In an example, the instructor can use the information to take attendance, view previous records, manipulate information, or to perform 15 any other action.

[0047] At 714, the instructor 106 can communicate with the audience 106 based on the mapped information. In an example, the instructor 104 can use the interactive user interface to transfer or manipulate any digital information to/from the audience 106 by dragging 20 and dropping the digital information in the location coordinates of the audience 106. In an example, the instructor 104 can perform an adaptive communication with the audience 106 based on the information displayed on the PED 208.

[0048] FIG. 8 depicts a flowchart 800 illustrating a method for 25 providing augmented reality environment, according to embodiments as disclosed herein. Various steps of the flowchart 800 are provided in blocks, where the steps are performed by the instructor 106, the PED 208, the server 402, and a combination thereof. The flowchart 800 starts at step 802. At step 804, the method includes capturing image of the audience 14/20 106. In an example, the instructor 104 uses the PED 208 to capture an image of the audience 106. The instructor 104 adjusts the position of the PED 208 in a way that the audience 106 is within the predetermined region 212 of the imaging sensor 210 and face of every individual in the audience 106 is clearly visible in the image5 .

[0049] At step 806, the method includes sending the image to the server 402. In an example, the instructor 104 uses the PED 208 to send the image to the server 402 through the communication network 404. In an example, the instructor 104 uses the PED 208 to further process the image 10 without sending it to the server 402. At step 808, the method includes recognizing the audience 106 in the image. In an example, the server 402 performs a facial recognition function on the image to determine the face portion of the audience 106. The server 402 recognizes the audience 106 by authenticating the determined face portions with the audience data 15 stored in the database.

[0050] At step 810, the method includes fetching information related to the audience 106. In an example, the server 402 fetches the information associated with the recognized audience 106 from the audience data stored in the database. At step 812, the method includes determining 20 location of the audience 106 in the image. In an example, the server 402 derives the location coordinates of the audience 106 using the standard location coordinate systems.

[0051] At step 814, the method includes providing the information and determined location of audience 106. In an example, the server 402 25 provides the determined location and the information associated with the audience 106 to the PED 208 through the communication network 404. At step 816, the method includes performing an adaptive communication with the audience 106 using the information and the determined location of the audience 106. In an example, the instructor 104 transfers the digital 15/20 information to the audience 106 by dragging and dropping the digital information in the location coordinates of the audience 106. In an example, the instructor 104 communicates with the audience 106 by mapping the received information with the determined location of the audience 106. At step 818, if the instructor 104 wants to perform the operation again, the5 n the method includes repeating the steps 804-818, else the flowchart 800 stops at step 820.

[0052] FIG. 9 depicts a flowchart 900 illustrating operations performed by the instructor 104, according to embodiments as disclosed 10 herein. The flowchart 900 starts at step 902. At step 904, the instructor 104 captures and sends an image to the server 402. In an example, the instructor 104 uses the PED 208 to capture the image of the audience 106 and send the image to the server 402 over the communication network 404. At step 906, the instructor 104 receives the location and information about 15 the audience 106. In an example, the server 402 recognizes the audience 106 and fetches the information associated with the recognized audience. In an example, the server 402 determines the location coordinates of the audience 106 in the image. Further, the server 402 sends the location coordinates and the information associated with the audience 106 to the 20 instructor 104. The instructor 106 uses the PED 208 to receive the location coordinates and the information about the audience 106 through the communication network 408.

[0053] At step 908, the instructor 104 communicates with the audience 106 by mapping the received information with the corresponding 25 location coordinates of the audience 106. At step 910, the instructor 104 uses the received information to take attendance, view previous records, manipulate information, or to perform any other task in the classroom or any other gathering. At step 912, if the instructor 104 wants to perform the 16/20 operations again, then the steps 904-912 of the flowchart 900 is repeated, else the flowchart 900 stops at step 914.

[0054] The various steps described with respect to the FIGS. 6-9 can be performed in sequential order, in random order, simultaneously, parallel, or a combination thereof. Further, in some embodiments, some o5 f the steps can be omitted, skipped, or added without departing from the scope of the invention.

[0055] FIG. 10 depicts a computing environment implementing the application, in accordance with various embodiments of the present 10 invention. As depicted, the computing environment comprises at least one processing unit that is equipped with a control unit and an Arithmetic Logic Unit (ALU), a memory, a storage unit, a clock chip, plurality of networking devices, and a plurality Input output (I/O) devices. The processing unit is responsible for processing the instructions of the algorithm. The processing 15 unit receives commands from the control unit in order to perform its processing. Further, any logical and arithmetic operations involved in the execution of the instructions are computed with the help of the ALU.

[0056] The overall computing environment can be composed of multiple homogeneous and/or heterogeneous cores, multiple CPUs of 20 different kinds, special media and other accelerators. The processing unit is responsible for processing the instructions of the algorithm. The processing unit receives commands from the control unit in order to perform its processing. Further, any logical and arithmetic operations involved in the execution of the instructions are computed with the help of the ALU. 25 Further, the plurality of process units may be located on a single chip or over multiple chips.

[0057] The algorithm comprising of instructions and codes required for the implementation are stored in either the memory unit or the storage or both. At the time of execution, the instructions may be fetched from the 17/20 corresponding memory and/or storage, and executed by the processing unit. The processing unit synchronizes the operations and executes the instructions based on the timing signals generated by the clock chip. The embodiments disclosed herein can be implemented through at least one software program running on at least one hardware device and performin5 g network management functions to control the elements. The elements shown in the FIGS. 1-10 include various units, blocks, modules, or steps described in relation with methods, processes, algorithms, or systems of the present invention, which can be implemented using any general purpose 10 processor and any combination of programming language, application, and embedded processor.

[0058] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various 15 applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of 20 limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments as described herein.

STATEMENT OF CLAIMS We claim: 1. A method for providing augmented reality based environment using a portable electronic device, the method comprising: capturing an image of at least one user; recognizing said at least one user in said image; fetching information associated with said at least one recognized user; determining location of said at least one user in said image; mapping said fetched information associated with said at least one user with said determined location of said at least one user; and communicating with said at least one user based on said mapping. 2. The method of claim 1, wherein said method further comprises adjusting position of said portable electronic device according to position of said at least one user. 3. The method of claim 2, wherein said position of said portable electronic device is adjusted in accordance to a predetermined region. 4. The method of claim 1, wherein said method further comprises sending said image to a server for recognizing said at least one user. 5. The method of claim 1, wherein recognizing said at least one user comprises: performing a facial recognition function on said image to determine face portion of said at least one user; and authenticating said determined face portion in said image to recognize said at least one user. 19/20 6. The method of claim 1, wherein determining location of said at least one user comprises deriving location coordinates of said at least one user in said image. 7. The method of claim 1, wherein said method further comprises transferring digital information to said at least one user using said information and said determined location of said at least one user. 8. The method of claim 7, wherein said digital information is transferred by dragging and dropping said digital information in said determined location of said at least one user. 9. The method of claim 1, wherein said method further comprises using said information and said determined location of said at least one user to take attendance of said at least one user in said environment. 10. The method of claim 1, said method further comprises performing an adaptive communication with said at least one user based on said fetched information.