Claims:1. A method for an automated real-time evaluation of unstructured input content, comprising:
assimilating data from one or more data sources, wherein the assimilating comprises:
identifying at least one of a candidate information, a primary domain associated with the candidate information, or one or more secondary domains that are inter-related with the primary domain from the data,
identifying at least one combination of one or more inter-related words and sentences in the data, and
vectorizing each word and each sentence in the combination of the one or more inter-related words and sentences;
automatically generating one or more questions based on the at least one combination of one or more inter-related words and sentences identified, wherein the one or more questions are related to at least one of the candidate information, the primary domain, the one or more secondary domains, or a combination of the primary domain and the one or more secondary domains from the assimilated data;
automatically generating one or more model answers corresponding to the one or more questions generated based on the at least one combination of one or more inter-related words and sentences identified, wherein the one or more model answers include the vectorized words and sentences;
receiving the unstructured input content from one or more unstructured answers provided by a candidate corresponding to the one or more questions, wherein the one or more questions are presented to the candidate digitally or offline;
evaluating the unstructured input content in real-time, wherein, for each unstructured answer, the evaluating comprises:
analyzing the unstructured answer,
mapping the unstructured answer with the one or more model answers based on the analysis and generating a combinatorial score,
vectorizing each word and each sentence in the unstructured answer;
mapping each vectorized word and sentence in the unstructured answer with each vectorized word and sentence in the one or more model answers, and generating a vectorization score, and
scoring the unstructured answer in real-time based on the vectorization score and the combinatorial score; and
presenting the evaluated score to a reviewer in real-time.
2. The method of claim 1, comprising:
initializing the automated real-time evaluation in response to at least one of a real-time audio, video, or text request.
3. The method of claim 1, wherein the receiving the unstructured input content is in at least one of a real-time audio, video, image, or text format.
4. The method of claim 1, wherein the automatically generating the one or more questions comprises generating:
one or more objective questions,
one or more subjective questions,
a combination of the one or more objective questions and the one or more subjective questions, or
one or more domain-related scenarios based on at least one of the primary domain, the one or more secondary domains, or a combination of the primary domain and the one or more secondary domains, wherein the one or more domain-related scenarios includes a scenario description and one or more objective, subjective or a combination of objective and subjective questions related to the scenario description,
the one or more questions automatically generated being at least one of un-related, inter-related, or interdependent.
5. The method of claim 1, comprising one of:
receiving the data from the one or more data sources via one or more input means, the one or more input means including at least one of text, images, audio, or video inputs, or
retrieving the data automatically via the one or more data sources available on a network in response to at least one of a real-time audio, video, or text request,
the one or more data sources including structured data or unstructured data related to at least one of the primary domain, the one or more secondary domains, and the candidate information.
6. The method of claim 1, wherein:
the assimilating of the data comprises:
assigning one or more pointers to the combination of the one or more inter-related words and sentences, wherein the pointers store information related to a starting point and an end point of each word and each sentence in the combination of one or more inter-related words and sentences respectively;
the automatically generating of the one or more model answers comprises:
associating the one or more pointers with the one or more model answers; and
the evaluating of each answer comprises:
the mapping of the unstructured answer with the one or more model answers via the one or more pointers.
7. The method of claim 6, comprising:
storing the assimilated data, the primary domain, the one or more secondary domains, the at least one combination of one or more inter-related words and sentences, the one or more pointers, the one or more questions, the one or more model answers, the unstructured answers, the vectorized words and sentences in the at least one combination of one or more inter-related words and sentences, the one or more model answers, and the unstructured answers, the combinatorial score, the vectorization score, and the evaluated score for training or retraining a model.
8. The method of claim 1, comprising:
varying at least one of a presentation order, difficulty, or complexity of the one or more questions in response to receiving at least one unstructured answer of the one or more unstructured answers from the candidate corresponding to at least one question of the one or more questions, the one or more unstructured answers being provided in at least one of a real-time text, video, audio, or image format.
9. The method of claim 1, wherein the scoring the answer comprises:
generating a deep learning model score for the unstructured answer based on the vectorization score and the combinatorial score, and
scoring the unstructured answer in real-time based on the vectorization score, the combination score, and the deep learning model score.
10. The method of claim 9, wherein the evaluating comprises:
generating a performance report associated with the candidate based on the evaluated score for each unstructured answer in real-time, wherein the generating of the performance report comprises:
generating a cumulative score for the one or more unstructured answers, the cumulative score including the vectorization score, the combinatorial score and the deep learning model score for each unstructured answer;
determining a proficiency score of the candidate in the primary domain and the one or more secondary domains based on the cumulative score; and
providing a recommendation score for a selection of the candidate based on the proficiency score.
11. The method of claim 9, wherein the assimilating, the automatically generating of the one or more questions and answers, and the evaluating is based on a model, the model being configured to generate the vectorization score, the combinatorial score, and the deep learning model score.
12. The method of claim 11, comprising:
retraining the model, the retraining including:
requesting and receiving a review score for the unstructured answer from a reviewer,
comparing the deep learning model score and the review score,
identifying the one or more model answers mapped with the unstructured answer, and
modifying the assimilated data and the one or more model answers based on the mapped unstructured answer when the deep learning model score is below a predefined threshold value in comparison with the review score; and
re-evaluating the unstructured answer based on the retrained model.
13. The method of claim 12, wherein the modifying of the assimilated data and the one or more model answers comprises:
matching one or more keywords in the unstructured answer with one or more keywords identified in the one or more model answers mapped with the unstructured answer;
determining a relatedness of each word and each sentence in the unstructured answer with each word and each sentence in the one or more model answers mapped with the unstructured answer;
determining a semantic and grammatical accuracy of the unstructured answer; and
determining a character length of the unstructured answer in comparison with a character length of the one or more model answers mapped with the unstructured answer.
14. A system for an automated real-time evaluation of unstructured input content, comprising:
a first electronic device of a candidate configured to receive the unstructured input content from the candidate;
a second electronic device of a reviewer configured to receive the automated real-time evaluation; and
a server in communication with the first electronic device and the second electronic device via a network, wherein the server is configured to perform one or more functions, the one or more functions including:
assimilating data from one or more data sources, wherein the assimilating comprises:
identifying at least one of a candidate information, a primary domain associated with the candidate information, or one or more secondary domains that are inter-related with the primary domain from the data,
identifying at least one combination of one or more inter-related words and sentences in the data, and
vectorizing each word and each sentence in the combination of the one or more inter-related words and sentences;
automatically generating one or more questions based on the at least one combination of one or more inter-related words and sentences identified, wherein the one or more questions are related to at least one of the candidate information, the primary domain, or a combination of the primary domain and the one or more secondary domains from the assimilated data;
automatically generating one or more model answers corresponding to the one or more questions generated based on the at least one combination of one or more inter-related words and sentences identified, wherein the one or more model answers include the vectorized words and sentences;
receiving the unstructured input content from one or more unstructured answers provided by a candidate corresponding to the one or more questions;
evaluating each unstructured answer of the one or more unstructured answers from the candidate in real-time, wherein the evaluating comprises:
analyzing the unstructured answer,
mapping the unstructured answer with the one or more model answers based on the analysis and generating a combinatorial score,
vectorizing each word and each sentence in the unstructured answer;
mapping each vectorized word and sentence in the unstructured answer with each vectorized word and sentence in the one or more model answers, and generating a vectorization score,
scoring the unstructured answer in real-time based on the vectorization score and the combinatorial score; and
presenting the evaluated score to the reviewer on the second electronic device in real-time.
15. An electronic device, comprising:
a bus;
a memory to store one or more predefined computer instructions; and
at least one processor coupled with the bus that is capable of executing the one or more predefined computer instructions stored in the memory in order to perform one or more functions, the one or more functions including:
assimilating data from one or more data sources, wherein the assimilating comprises:
identifying at least one of a candidate information, a primary domain associated with the candidate information, or one or more secondary domains that are inter-related with the primary domain from the data,
identifying at least one combination of one or more inter-related words and sentences in the data, and
vectorizing each word and each sentence in the combination of the one or more inter-related words and sentences;
automatically generating one or more questions based on the at least one combination of one or more inter-related words and sentences identified, wherein the one or more questions are related to at least one of the candidate information, the primary domain, or a combination of the primary domain and the one or more secondary domains from the assimilated data;
automatically generating one or more model answers corresponding to the one or more questions generated based on the at least one combination of one or more inter-related words and sentences identified, wherein the one or more model answers include the vectorized words and sentences;
receiving the unstructured input content from one or more unstructured answers provided by a candidate corresponding to the one or more questions;
evaluating each unstructured answer of the one or more unstructured answers from the candidate in real-time, wherein the evaluating comprises:
analyzing the unstructured answer,
mapping the unstructured answer with the one or more model answers based on the analysis and generating a combinatorial score,
vectorizing each word and each sentence in the unstructured answer;
mapping each vectorized word and sentence in the unstructured answer with each vectorized word and sentence in the one or more model answers, and generating a vectorization score, and
scoring the unstructured answer in real-time based on the vectorization score and the combinatorial score; and
presenting the evaluated score to a reviewer in real-time.
, Description:FIELD OF INVENTION
[0001] The present disclosure relates in general to performance evaluation. More particularly, the present disclosure relates to a method and system for automated real-time evaluation of unstructured input content.
BACKGROUND OF THE INVENTION
[0002] Various domains such as human resource recruitment, business process outsourcing (BPO) talent acquisition, customer service management and engagement, and student examination and evaluation involve several time-consuming and arduous processes that typically require manual involvement and monitoring at every step. For example, a BPO talent acquisition process involves identifying and/or short-listing a candidate for an evaluation, scheduling and conducting the evaluation, and selecting or rejecting the candidate based on the evaluation. Such a process is time consuming, cumbersome, and requires significant expenditure, human effort, expertise and experience.
[0003] In recent times, organizations and companies have adopted different online platforms and/or software solutions to conduct such processes. However, despite an ease and convenience of conducting such processes online, extended time and manual effort is still involved. More recently, with the advent of automation technologies, such as Artificial Intelligence (AI) and Machine Learning (ML), newer solutions have emerged to address and automate such processes. However, the newer solutions may only address a few aspects of such processes, such as, for example, automating scheduling of evaluation tests and recording of evaluation sessions. In addition, the newer solutions may still require human intervention to evaluate candidate performance accurately. For example, the candidate may provide verbal or written responses in an unstructured format to questions being asked during the evaluation sessions. The present automation technologies may not be capable of deciphering and/or evaluating the unstructured responses accurately and accordingly require a human reviewer to re-evaluate the candidate. However, the evaluation of the candidate by the human reviewer may also be prone to error and limited in scope based on a knowledge, expertise, and experience of the human reviewer.
[0004] Hence, there is a need for a method and system for an automated real-time evaluation of unstructured input content provided by the candidate during the evaluation process. As a result, an accurate evaluation of a proficiency and in-depth knowledge of the candidate may be automatically ascertained. Further, the time, the manual effort, and the cost involved in the evaluation of the candidate may also be minimized.
SUMMARY OF THE INVENTION
[0005] In an aspect of the present disclosure, a method for an automated real-time evaluation of unstructured input content is disclosed. The method includes a step of assimilating data from data sources. The assimilating includes a step of identifying a candidate information, a primary domain associated with the candidate information, and/or secondary domains that are inter-related with the primary domain from the data sources. Further, the assimilating includes a step of identifying a combination of inter-related words and sentences in the data sources. In addition, the assimilating also includes a step of vectorizing each word and each sentence in the combination of the inter-related words and sentences. The method also includes a step of automatically generating questions related to the candidate information, the primary domain, and/or a combination of the primary domain and the secondary domains from the assimilated data sources. Further, the method includes a step of automatically generating model answers corresponding to the questions generated based on the combination of inter-related words and sentences. The model answers may include the vectorized words and sentences. The method also includes a step of receiving the unstructured input content from unstructured answers provided by a candidate corresponding to the questions. In addition, the method includes a step of evaluating the unstructured input content in real-time. The evaluating includes a step of evaluating each unstructured answer received from the candidate in real-time. The evaluating also includes a step of analyzing the unstructured answer and mapping the unstructured answer with the model answers based on the analysis. The evaluating also includes a step of generating a combinatorial score based on the mapping. Further, the evaluating includes a step of vectorizing each word and each sentence in the unstructured answer. In addition, the evaluating includes a step of mapping each vectorized word and sentence in the unstructured answer with each vectorized word and sentence in the model answers. In addition, the evaluating includes a step of generating a vectorization score based on the vectorizing. The evaluating also includes a step of scoring the unstructured answer in real-time based on the vectorization score and the combinatorial score. Further, the method also includes a step of presenting the evaluated score to a reviewer in real-time.
[0006] In another aspect of the present disclosure, a system for the automated real-time evaluation of unstructured input content is disclosed. The system comprises a first electronic device of a candidate configured to receive the unstructured input content from the candidate. The system also comprises a second electronic device of a reviewer configured to receive the automated real-time evaluation. Further, the system comprises a server that is in communication with the first electronic device and the second electronic device via a network. The server may be configured to perform different functions including, but not limited to, the steps of the method for the automated real-time evaluation of unstructured input content.
[0007] In another aspect of the present disclosure, an electronic device for the automated real-time evaluation of unstructured input content is disclosed. The electronic device comprises a bus and a memory to store predefined computer instructions. Further, the electronic device comprises a processor coupled with the bus that is capable of executing the one or more predefined computer instructions stored in the memory in order to perform one or more functions. The functions performed by the processor, include, but are not limited to, the steps of the method for the automated real-time evaluation of unstructured input content.
BRIEF DESCRIPTION OF DRAWINGS
[0008] FIG. 1 is a schematic block diagram of an environment including a first electronic device of a candidate, a server, and a second electronic device of a reviewer, in accordance with which various embodiments of the present disclosure may be implemented;
[0009] FIG. 2 is a schematic block diagram of different components in the server of FIG. 1, in accordance with an embodiment of the present disclosure;
[0010] FIG. 3 is a schematic block diagram of a processor in the server of FIG. 2, in accordance with the embodiment of the present disclosure; and
[0011] FIG. 4 is a schematic block diagram of a method for automated real-time evaluation of unstructured input content, in accordance with the embodiment of the present disclosure.
DETAILED DESCRIPTION
[0012] Reference will now be made in detail to embodiments of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.
[0013] Referring to FIG. 1, a schematic block diagram of an environment including a first electronic device 105 of a candidate, a server 110, and a second electronic device 120 of a reviewer inter-connected via a network 115 is disclosed. Examples of the electronic devices 105, 120, and/or the server 110 include, but are not limited to, computers, laptops, mobile devices, handheld devices, personal digital assistants (PDAs), tablet personal computers, digital notebook, and similar electronic devices. The network 115 may include communication networks such as, but not limited to, a Local Area Network (LAN), a Wireless Local Area Network (WLAN), a Wide Area Network (WAN), internet, a Small Area Network (SAN), and the Internet.
[0014] The first electronic device 105 may be configured to receive the unstructured input content from the candidate and send the unstructured input content to the server 110 via the network 115. The second electronic device 120 of the reviewer may be configured to receive the automated real-time evaluation from the server 110 via the network 115. The server 110, herein referred to as “knowledge engine 110”, may be configured to perform different functions in order to perform the automated real-time evaluation of the unstructured input content provided by the candidate.
[0015] In an embodiment, the knowledge engine 110 may be configured to perform the automated real-time evaluation of the unstructured input content in real-time in response to a request received via different input means such as, but not limited to, real-time text, audio, video, and/or image inputs. In another embodiment, the knowledge engine 110 may be configured to perform the automated real-time evaluation in real-time in response to a request received from the second electronic device 120 of the reviewer or the first electronic device 105 of the candidate via the network 115. The candidate may send the request to the knowledge engine 110 from the first electronic device 105 via the different input means. Similarly, the reviewer may send the request to the knowledge engine 110 from the second electronic device 120 via the different input means. In an embodiment, the knowledge engine 110 may be configured to receive the request in a conversational natural language format from the first electronic device 105 or the second electronic device 120 via the network 115.
[0016] The knowledge engine 110 may also be configured to generate and present an evaluated score for each unstructured answer received from the candidate to a reviewer on the second electronic device 120 in real-time. In an embodiment, the knowledge engine 110 may also be configured to generate a performance report associated the candidate based on the evaluated scores corresponding to the unstructured answers in real-time. Further, knowledge engine 110 may be configured to present the performance report to the reviewer on the second electronic device 120 in real-time.
[0017] Referring to FIG. 2, schematic block diagram of different components in the knowledge engine 110 of FIG. 1 is disclosed. In an embodiment, the knowledge engine 110 includes a bus 205 or other communication mechanism for communicating information, and a processor 210 coupled with the bus 205 for processing information. The knowledge engine 110 also includes a memory 215, such as a random-access memory (RAM) or other dynamic storage device, coupled to the bus 205 for storing information and instructions to be executed by the processor 210. The memory 215 can be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 210. The knowledge engine 110 further includes a read only memory (ROM) 220 or other static storage device coupled to bus 205 for storing static information and instructions for processor 210. A storage unit 225, such as a magnetic disk or optical disk, is provided and coupled to the bus 205. The storage unit 225 may store data retrieved from different data sources and one or more machine-learning models. The machine-learning models may correspond to mathematical models generated from computer algorithms based on the data retrieved from different data sources. The data may include, but is not limited to, text, audio, images, and/or video content. The machine-learning models may be configured to perform an automated real-time evaluation of the unstructured input content.
[0018] The knowledge engine 110 can be coupled via the bus 205 to a display 230, such as a cathode ray tube (CRT), and liquid crystal display (LCD) for displaying information to the user. An input device 235, including alphanumeric and other keys, is coupled to bus 205 for communicating information and command selections to the processor 210. Another type of user input device is a cursor control 240, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to the processor 210 and for controlling cursor movement on the display 230. The input device 235 can also be included in the display 230, for example a touch screen and may also include compatible devices with the display 230 such as a stylus. Further, the input device 235 can also include audio-visual devices such as a microphone or a camera.
[0019] Various embodiments are related to the use of knowledge engine 110 for implementing the techniques described herein. In one embodiment, the techniques are performed by the knowledge engine 110 in response to the processor 210 executing instructions included in the memory 215. Such instructions can be read into the memory 215 from another machine-readable medium, such as the storage unit 225. Execution of the instructions included in the memory 215 causes the processor 210 to perform the process steps described herein.
[0020] The term “machine-readable medium” as used herein refers to any medium that participates in providing data that causes a machine to operate in a specific fashion. In an embodiment implemented using the knowledge engine 110, various machine-readable medium is involved, for example, in providing instructions to the processor 210 for execution. The machine-readable medium can be a storage media. Storage media includes both non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage unit 225. Volatile media includes dynamic memory, such as the memory 215. All such media must be tangible to enable the instructions carried by the media to be detected by a physical mechanism that reads the instructions into a machine.
[0021] Common forms of machine-readable medium include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper-tape, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge.
[0022] In another embodiment, the machine-readable medium can be a transmission media including coaxial cables, copper wire and fiber optics, including the wires that comprise the bus 205. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications. Examples of machine-readable medium may include but are not limited to a carrier wave as describer hereinafter or any other medium from which the knowledge engine 110 can read, for example online software, download links, installation links, and online links. For example, the instructions can initially be carried on a magnetic disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the knowledge engine 110 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on the bus 205. The bus 205 carries the data to the memory 215, from which the processor 210 retrieves and executes the instructions. The instructions received by the memory 215 can optionally be stored on storage unit 225 either before or after execution by the processor 210. All such media must be tangible to enable the instructions carried by the media to be detected by a physical mechanism that reads the instructions into a machine.
[0023] The knowledge engine 110 also includes a communication interface 245 coupled to the bus 205. The communication interface 245 provides a two-way data communication coupling to the network 115. For example, the communication interface 245 can be an integrated service digital network (ISDN) card or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the communication interface 245 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, the communication interface 245 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
[0024] In some embodiments, the processor 210 may be capable of executing the one or more predefined computer instructions in order to perform one or more functions. The processor 210 may also execute one or more computer modules 305 – 320 in order to perform the one or more functions. The computer modules 305 – 320 in the processor 210 may correspond to a combination of hardware circuitry and software components configured to the perform the functions.
[0025] Referring to FIGS. 2 – 3, the processor 210 may include a data assimilation unit 305, a question and answer (QA) generation unit 310, an unstructured input unit 315, and an evaluation unit 320.
[0026] The data assimilation unit 305 may be configured to assimilate data from different data sources such as, but not limited to, candidate information in curriculum vitae, text documents, blogs, scientific papers, textbooks, essays, videos, images, audio books, online content, and web sources having relevant information. The data assimilation unit 305 may be configured to receive the data sources via different input means such as, but not limited to, text, image, audio, and/or video inputs. In an embodiment, the second electronic device 120 of the reviewer may be configured to provide the data sources to the data assimilation unit 305 via the network 115. In another embodiment, data assimilation unit 305 may be configured to receive the data from the data sources directly via the different input means. In yet another embodiment, the data assimilation unit 305 may also be configured to retrieve the data automatically from the data sources from one or more web sources such as, but not limited to, online blogs, scientific papers, online information pages, images, videos, and audio via the network 115. In an embodiment, the data assimilation unit 305 may be configured to convert the data received from the different input means and/or the web sources into digital text in order to assimilate the data.
[0027] In an embodiment, the data assimilation unit 305 may be configured to assimilate the data from different data sources in real-time in response to a request received via different input means such as, but not limited to, real-time text, audio, video, and/or image inputs. In another embodiment, the data assimilation unit 305 may be configured to assimilate the data from different data sources in real-time in response to a request received from the second electronic device 120 of the reviewer or the first electronic device 105 of the candidate via the network 115. The candidate may send the request to the data assimilation unit 305 from the first electronic device 105 via the different input means. Similarly, the reviewer may send the request to the data assimilation unit 305 from the second electronic device 120 via the different input means. In an embodiment, the data assimilation unit 305 may be configured to receive the request in a conversational natural language format from the first electronic device 105 or the second electronic device 120 via the network 115.
[0028] Further, the data assimilation unit 305 may be configured to identify a candidate information from the data obtained from the data sources. Examples of the candidate information include, but are not limited to, legal identification number, phone number, name, email id, work experience, educational qualifications, interests, and hobbies of the candidate. In addition, the data assimilation unit 305 may be configured to identify a primary domain from the data obtained from the data sources. Examples of the primary domain include, but are not limited to, a programming language such as Java or Python, a subject such as geography or history, or a topic such as health or fitness. The data assimilation unit 305 may also be configured to identify different hierarchical sub-topics related to the primary domain. For example, the data assimilation unit 305 may be configured to identify different programming basics such as syntax, data types, functions, file handling, and/or memory management as the hierarchical sub-topics related to the programming language, such as Java or Python, identified as the primary domain. The data assimilation unit 305 may also be configured to identify secondary domains that are inter-related with the primary domain from the data obtained from the data sources. For example, the data assimilation unit 305 may be configured to identify different database management programming languages, such as Structured Query Language (SQL), as the secondary domains that are inter-related with the programming language, such as Java or Python, identified as the primary domain. In an embodiment, the data assimilation unit 305 may be configured to identify the hierarchical sub-topics and/or the secondary domains from the data sources. In another embodiment, the hierarchical sub-topics and the secondary domains associated with and inter-related with the primary domain respectively may be predefined in the data assimilation unit 305. Accordingly, the data assimilation unit 305 may be configured to perform the identification of the hierarchical sub-topics and/or the secondary domains automatically without a dependence on the data obtained from the data sources. In yet another embodiment, the data assimilation unit 305 may be configured to identify the primary domain, the hierarchical sub-topics, and/or secondary domains based on the candidate information.
[0029] The data assimilation unit 305 may also be configured to identify at least one combination of inter-related words and sentences in the data obtained from the data sources. The inter-related words and sentences may be sequentially or non-sequentially arranged in the data obtained from the data sources. The data assimilation unit 305 may identify the combination of inter-related words and sentences based on, but not limited to, keyword analysis, grammatical analysis, and/or semantic analysis. The data assimilation unit 305 may also be configured to assign one or more pointers to the combination of one or more inter-related words and sentences. The pointers assigned may store information related to a starting point and an end point of each word and each sentence in the combination of one or more inter-related words and sentences identified.
[0030] Further, the data assimilation unit 305 may be configured to vectorize each word and each sentence in the combination of the one or more inter-related words and sentences. In an embodiment, the data assimilation unit 305 may be configured to vectorize each word in each sentence in the combination of the one or more inter-related words and sentences. The vectorization of words and sentences may correspond to a conversion of the words and the sentences into numbers. Accordingly, the data assimilation unit 305 may be configured to associate each word with a corresponding word vector and each sentence with a corresponding sentence vector. In an embodiment, the data assimilation unit 305 may be configured to generate the sentence vector for each sentence by calculating a sum total of the word vector associated with each word in the sentence.
[0031] The data assimilation unit 305 may be also configured to store the assimilated data, the primary domain, the secondary domains, the combination of one or more inter-related words and sentences, the vectorized words and sentences, and the pointers in the storage unit 225 of the knowledge engine 110. Further, in an embodiment, the data assimilation unit 305 may be configured to perform the identifying of the candidate information, the primary domain, and the secondary domains, the identifying of the combination of inter-related words and sentences, and the vectorization simultaneously.
[0032] The QA generation unit 310 may be configured to automatically generate one or more questions based on the combination of one or more inter-related words and sentences identified in the assimilated data. The QA generation unit 310 may be also be configured to automatically generate the questions in real-time in response to a request received via different input means such as, but not limited to, real-time text, audio, video, and/or image inputs. In an embodiment, the QA generation unit 310 may be configured to generate the questions in real-time in response to a request received from the second electronic device 120 of the reviewer or the first electronic device 105 of the candidate via the network 115. The candidate may send the request to the QA generation unit 310 from the first electronic device 105 via the different input means. Similarly, the reviewer may send the request to the QA generation unit 310 from the second electronic device 120 via the different input means. In an embodiment, the QA generation unit 310 may be configured to receive the request in a conversational natural language format from the first electronic device 105 or the second electronic device 120 via the network 115.
[0033] The QA generation unit 310 may be configured to generate the questions that may be un-related, inter-related, or interdependent. In an embodiment, the one or more questions generated may be related to the candidate information, the primary domain, the secondary domains, and/or a combination of the primary domain and the secondary domains from the assimilated data. Further, the QA generation unit 310 may be configured to generate objective questions, subjective questions, a combination of the objective questions and the subjective questions. In an embodiment, the QA generation unit 310 may also be configured to generate domain-related scenarios based on the primary domain, the secondary domains, or the combination of the primary domain and the secondary domains. The domain-related scenarios may include a scenario description. The domain-related scenarios may also include objective, subjective or the combination of objective and subjective questions related to the scenario description.
[0034] In an embodiment, the evaluation unit 320 may be configured to automatically generate the questions having varying difficulty and complexity. The evaluation unit 320 may generate questions with increasing difficulty by increasing a number of the hierarchical sub-topics associated with the primary domain identified by the data assimilation unit 305. Further, the evaluation unit 320 may generate questions with increasing complexity by increasing a number of questions related to the hierarchical sub-topic identified by the data assimilation unit 305.
[0035] Further, the QA generation unit 310 may be configured to automatically generate one or more model answers corresponding to the questions generated. The model answers generated may be based on the combination of inter-related words and sentences identified. Accordingly, the model answers generated may also include the vectorized words and sentences generated by the QA generation unit 310. Further, the QA generation unit 310 may also be configured to associate the pointers previously assigned to the combination of inter-related words and sentences with the model answers. In an embodiment, the QA generation unit 310 may be configured to automatically generate one or more model answers in real-time in response to a request received via different input means such as, but not limited to, real-time text, audio, video, and/or image inputs. The QA generation unit 310 may receive the request directly or from the second electronic device 120 of the reviewer.
[0036] In one embodiment, the first electronic device 105 and the second electronic device 120 may be provided with an executable application respectively that facilitates the candidate and/or the reviewer to place the requests to the data assimilation unit 305 and/or the QA generation unit 310 respectively via the network 115. The executable application may be pre-installed or downloaded from an application distribution platform in the first electronic device 105 and the second electronic device 120 respectively. In another embodiment, the knowledge engine 110 may be configured to host a web application that be accessed by the candidate and/or the reviewer from the first electronic device 105 and the second electronic device 120 via the network 115 respectively in order to place the requests.
[0037] In an embodiment, the QA generation unit 310 may be configured to interactively present the automatically generated questions to the candidate on the first electronic device 105 via the network 115. For example, the QA generation unit 310 may be configured to interactively present the questions in real-time text, audio, video, and/or image format on the first electronic device 105 of the candidate. The QA generation unit 310 may also be configured to interactively present the questions as prompts or in a conversational natural language format. In another embodiment, the automatically generated questions by the QA generation unit 310 may also be presented to the candidate offline. In an embodiment, the QA generation unit 310 may be configured to interactively present the questions via the executable application or the web application provided on the first electronic device 105 of the candidate via the network 115.
[0038] In an embodiment, the QA generation unit 310 may be also configured to store the automatically generated questions and the model answers in the storage unit 225 of the knowledge engine 110.
[0039] The functions collectively performed by the data assimilation unit 305 and the QA generation unit 310 are described herein with an illustrative example. Referring to the illustrative example, the data assimilation unit 305 may identify the primary domain as “Python”, a programming language, the hierarchical sub-topic as “Garbage collection and Memory management” associated with “Python”, and the secondary domain as “SQL”, a database programming language inter-related with “Python. The data assimilation unit 305 may receive an unstructured text document related to “Python” and “Garbage collection and Memory management”. The data assimilation unit 305 may then assimilate and store the below stated data including the inter-related words and sentences related to “Python”, “Garbage collection and Memory management”, and “SQL”:
[0040] “Python memory is managed by Python private heap space. All Python objects and data structures are located in a private heap. The programmer does not have access to this private heap and interpreter takes care of this Python private heap. The allocation of Python heap space for Python objects is done by Python memory manager. The core API gives access to some tools for the programmer to code. Python also have an inbuilt garbage collector, which recycle all the unused memory and frees the memory and makes it available to the heap space. The following points are worth nothing for the garbage collector, Python maintains a count of how many references there are to each object in memory. When a reference count drops to zero, it means the object is dead and Python can free the memory it allocated to that object. The garbage collector looks for reference cycles and cleans them up. Python uses heuristics to speed up garbage collection Recently created objects might as well be dead. The garbage collector assigns generations to each object as it is created. It deals with the younger generations first.
Mirroring is a high availability solution. It is designed to maintain a hot standby server which is consistent with the primary server in terms of a transaction. Transaction Log records are sent directly from principal server to a secondary server which keeps a secondary server up to date with the principal server.
To execute a SQL query in Python, you’ll need to use a cursor, which abstracts away the access to database records. MySQL Connector/Python provides you with the MySQLCursor class, which instantiates objects that can execute MySQL queries in Python. An instance of the MySQLCursor class is also called a cursor. you need to pass this query to cursor.execute(), which accepts a MySQL query and executes the query on the connected MySQL database. By default, your MySQL connector doesn’t auto-commit transactions. In MySQL, modifications mentioned in a transaction occur only when you use a COMMIT command in the end. Always call this method after every transaction to perform changes in the actual table.”
[0041] Accordingly, the QA generation unit 310 may automatically generate the one or more questions related to the primary domain from the above-stated assimilated data. For example, the QA generation unit 310 may generate a domain-related scenario including the following scenario description:
[0042] 1. “Considering you are a senior python developer in X organization. Your technical manager asks you to explain python’s memory management to junior developer. Before explaining to junior developers, manager asks you to explain memory management in python to him.”
[0043] The QA generation unit 310 may then generate the following question related to the hierarchical sub-topic associated with the primary domain:
[0044] “So, what do you know about memory management in python?”
[0045] The QA generation unit 310 may also automatically generate a model answer corresponding to the above-stated question related to the primary domain. The model answer generated by the QA generation unit 310 may include a part of the above-stated assimilated data. Exemplarily, the QA generation unit 310 may assign pointers to the automatically generated model answer and thereby, demarcate the model answer from the above-stated assimilated data. Referring to the above-stated assimilated data, the QA generation unit 310 may assign a starting pointer of 0 and an end point of 94 to the model answer. The starting pointer of 0 and the end point of 94 may represent a count of a number of words from a beginning of the above-stated assimilated data to an end count of the number of words in the model answer generated by the QA generation unit 310. Accordingly, the model answer automatically generated from the above-stated assimilated data may be as described below:
[0046] “Python memory is managed by Python private heap space. All Python objects and data structures are located in a private heap. The programmer does not have access to this private heap and interpreter takes care of this Python private heap. The allocation of Python heap space for Python objects is done by Python memory manager. The core API gives access to some tools for the programmer to code. Python also have an inbuilt garbage collector, which recycle all the unused memory and frees the memory and makes it available to the heap space.”
[0047] Similarly, the QA generation unit 310 may also automatically generate the following question related to the hierarchical sub-topic associated with the primary domain identified from the above-stated assimilated data:
[0048] 2. “If you have to explain to the interns in your company about the garbage collections in python along with its importance. How will you explain?”
[0049] Accordingly, the QA generation unit 310 may generate a model answer from the above-stated quested related to the primary domain that includes a part of the above-stated assimilated data. Referring to the above-stated assimilated data, the QA generation unit 310 may assign a starting pointer of 94 and an end pointer of 192 to the model answer generated for the above-stated question.
[0050] Similarly, the QA generation unit 310 may also automatically generate the following question related to the secondary domain “SQL” identified from the above-stated assimilated data:
[0051] 3. “You need to maintain high availability of SQL server. Mirroring is one of the solutions. Can you explain what is mirroring and how it helps in maintaining high availability?”
[0052] Accordingly, the QA generation unit 310 may generate a model answer from the above-stated quested related to the secondary domain that includes a part of the above-stated assimilated data. Referring to the above-stated assimilated data, the QA generation unit 310 may assign a starting pointer of 189 and an end pointer of 240 to the model answer generated for the above-stated question.
[0053] Similarly, the QA generation unit 310 may also automatically generate the following question related to a combination of the primary domain “Python” and the secondary domain “SQL” identified from the above-stated assimilated data:
[0054] “Consider you are using mysql database and python-mql-connector for connecting to a database. You ran the following query, but changes are not reflected in the database? What could be the problem?
create_movies_table_query = """
CREATE TABLE movies(
id INT AUTO_INCREMENT PRIMARY KEY,
title VARCHAR(100),
release_year YEAR(4),
genre VARCHAR(100),
collection_in_mil INT
)
"""
with connection.cursor() as cursor:
cursor.execute(create_movies_table_query)
[0055] Accordingly, the QA generation unit 310 may generate a model answer from the above-stated quested related to the combination of the primary domain “Python” and the secondary domain “SQL” that includes a part of the above-stated assimilated data. Referring to the above-stated assimilated data, the QA generation unit 310 may assign a starting pointer of 320 and an end pointer of 340 to the model answer generated for the above-stated question.
[0056] The unstructured input unit 315 may be configured to receive the unstructured input content from unstructured answers provided by a candidate corresponding to the questions. The unstructured input unit 315 may be configured to receive the unstructured answers from the first electronic device 105 of the candidate via the network 115. In an embodiment, the unstructured input unit 315 may be configured to receive the unstructured input content in real-time audio, video, image, and/or text format from the candidate. In another embodiment, the unstructured input unit 315 may be configured to receive digital images and/or electronic documents including the unstructured answers that are hand-written by the candidate corresponding the questions presented interactively or offline. In an embodiment, the unstructured input unit 315 may be configured to individually segregate each question and each unstructured answer received corresponding to the question from the candidate. In an embodiment, the unstructured input unit 315 may be configured to receive the unstructured input content via the executable application or the web application provided on the first electronic device 105 of the candidate via the network 115.
[0057] In an embodiment, the unstructured input unit 315 may be also configured to store the unstructured input content including the unstructured answers in the storage unit 225 of the knowledge engine 110.
[0058] The evaluation unit 320 may also be configured to evaluate the unstructured input content in real-time. In an embodiment, for each unstructured answer provided by the candidate, the evaluation unit 320 may be configured to analyze the unstructured answer by identifying keywords in the unstructured answer, determining semantic and grammatical accuracy of the unstructured answer, and determining a character length of the unstructured answer.
[0059] The evaluation unit 320 may also be configured to map the unstructured answer with the one or more model answers based on the analysis. In an embodiment, the evaluation unit 320 may also be configured to map the unstructured answer with the one or more model answers based on the pointers assigned to the combination of inter-related words and sentences. In an embodiment, the evaluation unit 320 may be configured to determine a relatedness between the unstructured answer and the model answers mapped with the unstructured answer. The evaluation unit 320 may determine the relatedness by matching one or more keywords identified in the unstructured answer with one or more keywords identified in the model answers. The evaluation unit 320 may configured to generate a combinatorial score based on the mapping. The combinatorial score may correspond to a percentage of the relatedness between the unstructured answer and the model answers mapped with the unstructured answer.
[0060] Further, the evaluation unit 320 may be configured to vectorize each word and each sentence in the unstructured answer provided by the candidate. Accordingly, the evaluation unit 320 may be configured to associate each word in the unstructured answer with a corresponding word vector and each sentence in the unstructured answer with a corresponding sentence vector. In an embodiment, the evaluation unit 320 may be configured to generate the sentence vector for each sentence in the unstructured answer by calculating a sum total of the word vector associated with each word in the sentence. The evaluation unit 320 may be also configured to map each vectorized word and sentence in the unstructured answer with each vectorized word and sentence in the model answers. In an embodiment, the evaluation unit 320 may be configured to determine a variation of the each vectorized word and sentence in the unstructured answer in comparison with each vectorized word and sentence in the model answers. Accordingly, the evaluation unit 320 may be configured to generate a vectorization score based on the mapping. The vectorization score may correspond to a percentage of the variation determined.
[0061] The mapping of the vectorized words and sentences are described herein with an illustrative example. Referring to the illustrative example, the model answer generated by the QA generation unit 310 may be:
“Model Answer: Afzal Khan embrace Shivaji. Afzal Khan stabbed Shivaji. Shivaji protected by Chain mail. Shivaji Retaliated. Shivaji killed Afzal Khan.”
[0062] The unstructured answer received by the unstructured input unit 315 from the candidate may be:
“Afzal Khan came close Shivaji. Afzal Khan stab Shivaji. Shivaji saved by Chain mail. Shivaji Retaliated. Shivaji Killed Afzal Khan.”
[0063] Accordingly, based on the mapping of the vectorized words and sentences, the evaluation unit 320 may be configured to determine that the following words and/or phrases may be similar in the model answer as well as the unstructured answer respectively:
“embrace” and “came close”
“protected by” and “saved by”
“stabbed” and “stabbed”
“killed” and “killed”
[0064] In an embodiment, the evaluation unit 320 may be configured to generate a deep learning model score for the unstructured answer based on the vectorization score and the combinatorial score. The evaluation unit 320 may be configured to assign a percentage of priority or weightage to the vectorization score and the combinatorial score. Accordingly, the deep learning model score may correspond to a percentage of an overall score determined based on the percentage of weightage assigned to the vectorization score and the combinatorial score.
[0065] Further, the evaluation unit 320 may be configured to score the unstructured answer in real-time based on the vectorization score, the combinatorial score, and/or the deep learning score. In an embodiment, the evaluated score of the unstructured answer may correspond to the deep learning model score.
[0066] In an embodiment, for instances when the evaluation unit 320 is unable to map the unstructured answer with the model answers, the evaluation unit 320 may be configured to score the unstructured answer in real-time solely based on the analysis of the unstructured answer.
[0067] The evaluation unit 320 may be also configured to present the evaluated score corresponding to the unstructured answer to a reviewer on the second electronic device 120 in real-time. Similarly, the evaluation unit 320 may be also configured to present the evaluated scores of all the unstructured answers provided by the candidate from the first electronic device 105 to the reviewer on the second electronic device 120 via the network 115 in real-time. In an embodiment, evaluation unit 320 may also be configured to present the questions and the unstructured answers provided by the candidate to the reviewer on the second electronic device 120. In an embodiment, the evaluation unit 320 may be configured to present the evaluated score via the executable application or the web application provided on the second electronic device 120 of the reviewer via the network 115.
[0068] Further, in an embodiment, the evaluation unit 320 may also be configured to receive a review score of the reviewer from the second electronic device 120 corresponding to each unstructured answer provided by the candidate via the network 115. The second electronic device 120 may be configured to enable the reviewer to provide the review score based on a manual evaluation of the questions and the unstructured answers presented by the evaluation unit 320.
[0069] In an embodiment, the evaluation unit 320 may be configured to vary the presentation order, the difficulty, or the complexity of subsequent questions from the automatically generated questions based on the evaluation of the unstructured answer correspond to a question.
[0070] In an embodiment, the evaluation unit 320 may be configured to generate a performance report associated the candidate based on the evaluated score for each unstructured answer in real-time. In order generate the performance report, the evaluation unit 320 may be configured to generate a cumulative score for the unstructured answers. The cumulative score may include the vectorization score, the combinatorial score and the deep learning model score for each unstructured answer. Further, the evaluation unit 320 may be configured to determine a proficiency score of the candidate in the primary domain and the secondary domains based on the cumulative score. In addition, the evaluation unit 320 may be configured to provide a recommendation score for a selection of the candidate based on the proficiency score.
[0071] In an embodiment, the evaluation unit 320 may be also configured to store the vectorized words and sentences of the unstructured answers, the evaluated scores generated corresponding to the unstructured answers, the cumulative score, the proficiency score, the recommendation score, the performance report, and the review score in the storage unit 225 of the knowledge engine 110.
[0072] In an embodiment, the evaluation unit 320 may be configured to perform the analyzing, the mapping of the unstructured answer, the vectorization, and the mapping of the vectorized words and sentences simultaneously.
[0073] In an embodiment, the functions of the data assimilation unit 305, the QA generation unit 310, the unstructured input unit 315, and the evaluation unit 320 may be performed by the one or more machine learning models stored in the storage unit 225 of the knowledge engine 110. The one or more machine learning models are herein referred to as a “model” for purposes of clarity. The knowledge engine 110 may be configured to be retrain the model based on the evaluation of the unstructured input content. In an embodiment, in order to retrain the model, the processor 210 of the knowledge engine 110 may include a retraining unit 250 as shown in FIG. 2. The retraining unit 250 may be configured to request and receive the review score provided by the reviewer from the second electronic device 120 for the unstructured answer via the network 115. In an embodiment, the retraining unit 250 may be configured to retrain of the model automatically upon receiving the review score from the reviewer. In addition, the retraining unit 250 may be configured to compare the deep learning model score and the review score. Further, the retraining unit 250 may be configured to identify the model answers mapped with the unstructured answer. The retraining unit 250 may also be configured to modify the assimilated data and the model answers based on the mapped answer for instances when the deep learning model score is below a predefined threshold value in comparison with the review score. In an embodiment, in order modify the assimilated data and the model answers, the retraining unit 250 may be configured to match one or more keywords in the unstructured answer with one or more keywords identified in the model answers mapped with the unstructured answer. Further, the retraining unit 250 may be configured to determine a relatedness of each word and each sentence in the unstructured answer with each word and each sentence in the model answers mapped with the unstructured answer. In addition, the retraining unit 250 may be configured to determine a semantic and grammatical accuracy of the unstructured answer. The retraining unit 250 may also be configured to determine a character length of the unstructured answer in comparison with a character length of the model answers mapped with the unstructured answer. The retraining unit 250 may be configured to modify the assimilated data and the model answers only when the keywords, the relatedness, the semantic and grammatical accuracy, the character length of the unstructured answer meet predefined criteria in the model.
[0074] The retrained model, thus obtained from the retraining unit 250 by modifying the assimilated data and the model answers, may be configured to re-evaluate the unstructured answer based on the retrained model. In an embodiment, the retrained model may be also be configured to perform the functions of the data assimilation unit 305, the QA generation unit 310, the unstructured input unit 315, and the evaluation unit 320 in order to perform the re-evaluation. Accordingly, it may be apparent that the retraining unit 250 may be configured to retrain the model repeatedly based on the review score obtained over a period of time to improve an accuracy of the functions performed computer modules 305 – 320. In an embodiment, the retraining unit 250 may be configured to use information stored by the data assimilation unit 305, the QA generation unit 310, the unstructured input unit 315, and the evaluation unit 320 in the storage unit 225 of the knowledge engine 110 in order to train and/or retrain the model.
[0075] It may be apparent that, in an embodiment, the first electronic device 105 of the candidate may also include the different components of the knowledge engine 110 such as the storage unit 225 having the machine-learning models, the processor 210 having the modules 250, 305 – 320 such that the first electronic device 105 may be configured to perform the evaluation of the unstructured input content and present the evaluated scores to the reviewer directly without a dependence on the knowledge engine 110. The first electronic device 105 may be configured to store one or more executable applications in a storage unit of the first electronic device 105 that include the modules 250, 305 – 320 in order to perform the evaluation. The executable applications may be provided in-built or downloaded/installed from an application distribution platform provided in the first electronic device 105.
Industrial Applicability
[0076] Referring to FIG. 4, a method 400 for automated real-time evaluation of unstructured input content is disclosed. The method 400 includes a step 405 of assimilating data from data sources. The step 405 includes a step 406 of identifying a candidate information, a primary domain associated with the candidate information, and/or secondary domains that are inter-related with the primary domain from the data sources. Further, the step 405 includes a step 407 of identifying a combination of inter-related words and sentences in the data sources. In addition, the step 405 also includes a step 408 of vectorizing each word and each sentence in the combination of the inter-related words and sentences. The method 400 also includes a step 410 of automatically generating questions related to the candidate information, the primary domain, and/or a combination of the primary domain and the secondary domains from the assimilated data sources. Further, the method 400 includes a step 415 of automatically generating model answers corresponding to the questions generated based on the combination of inter-related words and sentences. The model answers may include the vectorized words and sentences. The method 400 also includes a step 420 of receiving the unstructured input content from answers provided by a candidate corresponding to the questions. In addition, the method 400 includes a step 425 of evaluating each answer from the candidate in real-time. The step 425 includes a step 426 of analyzing the answer and mapping the answer with the combination of inter-related words and sentences identified based on the analysis and generating a combinatorial score. Further, the step 425 includes a step 427 of vectorizing each word and each sentence in the answer and mapping each vectorized word and sentence in the answer with each vectorized word and sentence in the combination of inter-related words and sentences. In addition, the step 425 includes a step 427 of generating a vectorization score. Further, step 425 includes a step 428 of generating a deep learning score based on the combinatorial score and the vectorization score. The step 425 also includes a step 429 of scoring the answer in real-time based on the vectorization score and the combinatorial score. Further, the step 429 also includes presenting the score to a reviewer in real-time. The step 425 also includes a step 430 of generating a performance report associated with the candidate based on the scores generated for the unstructured answers.
[0077] In the preceding specification, the present disclosure and its advantages have been described with reference to specific embodiments. However, it will be apparent to a person of ordinary skill in the art that various modifications and changes can be made, without departing from the scope of the present disclosure, as set forth in the claims below. Accordingly, the specification and figures are to be regarded as illustrative examples of the present disclosure, rather than in restrictive sense. All such possible modifications are intended to be included within the scope of present disclosure.