Claims:WE CLAIM:
1. A method of annotating tokens for natural language processing, the method comprising:
segmenting, by an automatic annotation device, a plurality of corpus based on each of a plurality of instances;
deriving, by the automatic annotation device, a plurality of entities for each of the plurality of instances based on at least one of a machine learning technique and a deep learning technique;
determining, by the automatic annotation device, a word vector for each of the plurality of entities associated with each of the plurality of instances; and
labelling, by the automatic annotation device, a plurality of tokens for each of the plurality of instances, wherein the plurality of tokens associated with the plurality of entities are identified based on a frequency of each of the plurality of entities.

2. The method of claim 1, further comprising:
receiving a plurality of dataset for the plurality of instances from at least one of a website, a portable document format (PDF), a research paper, a journal, or an article; and
determining the plurality of corpus by clubbing each of the plurality of dataset.

3. The method of claim 1, further comprising:
determining a pattern for each of the plurality of tokens based on an alphabetic embedding; and
identifying a matching tokens of the plurality of tokens based on the pattern, wherein the matching tokens is used to identify at least one of a rhythm, a poetry, or a prose.

4. The method of claim 1, wherein labelling the plurality of tokens further comprises:
measuring an accuracy of each of the plurality of tokens for each of the plurality of instances using a confusion matrix; and
re-labelling the plurality of tokens for each of the plurality of instances when the accuracy is below an accuracy threshold, wherein the accuracy threshold can be scaled up or scaled down based on a mistake performed by a neural network.

5. The method of claim 4, further comprising training the neural network based on the plurality of tokens when the accuracy of each of the plurality of token is above the accuracy threshold.

6. The method of claim 5, further comprising:
determining an accuracy value of each of the plurality of entities based on n-fold cross validation technique; and
re-labelling each of the plurality of tokens when the accuracy value is not in a predefined range.

7. A system for annotating tokens for natural language processing, the system comprising:
an automatic annotation device comprising at least one processor and a computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
segmenting a plurality of corpus based on each of a plurality of instances;
deriving a plurality of entities for each of the plurality of instances based on at least one of a machine learning technique and a deep learning technique;
determining a word vector for each of the plurality of entities associated with each of the plurality of instances; and
labelling a plurality of tokens for each of the plurality of instances, wherein the plurality of tokens associated with the plurality of entities are identified based on a frequency of each of the plurality of entities.

8. The system of claim 7, wherein the operations further comprise:
receiving a plurality of dataset for the plurality of instances from at least one of a website, a portable document format (PDF), a research paper, a journal, or an article; and
determining the plurality of corpus by clubbing each of the plurality of dataset.

9. The system of claim 7, wherein the operations further comprise:
determining a pattern for each of the plurality of tokens based on an alphabetic embedding; and
identifying a matching tokens of the plurality of tokens based on the pattern, wherein the matching tokens is used to identify at least one of a rhythm, a poetry, or a prose.

10. The system of claim 7, wherein labelling the plurality of tokens further comprises:
measuring an accuracy of each of the plurality of tokens for each of the plurality of instances using a confusion matrix; and
re-labelling the plurality of tokens for each of the plurality of instances when the accuracy is below an accuracy threshold, wherein the accuracy threshold can be scaled up or scaled down based on a mistake performed by a neural network.

11. The system of claim 10, wherein the operations further comprise training the neural network based on the plurality of tokens when the accuracy of each of the plurality of token is above the accuracy threshold.

12. The system of claim 11, wherein the operations further comprise:
determining an accuracy value of each of the plurality of entities based on n-fold cross validation technique; and
re-labelling each of the plurality of tokens when the accuracy value is not in a predefined range.

Dated 12th day of March, 2019

R Ramya Rao
Of K&S Partners
Agent for the Applicant
IN/PA-1607
, Description:TECHNICAL FIELD
[001] This disclosure relates generally to natural language processing (NLP), and more particularly to method and system for annotating tokens for NLP applications.