DESC:FIELD OF THE INVENTION
The present invention relates to an automated system for identifying an optimal set for text labelling.
More particularly, the present invention relates to an active learning based system automated by reinforcement learning which automatically extracts the most informative and a minimal sample from a large unlabelled dataset for any sequence labelling problem in text applications.

BACKGROUND OF THE INVENTION
In machine learning, sequence labeling is a type of pattern recognition task that involves the algorithmic assignment of a categorical label to each member of a sequence of observed values. A common example of a sequence labeling task is part of speech tagging, which seeks to assign a part of speech to each word in an input sentence or document. Sequence labeling can be treated as a set of independent classification tasks, one per member of the sequence. However, accuracy is generally improved by making the optimal label for a given element dependent on the choices of nearby elements, using special algorithms to choose the globally best set of labels for the entire sequence at once.

The burgeoning demand for labelled data has seen the evolvement of data annotation industry as a billion dollar business. It has witnessed the growth of third-party companies that employ highly trained data labelers and develop advanced annotation tools for professional labelling services.

One of the most popular strategies used in the existing state of art is uncertainty based sampling which chooses instances with high uncertainty. The strategy was adaptable for a wide range of classifiers but is not always the most relevant manner of selecting data. The rest of the unlabeled space which is far away from the decision boundaries is always ignored. Another drawback is that the model often gets misguided by outliers.

Another popular active learning framework in the existing state of art is to query the instance that would impart the greatest change to the model if its label was known. The idea behind the framework is that it prefers instances that are likely to most influence the model, regardless of its label. Since discriminative probabilistic models are usually trained using gradient based optimization, the expected model change can be measured by the length of the training gradient. This approach is effective but computationally expensive if feature space is large and the number of labels is also high. Most of the experimented query strategies attempt exploitation, but strategic exploration is also required.

Supervised sequence labelling is a vital area of machine learning, encompassing tasks such as speech, handwriting and gesture recognition, apart-of-speech tagging etc. The supervised models are highly preferred as they are based on ground truth where we have prior knowledge of what the output values for our samples should be. A supervised learning algorithm learns from labelled training data where the goal is to learn a function that, given a sample of data and desired outputs, best approximates the relationship between input and output observable in the data. Supervised models are used for a wide range of applications including text summarization, object identification in images, sentiment classification, emotion recognition, machine translation, image classification models etc.

One of the major challenge imposed in implementing a supervised model is the large amount of training data required. The quality of the labels is also very crucial making Data annotation a cumbersome and time consuming task. Active Learning (AL) provides a solution for this problem. Active Learning is the process in which a highly informative subset of the data is chosen among the entire dataset such that the resulting subset can act as a representative of the original dataset. It is also known as machine learning with a human in the loop because said model actively queries the user whether a particular sample is to be kept in the final dataset or discarded. However, this model highly prone to errors and defeats the purpose of reducing labor cost.

A Reinforcement Learning environment can be used as a substitute for the human in the loop to reduce any possible error and reduce the decision. making faster and more efficient. Reinforcement Learning (RL) is a process in which a sequence of actions is taken so that reward obtained in following that sequence is maximized.

Literature in the existing state of art showcases various approaches and combinations were considered by researchers and practitioners for decision making regarding the informativeness of each instance selected or the stopping criterion. The problem arises in optimizing exploration and exploitation strategy. Although there has been active research in the area of active learning, the problem of expanding to high dimensional data also remains a challenge.
Reference is made to Patent application no. CN111666734B disclosing a sequence labeling method and device comprising the obtaining a sentence to be analyzed, inputting a sentence to be analyzed into the sequence labeling model, and outputting a word segmentation result of the sentence and a labeling result of each word. The sequence labeling model is obtained by training based on reinforcement learning based on the sample sentences and the participles and labeling data corresponding to the sample sentences and can be used for implicitly learning the word segmentation information of the text through the reinforcement learning technology, so that the mismatching of the word segmentation and the target in the Chinese sequence labeling can be relieved.

Another reference is made to Patent application no. CN112541339A disclosing a knowledge extraction method based on a random forest and a sequence labeling model, and particularly relates to an entity relationship joint extraction method based on the random forest and a BILSTM _ CRF. Firstly, acquiring an unstructured text, preprocessing the text and expressing the text in a sentence vectorization mode, then inputting a sentence sequence into a sentence selector to screen out high-quality sentences, inputting the selected sentences into a BILSTM _ CRF sequence labeling model to perform labeling training, and finally performing sentence-level sequence labeling on the input sentences by using the trained model. The invention is based on random forest, effectively extracts knowledge in the unstructured text and forms structured information through the BILSTM _ CRF sequence labeling model, and by adopting the extraction method, the extraction efficiency of the unstructured information is greatly improved, the existing knowledge map resources are enriched, and further, the invention can better serve various intelligent applications.

Another reference is made to “An Efficient Text Labeling Framework Using Active Learning Model", Intelligent Systems, Technologies and Applications, Springer, Singapore, 2020. 141-155 discloses a strategy to select the most informative samples using deep reinforcement learning models such as Actor-Critic. The performance of the sample selection strategy is determined by finding the accuracy of the model after a predefined number of iterations.

Another reference is made to "Active Learning Enhanced Sequence Labeling for Aspect Term Extraction in Review Data", Advanced Computing. Springer, Singapore, 2021. 141-155 discloses using Active Learning to select a highly informative subset of the data that needs to be labeled. The identification of aspect terms is defined as a sequence labelling problem with the help of BiLSTM network and CRF.

However, the uncertainty based sampling in the Active Learning which chooses instances with high uncertainty is not always the most relevant manner of selecting data and the rest of the unlabeled space which is far away from the decision boundaries is often ignored. Further, discriminative probabilistic sequence models such as CRFs are usually trained using gradient based optimization and expected model change can be measured by the length of the training gradient and are computationally expensive if feature space is large and the number of labels is also high. A possible solution for this is by automating the active learning process using Reinforcement Learning.
One of the major challenges posed in implementing a supervised model is the large amount of training data required from which the resulting subset can act as a representative of the original dataset as described in detail below. In the existing state of art this model is highly prone to errors.

OBJECT OF THE INVENTION
In order to obviate the drawbacks in the existing state of the art, the main object of the present invention is to provide an active learning based system automated by reinforcement learning which automatically extracts the most informative and minimal sample from a large unlabelled dataset for any sequence labelling problem in text applications.

Another object of the present invention is to provide a system capable of identifying the optimal i.e., minimal, and most informative sample set from the population that is guaranteed to provide maximum coverage for annotations and training for Machine Learning model creation.

Yet another object of the present invention is to provide a system capable of providing a coverage score that is indicative of the percentage coverage the sample set provides.

Yet another object of the present invention is to provide a system capable of selecting a seed set using business provided domain dictionary and/or gazetteers.

Yet another object of the present invention is to provide a confidence score associated with the coverage that is indicative of the accuracy of coverage prediction associated with the sample selection from the population.

ADVANTAGES OF THE PRESENT INVENTION OVER THE EXISTING STATE OF ART
Accordingly, the present invention reduces the amount of training data to be annotated by the user and reduces cost in model implementation by providing an active learning based system automated by reinforcement learning which automatically extracts the most informative and a minimal sample from a large unlabelled dataset for any sequence labelling problem in text applications. The sampling of the dataset reduces the load where the minimal and most informative is automatically extracted from the unlabelled data.

The present invention uses a Reinforcement Learning environment as a substitute for the human in the loop to reduce any possible error and reduce the decision making faster and more efficient.

The active learning model of the present invention is applied in the medical domain to enhance automatic labelling of text discharge summaries and on aspect term extraction in sentiment analysis, the novelty being the query strategy designed using deep reinforcement learning techniques. An appreciable increase in the classifier performance has been achieved.

The present system aids supervised models, thereby reducing the cost, effort
and time spent on creating a suitable annotated dataset for a machine learning (ML) model. The present system is a generic framework that can be applied on any application framework that operates on textual data.

SUMMARY OF THE INVENTION:
The present invention relates to a system for identifying an optimal set for text labelling. More particularly, the present invention relates to an active learning based system automated by reinforcement learning (RL) which automatically extracts the most informative and a minimal sample from a large unlabelled dataset for any sequence labelling problem in text applications.

The present system is automated through Reinforcement Learning (RL) that is able to work on unlabeled dataset, i.e., an unsupervised sample selection process, which is expected to extract the most informative samples with respect to the sequence labelling task being experimented.

The system of the present invention is capable of identify the optimal i.e. minimal and most informative sample set from the population that is guaranteed to provide maximum coverage for annotations and training for Machine Learning model creation and select a seed set using business provided domain dictionary or gazetteers and provides a confidence score associated with the coverage that is indicative of the accuracy of coverage prediction associated with the sample selection from the population.

The present system is economically significant since it is capable of aiding the supervised models, thereby reducing the cost, effort and time spent on creating a suitable annotated dataset for a machine learning (ML) model. The model is a generic framework that can be applied on any application framework that operates on textual data.

BRIEF DESCRIPTION OF DRAWINGS
Figure 1 depicts Aspect Term Extraction as a sequence labelling task.
Figure 2 depicts a schematic diagram of methodology for Active Learning based sample selection
Figure 3 depicts a flow diagram for Seed Model Generation.
Figure 4 depicts a schematic diagram of Reinforced Agent Training Environment.
Figure 5 depicts a flow diagram for Reinforced Active learning and Sample Extraction phase.
Figure 6 depicts a flow diagram for Enhanced Model generation phase.

DETAILED DESCRIPTION OF THE INVENTION ILLUSTRATIONS AND EXAMPLES
While the invention has been disclosed with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made, and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt to a particular situation or material to the teachings of the invention without departing from its scope.

Throughout the specification and claims, the following terms take the meanings explicitly associated herein unless the context clearly dictates otherwise. The meaning of “a”, “an”, and “the” include plural references. Additionally, a reference to the singular includes a reference to the plural unless otherwise stated or inconsistent with the disclosure herein.

The present invention provides an active learning based system automated by reinforcement learning to develop a model which automatically extracts the most informative and a minimal sample from a large unlabelled dataset for any sequence labelling problem in text applications. An example for a sequence labelling task i.e., Aspect Term Extraction in sentiment analysis is provided in Figure 1.

The Active Learning system of the present invention is capable of working on unlabeled dataset, i.e., an unsupervised sample selection process, which is expected to extract the most informative samples with respect to the sequence labelling task being experimented.

The present system comprises of a three phased architecture as depicted in Figure 2 representing the overview of the Active Learning enhanced Sequence Labeling process.

The three phased architecture of the present invention comprises of the following phases:
? Seed Model Generation for Sequential Text Labelling
? Reinforced Active Learning for Sample Extraction
? Enhanced Model Generation and Evaluation

Seed Model Generation for Sequential Text Labelling phase
The Active Learning model (AL) of the present invention is completely unsupervised. Therefore, in order to provide a starting point for the AL model, a base model trained on a seed set for the sequence labeling task is generated and the weights of the hidden layer in the final iteration are used to initialize the AL model. The idea is an implementation of transfer learning where instead of starting the learning process from scratch, it start from patterns that have been learned when solving a different problem that is identical in task or genre of datasets. The seed set is a set of minimal instances, yet representative of the semantic and structural diversity of the data population.

A sequence processing model such as Bi-LSTM based Deep Neural Network is used as the base model for text sequential labelling. In most of the text labelling scenarios, the label of a token is likely to depend on its prior or posterior sequences. The Bidirectional Long Short-Term Memory (Bi-LSTM) network is an apt choice for a sequential text labelling model as it retains information on past and future instances as the network runs from both sides. Therefore, the BiLSTM network coupled with a Conditional Random Field (CRF) annotator used as the seed or base model is trained using a minimal labelled seed set depicted as Input 1 in Figure 2.

The seed set for training is chosen using a seeded topic modelling approach such as Latent Dirichlet Allocation (LDA) which identifies the categories in the domain using domain dictionaries or gazetteers from the domain and ensures a uniform distribution across different categories. The decision regarding categories is based on the application task in hand. For example, if the sequence labelling task is Aspect Term Extraction, then the categories would correspond to the aspect categories mapped to aspect terms.

After the application of the LDA model, instances which have the highest probability of belonging to each category are chosen to the seed set. The seed instances thus selected using the unsupervised LDA approach would ensure a minimal coverage across all categories in the dataset rather than a random sampling. The input to the Bi-LSTM layer is given in the form of word embeddings of the seed data based on contextual word embeddings such as BERT model. The output generated from this layer is processed using Conditional Random Field to obtain the final labels or tags.

The training is done over iterations until there is an appreciable performance on the sequence labelling task. The final state of this Bi-LSTM network is saved and fed to the active learning agent as its initial state as depicted in Figure 2. The final state represents the weight matrix of the Bi-LSTM network obtained at the end of the last iteration. The sequence of steps that this module navigates through is represented using the flow diagram as depicted in Figure 3.

Reinforced Active Learning for Sample Extraction
The seed model input to the active learning model serves as a starting point to orient itself towards the specific text labelling application. Unlabelled instances are input to the AL model which is expected to extract the most informative instances for the sequence labelling task. The active learning strategy implemented in the present invention is a pool based or multi instance learning strategy as the decision regarding the informativeness of the instance with respect to the labelling task is not an individual decision but a collective division of all the instances in the bag at that point of time.

The active learning training of the present system has been designed as a Reinforcement Learning (RL) problem in which the decision-making is based on first order Markov Model assumption. This implies that during the process of adding instances to the bag, the Markov Decision Process can decide whether the next instance is to be added or discarded based on the current state of the bag.

The decision is based on a rewards system where the RL model tries to maximize this reward, i.e., if a new instance is to pick into the bag, there should be an increase in the reward after adding the instance into the bag. If there is a decrease in the reward, the corresponding instance is then discarded from the bag. Hence, the final set of instances in the bag will be the most suitable set of samples for training the sequence labelling model since the active learning model is initialized with the internal state of a sequence labelling model.
The Actor-Critic model which combines the advantages of both value based and policy based algorithms is the chosen model for the learning environment. The actor critic agent is implemented using a Self-Attention RNN model. The model is represented as a tuple (S, A, R) where S represents the current state of the agent, A the possible set of actions and R the calculated reward.

Figure 4 depicts the Actor-Critic model for agent training. The word embeddings of the unlabelled instances are given as the input to the agent. Two different outputs are extracted after each step.
The first output is in the form of [1 x 3] vector, corresponding to the three different actions that could be taken by the agent and each value in the vector is the probability of the action to be taken.

The three different actions include:
0-> add instance to the bag;
1->discard instance
2-> end episode (replace the bag of instances)

The second output is the state value derived from the action probabilities through linear compression. The next action to be taken is calculated from the probabilities and the reward is calculated based on the action and the current state.

The reward is dependent on three parameters:
- the average semantic distance between each instance in the bag,
- the parse tree distance based on tree kernel, and
- the size of the bag.

The semantic distance is calculated using the Word Mover Distance method and the parse tree distance which is a measure of structural dissimilarity is calculated using tree kernel concept.

The reward takes a positive effect from the distances, with the assumption that a set of instances that are far away from each other will probably contain information that are exclusive to that instance. The size of the bag negatively affects the reward because, at the end the aim is to minimize the training cost by reducing the required amount of training data.

After each step, the loss is calculated based on the state value, action and reward of that particular record and the loss is back-propagated to the Self-Attention RNN. The process is repeated for a fixed number of epochs over the given dataset to improve the agent. Thus, the agent learns over a period to include only significant documents with respect to the sequence labelling application. Once the agent is trained, it is used to extract a list of most informative instances from an unlabelled dataset input to the model. The sample set is passed on to the next phase for enhanced mode generation. The flow diagrammatic representation of the Reinforced Active Learning for Sample Extraction is represented in Figure 5.
Enhanced model Generation
In this phase, the sample set chosen by the active learning agent is added to the initial seed set and the sequence labeling base model is retrained. The training is continued iteratively until an acceptable loss value is reached. The model is then evaluated on a test dataset. The model is also compared with a model trained on randomly selected samples instead of Active learning guided sampling.

This reinforced active learning model can be implemented as a continuously learning model where the results of the enhanced model on the test data form new labelled instances that are appended to the seed set and the active learning model continues to extract informative instances from newly input unlabeled data. After a few iterations the seed model itself can be used to predict the labels of the sample selected by the active learning model so that it only needs a label verification and/or correction rather than manual labelling and therefore, reduces the human intervention over iterations.

The active learning enhanced model of the present system performs better than the model trained on random samples and almost in par with the model trained on the entire data input to the active learning model for sample extraction. The following findings ascertain the strength of the active learning model of the present system.
The model strength is expressed in terms of two metrics:
• a coverage score that is indicative of the % coverage the sample set would provide in comparison to the population size (the set from which the sample set is extracted)

• a confidence score that is indicative of the accuracy of coverage prediction associated with the sample selection from the population. This would be indicative of the minimum performed that would be showcased by the enhanced model in terms of F-measure which a weighted measure is ensuring a balanced performance.

The enhanced model generation and the iterative implementation is depicted in the flow diagram of Figure 6.
,CLAIMS:1. An automated system for identifying an optimal set for text labelling, wherein said system is based on an active learning model automated by Reinforcement Learning for identification and extraction of an optimal sample from a large unlabelled dataset for any sequence labelling problem in text applications

said system architecture comprising of:
? Seed Model Generation for Sequential Text Labelling phase
? Reinforced Active Learning for Sample Extraction phase
? Enhanced Model Generation and Evaluation phase

characterized in that,
said system being capable of
- identifying the optimal sample set from a population that is guaranteed to provide maximum coverage for annotations and training for Machine Learning model creation,
- providing a coverage score that is indicative of percentage coverage the sample set provides,
- selecting a seed set using a business provided domain dictionary and/or gazetteers,
- providing a confidence score associated with the coverage that is indicative of the accuracy of coverage predict ion associated with the sample selection from the population,
in order to provide a Reinforcement Learning environment which can be used as a substitute for the human in the loop to reduce any possible error and reduce the decision making time.

2. The automated system for identifying an optimal set for text labelling as claimed in claim 1 wherein said optimal sample set is minimal and most informative sample set from the population that is guaranteed to provide maximum coverage for annotations and training for Machine Learning model creation.

3. The automated system for identifying an optimal set for text labelling as claimed in claim 1 wherein said Seed Model Generation for Sequential Text Labelling phase comprises the steps of:
? providing a starting point for the Active Learning model,
? generating a base model trained on a seed set for the sequence labeling task
? using the weights of the hidden layer in the final iteration to initialize said active learning model
? using a sequence processing model such as Bidirectional Long Short-Term Memory (Bi-LSTM) for the base model for text sequential labelling
? training said sequence processing model such as BiLSTM network coupled with a Conditional Random Field (CRF) annotator used as the seed or base model using a minimal labelled seed set,
? choosing seed set for training using a seeded topic modelling approach such as Latent Dirichlet Allocation (LDA) which identifies categories in a domain using domain dictionaries or gazetteers from the domain and ensures a uniform distribution across different categories,
? choosing seed instances having the highest probability of belonging to each category,
? giving input to the sequence processing model in the form of word embeddings of the seed data based on contextual word embeddings such as BERT model,
? generating output from said sequence processing model, and
? processing said output using said Conditional Random Field (CRF) to obtain the final labels or tags

4. The automated system for identifying an optimal set for text labelling as claimed in claim 2 wherein said decision regarding categories is based on an application task in hand.

5. The automated system for identifying an optimal set for text labelling as claimed in claim 2 wherein said seed instances are selected using a topic modelling approach such as unsupervised Latent Dirichlet Allocation approach ensuring a minimal coverage across all categories in the dataset rather than a random sampling.

6. The automated system for identifying an optimal set for text labelling as claimed in claim 2 wherein a final state represents weight matrix of said sequence labelling network obtained at the end of last iteration.

7. The automated system for identifying an optimal set for text labelling as claimed in claim 6 wherein said final state of the sequence labelling network is saved and fed to an active learning agent as its initial state.

8. The automated system for identifying an optimal set for text labelling as claimed in claim 1 wherein in said Reinforced Active Learning for Sample Extraction phase, the active learning is a pool based or multi instance learning and designed as a Reinforcement Learning model wherein the decision-making is based on first order Markov Model assumption.

9. The automated system for identifying an optimal set for text labelling as claimed in claim 8 wherein said decision regarding the informativeness of the instance with respect to the labelling task is not an individual decision but a collective division of all the instances in a bag at that point of time.

10. The automated system for identifying an optimal set for text labelling as claimed in claim 8 wherein unlabelled instances are input to the active learning model which extracts the most informative instances for the sequence labelling task.

11. The automated system for identifying an optimal set for text labelling as claimed in claim 8 wherein the Markov decision process is used to decide whether a next instance is to be added or discarded based on current state of the bag.

12. The automated system for identifying an optimal set for text labelling as claimed in claim 11 wherein said decision is based on a rewards system wherein the Reinforcement Learning model tries to maximize reward.

13. The automated system for identifying an optimal set for text labelling as claimed in claim 11 wherein a final set of said instances in the bag are the most suitable set of samples for training the sequence labelling model.

14. The automated system for identifying an optimal set for text labelling as claimed in claim 8 wherein Reinforcement Learning models such as Actor-Critic model is the chosen model for learning environment.

15. The automated system for identifying an optimal set for text labelling as claimed in claim 14 wherein said Actor-Critic model combines the advantages of both value based and policy based algorithms.

16. The automated system for identifying an optimal set for text labelling as claimed in claim 14 wherein an actor critic agent is implemented using a Self-Attention RNN model.

17. The automated system for identifying an optimal set for text labelling as claimed in claim 16 wherein said RNN model is represented as a tuple (S, A, R) wherein S represents current state of the agent, A represents possible set of actions and R represents calculated reward.

18. The automated system for identifying an optimal set for text labelling as claimed in claim 16 wherein word embeddings of the unlabelled instances are given as the input to the agent and two different outputs are extracted after each step.

19. The automated system for identifying an optimal set for text labelling as claimed in claim 18 wherein the first output is in the form of [1 x 3] vector, where ‘n’ corresponds to actions that could be taken by the agent and each value in the vector is a probability of an action to be taken.

20. The automated system for identifying an optimal set for text labelling as claimed in claim 18 wherein the second output is a state value derived from said action probabilities through linear compression.

21. The automated system for identifying an optimal set for text labelling as claimed in claim 19 wherein next action is calculated from said probabilities and a reward is calculated based on said action and current state
characterized in that,
said reward is dependent on three parameters:
- average semantic distance between each instance in the bag,
- parse tree distance based on tree kernel, and
- size of the bag.

22. The automated system for identifying an optimal set for text labelling as claimed in claim 21 wherein the semantic distance is calculated using Word Mover Distance method and the parse tree distance is calculated using tree kernel concept.

23. The automated system for identifying an optimal set for text labelling as claimed in claim 21 wherein said reward takes a positive effect from the distances, characterized in that, the set of instances that are far away from each other contains information exclusive to that instance.

24. The automated system for identifying an optimal set for text labelling as claimed in claim 21 wherein the size of the bag negatively affects the reward with an aim is to minimize training cost by reducing required amount of training data.

25. The automated system for identifying an optimal set for text labelling as claimed in claims 8 to 24, wherein said Reinforced Active Learning phase further comprises the steps of
? calculating the loss based on the state value, action and reward of a particular record after each step
? back-propagating the loss to the Self-Attention RNN,
? repeating the process for a fixed number of epochs over the given dataset to improve the agent,
? extracting a list of most informative instances from the unlabelled dataset input to the model, and
? passing the sample set on to the next phase for enhanced mode generation.

26. The automated system for identifying an optimal set for text labelling as claimed in claims 1 wherein said Enhanced mode generation phrases comprises the steps of
? choosing a sample set chosen by the active learning agent
? added said sample set to the initial seed set
? restraining the sequence labeling base model
? continuing the training iteratively until an acceptable loss value is reached.
? evaluating the model is on a test dataset
? comparing the model with a model trained on randomly selected samples instead of Active learning guided sampling

characterized in that
- said reinforced active learning model can be implemented as a continuously learning model where the results of the enhanced model on the test data form new labelled instances that are appended to the seed set and the active learning model continues to extract informative instances from newly input unlabeled data.
- after a few iterations the seed model can be used to predict the labels of the sample selected by the active learning model so that it only needs a label verification and/or correction rather than manual labelling thereby reducing human intervention over iterations.

27. The automated system for identifying an optimal set for text labelling as claimed in claims 1 wherein said active learning enhanced model of performs better than the model trained on random samples and at par with a model trained on the entire data input to the active learning model for sample extraction.

characterized in that, the strength of said model is expressed in terms of two metrics
- a coverage score that is indicative of the percentage coverage the sample set provides in comparison to the population size
- a confidence score that is indicative of the accuracy of coverage prediction associated with the sample selection from a population