Description:METHOD AND SYSTEM FOR GENERATING AND EVALUATING PERSONALIZED MARKETING CONTENT
DESCRIPTION
TECHNICAL FIELD
This disclosure generally relates to personalized marketing, and more particularly to method and system for generating and evaluating personalized marketing content.
BACKGROUND
Marketing content campaigns (for example, marketing email campaigns, social media campaigns, etc.) are a critical component of modern marketing strategies. Personalized and targeted content campaigns may directly contribute to an increase in revenue for an organization. For example, a Forrester study claims that personalized, targeted email campaigns may increase revenue by as much as 15%. However, a process of creating the personalized marketing content requires time and cost (in some cases, as high as about 3 days and 300 USD). Upon creating the personalized marketing content, existing techniques lack an objective way to evaluate or predict the performance of the created personalized marketing content.
The existing techniques for assessing the performance of the personalized marketing content, such as, but not limited to, an A/B testing, are inherently time-consuming, resource-intensive, and may provide delayed output. The A/B testing requires sending multiple content variants to different customer segments and then analyzing the results, which may take days or even weeks. Additionally, the targeted customer segments may not even be available for A/B testing. Thus, the iterative process of A/B testing may result in delayed campaign launches and may hinder rapid adaptation to changing market conditions. Additionally, the existing techniques rely solely on post-deployment metrics which provide limited insights into underlying reasons for campaign success or failure.
There is, therefore, a need for a data-driven solution to generate, evaluate, and improve personalized marketing content.
SUMMARY
In one embodiment, a method for generating and evaluating personalized marketing content is disclosed. In one example, the method may include receiving, via a user interface, content information and details of a target customer segment. The method may further include generating, via a first Large Language Model (LLM), a set of content variants in response to a content generation prompt. The content generation prompt includes the content information, the details of the target customer segment, and a predefined set of content generation instructions. The method may further include determining, via the first LLM, a weighted score for each of the set of content variants based on a set of evaluation parameters and a corresponding set of weights. The method may further include predicting, via a second LLM, a persona response to each of the set of content variants based on a user-defined persona through a response prediction prompt. The persona response includes a set of response parameters. The user-defined persona corresponds to the target customer segment. The response prediction prompt includes the set of set of content variants, the user-defined persona, and a predefined set of response prediction instructions. The method may further include rendering, via the user interface, the set of content variants, and the weighted score and the persona response corresponding to each of the set of content variants.
In another embodiment, a system for generating and evaluating personalized marketing content is disclosed. In one example, the system may include a processor, and a memory communicatively coupled to the processor. The memory may store processor-executable instructions, which, on execution, may cause the processor to receive, via a user interface, content information and details of a target customer segment. The stored processor-executable instructions, on execution, may further cause the processor to generate, via a first Large Language Model (LLM), a set of content variants in response to a content generation prompt. The content generation prompt includes the content information, the details of the target customer segment, and a predefined set of content generation instructions. The processor-executable instructions, on execution, may further cause the processor to determine, via the first LLM, a weighted score for each of the set of content variants based on a set of evaluation parameters and a corresponding set of weights. The processor-executable instructions, on execution, may further cause the processor to predict, via a second LLM, a persona response to each of the set of content variants based on a user-defined persona through a response prediction prompt. The persona response includes a set of response parameters. The user-defined persona corresponds to the target customer segment. The response prediction prompt includes the set of content variants, the user-defined persona, and a predefined set of response prediction instructions. The processor-executable instructions, on execution, may further cause the processor to render, via the user interface, the set of content variants, and the weighted score and the persona response corresponding to each of the set of content variants.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and, together with the description, serve to explain the disclosed principles.
FIG. 1 is a block diagram of an exemplary system for generating and evaluating personalized marketing content, in accordance with some embodiments of the present disclosure.
FIG. 2 is a functional block diagram of various modules within a memory of a computing device configured to generate and evaluate personalized marketing content, in accordance with some embodiments of the present disclosure.
FIG. 3 is a flow diagram of an exemplary method for generating and evaluating personalized marketing content, in accordance with some embodiments of the present disclosure.
FIG. 4 is a flow diagram of an exemplary method for predicting the persona response to content variants, in accordance with some embodiments of the present disclosure.
FIG. 5 is a flow diagram of an exemplary method for fine-tuning of a first LLM and a second LLM, in accordance with some embodiments of the present disclosure.
FIG. 6A is a table representing experimental results for IV computation of readability parameter based on an exemplary primary dataset, in accordance with an embodiment of the present disclosure.
FIGS. 6B and 6C are tables representing experimental results for IV computations of readability parameter for individual customer segments within the exemplary primary dataset, in accordance with an embodiment of the present disclosure.
FIG. 7 is a schematic diagram of a detailed exemplary process for generating and evaluating personalized emails, in accordance with some embodiments of the present disclosure.
FIG. 8 is a layout of an exemplary generated email, in accordance with an embodiment of the disclosure.
FIG. 9 is a block diagram of an exemplary computer system for implementing embodiments consistent with the present disclosure.

DETAILED DESCRIPTION
Exemplary embodiments are described with reference to the accompanying drawings. Wherever convenient, the same reference numbers are used throughout the drawings to refer to the same or like parts. While examples and features of disclosed principles are described herein, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. It is intended that the following detailed description be considered as exemplary only, with the true scope and spirit being indicated by the following claims.
Referring now to FIG. 1, an exemplary system 100 for generating and evaluating personalized marketing content is illustrated, in accordance with some embodiments of the present disclosure. The system 100 may include a computing device 102 (for example, a server, a desktop, a laptop, a notebook, a netbook, a tablet, a smartphone, a mobile phone, or any other computing device), in accordance with some embodiments of the present disclosure. The computing device 102 may generate a set of content variants through a first Large Language Model (LLM). Further, the computing device 102 may select an optimal content variant from the set of content variants based on an optimal combination of a weighted score-based evaluation and a persona response-based evaluation.
As will be described in greater detail in conjunction with FIGS. 2-9, the computing device 102 may receive, via a user interface, content information and details of a target customer segment. The computing device 102 may further generate, via a first LLM, a set of content variants in response to a content generation prompt. The content generation prompt includes the content information, the details of the target customer segment, and a predefined set of content generation instructions. The computing device 102 may further determine, via the first LLM, a weighted score for each of the set of content variants based on a set of evaluation parameters and a corresponding set of weights. The computing device 102 may further predict, via a second LLM, a persona response to each of the set of content variants based on a user-defined persona through a response prediction prompt. The persona response may include a set of response parameters. The user-defined persona corresponds to the target customer segment. The response prediction prompt includes the set of content variants, the user-defined persona, and a predefined set of response prediction instructions. The computing device 102 may further render, via the user interface, the set of content variants, and the weighted score and the persona response correspond to each of the set of content variants.
In some embodiments, the computing device 102 may include one or more processors 104 and a memory 106. Further, the memory 106 may store instructions that, when executed by the one or more processors 104, cause the one or more processors 104 to generate and evaluate personalized marketing content, in accordance with some aspects of the present disclosure. The memory 106 may also store various data (for example, a content information, a predefined set of content generation instructions, a set of evaluation parameters, a corresponding set of weights, a set of personas, a persona response, a set of response parameters, a predefined set of response prediction instructions, an information value (IV), a set of customer segments, a real-world acceptance rate, a predicted acceptance rate, and the like) that may be captured, processed, and/or required by the system 100.
The system 100 may further include a display 108. The system 100 may interact with a user via a user interface 110 accessible via the display 108. The system 100 may also include one or more external devices 112. In some embodiments, the computing device 102 may interact with the one or more external devices 112 over a communication network 114 for sending or receiving various data. The external devices 112 may include, but may not be limited to, a remote server, a digital device, or another computing system.
Referring now to FIG. 2, a functional block diagram 200 of various modules within the memory 106 of the computing device 102 configured to generate and evaluate personalized marketing content is illustrated, in accordance with some embodiments of the present disclosure. The memory 106 may include a prompt creation module 202, a content variants generation module 204, a score determination module 206, an information value calculation module 208, a response prediction module 210, and a fine-tuning module 212. The content variants generation module 204 may include a first LLM 214. The response prediction module 210 may include a second LLM 216.
The prompt creation module 202 may receive content information 218 and customer details 220 (i.e., details of the target customer) from a user through a Graphical User Interface (GUI) (for example, the user interface 110). The user may be any individual that is drafting content. In an exemplary scenario, the user may be a marketing professional drafting marketing content (for example, promotional email, promotional blog, a newsletter, sales page, or the like). In a preferred embodiment, the marketing content may include textual content and may optionally include image content. The user may or may not be associated with an enterprise.
Upon receiving, the prompt creation module 202 may create the content generation prompt using the content information 218, the customer details 220, and a predefined set of content generation instructions. The content information 218 may include, but may not be limited to, product details, marketing offers associated with the product (such as promotional discounts or incentives), visual elements (i.e., specifications for images and graphics), customizable sections (i.e., tailored parts of the email, such as introductions and footers), image descriptions (i.e., details to align images with the marketing message), and the like. The customer details 220 may include details (such as associated demographic details and preferences) associated with a target customer segment. The target customer segment may be determined by the user or any other individual (or team of individuals) of the enterprise. In an embodiment, the target customer segment may be determined based on identification of a target demographic division, such as age group, gender, income level, geographic region, customer type (for example, new customer or returning customer), and the like. In another embodiment, the target customer segment may be determined based on identification of a target behavioral segment. Behavioral segments may be defined based on previous interactions, such as response rates (for example frequent responders or occasional responders).
Further, the prompt creation module 202 may input the content generation prompt to the first LLM 214. The content variants generation module 204 may then generate, via the first LLM 214, a set of content variants 222 in response to the content generation prompt. In other words, the first LLM 214 may craft ‘n’ number of tailored and engaging marketing content variants for target customer segments (where ‘n’ may be user-defined in the instructions of the content generation prompt). In one embodiment, the first LLM 214 may generate content variants with Hyper Text Markup Language (HTML) and text. This approach underscores the efficacy of AI-driven text generation in optimizing marketing content personalization and enhancing audience engagement.
The content variants generation module 204 may send the set of content variants 222 to the score determination module 206. Further, for each of the set of content variants 222, the score determination module 206 may determine a score corresponding to the content variant for each of a set of evaluation parameters. In an embodiment, the set of evaluation parameters may include, but may not be limited to, a readability parameter, a non-spam parameter, a sentiment parameter, an HTML parameter, an aesthetic parameter, a content engagement quality parameter, and the like.
The readability parameter is a composite metric that may evaluate an ease of comprehension of a given text in the content variant. The readability parameter integrates a Flesch Reading Ease with a normalized Flesch-Kincaid Grade Level and a Gunning Fog Index, providing a comprehensive assessment of text readability. The Flesch Reading Ease measures how easy the text is to read, with higher scores indicating simpler language. The Flesch-Kincaid Grade level reflects the grade level required for comprehension, providing a standardized measure of text complexity. The Gunning Fog Index measures text complexity based on sentence length and word difficulty, helping to identify areas that may be challenging for readers. Range of Flesch Reading Ease is from 0 to 100, range of Flesch-Kincaid Grade is from 0 to 20, and range of Gunning Fog is from 0 to 20. The readability parameter offers a balanced assessment, considering both comprehension ease and educational level required for understanding. Further, the readability parameter ensures that generated content is easily understandable by the target customer segment, text is suitable for the target audience education level and complexity of language is identified and adjusted for clarity in order to make the content variant accessible to all readers.
The non-spam parameter is tailored to protect the content variant from being classified as spam by accurately classifying text of the content variant. For determining the non-spam parameter, the system leverages a machine learning (ML) model, such as but not limited to a CatBoost Classifier. The CatBoost Classifier is a robust, high performance ML model which is trained on a diverse, publicly available dataset of content variant texts (such as email texts) categorized as spam and non-spam. The ML model is coupled with a Term Frequency- Inverse Document Frequency (TF-IDF) vectorized to convert text content into features that the ML model can analyze. In other words, the TF-IDF converts the input text into numerical features based on the term frequency-inverse document frequency, enabling the machine learning model to interpret the text.
A predictive model may assess the probability of the content variant text being detected as non-spam and may output this as a percentage, providing a clear and actionable result. For example, the non-spam parameter may be 0% when the text is highly likely to be spam and 100% when the text is highly likely to be non-spam. The non-spam parameter is used for email filtering, content moderation, and other applications where identifying spam is necessary to ensure a safe and productive environment. Further, the non-spam parameter ensures a clear and quantifiable probability of non-spam content to facilitate informed decision-making.
The sentiment parameter is used to identify an emotional tone of textual content. A supervised ML model from the TextBlob library is used to analyze the sentiment. In other words, the sentiment parameter may offer insights into content reception by different customer segments. An actual range obtained from the supervised ML model is from -1 to 1 but this range is normalized to a scale of 0 to 100 to ensure consistency and clarity in interpretation. For example, the sentiment parameter may be 0 when the sentiment is entirely negative, 50 when there is a neutral sentiment, and 100 when the sentiment is entirely positive.
The HTML parameter evaluates the quality of HTML in the content variant by focusing on defects that may lead to compromise in accessibility, search engine optimization, and maintainability. The HTML parameter may be determined by using BeautifulSoup for HTML parsing and then employ Cascading Style Sheets (CSS) selectors to identify and assess defects. The HTML parameter may be determined based on the number of defects in the content variant. For example, the HTML parameter is 100 if minimal defects (for example, 0-4) are detected in the text and 0 if significant defects (for example, 20 or more) are detected in the text.
By way of an example, the defects may be, but are not limited to, absence of alt attributes for images, use of inline styles instead of external CSS, employment of deprecated HTML tags, insecure attributes in links (e.g., missing rel="noopener noreferrer" for external links), missing or empty meta descriptions, incorrect use of aria-hidden attributes, and the like.
Upon determination, the defects are removed successfully from the content variant (such as the email) which makes the content variant well-structured and optimized. The optimized content variant may enhance the user experience and may improve the search engine visibility.
The aesthetic parameter may quantify the aesthetic appeal of images in order to align an image with the high marketing standards so that the content captures attention and effectively communicates the desired message. The aesthetic parameter is calculated using python libraries such as NumPy, PyTorch, Python Imaging Library (PIL) and the like. NumPy handles efficient array operations including cosine similarity calculations, PyTorch loads a pre-trained model for image feature extraction, while PIL processes the image and handling tasks such as loading from files or Uniform Resource Locator (URL). Further, an advanced AI model Contrastive Language-Image Pretraining (CLIP) is used to recognize and extract features from images. The CLIP model converts the image to a vector, which is a numerical representation that captures the essential visual elements of the image.
The aesthetic parameter may be determined based on a cosine similarity. Cosine similarity measures an angle between the two vectors to assess the alignment of the vector with a precomputed aesthetics vectors. Cosine similarity may be computed for both positive (or ideal) and negative (or undesirable) aesthetic qualities. The Aesthetic parameter is calculated by combining a positive cosine similarity and a negative cosine similarity. The positive cosine similarity is weighted positively and the negative cosine similarity is weighted negatively. An actual range of the aesthetic parameter is from -1 to 1 but is normalized to a scale of 0 to 100. In an embodiment, the aesthetic parameter 0 may indicate the least appealing score or mostly aligned with negative qualities and the aesthetic parameter 100 may indicate the most appealing score or mostly aligned with positive qualities.
The engagement quality parameter (or an email engagement quality parameter in case of emails) may be computed using three attributes: Entropy, Content Length and Call-to-Action (CTA). By combining the three attributes, it becomes easier for the content creators or drafters to identify areas for improvement and optimize online marketing campaigns (such as email campaigns) for maximum impact and engagement. Several python libraries may be used to compute the three attribute, such as, but not limited to, NumPy, BeautifulSoup, and OpenAI GPT-4 Turbo. NumPy may be used for high-performance numerical computations, BeautifulSoup may be used for parsing and processing HTML content, and Turbo may be integrated for advanced semantic analysis and evaluation of CTA elements. Additionally, some probabilistic techniques may be used such as but are not limited to Bidirectional Encoder Representations from Transformers (BERT) embeddings to compute entropy and assess the richness of information within the content.
The entropy may be calculated to ensure the balance between rich data and static or constant data. The entropy may be evaluated based on token embeddings obtained with the help of BERT. In other words, text is tokenized using BERT. The probabilities are derived via softmax. The entropy may be calculated using the equation (1).
H(P)=∑_(i=1)^N▒Pi⋅log⁡(Pⅈ) (1)
In the equation (1), pi represents the probability of the ith token from BERT embeddings. The result of the entropy is a normalized score between 0 and 100, providing insight into the information density of the content.
Additionally, a content length score may be calculated by comparing an actual content length to a predefined maximum limit (for example, 300 characters), and the score is normalized proportionally, resulting in a value between 0 and 100. The content length score determines whether length of the content aligns with optimal engagement thresholds. The content length score is calculated using equation (2).
Length Score=min((Content Length)/(Max Length) ,1.0)∙100 (2)
Further, a CTA score may be calculated for the clarity, placement, and effectiveness of CTAs. The CTA score is evaluated by embedding HTML content into a query for analysis by GPT-4 turbo, producing a structured JSON output that includes an overall CTA score and a detailed criteria specific evaluations.
Upon evaluating scores for all three attributes, a consolidated engagement quality score for the email is determined for evaluation of the engagement quality parameter which provides an overall assessment of the email quality. Using a weighted formula, the email engagement quality parameter may be calculated as, for example, 40% of the Entropy score, 20% of the Length score, and 40% of the CTA score. The resulting value is normalized to a range of 0 to 100. The consolidated email engagement quality score may be calculated using the equation (3).
Final score=(0.4∙Entropy Score)+(0.2∙Length Score)+(0.4∙CTA Score) (3)
The final score (i.e., the consolidated engagement quality score) may provide the email engagement quality parameter that helps the content creators or drafters to identify weak attributes in email design, to optimize engagement by adjusting content and CTA, to compare email performance across campaigns. The structured evaluation ensures that email content is both impactful and strategically optimized for maximum engagement.
Additionally, a check may be performed to validate grammatical structure and spelling accuracy of HTML content. The check ensures an error-free and professionally presented final email. The check may be performed using a language model such as but is not limited to a spaCy language model (or en_core_web_sm) to analyze grammatical structures within the text, focusing on specific dependencies such as passive subjects and verb-subject relationships. Specific issues, such as absence of a subject in passive constructions, will result in a failed check. Further, a SpellChecker library is used to identify and correct spelling errors. In other words, the text may be checked for punctuation and for misspelled words using the SpellChecker library. Any detected misspellings may lead to a failed check. A failed check in both cases refers to a requirement of changes in the content. The content may also be cleaned of punctuation before spell checking to ensure accuracy. This dual approach guarantees a thorough examination of the text for both grammar and spelling errors.
In some embodiments, the score determination module 206 may normalize the score for each of the set of evaluation parameters to a scale of 0 to 100, where a higher score is a favorable indication for the text (i.e., marketing content).
Further, for each evaluation parameter of the set of evaluation parameters, the IV calculation module 208 may calculate an IV corresponding to the evaluation parameter based on a number of responders and a number of non-responders. The number of responders and the number of non-responders are obtained from historical data of the set of response parameters.
As will be appreciated, IV is a statistical tool that measures the predictive power of an independent variable. The IV indicates the ability of the independent variable to distinguish between different outcomes (i.e., dependent variables) based on the information gain provided by that independent variable. In the present disclosure, the IV corresponds to a statistical technique that correlates the set of evaluation parameters (independent variables) with customer response (dependent variables) to determine the relative significance and contribution of each evaluation parameter. The dependent variable used to calculate the IV is the customer response, which is denoted as a binary outcome (i.e., 1 for a positive response and 0 for no response from a customer).
To calculate the IV, historical data of the set of response parameters may be retrieved. The historical email data may include data corresponding to each of a plurality of historical content variants. This historical data may include a customer response (i.e., responder or non-responder) for each historical content variant. In an embodiment, the customer response may be recorded in the form of binary values, for example a responder may correspond to 1 and a non-responder may correspond to 0. Thus, a number of responders (or a percentage of responders) and a number of non-responders (or a percentage of non-responders) may be obtained from the historical data. The historical data may also include scores of the set of evaluation parameters for each historical content variant.
For IV calculation of an evaluation parameter, a set of continuous bins (i.e., ranges or intervals) may be created for the historical scores of the evaluation parameter. By way of an example, the set of bins may be created based on criteria such as deciles, quartiles, business logic, and the like. Further, the historical data may be divided into appropriate bins. In other words, the number of responders and the number of non-responders in the historical email data may be separately counted for each of the set of bins. For example, if, for a historical email, the score of an evaluation parameter is 25 and the customer response corresponds to a responder, then the number of responders in an appropriate bin for the score (e.g., 21-40) may be increased by 1.
Further, a Weight of Evidence (WoE) may be calculated for each bin through equation (4).
WOE= log⁡((Percentage of Responders in Bin)/(Percentage of Non-Responders in Bin)) (4)
After calculating WoE, the conventional IV may be calculated for the evaluation parameter using the equation (5).
IV= ∑_1^n▒〖(Percentage of Responders-Percentage of Non Responders)×WOE〗 (5)
Where n corresponds to a number of bins in the set of bins.
In some embodiments, the historical data may be divided into historical data corresponding to a set of customer segments. The IV may then be calculated as a weighted sum of a set of IVs corresponding to a set of customer segments. As will be appreciated, each of the set of customer segments may be predefined with a unique set of characteristics. In an embodiment, the set of customer segments may be created based on demographic groups (such as age groups, gender, income levels, geographic regions, customer types (e.g., new vs. returning), or the like). In another embodiment, the set of customer segments may be created based on behavioral segments. The behavioral segments may be defined based on previous customer interactions (extracted from the historical email data), such as response rates (for example, frequent responders or occasional responders). The user may require the email subject line to be customized according to the targeted customer segment.
The calculated IV corresponding to the evaluation parameter may be used as weights to compute the weighted score for predicting whether an email will be opened or not. In a similar manner, the IV helps to evaluate the predicting power associated with each of the set of evaluation parameters. Here, the dependent variable may be the customer feedback in determining whether the email was opened.
It should be noted that behavior (or response) of a customer segment may change with time for an evaluation parameter. So, the IV calculation module 208 may iteratively calculate the customer segment weighted IV at predefined time intervals. Further, the IV calculation module 208 may modify the IV of the evaluation parameter based on a time decay. In other words, for each evaluation parameter of the set of evaluation parameters, the IV calculation module 208 may modify a current IV of the evaluation parameter based on a decay factor and a previous IV of the evaluation parameter. The decay factor may be indicative of an impact of the previous IV on the current IV. The previous IV may be the IV of the evaluation parameter calculated at a previous time interval. The modified IV (IVmodified) for an evaluation parameter may be computed using equation (6).
IVmodified = α * IVprevious + (1 - α) * IVcurrent (6)
Where IVprevious is a previous IV,
IVcurrent is a current IV, and
α is a decay factor.
Thus, the decay factor (α) is directly correlated to the impact of previous information (or previous IV). A higher α value increases the impact of previous information, giving less weight to the current IV (IVcurrent), indicating a slower responsiveness to recent trends. Conversely, a lower α value reduces the influence of the previous IV (IVprevious), giving more weight to the current IV (IVcurrent), reflecting a quicker rate of change in customer behavior. Further, the IV calculation module 208 may send the IV of each of the set of evaluation parameters to the quality score determination module 206. By dynamically adjusting the IV for each of the set of evaluation parameters, an email personalization engine (or the first LLM 214) may be continuously fine-tuned to align the generated content recommendations with evolving customer preferences.
The IV calculation module 208 may send the IV associated with each of the set of evaluation parameters to the score determination module 206 for further processing. The score determination module 206 may assign the IV, for each evaluation parameter of the set of evaluation parameters, as a corresponding weight to the evaluation parameter. The score determination module 206 may further determine a weighted score for each of the set of content variants based on a set of evaluation parameters and a corresponding set of weights. The weighted score may be calculated using an equation (7).
Weighted Score=w1*Readability Score+w2*Non-Spam Score+w3*Sentiment Score+w4*HTML Score+w5*Aesthetic Score+w6* Email engagement Quality Score (7)
where w1, w2, w3, w4, w5, and w6 are weights corresponding to the IV of respective parameter, which may be derived from historical data. The weights are indicative of the importance and relevance of each factor in determining the overall quality and effectiveness of generated emails or the set of content variants.
Upon evaluating the weighted score, the response prediction module 210 may predict, via the second LLM 216, a persona response 224 to each of the set of content variants 222 based on a user-defined persona through a response prediction prompt. The persona response may include a set of response parameters. The response parameters may include an open rate, a click-through rate (click and read through the purchase links), conversion rates (purchase the product), a bounce rate, an unsubscribe rate and the like. The user-defined persona corresponds to the target customer segment. The response prediction prompt may include the set of content variants 222, the user-defined persona, and a predefined set of response prediction instructions.
The second LLM 216 may work as a marketing content testing simulator. In other words, the second LLM 216 may simulate an email campaign scenario and may then evaluate the effectiveness of the email or content before actual deployment based on the set of response prediction instructions.
Upon response prediction, the fine-tuning module 212 may fine tune the first LLM 214 and the second LLM 216. In some embodiments, the fine-tuning module 212 may fine tune the second LLM 216 using the historical data of the customer segment based on the comparison. In other embodiments, the fine-tuning module 212 may fine tune the second LLM 216 using a user feedback through reinforcement learning based on the comparison. Further, in some embodiments, the fine-tuning module 212 may include generating, via the first LLM 214, a set of modified content variants for the customer segment based on the comparison. In other embodiments, the fine-tuning module 212 may fine-tune the first LLM 214 using a fine-tuning dataset 226 based on a reinforcement learning technique. The fine-tuning dataset 226 may include a content variant selected based on an optimal combination of the weighted score and the persona response, a randomly selected content variant from remaining of the set of content variants 222, and the weighted score and the persona response corresponding to each of the content variant and the randomly selected content variant.
It should be noted that all such aforementioned modules 202 – 212 may be represented as a single module or a combination of different modules. Further, as will be appreciated by those skilled in the art, each of the modules 202 – 212 may reside, in whole or in parts, on one device or multiple devices in communication with each other. In some embodiments, each of the modules 202 – 212 may be implemented as dedicated hardware circuit comprising custom application-specific integrated circuit (ASIC) or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. Each of the modules 202 – 212 may also be implemented in a programmable hardware device such as a field programmable gate array (FPGA), programmable array logic, programmable logic device, and so forth. Alternatively, each of the modules 202 – 212 may be implemented in software for execution by various types of processors (e.g., processor 104). An identified module of executable code may, for instance, include one or more physical or logical blocks of computer instructions, which may, for instance, be organized as an object, procedure, function, or other construct. Nevertheless, the executables of an identified module or component need not be physically located together but may include disparate instructions stored in different locations which, when joined logically together, include the module and achieve the stated purpose of the module. Indeed, a module of executable code could be a single instruction, or many instructions, and may even be distributed over several different code segments, among different applications, and across several memory devices.
As will be appreciated by one skilled in the art, a variety of processes may be employed for generating and evaluating personalized marketing content. For example, the exemplary system 100 and the associated computing device 102 may generate subject lines for emails by the processes discussed herein. In particular, as will be appreciated by those of ordinary skill in the art, control logic and/or automated routines for performing the techniques and steps described herein may be implemented by the system 100 and the associated computing device 102 either by hardware, software, or combinations of hardware and software. For example, suitable code may be accessed and executed by the one or more processors on the system 100 to perform some or all of the techniques described herein. Similarly, application specific integrated circuits (ASICs) configured to perform some, or all of the processes described herein may be included in the one or more processors on the system 100.
Referring now to FIG. 3, an exemplary method 300 for generating and evaluating personalized marketing content is depicted via a flowchart, in accordance with some embodiments of the present disclosure. The exemplary method 300 may be implemented by the computing device 102 of the system 100. The method 300 may include receiving, via a user interface (such as the user interface 110), content information (such as the content information 218) and details of a target customer segment (such as customer details 220), at step 302. It may be noted that the target customer segment may be one of the set of customer segments. Upon receiving, the method 300 may include creating, by a prompt creation module (such as the prompt creating module 202) may create a content generation prompt using the content information, the details of the target customer segment, and the predefined set of content generation instructions, at step 304. Further, the method 300 may include inputting, by the prompt creation module, the content generation prompt into a first LLM (such as the first LLM 214), at step 306. Further, the method 300 may include generating, by a content variants generation module (such as the content variants generation module 204), via the first LLM 214, a set of content variants (such as the set of content variants 222) in response to the content generation prompt, at step 308.
Further, the method 300 may include determining by a score determination module (such as the score determination module 206) and a IV calculation module (such as the IV calculation module 208), via the first LLM, a weighted score for each of the set of content variants based on a set of evaluation parameters and a corresponding set of weights, at step 310. By way of an example, the set of evaluation parameters may include, but may not be limited to, a readability parameter, a non-spam parameter, a sentiment parameter, an HTML parameter, an aesthetic parameter, an email engagement quality parameter, and the like. Upon evaluating the weighted score, the method 300 may include creating, by the prompt creation module, a response prediction prompt using the set of content variants, the user-defined persona, and the predefined set of response prediction instructions, at step 312. Further, the method 300 may include inputting, by the prompt creation module, the response prediction prompt to a second LLM (such as the second LLM 216), at step 314.
Further, the method 300 may include predicting, by a response prediction module (such as the response prediction module 210), via the second LLM 216, a persona response (such as the persona response 224) to each of the set of content variants based on a user-defined persona through the response prediction prompt, at step 316. Further, the method 300 may include rendering, by the response prediction module, via the user interface, the set of content variants, and the weighted score and the persona response corresponding to each of the set of content variants, at step 318.
Referring now to FIG. 4, an exemplary method 400 for predicting the persona response to content variants (for example, the set of content variants 222) is depicted via a flowchart, in accordance with some embodiments of the present disclosure. The method 400 may include retrieving, by a response prediction module (such as the response prediction module 210), historical data of the set of response parameters corresponding to a plurality of customers, at step 402. Further, the method 400 may include clustering, by the response prediction module, the plurality of customers into a set of customer segments, at step 404. In other words, the target audience may be divided into ‘n’ distinct customer segments with the help of statistical clustering methods based on demography and other variables (for example, age, location, interests and the like). Further, the method 400 may include selecting, by the response prediction module, a representative customer from each of the set of customer segments, at step 406. Further, the method 400 may include creating, by the response prediction module, via a second LLM 216(such as the second LLM 216), a set of personas corresponding to the set of customer segments based on the historical data of the set of response parameters of the representative customer, at step 408.
Referring now to FIG. 5, an exemplary method 500 for fine-tuning of a first LLM 214(such as the first LLM 214) and a second LLM (such as the second LLM 216) is depicted via a flowchart, in accordance with some embodiments of the present disclosure. For each customer segment of the set of customer segments, by a fine-tuning module (such as the fine tuning module 212), the method 500 may include retrieving, by the fine tuning module, a real-world acceptance rate of the customer segment from the historical data, at step 502. Further, the method 500 may include determining, by the fine tuning module, a predicted acceptance rate of the customer segment corresponding to a set of content variants (such as the set of content variants 222), at step 504. Further, the method 500 may include comparing, by the fine tuning module, the real-world acceptance rate with the predicted acceptance rate using a statistical test, at step 506.
Upon comparing, the method 500 may further include fine-tuning, by the fine tuning module, the second LLM using at least one of: the historical data of the customer segment based on the comparison, at step 508, or through reinforcement learning based on the comparison, at step 510.
Additionally, upon comparing, the method 500 may include at least one of: generating, by the fine tuning module, a set of modified content variants for the customer segment based on the comparison, at step 512, or fine-tuning, by the fine tuning module, the first LLM 214 using a dataset based on a reinforcement learning, at step 514.
Referring now to FIG. 6A, a table 600A representing experimental results for IV computation of readability parameter based on an exemplary primary dataset is illustrated, in accordance with an embodiment of the present disclosure. The primary dataset may be a simulated dataset of 1000 users that are not divided into further customer segments.
The table 600A may include a column for readability bin 602, a column for count 604 of users, a column for number of responders 606, a column for number of non-responders 608, a column for percentage of responders 610 in the bin, a column for percentage of non-responders 612 in the bin, a column for WoE 614, and a column for IV 616. The column for readability bin 602 includes the set of bins (i.e., ranges) of readability scores. The column for WoE 614 includes the WoE value calculated for the bin using equation (1). The column for IV includes IV values computed for the bin using equations (2), (3) and (5).
In the table 600A, for the readability bins 602 ‘1-20’, ’21-40’, ’41-60’, ’61-80’, and ‘81-100’, the corresponding IV values 616 are ‘0.0056’, ‘0.0035’, ‘0.0127’, ‘0.0042’, and ‘0.0014’, respectively. The total IV value may be calculated as a sum of the IV values 616 of all the readability bins 602. Thus, the total IV value is ‘0.0275’.
Referring now to FIGS. 6B and 6C, tables representing experimental results for total IV computations of readability parameter for individual customer segments within the exemplary primary dataset are illustrated, in accordance with an embodiment of the present disclosure. FIGS. 6B and 6C are explained in conjunction with FIG. 6A.
The primary dataset may include simulated data based on some assumptions. The assumptions are based on three observations from customer data of an enterprise. Firstly, approximately 30-40% of the customers are classified as belonging to the high-income group, while the remaining 60-70% are classified as low income, based on predefined income thresholds. Thus, two customer segments may be created based on income levels of the users. A first customer segment may correspond to a high-income group and a second customer segment may correspond to a low-income group. Secondly, the response rate is slightly higher among customers in the high-income group compared to those in the low-income group. Thirdly, for the low-income group, easier readability (higher readability score) attracts more response. On the other hand, for the high-income group, standard readability (lower readability score) has more responders. Based on the above observations and assumptions, the primary dataset is generated to analyze the IV across different customer segments.
In FIG. 6B, a table 600B is shown. The table 600B may be based on a high-income dataset derived from the primary dataset. The high-income dataset may include 382 users from the 1000 users in the primary dataset. The table 600B may include the column for readability bin 602, the column for count 604 of users, the column for number of responders 606, the column for number of non-responders 608, the column for percentage of responders 610 in the bin, the column for percentage of non-responders 612 in the bin, the column for WoE 614, and the column for IV 616.
In the table 600B, for the readability bins 602 ‘1-20’, ’21-40’, ’41-60’, ’61-80’, and ‘81-100’, the corresponding IV values 616 are ‘0.0406’, ‘0.0033’, ‘0.0181’, ‘0.0101’, and ‘0.0000’, respectively. The total IV value may be calculated as a sum of the IV values 616 of all the readability bins 602. Thus, the total IV value is ‘0.0721’.
In FIG. 6C, a table 600C is shown. The table 600C may be based on a low income dataset 600C. derived from the primary dataset. The low-income dataset may include 618 users from the 1000 users in the primary dataset. The table 600C may include the column for readability bin 602, the column for count 604 of users, the column for number of responders 606, the column for number of non-responders 608, the column for percentage of responders 610 in the bin, the column for percentage of non-responders 612 in the bin, the column for WoE 614, and the column for IV 616.
In the table 600C, for the readability bins 602 ‘1-20’, ’21-40’, ’41-60’, ’61-80’, and ‘81-100’, the corresponding IV values 616 are ‘0.0001’, ‘0.0035’, ‘0.0108’, ‘0.0020’, and ‘0.0043’, respectively. The total IV value may be calculated as a sum of the IV values 616 of all the readability bins 602. Thus, the total IV value is ‘0.0207’.
The overall IV (obtained from the table 600A) is 0.0275, whereas for the low-income group, the IV (obtained from the table 600C) is 0.0207, and for the high-income group, the IV (obtained from the table 600B) is significantly higher, at 0.0721. This disparity suggests that the overall IV may not accurately represent the influence of the readability score within the high-income group. Therefore, employing a weighted IV may be a more effective method to accurately reflect the distinct impacts of different income segments on customer responses. The customer segment weighted IV (using weights 0.6 for the high-income group and 0.6 for the low-income group) may be calculated using an equation (8).
IV_(readability,weighted)=w_low.IV_(readability,low)+w_high.IV_(readability,high) (8)
= 0.6*0.0721 + 0.6*0.0207 = 0.052
Referring now to FIG. 7, a schematic diagram of a detailed exemplary process for generating and evaluating personalized marketing content is illustrated, in accordance with some embodiments of the present disclosure. In an embodiment, the process may be implemented by the system 200 to generate emails. The process may be implemented via a simulation device 700. The simulation engine 700 may receive information 702 as an input from a user that includes a product information 702a, an offer information 702b, and a demographic information 702c associated with a targeted customer segment. The simulation device 700 may be analogous to the computing device 102. The simulation device 700 may include a content generation prompt 704, an email generator 706, a score engine 708, and a simulator 710.
The information 702 may be used to generate content generation prompt 704. The prompt creation module 202 may then input the content generation prompt 704 to the email generator 706. The email generator 706 may be analogous to the first LLM 214. The email generator 706 may receive the content generation prompt 704 and may generate a set of email variants. In other words, the content generation prompt 704 may be used by the email generator 706 to generate ‘n’ number of email variants (such as, an email variant 712a, an email variant 712b, …., an email variant 712n). By way of an example, the email variant 712a, the email variant 712b, …, the email variant 712n (collectively referred to as the set of email variants 712) may be HTML enabled emails. In an embodiment, the open-source Mistral-7B-Instruct-v0.1 Generative AI model may be selected as the email generator 706first LLM 214. As will be appreciated, the open-source Mistral-7B-Instruct-v0.1 Generative AI model possesses a capacity to generate emails in compliance with the content generation prompt 702. Further, the email generator 706 may send the generated email variants (i.e., the email variant 712a, the email variant 712b, …, the email variant 712n) to the score engine 708.
The score engine 708 (for example, the score determination module 206) may determine the weighted score for each of the ‘n’ number of email variants 712based on the set of evaluation parameters and the corresponding set of weights as described in greater detail in conjunction with the FIG. 2. In other words, the scores corresponding to the email variant 712a, the email variant 712b, …, the email variant 712n may be a score 714a, a score 714b, …, a score 714n, respectively.
Upon evaluating scores, the score engine 708 may send the email variant 712a, the email variant 712b, …, the email variant 712n to the simulator 710. The simulator 710 may be a tool designed to simulate various email campaign scenarios and evaluate the effectiveness of the scenarios before actual deployment. The simulator 710 may be analogous to the second LLM 216. The simulator 710 mimics real-world user behaviour and is used to compute metrics (such as open rates, click-through rates, and conversion rates) for the set of email variants 712. By way of an example, the metrics corresponding to the email variant 712a, the email variant 712b, …, the email variant 712n may be metrics 714a, metrics 714b, …, metrics 714n, respectively. In an embodiment, the simulator 710 may focus on multivariate A/B testing of the email variants 712, on setup, execution, and analysis of real-time experiments to optimize marketing strategies. The multivariate A/B testing enhances email marketing by comparing the email variants 712 to identify the best performer based on a set of response parameters (i.e., metrics). The multivariate A/B testing may enable marketers to make data-driven decisions for optimization based on the identification of the most effective elements of the campaign.
Upon collecting real-world data through multivariate A/B testing, the simulator 710 may focus on Persona-Driven Marketing Email Campaign Data Synthesis and may function as an email marketing tester by retrieving data from the second LLM 216. In other words, the LLM (for example, the second LLM 216) may be used with the set of personas to simulate user response to marketing campaign. By combining the LLM with the set of personas, marketers can simulate the user responses of their target customer segment to different marketing campaigns and test the effectiveness of different campaign strategies. A persona is a fictional representation of ideal customers, where each customer may possess unique traits, used in marketing to understand and engage target customer segment. By way of an example, a marketer may create the persona like ‘impulsive buyer’ to generate tailored campaigns that resonate with the persona.
By way of another example, a prompt for the user-defined persona for the second LLM 216 may be, “Now behave like a 30-year-old male. And then I'm going to ask you some question and tell me whether you would like this marketing to offer or not’ for simulating user response to marketing campaigns.” In this example, the second LLM 216 is behaving as a digital twin of the user in a supervised machine learning model. In other words, the second LLM 216 may act as a digital replica of user while providing insights and predictions based on their behavior patterns. The second LLM 216 may generate realistic simulations of user interactions throughout the campaign lifecycle, predicting behaviors such as engagement, conversion, and retention. Digital twining may enable marketers to predict user behaviors in various stages including whether the user may notice or see an impression (for example, an email), click on the content (for example links within the email), engage with the content (for example, reading the message or interacting with interactive elements), and finally conversion rate (for example, making a purchase, signing up for a service, etc.). Here, the open rate and the click-through rate may be considered as initial engagement, and conversion, and retention may be considered as long-term engagement.
By analyzing historical data and behavioral patterns, users are clustered into segments, a representative individual is selected from each segment to create digital twins. A pseudo prompt for the simulator 710 for multivariate A/B testing using the second LLM 216 as digital twin may be as follows.
“The pseudo prompt: Email Campaign: Get the most out of your Designer Handbag with this style tip: Pair it with a monochrome outfit for a pop of luxury. Check out more tips here [video link].’persona_list (the set of personas) = 'Impulsive Buyer', 'Bargain Hunter', 'Loyal Customer', 'Brand Advocate', 'Occasional Buyer'.”
For generating the set of user responses, the pseudo prompt may be sent with email campaign and persona type to the second LLM 216 to generate a response. Further, the response of the second LLM 216 (which acts as a digital twin to the real-world persona) may be measured. Further, the set of response parameters of the response mentioned in the output may be calculated.
In the above example, the inputs are the set of email variants 712 (email campaign) and the persona type while the output is the simulated user response in form of the response parameters. Further, the simulated user response corresponding to the persona may help the marketers to make data-driven decisions for which the campaigns are most likely to resonate with their target customer segment and drive the best results. It is to be noted that user-defined persona may be one of the set of personas. Unlike traditional A/B testing or multivariate testing, which require emails to be sent to a live audience, the email marketing tester creates a controlled, virtual environment to analyze and predict campaign performance using machine learning and historical data. In other words, the email marketing tester may perform tests on the set of personas. The multivariate A/B testing technique may tackle scalability by simultaneously testing multiple email variants without waiting for live responses from the real-world users that reduces cost related to email list segmentation and data collection. Further, the technique avoids high unsubscribe rates or brand damage by optimizing the campaign before launch.
The simulation device 700 may then output the email variants (i.e., the email variant 712a, the email variant 712b, …, the email variant 712n), the corresponding scores (i.e., the score 714a, the score 714b, …, the score 714n), and the corresponding metrics (i.e., the metrics 716a, the metrics 716b, …, the metrics 716n). At step 718, a marketer may review the set of email variants 712 based on the corresponding scores and metrics. The marketer may then select an email variant from the email variants 712. Further, at step 720, the marketer may launch the campaign using the selected email variant.
Upon testing, the simulator 710 may retrieve, for each customer segment of the set of customer segments, a real-world acceptance rate from the historical data. In other words, for each segment i (where 1 ≤ i ≤ n), historical record (data) of real-world acceptance rates (RWAR_i) is maintained based on past campaigns. Further, the simulator 710 may retrieve, via the second LLM 216, a predicted acceptance rate of the customer segment corresponding to the set of email variants. In other words, the simulator 710 predicts acceptance rates (SPAR_i) for each segment i based on the current email variants 712. These email variants 712 may be customized for the targeted customer segment. Upon retrieving, the simulator 710 may compare the real-world acceptance rate with the predicted acceptance rate using a statistical test. In other words, the RWAR_i and SPAR_i may be compared for each demographic. In order to determine whether the simulator 710 accurately mimics real-world performance, the statistical test may be necessary to compare two sets of proportions (acceptance rates). In some embodiments, the suitable statistical test may be a two-proportion z-test.
The z-test may determine whether the difference between the real-world acceptance rate and the simulated acceptance rate for the demography is statistically significant or not, and the z-test may be performed for all demographics. The z-test may include a null hypothesis (H0) when the difference the real-world and simulated acceptance rates i.e. RWAR_i = SPAR_i. In case of the null hypothesis there may be no need for fine-tuning of the first LLM 214 or the second LLM 216 and the set of email variants 712 are stored in a cloud storage 722.
Further, the z-test may include an alternative hypothesis (H1) when the difference is significant (RWAR_i ≠ SPAR_i). The test statistic is calculated by using equation (9).
z=((p1-p2))⁄sqrt(p(1-p)(1⁄n1+1⁄n2)) (9)
where:
p1 = RWAR_i
p2 = SPAR_i
n1 = Number of real-world observations for demographic i
n2 = Number of simulated observations for demographic i
p = (x1 + x2) / (n1 + n2), where x1 and x2 are the number of successes (acceptances) in the real-world and simulated data, respectively.
When the absolute value of the calculated z-score exceeds a threshold value (based on a threshold significance level, typically 0.05), then the null hypothesis is rejected, and a statistically significant difference is concluded between the real-world acceptance rate and the simulated acceptance rate.
The significant difference leads to the fine-tuning of the second LLM 216 based on a reinforcement learning from human feedback (RLHF), at step 724. In RLHF, human feedback may be used to identify areas where the simulator deviates from real-world behavior and adjust accordingly. Additionally, the internal parameters of the simulator 710 may be redetermined that influence acceptance rate predictions for that demographic, i.e., assigning weights certain content features differently. In other words, a weighted scoring mechanism may be applied as the reward function within the RL framework, enabling model optimization. In the off-policy method, data may be collected from the operational environment and used for model training in an offline setting.
Any reinforcement learning algorithm may be employed to fine-tune the second LLM 216 using the weighted scores such as but is not limited to Direct Preference Optimization (DPO). DPO technique may be selected due to its recent popularity and straightforward implementation. DPO technique may be particularly suited for this use due to the robustness against variations in data and inherent biases, thereby minimizing the risk of overfitting to noisy or non-representative data. DPO may ensure model reliability and maintain consistent performance across diverse datasets.
Further, for fine-tuning of the first LLM 214 may be done using a dataset based on a reinforcement learning. The fine-tuning process begins with supervised training, followed by a secondary fine-tuning stage utilizing the weighted scores in a DPO framework. The integration of weighted scores may enable data preparation that aligns with DPO training protocols, thus allowing customization of the LLMs (the first LLM 214 and the second LLM 216) to align with predefined campaign goals. By assigning priority to email content based on the scores, the model may be optimized to account for factors crucial to the success of various email marketing strategies.
Further, the dataset for fine-tuning may include a chosen email variant 722 selected based on an optimal combination of the weighted score and the persona response, a randomly selected email variant from remaining of the set of email variants 712, and the weighted score and the persona response corresponding to each of the email variant and the randomly selected email variant. In other words, the dataset may include a list of input parameters such as, but not limited to, a list of email content inputs provided by the user for a configurable time interval (Δt), email content associated with the highest weighted score for each input, and a randomly selected generated email from a set of m options, where the selected email does not have the highest weighted score.
For the fine-tuning, pairs of email content may be generated from input parameters 2 and 3.a prompt for model fine-tuning may be developed. Further, the model may be fine-tuned using the provided data.
The output of the fine-tuning may be a fine-tuned first LLM 214 optimized for subsequent content generation tasks. Additionally, due to the significant difference upon comparison the first LLM 214 may generate a set of modified content variants for the customer segment. By way of an example, the modification in an email may be the email content, subject line, or call to action specifically for that demographic.
Referring now to FIG. 8, a layout 800 of email that is to be generated using the schematic 700 is illustrated, in accordance with an embodiment of the disclosure. The structure 800 may include three major sections such as a header section 802, a body section 802 and a footer section 806. Some sections may further be divided into sections. The header section 802 may include a subject line 808 and a preheader 810.
The body section 804 may include an image 812, a content 814, a cta 816, and an additional info 818. The content 814 may include greetings, introduction to the product and main content. The structure 800 may be considered as a set of HTML tags such as, but not limited to, ‘’, ‘’, ‘’, ‘’, ‘’, ‘’, ‘’, ‘’, and ‘’. Further, the output may include a response to the input. The output may include the subject line, the preheader, an engaging image, the email content, additional information, and a personalized message in the structured grid layout 800 in a user-friendly format.
The subject line 808 may capture attention of the targeted customer segment and summarize the email content, the preheader may set the tone for the email. In the body section 804, the engaging image (image 812) may visually attract the recipient’s interest, the email content may provide relevant information for the email generation such as product details and personalized messages. The personalized messages may include tailored content as per the recipient’s preferences. The additional info may include details such as but is not limited to shipping offers and flash sales. The content may be in a user-friendly format to ensure easy navigation and accessibility of information.
The structure 800 may be followed by the first LLM for generating the set of emails based on the content generation prompt 702. By way of an example, the content generation prompt may be described below.
Prompt example: product = "Samsung electronic gadgets cutting-edge technology with versatile features and vast selection"
customer = "Veeresh, 20-year-old, male, new customer"
advanced = "Enjoy our Tech Extravaganza Sale with up to 25% off on selected items for a limited time. Buy any Samsung smartphone and get 50% off on selected accessories, plus receive a free extended warranty for 1 year on all purchases. Benefit from free shipping on orders over $50 and exclusive bundle deals on Samsung tablets and wearables. Trade in your old device and get an extra $100 off your new purchase. Earn double loyalty points with every purchase and sign up for early access to the sale. Students can enjoy a special 15% discount and refer a friend to receive a $25 gift card each. Additionally, take advantage of an extra 10% off during hourly flash sales.",
sections_provided = "Introduction: We're thrilled to have you with us. As a tech enthusiast, you're going to love our Tech Extravaganza Sale, Footer: Please unsubscribe, if you do not want to receive this emails."
image = "Unleash savings! Shop top deals during Big Billion Day. Limited time!"
header_color_code = "336699"
main_content_color_code = "FFFFFF"
cta_color_code = "333333"
footer_color_code = "CCCCCC"
The prompt may be received by the first LLM 214 for generation of the set of emails in this embodiment. Upon generating, a weighted score may be determined for each email of the set of emails by the score determination module 206, based on a set of evaluation parameters and a corresponding set of weights. Further, a response prediction prompt may be created using the set of emails, a user-defined persona, and a predefined set of response prediction instructions. The response prediction prompt may then be inputted to the second LLM 216.
The second LLM 216 may then predict a persona response to each of the set of emails based on a user-defined persona through the response prediction prompt. The persona response may include a set of response parameters. The user-defined persona corresponds to the target customer segment. A user interface may then render the set of emails, and the weighted score and the persona response corresponding to each of the set of content variants. Then, historical data may be retrieved of the set of response parameters corresponding to a plurality of customers. The plurality of customers may be clustered into a set of customer segments. The target customer segment is one of the set of customer segments. A customer representative may be selected from each of the set of customer segments.
Then, a set of personas may be created via the second LLM 216 corresponding to the set of customer segments based on the historical data of the set of response parameters of the representative customer. Further, for each customer segment of the set of customer segments, a real-world acceptance rate may be retrieved from the historical data, a predicted acceptance rate of the customer segment may be determined corresponding to the set of content variants. Then, the real-world acceptance rate is compared with the predicted acceptance rate using a statistical test.
Further, the second LLM 216 may be fine-tuned using the historical data of the customer segment or using a user feedback through reinforcement learning based on the comparison. Further, a set of modified content variants may be generated via the first LLM 214 for the customer segment based on the comparison or the first LLM 214 may be fine-tuned using a dataset based on a reinforcement learning. The dataset may include a content variant selected based on an optimal combination of the weighted score and the persona response, a randomly selected content variant from remaining of the set of content variants, and the weighted score and the persona response corresponding to each of the content variant and the randomly selected content variant.
The disclosed methods and systems may be implemented on a conventional or a general-purpose computer system, such as a personal computer (PC) or server computer. Referring now to FIG. 9, an exemplary computing system 900 that may be employed to implement processing functionality for various embodiments (e.g., as a SIMD device, client device, server device, one or more processors, or the like) is illustrated. Those skilled in the relevant art will also recognize how to implement the invention using other computer systems or architectures. The computing system 900 may represent, for example, a user device such as a desktop, a laptop, a mobile phone, personal entertainment device, DVR, and so on, or any other type of special or general-purpose computing device as may be desirable or appropriate for a given application or environment. The computing system 900 may include one or more processors, such as a processor 902 that may be implemented using a general or special purpose processing engine such as, for example, a microprocessor, a microcontroller or other control logic. In this example, the processor 902 is connected to a bus 904 or other communication medium. In some embodiments, the processor 902 may be an Artificial Intelligence (AI) processor, which may be implemented as a Tensor Processing Unit (TPU), or a graphical processor unit, or a custom programmable solution Field-Programmable Gate Array (FPGA).
The computing system 900 may also include a memory 906 (main memory), for example, Random Access Memory (RAM) or other dynamic memory, for storing information and instructions to be executed by the processor 902. The memory 906 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 902. The computing system 900 may likewise include a read only memory (“ROM”) or other static storage device coupled to bus 904 for storing static information and instructions for the processor 902.
The computing system 900 may also include storage devices 908, which may include, for example, a media drive 910 and a removable storage interface. The media drive 910 may include a drive or other mechanism to support fixed or removable storage media, such as a hard disk drive, a floppy disk drive, a magnetic tape drive, an SD card port, a USB port, a micro-USB, an optical disk drive, a CD or DVD drive (R or RW), or other removable or fixed media drive. A storage media 912 may include, for example, a hard disk, a magnetic tape, a flash drive, or other fixed or removable medium that is read by and written to by the media drive 910. As these examples illustrate, the storage media 912 may include a computer-readable storage medium having stored therein particular computer software or data.
In alternative embodiments, the storage devices 908 may include other similar instrumentalities for allowing computer programs or other instructions or data to be loaded into the computing system 900. Such instrumentalities may include, for example, a removable storage unit 914 and a storage unit interface 916, such as a program cartridge and cartridge interface, a removable memory (for example, a flash memory or other removable memory module) and memory slot, and other removable storage units and interfaces that allow software and data to be transferred from the removable storage unit 914 to the computing system 900.
The computing system 900 may also include a communications interface 918. The communications interface 918 may be used to allow software and data to be transferred between the computing system 900 and external devices. Examples of the communications interface 918 may include a network interface (such as an Ethernet or other NIC card), a communications port (such as for example, a USB port, a micro-USB port), Near field Communication (NFC), etc. Software and data transferred via the communications interface 918 are in the form of signals which may be electronic, electromagnetic, optical, or other signals capable of being received by the communications interface 918. These signals are provided to the communications interface 918 via a channel 920. The channel 920 may carry signals and may be implemented using a wireless medium, wire or cable, fiber optics, or another communications medium. Some examples of the channel 920 may include a phone line, a cellular phone link, an RF link, a Bluetooth link, a network interface, a local or wide area network, and other communications channels.
The computing system 900 may further include Input/Output (I/O) devices 922. Examples may include, but are not limited to a display, keypad, microphone, audio speakers, vibrating motor, LED lights, etc. The I/O devices 922 may receive input from a user and also display an output of the computation performed by the processor 902. In this document, the terms “computer program product” and “computer-readable medium” may be used generally to refer to media such as, for example, the memory 906, the storage devices 908, the removable storage unit 914, or signal(s) on the channel 920. These and other forms of computer-readable media may be involved in providing one or more sequences of one or more instructions to the processor 902 for execution. Such instructions, generally referred to as “computer program code” (which may be grouped in the form of computer programs or other groupings), when executed, enable the computing system 900 to perform features or functions of embodiments of the present invention.
In an embodiment where the elements are implemented using software, the software may be stored in a computer-readable medium and loaded into the computing system 900 using, for example, the removable storage unit 914, the media drive 910 or the communications interface 918. The control logic (in this example, software instructions or computer program code), when executed by the processor 902, causes the processor 902 to perform the functions of the invention as described herein.
Various embodiments provide method and system for generating and evaluating personalized marketing content. The disclosed method and system may receive, via a user interface, content information and details of a target customer segment. Further, the disclosed method and system may generate, via a first LLM, a set of content variants in response to a content generation prompt. The content generation prompt may include the content information, the details of the target customer segment, and a predefined set of content generation instructions. Further, the disclosed method and system may determine, via the first LLM, a weighted score for each of the set of content variants based on a set of evaluation parameters and a corresponding set of weights. Further, the disclosed method and system may predict, via a second LLM, a persona response to each of the set of content variants based on a user-defined persona through a response prediction prompt. The persona response may include a set of response parameters. The user-defined persona may correspond to the target customer segment. The response prediction prompt may include the set of content variants, the user-defined persona, and a predefined set of response prediction instructions. Further, the disclosed method and system may render, via the user interface, the set of content variants, and the weighted score and the persona response corresponding to each of the set of content variants.
Thus, the disclosed techniques try to overcome the logical problem for generating and evaluating personalized marketing content. The techniques provide an enhanced content engagement by improving the effectiveness of marketing campaigns. Further, the techniques may allow proactive testing and refinement of the content variants that eliminates the need for costly post-deployment adjustments. The proactive testing may also save time in the long run due to the trial-and-error approach. Further, the techniques may facilitate more informed, data-driven decisions regarding marketing content strategies and may optimize marketing efforts. Further, the techniques may tackle scalability, accommodating a wide range of marketing strategies. Further, the techniques provide a straightforward yet effective method for assessing the quality of a set of content variants, facilitating rapid implementation and immediate improvements in marketing communications. The improvements may be made in response to the human feedback and the techniques may continually adapt to the user preferences that leads to improvement in relevance and personalization of the content over time. Further, the techniques provide optimized AI model adaptability. The weighted scores are incorporated into the fine-tuning of the Generative AI model or LLM for creating content variants, which enhances the model’s accuracy, relevance, and generates more effective content.
In light of the above-mentioned advantages and the technical advancements provided by the disclosed method and system, the claimed steps as discussed above are not routine, conventional, or well understood in the art, as the claimed steps enable the following solutions to the existing problems in conventional technologies. Further, the claimed steps clearly bring an improvement in the functioning of the device itself as the claimed steps provide a technical solution to a technical problem.
The specification has a described method and system for generating and evaluating personalized marketing content. The illustrated steps are set out to explain the exemplary embodiments shown, and it should be anticipated that ongoing technological development will change the manner in which particular functions are performed. These examples are presented herein for purposes of illustration, and not limitation. Further, the boundaries of the functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternative boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed. Alternatives (including equivalents, extensions, variations, deviations, etc., of those described herein) will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein. Such alternatives fall within the scope and spirit of the disclosed embodiments.
Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein. The term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.
It is intended that the disclosure and examples be considered as exemplary only, with a true scope and spirit of disclosed embodiments being indicated by the following claims.
, Claims:CLAIMS
I/WE CLAIM:
1. A method (300) for generating and evaluating personalized marketing content, the method (300) comprising:
receiving (302), by a processor (104) via a user interface (110), content information (218) and details of a target customer segment;
generating (308), by the processor (104) and via a first Large Language Model (LLM) (214), a set of content variants (222) in response to a content generation prompt, wherein the content generation prompt comprises the content information (218), the details of the target customer segment, and a predefined set of content generation instructions;
determining (310), by the processor (104) and via the first LLM (214), a weighted score for each of the set of content variants (222) based on a set of evaluation parameters and a corresponding set of weights;
predicting (316), by the processor (104) via a second LLM (216), a persona response (224) to each of the set of content variants (222) based on a user-defined persona through a response prediction prompt, wherein the persona response (224) comprises a set of response parameters, wherein the user-defined persona corresponds to the target customer segment, and wherein the response prediction prompt comprises the set of content variants (222), the user-defined persona, and a predefined set of response prediction instructions; and
rendering (318), by the processor (104) via the user interface (110), the set of content variants (222), and the weighted score and the persona response (224) corresponding to each of the set of content variants (222).

2. The method (300) as claimed in claim 1, comprising:
creating (304) the content generation prompt using the content information (218), the details of the target customer segment, and the predefined set of content generation instructions; and
inputting (306) the content generation prompt to the first LLM (214).

3. The method (300) as claimed in claim 1, wherein determining the weighted score for each of the set of content variants (222) comprises:
for each evaluation parameter of the set of evaluation parameters,
calculating an information value (IV) corresponding to the evaluation parameter based on a number of responders and a number of non-responders, wherein the number of responders and the number of non-responders are obtained from historical data of the set of response parameters; and
assigning the IV as a corresponding weight to the evaluation parameter.

4. The method (300) as claimed in claim 3, comprising modifying the IV of the evaluation parameter based on a decay factor, wherein the decay factor is indicative of an impact of a previous IV on a current IV.

5. The method (300) as claimed in claim 1, wherein predicting (308), via the second LLM (216), the persona response (224) to each of the set of content variants (222) comprises:
creating (312) the response prediction prompt using the set of content variants (222), the user-defined persona, and the predefined set of response prediction instructions; and
inputting (314) the response prediction prompt to the second LLM (216).

6. The method (300) as claimed in claim 1, comprising:
retrieving (402) historical data of the set of response parameters corresponding to a plurality of customers;
clustering (404) the plurality of customers into a set of customer segments, wherein the target customer segment is one of the set of customer segments;
selecting (406) a representative customer from each of the set of customer segments; and
creating (408), via the second LLM (216), a set of personas corresponding to the set of customer segments based on the historical data of the set of response parameters of the representative customer, wherein the user-defined persona is one of the set of personas.

7. The method (300) as claimed in claim 6, comprising:
for each customer segment of the set of customer segments,
retrieving (502) a real-world acceptance rate of the customer segment from the historical data;
determining (504), via the second LLM (216), a predicted acceptance rate of the customer segment corresponding to the set of content variants (222); and
comparing (506) the real-world acceptance rate with the predicted acceptance rate using a statistical test.

8. The method (300) as claimed in claim 7, comprising at least one of:
fine-tuning (508) the second LLM (216) using the historical data of the customer segment based on the comparison; or
fine-tuning the second LLM (216) using a user feedback through reinforcement learning based on the comparison.

9. The method (300) as claimed in claim 7, comprising, at least one of:
generating (512), via the first LLM (214), a set of modified content variants for the customer segment based on the comparison; or
fine-tuning (514) the first LLM (214) using a dataset based on a reinforcement learning, wherein the dataset comprises a content variant selected based on an optimal combination of the weighted score and one of the persona response (224) or a real-world persona response (224), a randomly selected content variant from remaining of the set of content variants (222), and the weighted score and the persona response (224) corresponding to each of the content variant and the randomly selected content variant.

10. A system (100) for generating and evaluating personalized marketing content, the system (100) comprising:
a processor (104); and
a memory (106) communicatively coupled to the processor (104), wherein the memory (106) stores processor-executable instructions, which when executed by the processor (104), cause the processor (104) to:
receive (302) content information (218) and details of a target customer segment;
generate (308) a set of content variants (222) in response to a content generation prompt, wherein the content generation prompt comprises the content information (218), the details of the target customer segment, and a predefined set of content generation instructions;
determine (310) a weighted score for each of the set of content variants (222) based on a set of evaluation parameters and a corresponding set of weights;
predict (316) a persona response (224) to each of the set of content variants (222) based on a user-defined persona through a response prediction prompt, wherein the persona response (224) comprises a set of response parameters, wherein the user-defined persona corresponds to the target customer segment, and wherein the response prediction prompt comprises the set of content variants (222), the user-defined persona, and a predefined set of response prediction instructions; and
render (318) the set of content variants (222), and the weighted score and the persona response (224) corresponding to each of the set of content variants (222).

11. The system (100) as claimed in claim 10, wherein the processor (104) instructions, on execution, cause the processor (104) to:
create (304) the content generation prompt using the content information (218), the details of the target customer segment, and the predefined set of content generation instructions; and
input (306) the content generation prompt to the first LLM (214).

12. The system (100) as claimed in claim 10, wherein determining the weighted score for each of the set of content variants (222) comprises:
for each evaluation parameter of the set of evaluation parameters,
calculate an information value (IV) corresponding to the evaluation parameter based on a number of responders and a number of non-responders, wherein the number of responders and the number of non-responders are obtained from historical data of the set of response parameters; and
assign the IV as a corresponding weight to the evaluation parameter.

13. The system (100) as claimed in claim 12, wherein the processor (104) instructions, on execution, cause the processor (104) to modify the IV of the evaluation parameter based on a decay factor, wherein the decay factor is indicative of an impact of a previous IV on a current IV.

14. The system (100) as claimed in claim 10, wherein predicting (308), via the second LLM (216), the persona response (224) to each of the set of content variants (222) comprises:
create (312) the response prediction prompt using the set of content variants (222), the user-defined persona, and the predefined set of response prediction instructions; and
input (314) the response prediction prompt to the second LLM (216).

15. The system (100) as claimed in claim 10, wherein the processor (104) instructions, on execution, cause the processor (104) to:
retrieve (402) historical data of the set of response parameters corresponding to a plurality of customers;
cluster (404) the plurality of customers into a set of customer segments, wherein the target customer segment is one of the set of customer segments;
select (406) a representative customer from each of the set of customer segments; and
creating (408), via the second LLM (216), a set of personas corresponding to the set of customer segments based on the historical data of the set of response parameters of the representative customer, wherein the user-defined persona is one of the set of personas.

16. The system (100) as claimed in claim 15, wherein the processor (104) instructions, on execution, cause the processor (104) to:
for each customer segment of the set of customer segments,
retrieve (502) a real-world acceptance rate of the customer segment from the historical data;
determine (504) a predicted acceptance rate of the customer segment corresponding to the set of content variants (222); and
compare (506) the real-world acceptance rate with the predicted acceptance rate using a statistical test.

17. The system (100) as claimed in claim 16, wherein the processor (104) instructions, on execution, cause the processor (104) to, at least one of:
fine-tune (508) the second LLM (216) using the historical data of the customer segment based on the comparison; or
fine-tune (510) the second LLM (216) using a user feedback through reinforcement learning based on the comparison.

18. The system (100) as claimed in claim 16, wherein the processor (104) instructions, on execution, cause the processor (104) to, at least one of:
generate (512) a set of modified content variants for the customer segment based on the comparison; or
fine-tune (514) the first LLM (214) using a dataset based on a reinforcement learning, wherein the dataset comprises a content variant selected based on an optimal combination of the weighted score and one of the persona response (224) or a real-world persona response (224), a randomly selected content variant from remaining of the set of content variants (222), and the weighted score and the persona response (224) corresponding to each of the content variant and the randomly selected content variant.