{"id":4277,"date":"2016-06-30T08:00:18","date_gmt":"2016-06-30T15:00:18","guid":{"rendered":"https:\/\/www.kochava.com\/?p=4277"},"modified":"2024-03-20T10:30:20","modified_gmt":"2024-03-20T17:30:20","slug":"predicting-ad-click-rate-strategies-incorporating-state-dependence-random-effects","status":"publish","type":"post","link":"https:\/\/www.kochava.com\/ko\/blog\/predicting-ad-click-rate-strategies-incorporating-state-dependence-random-effects\/","title":{"rendered":"Predicting Ad Click-Through Rate: Strategies For Incorporating State Dependence and Random Effects"},"content":{"rendered":"[vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left” row_border_radius=”none” row_border_radius_applies=”bg” overlay_strength=”0.3″ gradient_direction=”left_to_right” shape_divider_position=”bottom” bg_image_animation=”none”][vc_column column_padding=”no-extra-padding” column_padding_tablet=”inherit” column_padding_phone=”inherit” column_padding_position=”all” background_color_opacity=”1″ background_hover_color_opacity=”1″ column_shadow=”none” column_border_radius=”none” column_link_target=”_self” gradient_direction=”left_to_right” overlay_strength=”0.3″ width=”1\/1″ tablet_width_inherit=”default” tablet_text_alignment=”default” phone_text_alignment=”default” column_border_width=”none” column_border_style=”solid” bg_image_animation=”none”][vc_column_text]Kochava brings\u00a0advertisers the best tools and intel. The following post from Aarki\u00a0dives deep into methodologies and benefits for CTR Prediction.\r\n\r\nAarki is a Kochava-certified network<\/a>. Click here to learn how to create an install campaign<\/a>\u00a0or\u00a0reengagement campaign<\/a>\u00a0within Kochava.\r\n\r\n
\r\n

The success of a mobile ad is determined by its ability to drive audience action — typically measured by the click-through rate (CTR). Accurate CTR prediction resulting from campaign actions is critical to the success of any mobile app advertising campaign.<\/em><\/p>\r\nAn ad can be deemed successful if it piques audience interest enough to incite them to interact with the call-to-action. To measure how well the ad does in capturing interest, ad click-through rate (CTR) is typically used. The higher the CTR, the more successful the ad is in generating interest amongst the target audience. In addition, CTR prediction can be helpful in setting campaign goals. The more accurate the prediction is, the better it can help advertisers set realistic expectations. This prediction can also be used to make better media buying decisions. Thus, the ability to accurately predict ad CTR is essential in mobile app advertising.\r\n

Benefits of Accurate Click Prediction<\/strong><\/h4>\r\nThis is especially true in a real-time bidding (RTB) situation, where the bid amount is derived from the predicted probability of a click, and it is not sufficient to simply rank ads ordinarily. The predicted click probability is used to determine which ad impressions to bid on and the bid amount. The accuracy of CTR prediction not only determines the placement of the ad but also the ad performance.\r\n\r\nThere are several methods that can be used to predict click probability.\r\n

Logistic Regression<\/strong><\/h4>\r\nA logistic regression model, and variants thereof, is commonly used to analyze the performance of ad campaigns. It is generally a common choice for predicting the probability of a binary (yes\/no) outcome based on a set of independent variables\u00a0X<\/span><\/span><\/span><\/span>X<\/span><\/span>.\r\n\r\nThis model assumes that for a set of coefficients \u03b2<\/span><\/span><\/span><\/span>\u03b2<\/span><\/span>, the dependent variable (y<\/span><\/span><\/span><\/span>y<\/span><\/span>) takes on the value 1 with probability given by\r\n\r\np<\/span><\/span>(<\/span><\/span>y<\/span><\/span>=<\/span><\/span>1<\/span><\/span>\u2223<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span>=<\/span><\/span>\u03c3<\/span><\/span>(<\/span><\/span>\u03b2<\/span><\/span><\/span>T<\/span><\/span><\/span><\/span>X<\/span><\/span>)<\/span><\/span><\/span><\/span>p(y=1\u2223\u03b2)=\u03c3(\u03b2TX)<\/span><\/span>\r\n\r\nRepresented differently,\u00a0y<\/span><\/span><\/span><\/span>y<\/span><\/span>\u00a0can be specified as a Bernoulli random variable with distribution\r\n\r\np<\/span><\/span>(<\/span><\/span>y<\/span><\/span>\u2223<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span>=<\/span><\/span>Bernoulli<\/span><\/span>(<\/span><\/span>\u03c3<\/span><\/span>(<\/span><\/span>\u03b2<\/span><\/span><\/span>T<\/span><\/span><\/span><\/span>X<\/span><\/span>)<\/span><\/span>)<\/span><\/span><\/span><\/span>p(y\u2223\u03b2)=Bernoulli(\u03c3(\u03b2TX))<\/span><\/span>\r\n

Naive Maximum Likelihood Estimation<\/strong><\/h4>\r\nIn the simplest case, \u03b2<\/span><\/span><\/span><\/span>\u03b2<\/span><\/span>\u00a0is chosen to maximize p<\/span><\/span>(<\/span><\/span>y<\/span><\/span>\u2223<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span><\/span><\/span>p(y\u2223\u03b2)<\/span><\/span>, the probability of observing the training data using maximum-likelihood estimate (MLE). This estimation approach assumes each observation to be an independent event and the model parameters to be constants.\r\n\r\nAnything we know about the model parameters a priori is ignored. While it is tempting to assume that we know nothing and let the model do the work, feeding prior knowledge into the model helps to minimize the generalization error.\r\n\r\nOne consequence of naive MLE is a tendency to overfit, i.e., exaggerate relatively small fluctuations in observed data. This can be mitigated by techniques such as regularization, but this leads to miscalibrated probabilities, which require additional adjustment.\r\n\r\nIt is also important to remember that the probability of success is a random variable and is influenced by various exogenous factors not included in the model. For example, the basic model excludes randomness in usage patterns that may have occurred over time. MLE only gives an average point estimate of this random variable. While this estimate is a useful measure of the central tendency, we cannot be certain that it is representative of the entire distribution for prediction purposes.\r\n

Maximum A Posteriori Estimation<\/strong><\/h4>\r\nA somewhat more sophisticated model estimation approach is to choose the coefficient vector \u03b2<\/span><\/span><\/span><\/span>\u03b2<\/span><\/span>\u00a0that maximizes the posterior probability p<\/span><\/span>(<\/span><\/span>\u03b2<\/span><\/span>\u2223<\/span><\/span>y<\/span><\/span>)<\/span><\/span><\/span><\/span>p(\u03b2\u2223y)<\/span><\/span>\u00a0using maximum a posteriori (MAP) estimate. This posterior probability is proportional to p<\/span><\/span>(<\/span><\/span>y<\/span><\/span>\u2223<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span>p<\/span><\/span>(<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span><\/span><\/span>p(y\u2223\u03b2)p(\u03b2)<\/span><\/span>.\r\n\r\nThis approach does incorporate our prior knowledge about the model parameters p<\/span><\/span>(<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span><\/span><\/span>p(\u03b2)<\/span><\/span>\u00a0and mitigates the overfitting issue. The result, however, is still a point estimate of the probability, albeit a more informed one that includes state dependence.\r\n

Bayesian Inversion Estimation<\/strong><\/h4>\r\nIdeally, we would like to get a picture of the full posterior distribution, one that is aware of both state dependence and random effects. This posterior distribution is given by Bayes\u2019 theorem\r\n\r\np<\/span><\/span>(<\/span><\/span>\u03b2<\/span><\/span>\u2223<\/span><\/span>y<\/span><\/span>)<\/span><\/span>=<\/span><\/span>p<\/span><\/span>(<\/span><\/span>y<\/span><\/span>\u2223<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span>p<\/span><\/span>(<\/span><\/span>\u03b2<\/span><\/span>)<\/span><\/span><\/span><\/span>p<\/span><\/span>(<\/span><\/span>y<\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>p(\u03b2\u2223y)=p(y\u2223\u03b2)p(\u03b2)p(y)<\/span><\/span>\r\n\r\nHowever, in all but the simplest of cases, this model needs to rely on numerical estimation such as the Markov Chain Monte Carlo (MCMC) method. This model estimation is able to capture both prior knowledge as well as random changes to the system in a robust manner. As a result, the team can make campaign decisions that are statistically sound and more trackable.\r\n

Experiment<\/strong><\/h4>\r\nTo illustrate the differences between the three approaches outlined above, we trained and tested three model specifications on a set of 2.5 million historical impressions spanning 23 campaigns and over 10,000 publishers.\r\n\r\nEach model was trained on a random 75% subsample of the dataset, and then tested over the remaining 25% using average element log-loss, and an R-squared statistic computed by averaging the true and predicted CTR for each campaign. The baseline case represents a \u201cmarket share\u201d model that predicts the average CTR from the training set for every impression in the test set.\r\n

Results<\/strong><\/h4>\r\nThe results of the analysis are given below.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
<\/td>\r\nAvg. Predicted CTR<\/strong><\/td>\r\nAvg. Error<\/strong><\/td>\r\nR-squared<\/strong><\/td>\r\nLog-loss<\/strong><\/td>\r\n<\/tr>\r\n
Baseline<\/td>\r\n0.099200<\/td>\r\n5.01<\/td>\r\n-0.02<\/td>\r\n0.312732<\/td>\r\n<\/tr>\r\n
Naive MLE<\/td>\r\n0.098258<\/td>\r\n4.01<\/td>\r\n0.53<\/td>\r\n0.298476<\/td>\r\n<\/tr>\r\n
MAP<\/td>\r\n0.095153<\/td>\r\n0.71<\/td>\r\n0.67<\/td>\r\n0.221826<\/td>\r\n<\/tr>\r\n
Bayesian Inversion<\/td>\r\n0.094219<\/td>\r\n-0.26<\/td>\r\n0.81<\/td>\r\n0.211702<\/td>\r\n<\/tr>\r\n
Actual<\/td>\r\n0.094467<\/td>\r\n0.00<\/td>\r\n1.00<\/td>\r\n0.000000<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n \r\n

Summary<\/strong><\/h4>\r\nThe Naive MLE model results in the least accurate predictions. Model fit metrics improve significantly when we include state dependence in model estimation. In particular, the MAP estimate, which incorporates a shrinkage prior on the regression coefficients as a form of regularization, results in a significantly better model, which generalizes better to the test set than MLE.\r\n\r\nThe most accurate estimate, however, is the Bayesian Inversion model. The high accuracy of this estimate can be credited to the fact that the model is aware of both state dependence and random effects.\r\n\r\nThis analysis indicates that choice of model specification and estimation methodology can have a significant impact on the accuracy and robustness of the prediction. This, in turn, impacts campaign performance and the velocity of achieving optimal results.\r\n\r\n
\r\n\r\nAarki is a Kochava-certified network<\/a>. Click here to learn how to create an install campaign<\/a>\u00a0or\u00a0reengagement campaign<\/a>\u00a0within Kochava.[\/vc_column_text][\/vc_column][\/vc_row]","protected":false},"excerpt":{"rendered":"

[vc_row type=”in_container” full_screen_row_position=”middle” column_margin=”default” column_direction=”default” column_direction_tablet=”default” column_direction_phone=”default” scene_position=”center” text_color=”dark” text_align=”left”…<\/p>\n","protected":false},"author":1,"featured_media":9244,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[27,11,201],"tags":[],"class_list":{"0":"post-4277","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-education","8":"category-news-and-updates","9":"category-product-updates"},"_links":{"self":[{"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/posts\/4277","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/comments?post=4277"}],"version-history":[{"count":7,"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/posts\/4277\/revisions"}],"predecessor-version":[{"id":45763,"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/posts\/4277\/revisions\/45763"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/media\/9244"}],"wp:attachment":[{"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/media?parent=4277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/categories?post=4277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kochava.com\/ko\/wp-json\/wp\/v2\/tags?post=4277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}