Model Validation

Each time the model is built for each campaign, the model is validated before it is used on live traffic.

This validation step ensures that the new model does not overfit to the training data. To ensure that the model is indeed predictive of new, unseen traffic, instead of simply describing what happened before, the model is tested. First, the raw training data traffic is split into two sections. Most goes into building the model, and the model runs against the rest of the raw traffic. The correlation between the actual response values from the test group is measured against the model's scores for those records. The new model must pass certain thresholds to become the new model for the campaign.