aditagar/ads-ml-lab-week03

ads-ml-lab — Week 03: Calibration

The data for week 03 of ads-ml-lab, a twelve-week course that rebuilds the core model classes of ads measurement from scratch on public data. This repo exists so the leaderboard can serve downloads from a CDN instead of from a small VM.

This is a derived dataset, not a copy. It is built by `tools/prepare_data.py` from `criteo/criteo-attribution-dataset`.

The task

Predict, per impression, the probability that the user converted within 30 days.

conversion is an impression-level label meaning "the user who saw this impression converted within 30 days" — not "this impression caused the conversion". Every impression in a converting journey carries a 1. It is a legitimate CVR target; it is not attribution.

Files

impression_id is assigned during preparation and has no meaning upstream. It is the join key for scoring.

How it was built

• —Time-ordered split. The boundary is the 70th percentile of timestamp, taken before subsampling. Subsampling first would let a test row predate a train row. Train is strictly earlier than test; there is no overlap and no shuffling. A random split on this data inflates AUC by an amount that looks like progress.
• —Subsampled to 1,500,000 train and 400,000 test rows so a submission is a few MB rather than fifty.
• —Leaky columns removed from the test file: conversion, conversion_timestamp, conversion_id, cpo, and attribution. That last one is the subtle one — it is Criteo's own last-click flag and is nonzero only on converting journeys, so shipping it would hand over the label under a different name.
• —The held-out labels are not in this repo.

Evaluation

The leaderboard ranks on normalised entropy (log-loss relative to a base-rate predictor), not AUC. AUC is invariant to monotone rescaling, so a model calibrated three times too high scores identically on AUC and would overbid every auction threefold. NE is a proper scoring rule and sees both ranking and calibration.

Base rate is about 4.9%, and it drifts slightly downward across the window — the test half converts a little less often than the train half. That drift is part of the problem.

Licence and attribution

CC-BY-NC-SA 4.0, inherited from the upstream dataset. Non-commercial use only, and derived works must carry the same licence.

@inproceedings{DiemertMeynet2017,
  author    = {{Diemert Eustache, Meynet Julien} and Galland, Pierre and Lefortier, Damien},
  title     = {Attribution Modeling Increases Efficiency of Bidding in Display Advertising},
  booktitle = {Proceedings of the AdKDD and TargetAd Workshop, KDD, Halifax, NS, Canada, August, 14, 2017},
  year      = {2017},
  publisher = {ACM}
}