How Proofseal checks

The road a measurement question travels

Every question goes through the same steps. What each step did, and what it was based on, is recorded.

1. 1DataWeekly spend and outcomes come in and are saved as a versioned copy of the data
2. 2ChecksBefore modeling, confirm the data can answer the question
3. 3MeasurementEstimate each channel's effect, with a likely range around every number
4. 4ValidationTest the model on simulated data with known answers to see how accurate its estimates are
5. 5VerdictVerdict rules written in advance turn the evidence into a verdict, with reasons and limitations

Four groups of checks

The checks fall into four groups. Each group is a set of concrete questions you can read one by one, and none of it is reduced to a single score.

Data checks

How many weeks of data there are, which weeks are missing, whether spend actually goes up and down, which channels always rise and fall together, and which have had no spend for a long time. If the data can't answer a question, the later steps say so.

• History length and missing weeks
• Spend variation per channel
• Whether channels rise and fall together
• Zero-spend periods

Model diagnostics

After the model runs, a few basic numbers are checked first: how much it explains in a period it wasn't trained on, whether channels can be told apart, how many times the ranges were recomputed, and whether seasonal swings were removed. Each number shows its unit, its threshold and where the threshold comes from.

• Fit on a period not used for training
• Whether channel effects can be told apart
• Whether the ranges are computed stably
• Whether seasonal swings are accounted for

Simulation validation

The model is rerun on simulated data that resembles your data's conditions. Each channel's answer in the simulation is set by us, so it is known. Three things are checked: when the model says it is 90% sure, how often its range contains the known answer; how far each channel's estimate is from the known answer; and whether the channel ranking is correct. The next section lists which settings come from your data and which are assumptions.

• Stated confidence vs actual hit rate
• How far off each channel's estimate is
• Whether the channel ranking is right
• Simulation conditions matched to your data

Verdict rules

Verdicts come from rules written in advance, not from a person deciding on the spot. Every rule states where its threshold comes from, and the same evidence under the same rule version always gets the same verdict. Verdicts are never edited; updated data or rules produce new verdicts.

• Rules have version numbers (currently v0.1.0)
• Every threshold names its source
• Verdicts are immutable
• Re-runs are comparable, before against after

How the simulation test is set up

The simulated data is set partly from your data's conditions and partly from our assumptions. The two are listed separately below, and the verdict section shows the values actually used each time.

So this test cannot tell you

• · How much your ads actually brought in: real data has no known answer
• · How the model performs outside these assumptions, such as with carryover, saturation or promotions

Current verdict thresholds (v0.1.0)

Apart from the weeks of data (based on Meridian's data guidance) and channel correlation (product policy), these thresholds are provisional and not yet validated by research. They will be updated with a new version when the research simulations are done, and never adjusted to make verdicts look better.

• · At least 26 weeks of data; 52 or more recommended
• · Share of ranges containing the known answer: 70% or more for Supported, below 45% for Not supported
• · Average gap from known answers: within 15% for Supported, above 40% for Not supported
• · Channel comparison: ranking agreement of 0.7 or more and the best channel picked in 75% or more of runs; direction of a shift: 60% or more
• · Spend correlation of 0.8 or more between two channels is listed as a limitation

Where the thresholds come from

Every threshold, such as "must be above 0.9", names its source. There are seven kinds of source:

Literature

Values recommended by published research

Standards

Industry or methodological standards

Model authors

Usage advice written by the model's developers

Simulation tests

Values measured on simulated data where the answer is known

Empirical

Experience values from real data

Product policy

A conservative trade-off the product states openly

Provisional

Not enough grounding yet, so marked provisional

The verdict vocabulary is fixed

Every measurement question ends with one verdict. Verdicts use only these words, and their meaning doesn't change:

• SupportedGiven the current data, methods and assumptions, the evidence supports this conclusion
• ConditionalThe evidence offers support, with a few things to watch; the verdict lists each one
• Not supportedThe current evidence does not support this conclusion; the verdict explains why
• More evidence neededThere isn't enough evidence to conclude yet; the verdict says what is missing and what would help most
• Out of dateThe data or conditions have changed, so the earlier verdict no longer applies and needs re-validating

How regional experiments are judged

Besides the model, a regional experiment gives a separate estimate of the ads' effect. Proofseal analyzes it with synthetic control; the verdict rules are currently provisional v0.1.0:

1. 1At least 8 weeks of data before the experiment and at least 3 unchanged regions to compare; otherwise the verdict is More evidence needed.
2. 2Before the experiment, the comparison group must stay within 10% of the test region on average for the result to be used; within 5% counts as a tight match.
3. 3The test region's gap must exceed the natural drift of every unchanged region, with at least 9 regions to compare, for a Supported verdict. With fewer regions the verdict is at most Conditional.

Where we differ from the original paper: we first divide each region by its own pre-experiment average and then build the comparison group, so regions of very different size can be compared. The original paper combines the raw numbers. This adjustment means the result reads as a percentage change rather than an amount.

When a verdict needs redoing

A verdict records the data version, method version and checks it was made with. After the data updates, the market changes, or the verdict rules are revised, the earlier verdict may no longer apply.

After data is updated, the verdict section notes that the current verdicts rest on earlier data; pressing "Recompute verdicts" produces new ones and keeps the old ones for comparison. Automatically marking old verdicts as Out of date is still in development.

Where the methods come from

Proofseal's methods come from published research; the verdict thresholds and some adjustments are our own and are marked provisional. The main references are listed below, with where each is used in Proofseal.

1. Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California's Tobacco Control Program. Journal of the American Statistical Association, 105(490).↗

   Used for: How regional experiments are analyzed (synthetic control) and the placebo test

2. Abadie, A. (2021). Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects. Journal of Economic Literature, 59(2).↗

   Used for: When synthetic control fits and what data it needs

3. Vaver, J., & Koehler, J. (2011). Measuring Ad Effectiveness Using Geo Experiments. Google Inc.↗

   Used for: Designing and analyzing geo experiments for advertising

4. Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The Augmented Synthetic Control Method. Journal of the American Statistical Association, 116(536).↗

   Used for: An extension of synthetic control, the approach GeoLift uses

5. Meta Open Source. GeoLift: an end-to-end solution for geo experiments (R package).↗

   Used for: Planning geo experiments: choosing regions, estimating detectable effects

6. Jin, Y., Wang, Y., Sun, Y., Chan, D., & Koehler, J. (2017). Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects. Google Inc.↗

   Used for: Bayesian marketing mix modeling, the basis of Meridian

7. Google. Meridian: an open-source marketing mix modeling framework.↗

   Used for: One of the models we measure with

8. Morris, T. P., White, I. R., & Crowther, M. J. (2019). Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11).↗

   Used for: Testing statistical methods on simulated data with known answers (the ADEMP framework), the basis of model validation

Links go to the official journal or publisher pages; some journals need a subscription for the full text, and university or company libraries usually provide free access. These references show where the methods come from; they do not mean the authors, Google or Meta endorse Proofseal.

Thanks to open source

Proofseal relies on published research and open-source projects. Measurement uses Google's Meridian, regional experiment planning draws on Meta's GeoLift, and the system itself is built on Next.js, NestJS, FastAPI, NumPy, PostgreSQL and other open-source projects. Thank you to the teams who share their work openly.