Granthara03.03.06Where to source clean data, and how to verify it

03.03.06 · Where to source clean data, and how to verify it

PreviouslyLook-ahead bias is data used on a date it was not yet public.

Six checks, and who to believe.

Five lessons of ways a number can be wrong while looking perfectly reasonable. The useful response is not suspicion of everything. It is knowing which body published the number first, and owning six checks that take a morning once and a minute thereafter.

A source that will not state its adjustment policy is not a source. It is a rumour with decimal places.

Updated

6
Checks that settle whether a series can be measured
Part 03 · Data Literacy: Trusting What You See
  1. 03.03.01 Adjusted vs unadjusted prices
  2. 03.03.02 Split, bonus, and dividend adjustment errors
  3. 03.03.03 Continuous vs contract data (futures)
  4. 03.03.04 Survivorship bias in historical screens
  5. 03.03.05 Look-ahead bias: the mistake you cannot see
  6. 03.03.06 Where to source clean data, and how to verify it
0 of 6 read
Learn

Who publishes the number first

Every figure in this Part has an origin, and it is never an app. The exchanges publish the prices: NSE and BSE each put out a daily record of every scrip's open, high, low, close and volume, along with the corporate-action record that fixes every ex-date. That is the statement against which any price history is checked.

The company publishes the fundamentals, on the clock 01.08.02 set out, through the exchange's corporate-announcements feed and its own website. The registrar and the depositories hold the entitlement detail behind a corporate action.

Everything else — a broker's terminal, a screener, a charting site, a data vendor — is a copy of these with processing applied, and the processing is exactly what chapter 03.03 has been about. A copy is not worse than the original. It is just not the thing you reconcile to.

See it

The six checks

Run these once on any series before you measure anything with it. Tap each.

These check whether a series can be measured. None of them says anything about whether a company is worth owning.

Learn

What clean does and does not mean

Clean data is not accurate prediction and it is not a good company. It means one narrow thing: the series says what it claims to say, so a number measured from it is a measurement rather than an artefact.

It is also not a one-off state. A source that passed last year can change its adjustment policy, its vendor or its coverage without telling anyone, and a series that was complete can acquire a gap. The six checks are cheap enough to repeat when something surprises you.

And when a check fails, the answer is rarely to abandon the source. It is to know the failure and work around it: use the raw series and adjust yourself, or exclude the affected span, or carry the difference as a stated uncertainty. A known flaw is usable. An unknown one is not.

Where each kind of number comes from first
Daily open, high, low, close and volume
the exchange's own daily record
Ex-dates, ratios and entitlements
the exchange's corporate-action record
Results, shareholding, material events
the company, filed to the exchange (01.08.02)
Index levels and constituents
the index provider's published series
Anything in an app or a screener
a processed copy of the above
freewhat every primary source above costs to read
1check that needs no second source
Try it

Two reputable sources show different five-year returns for the same company. Both handle its split correctly. What is the most likely explanation?

Try it

Primary source, or a processed copy?

The exchange's daily record of a scrip's OHLC and volume
A charting site's adjusted price history
A company's results as filed to the exchange
A screener's list of companies meeting a condition
The exchange's corporate-action record of an ex-date
A news article quoting a quarter's profit

What Part 03 was actually for

A chart cannot be read until four things are settled: the chart type chose what to keep (03.01), the scale and frame chose how it is drawn (03.02), the data decided whether the numbers deserve drawing at all (03.03), and the workspace saves all of it so the decisions are made once rather than in the moment (03.04). None of it forecasts anything. All of it determines whether what you are looking at is worth being surprised by.

The instrument is now built and checked. What remains is the thing it was built to show: a single mark, four numbers wide, and everything people claim to read in it.

Remember

The exchange publishes the prices and ex-dates and the company publishes its results; everything else is a processed copy — and six checks, starting with one known corporate action, settle whether a series says what it claims to say before any number is measured from it.

Everything this chapter has told you, in order

  1. 03.03.01A bonus leaves two true histories — the prices that traded and the prices restated into today's units — and neither announces itself.
  2. 03.03.02A fall of exactly one half on a quiet day is arithmetic, not news: the ex-date is the only boundary an adjustment may use.
  3. 03.03.03Every price on a continuous futures chart is real; the line joining them is not, and its old levels never traded.
  4. 03.03.04Today's list was selected by history for surviving, so a test on it cannot fail on a company that no longer has a name.
  5. 03.03.05Every number is correct and the result improves, so the only tell is that it collapses once the lag is added.
  6. 03.03.06The exchange publishes the prices and the company its results; everything else is a processed copy, and six checks tell you how it was processed.
Next
03.04.01 — A minimal, repeatable chart template

Next: save every decision this Part has made, so you make them once.