When a first delivery lands and disappoints, the extraction is almost never the reason. The rows are there. The scraper worked. The problem is that two people had different pictures of what a column meant, and neither wrote it down.
A schema agreed up front costs an hour and prevents most of that. Here is what it needs to contain.
Four things every field needs
Not a type. Four things. A type on its own is the smallest and least useful part of a field definition.
| What | Why it matters |
|---|---|
| Name and type | `price` as decimal(10,2), not float. Money in a float is a rounding bug waiting for volume. |
| Unit | Currency, timezone, measure. `price: 129` is meaningless. `129 USD` is a value. |
| Null policy | Does an absent value mean not offered, not found, or not collected? These are three different facts and they need three different representations. |
| Key | What makes a row unique. Without it, deduplication is guesswork and every count you produce is approximate. |
A field defined on all four is one a downstream system can consume without a human interpreting it. A field defined on one is a support conversation.
What a good definition looks like
{
"sku": { "type": "string", "key": true },
"title": { "type": "string", "nullable": false },
"price": { "type": "decimal(10,2)", "unit": "currency" },
"currency": { "type": "iso4217" },
"in_stock": { "type": "boolean", "null_means": "not_stated" },
"collected_at": { "type": "timestamp", "unit": "UTC" },
"market": { "type": "string", "note": "vantage point of collection" }
}Two fields there are worth pointing at, because they are the ones most often missing and most often regretted.
collected_at, not scraped_at
The time the observation was true, in a stated timezone, on every row. Without it you cannot build a history, reconcile two sources, or answer the question "was this already wrong on Tuesday?", and you will be asked that question.
market, or whatever your vantage point is
A great deal of the web answers the same URL differently depending on where the request came from. If a row does not record where it was collected from, two rows that disagree are indistinguishable from one row that changed.
The questions that surface the disagreements
These are the ones we ask during scoping, because each has caught a real mismatch:
- 01When a product has three prices on the page (list, promotional, checkout) which one is `price`?
- 02If a field is missing on the page, do you want a null, an omitted row, or the previous known value?
- 03Is a variant a separate row, or a nested field on the parent?
- 04What should happen if today's batch has 40% fewer rows than yesterday's: ship it, or halt?
- 05Do you want history, or current state? They are different products and only one of them can be answered later.
Sign it before anyone writes extraction
The order matters and it is not just process hygiene. A schema agreed first is something the pipeline can be validated against: every batch checked for type conformance, null rates and drift before it ships. A schema inferred from whatever the scraper happened to produce validates nothing, because it is a description of the output rather than a contract for it.
An hour on the schema, before a line of extraction logic exists. It is the cheapest hour in the whole project.
This is how we work, not a pitch. If you want the same thing run for your sources, we'll scope it and send a real sample from your own targets before you commit to anything.