What to know before buying web data.
Reference material we'd want a client to have, including the parts that make our own job harder to sell. All of it is on this page, no gated downloads, no sign-up.
The gate every batch passes before it leaves us.
- Type conformanceEvery field matches its declared type
- Range and sanityValues fall inside agreed bounds
- Null-rate thresholdsCoverage flagged when it drops
- Duplicate resolutionResolved against your chosen key
- Schema driftSource structure changes caught early
- Volume varianceUnexpected record counts halt delivery
Six questions to ask any data vendor.
Including us. The answers tell you more about how a supplier operates than any capability list will.
What happens when a source changes its layout?
This is the single biggest cost in web data and the question most vendors answer vaguely. You want a specific answer about detection and repair, not reassurance that it 'rarely happens'.
“We detect it through validation failures and volume variance, and repairing it is included.”
“Our scrapers are very robust.”
What does a delivery actually contain?
A dataset without quality metadata is a dataset you have to audit yourself. Ask what ships alongside the records.
“Records plus a report with coverage, null rates per field, and duplicate counts.”
“A CSV file.”
Which of my sources won't work, and why?
Any vendor who says all your sources are fine hasn't looked at them. Some sites genuinely can't be collected reliably.
“These four are straightforward, this one paginates unreliably, and we'd advise against that one.”
“We can handle any website.”
What's your null policy per field?
Whether a missing price arrives as null, zero, or an empty string determines how much cleanup you write downstream.
“Agreed per field during scoping and enforced by validation.”
“We deliver whatever's on the page.”
Who do I talk to when something's wrong?
The distance between you and the person who can actually fix a pipeline predicts how long problems take to resolve.
“The engineer who built it.”
“Submit a ticket and we'll route it.”
What's the legal basis for collecting this?
You inherit the compliance posture of your data supplier. A vendor who has never thought about it is a liability on your side.
“Publicly accessible data, rate-limited collection, and here's what we won't take on.”
“Everyone does this, it's fine.”
When building it yourself is the right call.
Sometimes it is. If collection is core to what you sell, owning it is usually correct, and we'll say so rather than talk you out of it.
Building in-house
- Initial build
- Days to weeks per source, depending on complexity
- Ongoing maintenance
- The real cost. Sources change continuously
- Infrastructure
- Proxies, headless browsers, scheduling, monitoring, storage
- Quality tooling
- Validation and drift detection, built separately
- Key-person risk
- Usually one engineer who understands the whole thing
- Makes sense when
- Data collection is core to your product, not a supporting input
Managed collection
- Initial build
- Scoping and sample, then a live pipeline
- Ongoing maintenance
- Included. It's what you're paying for
- Infrastructure
- Ours, not on your cloud bill
- Quality tooling
- Validation suite runs before every delivery
- Key-person risk
- Moves off your team
- Makes sense when
- You need the data to be reliable, not to own the pipeline
Six things to measure on every delivery.
Useful whether the data comes from us, another vendor, or your own pipeline. If your supplier doesn't report these, measure them yourself.
Coverage
What percentage of expected records actually arrived? A drop from 99% to 82% usually means a source changed, not that the market did.
Null rate per field
Track it per field, not per dataset. One field silently emptying is invisible in an aggregate number.
Duplicate rate
Duplicates inflate counts and skew averages. Check what key deduplication runs on, and whether it's the key you'd have chosen.
Type conformance
Prices as strings, dates in mixed formats, and booleans as 'Yes'/'No' are the most common causes of downstream cleanup.
Freshness
When was each record actually collected? A daily delivery containing week-old data is a daily delivery of stale data.
Volume variance
A batch 40% smaller than yesterday's is a signal, not a coincidence. Variance thresholds should halt delivery, not pass it through.
Picking a format that fits your stack.
Where we draw lines.
Public means publicly accessible
We collect data available to any visitor without logging in. Working around authentication to reach data that isn't public is a line we don't cross, regardless of what a project is worth.
Personal data needs a basis
If a dataset would contain personal data, there needs to be a lawful basis for collecting and processing it. We'll raise this during scoping rather than after delivery.
Rate limits are respected
Collection is paced so it doesn't degrade the source. Beyond being the right thing to do, aggressive collection is also the fastest way to get a pipeline blocked.
Terms of service vary
Site terms, jurisdiction, and intended use all matter, and the picture differs by case. We'll tell you what we're comfortable with; your legal team owns the final call for your use.
This is how we approach collection, not legal advice. Your own counsel should assess your specific use case, jurisdiction, and the terms of the sources involved.
Terms that come up in scoping.
- Schema drift
- When a source changes its structure and extraction silently starts producing wrong or empty fields.
- Normalization
- Converting inconsistent source values (currencies, units, date formats, category names) into one agreed representation.
- Deduplication key
- The field or combination of fields used to decide whether two records describe the same thing.
- Coverage
- The share of expected records that a delivery actually contains, measured against a known or estimated total.
- Cadence
- How often a pipeline runs and delivers, from quarter-hourly to monthly.
- Provenance
- The source URL and collection timestamp carried on each record, so any value can be traced back and re-checked.
- Backfill
- Collecting historical data to fill a period before a pipeline went live. Often only partially possible.
- Rate limiting
- Deliberately pacing requests so collection doesn't place unreasonable load on a source.
Got a question this didn't answer?
Ask it directly. We'd rather give you a straight answer about feasibility than have you guess from a resources page.
No retainer required to find out whether your sources are feasible.