Anyone can write a scraper. Point a parser at a page, pull the fields you want, write them to a file. A competent engineer does that before lunch. The demo works, the CSV looks right, and the project feels finished.
It isn't finished. It has barely started. The interesting engineering in web data is not extraction, it is staying correct while the thing you are extracting from changes underneath you, every week, forever, without telling you.
Our longest-running pipeline has now delivered every single day for over 600 consecutive days. Here is what actually breaks over that span, roughly in the order you meet it.
Week 2: the markup moves
This is the failure everyone predicts, and it is the least dangerous one. A class name changes, a wrapper div appears, a product card gets restructured. Your selector returns nothing, your parser throws, and your alerting tells you within the hour.
It is easy to fix and easy to notice, which is exactly why it is not the problem. Loud failures get fixed. The dangerous ones are quiet.
Month 2: a field goes quietly null
The site adds a second price element for a promotional variant. Your selector still matches. It just matches the wrong one now, or matches nothing while the surrounding structure stays valid. No exception is raised. The row still writes. The file still lands. The transfer still succeeds.
And every dashboard downstream is now subtly wrong, in a direction nobody will question, until somebody eventually notices a number that cannot be right and traces it back three months.
The defence is not better selectors. It is refusing to trust a delivery that has no error in it. Every batch should be compared against what the last batch looked like:
- Null rate per field, with a threshold that halts delivery rather than warning about it
- Record count variance against a rolling window. A 40% drop is a failure even when nothing threw
- Type conformance, so a price that starts arriving as a string is caught at the boundary
- Distribution sanity: if yesterday's median price was 129 and today's is 1.29, someone changed a decimal separator
None of these are clever. All of them are the difference between a dataset you can build on and one you have to re-verify by hand.
Month 4: the shape of the block changes
Bot management is not a wall you get over once. Cloudflare, DataDome and Akamai are continuously tuned services, and the thing that worked in March is a fingerprint by June. What breaks is rarely a hard block. It is a slow degradation: more challenges, more retries, longer runs, and a collection window that starts creeping past the time you promised the data would land.
The signal to watch is not the error rate. It is the run duration. A pipeline whose 40-minute window has quietly become 90 minutes is a pipeline that will miss its slot within a month, and it will have been telling you so the whole time.
Month 6: pagination lies
Deep pagination is where a lot of long-running collection quietly loses coverage. Result sets shift between page requests, cursors expire mid-run, and page 340 of a 500-page walk returns items you already have while the ones you needed slid past. Your record count looks healthy. Your coverage is not.
The fix is to stop treating a run as a sequence of pages and start treating it as a set of identities. Deduplicate on a stable key, then measure what fraction of the expected key space you actually saw. That number, not the row count, is your coverage.
Month 9: geography and time drift in
Prices differ by region, availability differs by warehouse, and a lot of sites will happily serve a different answer to the same URL depending on where the request appears to come from. A pipeline that started in one market and quietly acquired a proxy pool in another is now mixing two datasets that look identical and are not.
Record the vantage point with the observation. If a row does not carry the market it was collected from and the timestamp it was collected at, you cannot reconstruct what happened later, and eventually you will need to.
The thing that does not break
Something worth saying, because it cuts against the rest of this post: transfers are the reliable part. Across 600+ days of scheduled delivery, our failed-transfer count is zero. Moving a file from one place to another, verified byte-for-byte, is a solved problem.
That is worth being precise about, because "zero failed transfers" and "zero bad records" are very different claims, and vendors are not always careful about which one they are making. The first is table stakes. The second is the actual work, and nobody can honestly promise it, what they can promise is that every batch is checked before it reaches you, and that a batch which fails those checks does not ship.
What 600 days actually requires
Not heroics. A short list of unglamorous commitments, held for a long time:
- 01A schema agreed in writing before any extraction is written, so there is something to validate against
- 02Validation that halts delivery rather than logging a warning nobody reads
- 03Monitoring of content, not just of uptime, the run that succeeds is the one to check
- 04Run duration tracked as a first-class metric, because degradation shows up there first
- 05A named person who owns the pipeline, so a source change is repaired rather than discussed
The first version really does take an afternoon. The 600th delivery takes all of the above, every week, for two years. That gap is the entire business.
This is how we work, not a pitch. If you want the same thing run for your sources, we'll scope it and send a real sample from your own targets before you commit to anything.