Almost every build-vs-buy conversation about web data starts from a number that is real but irrelevant: how long the first scraper took. It took an afternoon. It genuinely did. That is not the number the decision should be made on.
Here is what the rest of the invoice looks like, from having built and run this for other people.
The line items nobody quotes
| Line item | What it actually is |
|---|---|
| Initial build | Days to weeks per source. The only part anyone estimates. |
| Maintenance | The real cost. Sources change continuously and repairs are unplanned work. |
| Proxy infrastructure | Residential or mobile pools, rotation logic, and the bill that scales with volume. |
| Headless browsers | Compute that is an order of magnitude more expensive than plain HTTP. |
| Scheduling & retries | Backoff, partial-failure recovery, idempotency. A distributed systems problem. |
| Validation tooling | Type, range, null-rate and drift checks. Built separately, or you have no idea what shipped. |
| Storage & delivery | Formats, compression, transfer, verification. |
| On-call | Because a daily 06:00 window fails at 06:00, not at 10:00. |
| Key-person risk | Usually one engineer understands the whole thing. Then they take a holiday. |
Read that list back as a job description and it is a full-time infrastructure role, one that is entirely orthogonal to whatever your product actually does. That is the honest comparison: not "an afternoon versus an invoice", but "a standing engineering commitment versus an invoice".
The four-month cliff
There is a fairly consistent pattern to in-house scraping projects, and the timeline is unkind.
- Weeks 1–2: it works, it is cheap, everyone is pleased
- Weeks 3–8: two sources break, one engineer becomes the scraping person
- Months 3–4: coverage has quietly degraded, nobody is sure when, and trust in the dataset drops
- Month 5 onwards: either it gets real ownership and real budget, or it gets abandoned and quietly replaced
The cliff is not technical. It is that maintenance work is invisible when it succeeds and urgent when it fails, which makes it the first thing deprioritised and the worst thing to deprioritise.
When building it yourself is the right call
This is not a post that concludes you should always buy. There are cases where in-house is clearly correct, and we say so during scoping rather than after an invoice:
- Collection is core to your product, not an input to it. If the extraction is the thing you sell, own it.
- One stable source, low cadence. A weekly pull from an API that has not changed in three years does not need a vendor.
- You already run the infrastructure. If proxy pools and headless fleets are already yours for another reason, the marginal cost is genuinely low.
- The data is so sensitive that no third party is acceptable. That is a legitimate constraint and it settles the question.
What buying should get you
Buying badly is worse than building. A vendor who hands over a dataset and disappears has sold you the afternoon, not the two years. What a managed pipeline should mean, concretely:
- 01A schema agreed in writing before extraction is written
- 02Repair when a source changes: included, not re-quoted as new work
- 03Validation reports per batch, so you can see what shipped rather than trust that it did
- 04Your data in open formats, yours to keep if you leave
- 05A named engineer, not a ticket queue
If a quote does not cover those five, it is a build project with extra steps.
This is how we work, not a pitch. If you want the same thing run for your sources, we'll scope it and send a real sample from your own targets before you commit to anything.