The web you crawl is not the web you think
Most scraping failures are not caused by clever defenses but by mismatched assumptions. The public web is overwhelmingly dynamic and layered with assets, redirects, and templates that shift without notice. JavaScript runs on about 98% of websites, jQuery still appears on roughly three quarters of them, and close to 43% of all sites are powered by a single CMS family. Add to that the modern reality that well over 90% of page loads use HTTPS, and you get a practical picture: scrapers must behave like real browsers, negotiate encrypted sessions cleanly, and cope with template-driven markup that moves around more than hand-coded pages.
A typical desktop page also pulls in a large bundle of resources. Expect roughly 70 network requests and about 2 MB transferred for a median page, with scripts and images dominating. That matters for capacity planning: if your collector opens thousands of pages per minute, you are effectively coordinating hundreds of thousands of downstream requests. Bandwidth, connection pooling, and retries are not afterthoughts; they are the backbone of delivery.
What the numbers imply for scraper architecture
Render strategy: With near-universal JavaScript, assume you will need headless rendering for at least a portion of targets. A hybrid model works well: attempt fast HTML-first extraction, promote domains to a headless queue only when selectors miss required data.
Selector design: CMS-heavy ecosystems produce recurring markup patterns. Using layout-agnostic selectors (data attributes, JSON-LD blocks, or stable IDs) reduces breakage compared to brittle nth-child chains tied to cosmetic structure.
Network efficiency: Page weight and request counts push you to aggressive caching. Cache static assets by content hash. Use HTTP/2 or HTTP/3 clients to multiplex requests and minimize connection overhead.
Compression and parsing: Because most traffic is compressed, ensure your fetch layer supports gzip and brotli. Parse compressed JSON and streaming HTML to start extraction before full download completes.
Protocol hygiene: With encrypted traffic dominant, align TLS settings with modern servers, reuse connections, and respect HTTP cache headers to avoid unnecessary fetches.
Measure scraper quality with three simple metrics
Coverage: the share of intended pages successfully fetched and parsed. Track coverage by domain and by template. When coverage dips, check whether a template changed, a script-gated section appeared, or a login expired.
Freshness: the lag between a source change and your stored copy. On dynamic catalogs, aim for hours, not days. You can approximate change probability by monitoring ETag or Last-Modified headers and focusing crawls where deltas are highest.
Fidelity: the percentage of required fields captured without manual correction. Validate fields against type and range rules, and flag spikes in nulls or defaults as a signal that a selector needs attention.
Practical workflow that holds up under change
Start discovery with a lightweight pass to map pagination, detail pages, and any in-page JSON. Schema markup often carries clean fields; harvesting JSON-LD shortens post-processing. For fast prototyping of selectors, a browser helper like instant data scrapper can accelerate the first mile before you codify a production spider.
Promote promising patterns into a modular extractor: one fetcher, multiple parsers keyed by URL rules, and a validator that enforces required fields. Keep render decisions policy-driven rather than hardcoded per site. For example, if a page returns empty on a static fetch but includes script tags and no critical content, escalate to headless automatically and record the reason.
Resilience tactics that pay for themselves
Change detection: Store compact hashes of the DOM regions you rely on. If a region’s hash shifts while the rest is stable, you likely have a cosmetic change; if everything shifts, expect a template revamp or A/B test.
Retry budgets: Separate network retries from parser retries. Network errors tend to be bursty; parser errors cluster around template changes. Throttle network retries quickly; send parser errors to a canary queue for human review.
Robots and rate limits: Read and respect robots.txt and crawl-delay directives. Even modest pacing often improves reliability by avoiding auto-mitigation triggers.
Data sanity checks: Validate IDs, prices, dates, and currencies at the edge. Reject impossible values early rather than letting them poison downstream analytics.
Observability: Capture per-domain latency, bytes, request counts, and status codes. Alert on shifts in median or p95 latency, not just hard failures.
Turning raw HTML into dependable data
After extraction, standardize fields and deduplicate with stable keys. Normalize units, currencies, and encodings immediately. Maintain a simple lineage record: source URL, fetch timestamp, parser version, and transform version. This thin layer makes rollbacks and audits trivial when a site changes or a mapper ships a bad rule.
Finally, keep humans in the loop where they add leverage. A short daily review of low-fidelity samples uncovers silent failures fast. Pair that with automatic backfills when a parser is fixed, and your pipeline recovers without scrambling.
Scraping at scale is less about brute force and more about engineering to the web you actually face: dynamic, encrypted, template-driven, and chatty. If your design reflects those realities, reliability stops being a guessing game and becomes a set of measurable, repeatable practices.
