Your Scraper Just Broke — Again: Hard-Won Lessons from 100+ Ruby Spiders in Production

Maintaining one web scraper is easy. Maintaining over 100, each targeting a different financial institution's website, is a completely different beast.

This talk is drawn from a real production Rails system that continuously downloads
monthly report PDFs from Japanese asset management companies. When DWS silently migrated their REST API to a Nuxt.js SPA, when JPMorgan's ReactVirtualized table only rendered 12 visible rows and your target fund wasn't one of them, when Capital Group stopped offering PDF files and switched to HTML-only reports — your spiders just
quietly broke.

I'll walk through the three most common failure patterns and their Ruby solutions, how Module and Concern help organize shared logic across a large spider fleet, and the most important engineering judgment of all: when to fix, and when to just delete.