Check These 9 Free Data Sources Before You Write a Scraper
9 verified free bulk sources the number that matters here
By Stan K.
SEC filings, federal spending, census data, global maps and a petabyte-scale web crawl — all free, all bulk-downloadable, all verified live. A surprising share of scraping projects are re-downloading data that's already published.
Before you budget proxies, captcha solvers and headless browsers, run this test: is the data you’re about to scrape already published as a bulk download? Governments and research infrastructure give away astonishing amounts of data for free, and scraping a website that republishes a free dataset is paying to solve the wrong problem.
All nine sources below were verified live in November 2026. Details and limits matter, so they’re included.
The nine, and what each is actually good for
| Source | What you get | Access | Fine print |
|---|---|---|---|
| SEC EDGAR | Every public company’s filings — 10-K, 10-Q, insider trades — as nightly bulk ZIPs (companyfacts, submissions) plus free JSON APIs | No key; custom User-Agent required; ~10 req/s polite limit | Bulk zips live under the daily-index archive paths — the old root URLs 404 |
| Data.gov | 300,000+ US federal datasets: crime, agriculture, climate, energy | Mostly direct downloads | Catalog links out to agencies — some endpoints drift |
| USASpending.gov | Every federal contract, grant and loan, as bulk CSV/PostgreSQL dumps plus an API | Free, testable | The download API builds zips on request — plan for async |
| US Census Bureau API | Demographics, housing, business patterns by geography | Key optional (rate-limited without; free key lifts limits) | Endpoints stable for years |
| FRED | Half a million economic time series from the St. Louis Fed | Free key required | One of the rare ones that gates access, but the key is instant |
| OpenStreetMap (Geofabrik) | Daily .osm.pbf extracts by region — POIs, buildings, addresses, roads | Direct download, no key | Parse, don’t scrape: the extracts are the source of truth |
| Overture Maps | Cloud-native GeoParquet: global buildings, places, addresses | S3/HTTPS, no key | Heavy — query selectively, don’t download the world |
| Common Crawl | Petabytes of pre-crawled web pages as WARC/Parquet archives | Hosted free on S3 | The crawl already happened — you pay nothing to read it |
| OpenAlex | 250M+ academic papers, authors, institutions | Free REST API + full bulk snapshot, no key, polite pool | A model of how open data infrastructure should work |
If your target site publishes “as reported from public records,” or aggregates government data, or mirrors filings — the upstream source is on this list, and the upstream source is free, complete and legal to redistribute in most cases.
Three patterns worth internalizing
1. Public records are upstream of every records-aggregator. Our own public-records platform exists because county data is genuinely scattered. But the federal layer — spending, filings, demographics — is almost uniformly bulk-downloadable. Scraping a federal-data aggregator is pure waste; scraping a county portal may still be the only way.
2. “Crawl the web” has already been done. Common Crawl and OpenAlex answer the two most common large-scale scraping requests (web pages at scale, academic metadata at scale) with data that is sitting there, free, with no anti-bot anywhere. If your project starts with “fetch a million pages,” check whether those million pages are in an archive first.
3. Rate limits replace proxies. The polite limits on these APIs (10 req/s here, a key there) are the whole cost of access. There is no block rate, no captcha, no retry multiplier — the failure modes from our cost math simply don’t exist.
When you still need to scrape
None of this replaces scraping for: data that lives only in a live interface (auction calendars, availability, prices in real time), data behind interaction (wizard-driven portals), or genuinely private aggregation. That’s when the escalation path from our own stack applies — TLS impersonation first, browsers last — and the Data & scraping hub has the verified vendor tables for the volume cases.
But run the free check first. It’s the highest ROI five minutes in any data project.