ToolScore
Data & Files 4 min read

Check These 9 Free Data Sources Before You Write a Scraper

9 verified free bulk sources the number that matters here

By Stan K.

SEC filings, federal spending, census data, global maps and a petabyte-scale web crawl — all free, all bulk-downloadable, all verified live. A surprising share of scraping projects are re-downloading data that's already published.

Before you budget proxies, captcha solvers and headless browsers, run this test: is the data you’re about to scrape already published as a bulk download? Governments and research infrastructure give away astonishing amounts of data for free, and scraping a website that republishes a free dataset is paying to solve the wrong problem.

All nine sources below were verified live in November 2026. Details and limits matter, so they’re included.

The nine, and what each is actually good for

SourceWhat you getAccessFine print
SEC EDGAREvery public company’s filings — 10-K, 10-Q, insider trades — as nightly bulk ZIPs (companyfacts, submissions) plus free JSON APIsNo key; custom User-Agent required; ~10 req/s polite limitBulk zips live under the daily-index archive paths — the old root URLs 404
Data.gov300,000+ US federal datasets: crime, agriculture, climate, energyMostly direct downloadsCatalog links out to agencies — some endpoints drift
USASpending.govEvery federal contract, grant and loan, as bulk CSV/PostgreSQL dumps plus an APIFree, testableThe download API builds zips on request — plan for async
US Census Bureau APIDemographics, housing, business patterns by geographyKey optional (rate-limited without; free key lifts limits)Endpoints stable for years
FREDHalf a million economic time series from the St. Louis FedFree key requiredOne of the rare ones that gates access, but the key is instant
OpenStreetMap (Geofabrik)Daily .osm.pbf extracts by region — POIs, buildings, addresses, roadsDirect download, no keyParse, don’t scrape: the extracts are the source of truth
Overture MapsCloud-native GeoParquet: global buildings, places, addressesS3/HTTPS, no keyHeavy — query selectively, don’t download the world
Common CrawlPetabytes of pre-crawled web pages as WARC/Parquet archivesHosted free on S3The crawl already happened — you pay nothing to read it
OpenAlex250M+ academic papers, authors, institutionsFree REST API + full bulk snapshot, no key, polite poolA model of how open data infrastructure should work
The litmus test

If your target site publishes “as reported from public records,” or aggregates government data, or mirrors filings — the upstream source is on this list, and the upstream source is free, complete and legal to redistribute in most cases.

Three patterns worth internalizing

1. Public records are upstream of every records-aggregator. Our own public-records platform exists because county data is genuinely scattered. But the federal layer — spending, filings, demographics — is almost uniformly bulk-downloadable. Scraping a federal-data aggregator is pure waste; scraping a county portal may still be the only way.

2. “Crawl the web” has already been done. Common Crawl and OpenAlex answer the two most common large-scale scraping requests (web pages at scale, academic metadata at scale) with data that is sitting there, free, with no anti-bot anywhere. If your project starts with “fetch a million pages,” check whether those million pages are in an archive first.

3. Rate limits replace proxies. The polite limits on these APIs (10 req/s here, a key there) are the whole cost of access. There is no block rate, no captcha, no retry multiplier — the failure modes from our cost math simply don’t exist.

When you still need to scrape

None of this replaces scraping for: data that lives only in a live interface (auction calendars, availability, prices in real time), data behind interaction (wizard-driven portals), or genuinely private aggregation. That’s when the escalation path from our own stack applies — TLS impersonation first, browsers last — and the Data & scraping hub has the verified vendor tables for the volume cases.

But run the free check first. It’s the highest ROI five minutes in any data project.

Numbers change — the math doesn't
Token prices and plan limits shift monthly; the structural arguments in this guide age slowly. For current per-tool numbers see the live rankings.