What Common Crawl covers and misses compared with Google

Common Crawl is a free, open copy of part of the web. Its own FAQ says it is 'a randomly selected subset', and a Mozilla study says it is not representative. This is what its web graph holds, what it leaves out, and how we have used it.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

Contents

Three of our posts and one research page counted businesses or links using Common Crawl's web graph. A reader checking those counts should know what the graph can and cannot see. This page says what Common Crawl publishes about itself, what an outside study found, and what happened when we used it ourselves.

What it is

Common Crawl is a non-profit that runs a crawler called CCBot and publishes what it fetches, free. Its web graph release for May, June and July 2026 has a host-level graph of 240.4 million nodes and 3.7 billion edges, and a domain-level graph of 118.0 million nodes and 2.8 billion edges.[4] Those are the files our scans read.

The release page says 74.05 percent of the host-level nodes and 63.2 percent of the domain-level nodes are "dangling nodes".[4] The page does not define the term. Synthesis: in a link graph a dangling node has no links going out, which for a crawl means a site the graph knows only because some other page pointed at it. If that is the meaning, most of the nodes in the graph are sites it never read.

What it covers and what it leaves out

Common Crawl's own FAQ says: "Common Crawl's dataset is a sample of the web, and we do not generally archive any entire website but a randomly selected subset of it."[2]

Stefan Baack's study for the Mozilla Foundation (February 2024) looked at how the crawl chooses what to fetch. Its summary says that Common Crawl "does not contain the 'entire web,' nor a representative sample of it". It says the crawl is "almost entirely automated to prioritize pages on domains that are frequently linked to", which makes domains linked to "digitally marginalized communities" less likely to be included, and that "the language and regional coverage is strongly skewed toward English content".[1]

The study's description of the mechanism: Common Crawl combines a harmonic centrality score (a measure of how well linked a domain is) with points for how long ago a URL was last fetched, and about half of each crawl is URLs fetched before. Sites with a high score "are always included, while those with lower and varying scores may or may not be."[1]

Two ways a site can be missing, both from Common Crawl's pages:

  1. Blocked. A site that puts User-agent: CCBot and Disallow: / in its robots.txt is not crawled. The study names Facebook and the New York Times as large domains that block most or all of Common Crawl's crawling.[3][1]
  2. Not picked. A site that allows the crawler can still be left out in a given crawl, since the dataset is a subset (the FAQ's word).[2]

Is it a fair proxy for Google, Bing and DuckDuckGo?

Mackenzie's view, from the video where he reviewed the content gaps on 2026-11-06:

However, it matches Google about as well as it matches all of the other things that we do have access to, like DuckDuckGo and Bing.

Mackenzie,

He added that it is "a really good way" to form a hypothesis and test it against a data set. He also gave a picture of search use: Google at "90% plus", then Brave, then DuckDuckGo, with "almost nobody" on Bing, which sends us little traffic even though every page we publish goes to it through IndexNow.

What we checked:

  1. Google's share. Statcounter's worldwide search engine share for October 2026 is Google 91.05 percent, Bing 4.8, Yahoo 1.18, Yandex 1.14, DuckDuckGo 0.64 and Baidu 0.42.[5] That bears out the Google figure. It does not match the order he gave: on this source Bing is well ahead of DuckDuckGo, and Brave is not listed.
  2. The proxy claim. We found no source that compares Common Crawl's coverage with Google's, Bing's or DuckDuckGo's index. The claim that it matches Google about as well as it matches the others is Mackenzie's expert opinion and is unchecked here.

Synthesis: the Mozilla study's two findings (a crawl weighted toward well-linked, English-language domains, and no claim to represent the web) fit the use he describes. Search engines also rank by links, so a crawl that favours well-linked domains may resemble what they surface. That is a reason the proxy might hold, not a measurement that it does. It would not hold for small, newly built sites that few pages link to.

What we found using it

  1. Counting links into three vendors. For accounting firm software we streamed the May-July 2026 domain graph once, looking for links to TaxDome, Karbon and Canopy. The scan took about 26 minutes and cost nothing. It gives who links to those vendors; it cannot say who uses them.
  2. Counting BC dentists and law firms. Our dental scan and law firm scan took every .ca domain in the graph with a dental or law word in its name, added the OpenStreetMap businesses, and kept the sites whose home page showed a BC postal code. The dental post says its own caveat: the frame "favours practices whose .ca domain says 'dental' or 'smile'". That limit comes from how we built the frame, not from Common Crawl, and the graph shapes it too, because a practice missing from the graph is missing from the count.
  3. Looking for ourselves. On 2026-11-06 we asked Common Crawl's index for mackenziebowes.com in the October 2026 crawl (CC-MAIN-2026-43) and the July one (CC-MAIN-2026-30). It answered "No Captures found" for the October crawl and none for the July one. The site's robots.txt allows all crawlers and its sitemap lists 351 URLs. The index answered for commoncrawl.org, so the query worked. For our four other sites the index returned empty replies or timeouts, so we make no claim about them. The finding fits the FAQ's word "subset": our own site is not in the crawl, and a new, little-linked site may need time to be picked.[6]

What we could not find

  1. Any measurement of how closely Common Crawl overlaps Google's, Bing's or DuckDuckGo's index.
  2. What share of small-business sites in Canada, or in British Columbia, appear in a given crawl.
  3. Whether our other four sites are in the crawl (the index would not answer).

If you have compared Common Crawl against a search engine's index, or know how often new sites enter it, I would like to compare notes: get in touch, or write to mackenzie@mackenziebowes.com.

Sources

  1. Stefan Baack and Mozilla Insights, Training Data for the Price of a Sandwich: Common Crawl's Impact on Generative AI, Mozilla Foundation, February 2024. Read in full 2026-11-06. www.mozillafoundation.org/en/research/library/generative-ai-training-data/common-crawl/
  2. Common Crawl, FAQ. Read 2026-11-06. commoncrawl.org/faq
  3. Common Crawl, CCBot. Read 2026-11-06. commoncrawl.org/ccbot
  4. Common Crawl, Web Graph Release cc-main-2026-may-jun-jul. Read 2026-11-06. data.commoncrawl.org/projects/hyperlinkgraph/cc-main-2026-may-jun-jul/index.html
  5. Statcounter Global Stats, Search Engine Market Share Worldwide, October 2026. Read 2026-11-06. gs.statcounter.com/search-engine-market-share
  6. Common Crawl index, list of crawls and the CDX query we ran against CC-MAIN-2026-43 and CC-MAIN-2026-30, 2026-11-06. Our own check. index.commoncrawl.org/collinfo.json

Terms on this page

  • robots.txt: A file at the root of a site telling crawlers what they may fetch.