Greg Lindahl
Chief Technology Officer of the Common Crawl Foundation, which publishes an open web crawl archive
- Chief Technology Officer, Common Crawl13
Greg Lindahl is the Chief Technology Officer of the Common Crawl Foundation, the nonprofit that crawls the web and makes its archives and datasets freely available. Lindahl is the author of several of Common Crawl's technical posts, including the announcement of the Host Index dataset and an explainer on measuring how much of a website the crawl has covered.1235
Common Crawl
Datasets and writing
In April 2025 Lindahl announced a public test of the Host Index, which gathers per-host summaries from the crawl, its indexes, the web graph and raw crawler logs, and can be queried with SQL tools such as Amazon Athena or DuckDB. A July 2026 post by Lindahl explains why counting a site's pages in Common Crawl is harder than it looks, using CrawlDb states from the Nutch crawler.35
Lindahl also compiled the foundation's December 2025 roundup of a dozen research papers from that year that use or reference Common Crawl data.6
Work
Common Crawl Host Index (external site: github.com)Open data
Authored the April 2025 announcement of the Host Index, a dataset with one row per web host per crawl that combines crawl statistics, status codes, language summaries and robots.txt data, with tools published in the cc-host-index repository.34
Common Crawl corpusOpen data
As CTO, works on the technical side of the open repository of web crawl data that Common Crawl has collected since 2008 and publishes roughly once a month.12
Career timeline
Related people
More sourced profiles at the same organization, in the same field, or from the same school.
- Gil ElbazAlso at Common Crawl
- Abhimanyu DasDatasets and data
- Rich SkrentaAlso at Common Crawl
- Adina WilliamsDatasets and data
- Albert JiangDatasets and data
- Alex FrebergDatasets and data
- Alexander KarpDatasets and data
- Alexander ToshevDatasets and data
- Alexander WettigDatasets and data
- Ali GhodsiDatasets and data
- Andriy MulyarDatasets and data
- Andy KonwinskiDatasets and data
Is this you?
Profiles cover public professional work only, and every line cites a source. If something is out of date, missing or wrong, send a correction with a link to an official page and we will update it. Suggest a correction
Add this badge to your website or GitHub profile README. It links back to this page. It is not an endorsement or a rating.
Markdown (for a README)
[](https://unitedstatesofamericasuperintelligence.com/people/greg-lindahl/)
HTML
<a href="https://unitedstatesofamericasuperintelligence.com/people/greg-lindahl/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/profile.svg" alt="Greg Lindahl — Profile on USASI" height="20"></a>
This profile covers public professional work only, from the sources below. It is not an endorsement or a ranking. To correct or remove it, use the correction form.