Skip to content

#

common-crawl

Here are 46 public repositories matching this topic...

Language: All

Filter by language

All 46 Python 19 Java 7 Jupyter Notebook 6 Go 2 Rust 2 C 1 C# 1 Clojure 1 HTML 1 JavaScript 1

Sort: Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

MigoXLab / dingo

Dingo: A Comprehensive AI Data Quality Evaluation Tool

spark data-validation datascience openai gpt data-quality vlm common-crawl data-evaluation dataquality hallucination data-quality-report data-quality-assessment llm qwen data-agent deepseek hallucination-detection llm-as-a-judge opencompass

Updated Oct 16, 2025
JavaScript

commoncrawl / cc-pyspark

Process Common Crawl data with Python and Spark

spark pyspark sparksql wet commoncrawl common-crawl warc-files wat-files

Updated Sep 24, 2025
Python

commoncrawl / news-crawl

News crawling with StormCrawler - stores content as WARC

crawler news web-crawler apache-storm warc commoncrawl common-crawl storm-crawler

Updated Feb 19, 2025
Java

commoncrawl / cc-crawl-statistics

Statistics of Common Crawl monthly archives mined from URL index files

statistics commoncrawl common-crawl

Updated Oct 14, 2025
Python

oscar-project / ungoliant

🕷️ The pipeline for the OSCAR corpus

nlp crawler corpus-linguistics fasttext oscar commoncrawl common-crawl language-classification

Updated Dec 18, 2023
Rust

crissyfield / troll-a

Drill into WARC web archives

security internet-archive command-line-tool warc security-tools common-crawl

Updated Oct 16, 2024
Go

commoncrawl / cc-webgraph

Tools to construct and process Common Crawl webgraphs

pagerank webgraph commoncrawl common-crawl centrality-measures webgraph-framework

Updated Oct 5, 2025
Java

oscar-project / goclassy

An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.

nlp corpus-linguistics fasttext common-crawl language-classification

Updated Apr 21, 2021
Go

commoncrawl / cc-notebooks

Various Jupyter notebooks about Common Crawl data

jupyter-notebook aws-athena commoncrawl common-crawl webarchiving webgraph-framework

Updated Apr 1, 2025
Jupyter Notebook

IBM / cc-dbp

A dataset for knowledge base population research using Common Crawl and DBpedia.

dbpedia common-crawl ibm-research-ai knowledge-base-population

Updated Jan 27, 2022
Java

cisnlp / GlotCC

🕸 GlotCC Dataset and Pipline -- NeurIPS 2024

crawler multlingual corpus-linguistics glot language-identification commoncrawl common-crawl glotcc multilingual-dataset glotlid

Updated Apr 6, 2025
Jupyter Notebook

bminixhofer / gerpt2

German small and large versions of GPT2.

nlp machine-learning german language-model common-crawl gpt2

Updated May 11, 2022
Python

oscar-project / oscar-website

The website of the Oscar Project

nlp website machine-learning hugo language-model common-crawl

Updated Mar 27, 2025
TeX

toimik / CommonCrawl

Common Crawl's processing tools

warc wat wet commoncrawl common-crawl warc-files wat-files common-crawl-data wet-files

Updated Oct 15, 2024
C#

Mgosi / Big-Data-Analysis-using-MapReduce-in-Hadoop

We explore data by using Big Data Analysis and Visualization skills. To obtain this, we perform 3 main operations. i.e. i)Data Aggregation through different sources. ii) Big Data Analysis using MapReduce and iii) Visualization through Tableau. Data Analysis is very critical in understanding the data, and what we can do with the data. For small d...

docker big-data twitter-api hdfs tableau data-processing data-pipeline hadoop-docker common-crawl big-data-analytics tweet-collector

Updated Oct 5, 2019
Jupyter Notebook

code402 / warc-benchmark

Sample code to grep Common Crawl WARC files in Go, Java, Node and Python.

warc commoncrawl common-crawl

Updated Apr 30, 2021
Shell

alumik / common-crawl-downloader

Distributed download scripts for Common Crawl data

downloader common-crawl

Updated May 12, 2025
Python

tokenmill / common-crawl-utils

Various Common Crawl utilities in Clojure.

clojure clojure-library warc common-crawl cdx-api

Updated Dec 5, 2023
Clojure

thunderpoot / cc-getpage

Lightweight Python utility for retrieving individual pages from the Common Crawl archives.

commoncrawl common-crawl common-crawl-with-python common-crawl-python common-crawl-data

Updated Mar 2, 2025
Python

HRN-Projects / common_crawl_with_scrapy

Parsing Huge Web Archive files from Common Crawl data index to fetch any required domain's data concurrently with Python and Scrapy.

python data-mining python3 web-scraping scrapy web-crawling webarchive common-crawl common-crawl-with-scrapy parse-common-crawl common-crawl-with-python common-crawl-scrapy common-crawl-python common-crawl-data webarchive-data-scraping

Updated Jul 14, 2021
Python

Improve this page

Add a description, image, and links to the common-crawl topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the common-crawl topic, visit your repo's landing page and select "manage topics."