taher-ghaleb/DEPosit

DEPosit — Replication Package

DEPosit (Data Engineering Pipeline Repositories) is a dataset of open-source GitHub repositories in the data-engineering pipeline ecosystem.

The dataset contains a master cohort of 1,952 repositories together with repository activity, commits, pull requests, issues, CI-service information, contributor metrics, and pipeline discovery and feature data.

Dataset (Data/)

The Hugging Face Hub exposes each heterogeneous CSV table as a separate configuration. This prevents tables with different schemas from being incorrectly concatenated by the Dataset Viewer.

The SQLite file DE_pipeline_artifacts.db is retained as a downloadable replication artifact and is not used by the Hugging Face Dataset Viewer.

Join Keys

• —full_name or repo_name (owner/repo) for repository-level and process tables.
• —repo_full_name for tables in pipeline_artifacts/.
• —(repo_full_name, file_path) links DE_pipeline_files.csv to the pipeline feature tables.

Loading Data from Hugging Face

Each table can be loaded independently using its configuration name:

from datasets import load_dataset

repos = load_dataset("taher-ghaleb/DEPosit", "repositories")
commits = load_dataset("taher-ghaleb/DEPosit", "commits")
issues = load_dataset("taher-ghaleb/DEPosit", "issues")
ci = load_dataset("taher-ghaleb/DEPosit", "ci_services")

To load pipeline feature tables:

from datasets import load_dataset

airflow = load_dataset("taher-ghaleb/DEPosit", "pipeline_airflow")
dbt = load_dataset("taher-ghaleb/DEPosit", "pipeline_dbt_model")
prefect = load_dataset("taher-ghaleb/DEPosit", "pipeline_prefect")

Each configuration represents one logical table with a consistent schema. The tables are intended to be joined for analysis using the keys described above.

Scripts (Scripts/)

Quick start (analyze the shipped data)

py -m pip install -r requirements.txt

Open Data/filtered_DE_repositories.csv and join to other Data/DE_*.csv files on full_name / repo_name.

GitHub authentication (re-collection only)

Do not commit personal access tokens. Scripts load credentials from (in order):

1. 1.--token / --tokens CLI flags (where supported)
2. 2.Environment variables: GITHUB_TOKEN_SE4DE (preferred) or GITHUB_TOKEN
3. 3.Optional local .env file — copy .env.example to .env and set your PAT

copy .env.example .env
# Edit .env and set GITHUB_TOKEN_SE4DE=<your-pat>

# Or set for the current shell only:
$env:GITHUB_TOKEN_SE4DE = "<your-pat>"

For heavy REST collection, rotate rate limits with comma-separated PATs:

$env:GITHUB_TOKENS = "<pat1>,<pat2>"

Example:

py Scripts/collect_pipeline_artifacts.py --token $env:GITHUB_TOKEN_SE4DE
py Scripts/get_gha_workflow_runs_for_SE4DE_repos.py --token $env:GITHUB_TOKEN_SE4DE

Typical collection order: topic discovery → development history → derived metrics → pipeline artifacts → optional backfill_ci_gha_from_runs.py.

Historical commit/PR text in Data/*.csv may contain third-party leaked tokens from upstream repos; these were redacted where detected. Re-run Scripts/redact_leaked_tokens_in_data.py after adding new CSV exports.

Layout

ReplicationPackage/
├── README.md
├── LICENSE
├── DATA_LICENSE
├── .env.example
├── requirements.txt
├── Data/
│   ├── filtered_DE_repositories.csv
│   ├── DE_*.csv
│   ├── DE_pipeline_artifacts.db
│   └── pipeline_artifacts/
└── Scripts/
    ├── collect_*.py, get_*.py, compute_*.py
    ├── collect_pipeline_artifacts.py
    ├── backfill_ci_gha_from_runs.py
    ├── github_api.py, github_tokens.py
    └── pipeline_parsers/

Citation

If you use DEPosit, please cite:

@inproceedings{deposit2026,
  title     = {DEPosit: A Dataset of Open-Source Repositories for Data Engineering Pipelines},
  author    = {Gorjala, Bhavyalatha and Taher A. Ghaleb},
  booktitle = {Proceedings of the 42nd IEEE International Conference on Software Maintenance and Evolution (ICSME)},
  year      = {2026},
  organization={IEEE}
}