CleverDatasets

RDF dataset processing and upload pipeline for HuggingFace Hub.

Quick Start

1. Clone Repository

git clone <repository-url>
cd datasets

2. Setup Credentials

Copy example files and fill in your values:

cp .env.example .env
cp creds-example.json creds.json

Edit .env:

nano .env

Required values:

• GOOGLE_SHEETS_ID: Your Google Sheets spreadsheet ID from the URL
• HF_TOKEN: Your HuggingFace token from https://huggingface.co/settings/tokens

Edit creds.json with your Google Cloud service account credentials:

nano creds.json

3. Open in DevContainer

VS Code

code .

When prompted: "Reopen in Container?" → Click "Reopen in Container"

Wait 6-10 minutes for first-time setup.

PyCharm Professional

1. Open PyCharm and select File → Open → Navigate to the project directory
2. PyCharm will detect the devcontainer configuration automatically
3. When prompted: "Dev Container Configuration Found" → Click "Use Dev Container"
4. Wait 6-10 minutes for first-time setup

Note: PyCharm Professional (2023.1+) is required. Community Edition does not support devcontainers.

See .devcontainer/DevContainer.md for detailed troubleshooting and alternative setup methods.

4. Run Scripts

Inside the container:

python scripts/upload_all_datasets.py --list
python scripts/upload_all_datasets.py --dataset wordnet --sheet

Alternative: Using Environment Variables

Instead of creating .env, you can export variables directly:

export GOOGLE_SHEETS_ID="your_sheet_id"
export HF_TOKEN="your_token"
export GOOGLE_SHEETS_CREDENTIALS_PATH="creds.json"

python scripts/upload_all_datasets.py --list

Environment variables take precedence over .env file.

Running Outside Container

The scripts work on your host machine too:

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
python scripts/upload_all_datasets.py --list

Important Notes

• Never commit .env or creds.json - they are in .gitignore
• Always verify with git status before pushing
• Keep your credentials secure

Development

nox -s format lint typecheck tests

Documentation

nox -s docs
nox -s serve_docs