CleverDatasets

RDF dataset processing and upload pipeline for HuggingFace Hub.

Quick Start

1. Clone Repository

git clone dataset-uploader
cd dataset-uploader

2. Setup Credentials

Copy example files and fill in your values:

cp .env.example .env
cp creds-example.json creds.json

Edit .env:

nano .env

Required values:

• GOOGLE_SHEETS_ID: Your Google Sheets spreadsheet ID from the URL
• HF_TOKEN: Your HuggingFace token from https://huggingface.co/settings/tokens
• GOOGLE_SHEETS_WORKSHEET_NAME: Worksheet tab name (default: Sheet1)
• GOOGLE_SHEETS_CREDENTIALS_PATH: Path of the creds.json file

Edit creds.json with your Google Cloud service account credentials:

nano creds.json

How to create Google Cloud service account credentials:

Follow Google's official documentation:

• Create a service account
• Create service account keys

Quick steps:

1. Create a Google Cloud project at https://console.cloud.google.com
2. Enable the Google Sheets API for your project
3. Create a service account with Sheets API access
4. Download the service account key as JSON
5. Save the downloaded file as creds.json in your project root

Important: Note the service account email (format: name@project.iam.gserviceaccount.com) from the client_email field in creds.json - you'll need it to share your spreadsheet.

Google Sheets Setup (Optional)

If you want to track dataset processing status in Google Sheets:

1. Create Google Sheets spreadsheet with exactly these 3 columns in the first row:

   • Column A: DATASET NAME - The dataset identifier (e.g., wordnet, dbpedia)
   • Column B: STATUS - Processing status (will be auto-updated by script)
   • Column C: ERROR - Error messages (if any, auto-filled by script)

   Example header row:

   DATASET NAME | STATUS | ERROR
   

2. Pre-fill dataset names you want to process (fill Column A, leave B and C empty):

   DATASET NAME | STATUS       | ERROR
   wordnet      | NOT STARTED  |
   dbpedia      | NOT STARTED  |
   geonames     | NOT STARTED  |
   

   The script will automatically update STATUS (IN PROGRESS → DONE) and ERROR columns.

3. Share spreadsheet with service account:

   • Open creds.json → find client_email field
   • Click "Share" in Google Sheets
   • Add the service account email
   • Grant "Editor" permissions

4. Copy spreadsheet ID from URL to .env:

   https://docs.google.com/spreadsheets/d/[THIS_IS_THE_ID]/edit
   

Note: Column names are case-insensitive and the tracker will detect them automatically.

3. Open in DevContainer

VS Code

code .

When prompted: "Reopen in Container?" → Click "Reopen in Container"

Wait 6-10 minutes for first-time setup.

PyCharm Professional

1. Open PyCharm and select File → Open → Navigate to the project directory
2. PyCharm will detect the devcontainer configuration automatically
3. When prompted: "Dev Container Configuration Found" → Click "Use Dev Container"
4. Wait 6-10 minutes for first-time setup

Note: PyCharm Professional (2023.1+) is required. Community Edition does not support devcontainers.

See .devcontainer/DevContainer.md for detailed troubleshooting and alternative setup methods.

4. Run Scripts

Inside the container:

python scripts/upload_all_datasets.py --list
python scripts/upload_all_datasets.py --dataset wordnet --sheet

Alternative: Using Environment Variables

Instead of creating .env, you can export variables directly:

export GOOGLE_SHEETS_ID="your_sheet_id"
export HF_TOKEN="your_token"
export GOOGLE_SHEETS_CREDENTIALS_PATH="creds.json"

python scripts/upload_all_datasets.py --list

Environment variables take precedence over .env file.

Running Outside Container

The scripts work on your host machine too:

python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
python scripts/upload_all_datasets.py --list

Important Notes

• Never commit .env or creds.json - they are in .gitignore
• Always verify with git status before pushing
• Keep your credentials secure

Development

nox -s format lint typecheck tests

Documentation

nox -s docs
nox -s serve_docs