# CleverDatasets

RDF dataset processing and upload pipeline for HuggingFace Hub.

## Quick Start

### 1. Clone Repository

```bash
git clone <repository-url>
cd datasets
```

### 2. Setup Credentials

Copy example files and fill in your values:

```bash
cp .env.example .env
cp creds-example.json creds.json
```

Edit `.env`:
```bash
nano .env
```

Required values:
- `GOOGLE_SHEETS_ID`: Your Google Sheets spreadsheet ID from the URL
- `HF_TOKEN`: Your HuggingFace token from https://huggingface.co/settings/tokens

Edit `creds.json` with your Google Cloud service account credentials:
```bash
nano creds.json
```

### 3. Open in DevContainer

#### VS Code

```bash
code .
```

When prompted: **"Reopen in Container?"** → Click **"Reopen in Container"**

Wait 6-10 minutes for first-time setup.

#### PyCharm Professional

1. Open PyCharm and select **File → Open** → Navigate to the project directory
2. PyCharm will detect the devcontainer configuration automatically
3. When prompted: **"Dev Container Configuration Found"** → Click **"Use Dev Container"**
4. Wait 6-10 minutes for first-time setup

**Note:** PyCharm Professional (2023.1+) is required. Community Edition does not support devcontainers.

See `.devcontainer/DevContainer.md` for detailed troubleshooting and alternative setup methods.

### 4. Run Scripts

Inside the container:

```bash
python scripts/upload_all_datasets.py --list
python scripts/upload_all_datasets.py --dataset wordnet --sheet
```

## Alternative: Using Environment Variables

Instead of creating `.env`, you can export variables directly:

```bash
export GOOGLE_SHEETS_ID="your_sheet_id"
export HF_TOKEN="your_token"
export GOOGLE_SHEETS_CREDENTIALS_PATH="creds.json"

python scripts/upload_all_datasets.py --list
```

Environment variables take precedence over `.env` file.

## Running Outside Container

The scripts work on your host machine too:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
python scripts/upload_all_datasets.py --list
```

## Important Notes

- **Never commit `.env` or `creds.json`** - they are in `.gitignore`
- Always verify with `git status` before pushing
- Keep your credentials secure

## Development

```bash
nox -s format lint typecheck tests
```

## Documentation

```bash
nox -s docs
nox -s serve_docs
```