112 lines
2.3 KiB
Markdown
112 lines
2.3 KiB
Markdown
# CleverDatasets
|
|
|
|
RDF dataset processing and upload pipeline for HuggingFace Hub.
|
|
|
|
## Quick Start
|
|
|
|
### 1. Clone Repository
|
|
|
|
```bash
|
|
git clone <repository-url>
|
|
cd datasets
|
|
```
|
|
|
|
### 2. Setup Credentials
|
|
|
|
Copy example files and fill in your values:
|
|
|
|
```bash
|
|
cp .env.example .env
|
|
cp creds-example.json creds.json
|
|
```
|
|
|
|
Edit `.env`:
|
|
```bash
|
|
nano .env
|
|
```
|
|
|
|
Required values:
|
|
- `GOOGLE_SHEETS_ID`: Your Google Sheets spreadsheet ID from the URL
|
|
- `HF_TOKEN`: Your HuggingFace token from https://huggingface.co/settings/tokens
|
|
|
|
Edit `creds.json` with your Google Cloud service account credentials:
|
|
```bash
|
|
nano creds.json
|
|
```
|
|
|
|
### 3. Open in DevContainer
|
|
|
|
#### VS Code
|
|
|
|
```bash
|
|
code .
|
|
```
|
|
|
|
When prompted: **"Reopen in Container?"** → Click **"Reopen in Container"**
|
|
|
|
Wait 6-10 minutes for first-time setup.
|
|
|
|
#### PyCharm Professional
|
|
|
|
1. Open PyCharm and select **File → Open** → Navigate to the project directory
|
|
2. PyCharm will detect the devcontainer configuration automatically
|
|
3. When prompted: **"Dev Container Configuration Found"** → Click **"Use Dev Container"**
|
|
4. Wait 6-10 minutes for first-time setup
|
|
|
|
**Note:** PyCharm Professional (2023.1+) is required. Community Edition does not support devcontainers.
|
|
|
|
See `.devcontainer/DevContainer.md` for detailed troubleshooting and alternative setup methods.
|
|
|
|
### 4. Run Scripts
|
|
|
|
Inside the container:
|
|
|
|
```bash
|
|
python scripts/upload_all_datasets.py --list
|
|
python scripts/upload_all_datasets.py --dataset wordnet --sheet
|
|
```
|
|
|
|
## Alternative: Using Environment Variables
|
|
|
|
Instead of creating `.env`, you can export variables directly:
|
|
|
|
```bash
|
|
export GOOGLE_SHEETS_ID="your_sheet_id"
|
|
export HF_TOKEN="your_token"
|
|
export GOOGLE_SHEETS_CREDENTIALS_PATH="creds.json"
|
|
|
|
python scripts/upload_all_datasets.py --list
|
|
```
|
|
|
|
Environment variables take precedence over `.env` file.
|
|
|
|
## Running Outside Container
|
|
|
|
The scripts work on your host machine too:
|
|
|
|
```bash
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -e '.[dev]'
|
|
python scripts/upload_all_datasets.py --list
|
|
```
|
|
|
|
## Important Notes
|
|
|
|
- **Never commit `.env` or `creds.json`** - they are in `.gitignore`
|
|
- Always verify with `git status` before pushing
|
|
- Keep your credentials secure
|
|
|
|
## Development
|
|
|
|
```bash
|
|
nox -s format lint typecheck tests
|
|
```
|
|
|
|
## Documentation
|
|
|
|
```bash
|
|
nox -s docs
|
|
nox -s serve_docs
|
|
```
|