cleverdatasets/dataset-uploader
cleverdatasets/dataset-uploader
0
0
Fork 0
Code Issues 17 Pull requests 6 Projects Releases Packages Wiki Activity Actions

Feat: add initial Dockerfile and compose file for the repo #24

Open
aditya wants to merge 2 commits from 7-dockerfile-setup into mypy-and-pylint
pull from: 7-dockerfile-setup
merge into: cleverdatasets:mypy-and-pylint
Conversation 3 Commits 2 Files changed 3 +196 -39
aditya commented 2025-12-03 14:36:41 +00:00
Member
Copy link

Initial docker setup

Initial docker setup
aditya changed target branch from mypy-and-pylint to 16-error-messages-and-logging 2025-12-03 14:38:40 +00:00
aditya changed target branch from 16-error-messages-and-logging to mypy-and-pylint 2025-12-03 14:39:55 +00:00
brent.edwards requested changes 2025-12-04 01:34:45 +00:00
brent.edwards left a comment
Member
Copy link

Is this the expected behavior?

Is this the expected behavior?
Dockerfile
@ -36,2 +12,2 @@
RUN pip install --no-cache-dir /tmp/*.whl && \
rm -rf /tmp/*
# Install uv for fast dependency management
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
brent.edwards commented 2025-12-04 01:31:01 +00:00
Member
Copy link

Is this the best way to install uv? https://docs.astral.sh/uv/guides/integration/docker/#installing-uv recommends:

# Download the latest installer
ADD https://astral.sh/uv/install.sh /uv-installer.sh

# Run the installer then remove it
RUN sh /uv-installer.sh && rm /uv-installer.sh

# Ensure the installed binary is on the `PATH`
ENV PATH="/root/.local/bin/:$PATH"
Is this the best way to install `uv`? https://docs.astral.sh/uv/guides/integration/docker/#installing-uv recommends: ``` # Download the latest installer ADD https://astral.sh/uv/install.sh /uv-installer.sh # Run the installer then remove it RUN sh /uv-installer.sh && rm /uv-installer.sh # Ensure the installed binary is on the `PATH` ENV PATH="/root/.local/bin/:$PATH" ```
aditya commented 2025-12-04 13:29:43 +00:00
Author
Member
Copy link

Both methods are documented and valid, but COPY --from= is actually the preferred approach in the uv docs

Both methods are documented and valid, but COPY --from= is actually the preferred approach in the uv docs
brent.edwards marked this conversation as resolved
docker-compose.yml
@ -0,0 +1,49 @@
version: '3.8'
brent.edwards commented 2025-12-04 01:34:27 +00:00
Member
Copy link

When I try running docker compose up, I get the following:

❯ docker compose up
WARN[0000] /home/brent.edwards/Workspace-2/dataset-uploader/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 1/1
 ! cleverdatasets Warning pull access denied for cleverdatasets, repository does not exist or may require 'docker login': denied: requested access to the resource is denied                                           1.6s 
WARN[0001] Docker Compose is configured to build using Bake, but buildx isn't installed 
[+] Building 18.0s (21/21) FINISHED                                                                                                                                                                          docker:default
 => [cleverdatasets internal] load build definition from Dockerfile                                                                                                                                                    0.2s
 => => transferring dockerfile: 1.67kB                                                                                                                                                                                 0.0s
 => [cleverdatasets] resolve image config for docker-image://docker.io/docker/dockerfile:1                                                                                                                             1.3s
 => CACHED [cleverdatasets] docker-image://docker.io/docker/dockerfile:1@sha256:b6afd42430b15f2d2a4c5a02b919e98a525b785b1aaff16747d2f623364e39b6                                                                       0.0s
 => [cleverdatasets internal] load metadata for docker.io/library/python:3.13-slim                                                                                                                                     0.5s 
 => [cleverdatasets internal] load metadata for ghcr.io/astral-sh/uv:latest                                                                                                                                            0.7s
 => [cleverdatasets internal] load .dockerignore                                                                                                                                                                       0.0s
 => => transferring context: 980B                                                                                                                                                                                      0.0s 
 => [cleverdatasets stage-0  1/11] FROM docker.io/library/python:3.13-slim@sha256:05b118ecc93ea09e30569706568fb251c71b77d2a3908d338b77be033e162b59                                                                     0.0s 
 => [cleverdatasets internal] load build context                                                                                                                                                                       0.1s 
 => => transferring context: 2.09kB                                                                                                                                                                                    0.0s 
 => [cleverdatasets] FROM ghcr.io/astral-sh/uv:latest@sha256:4c1ad814fe658851f50ff95ecd6948673fffddb0d7994bdb019dcb58227abd52                                                                                          0.0s 
 => CACHED [cleverdatasets stage-0  2/11] RUN apt-get update && apt-get install -y --no-install-recommends     bzip2     gzip     curl     ca-certificates     && rm -rf /var/lib/apt/lists/*                          0.0s 
 => CACHED [cleverdatasets stage-0  3/11] COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv                                                                                                                0.0s 
 => [cleverdatasets stage-0  4/11] WORKDIR /app                                                                                                                                                                        0.1s 
 => [cleverdatasets stage-0  5/11] COPY pyproject.toml ./                                                                                                                                                              0.2s 
 => [cleverdatasets stage-0  6/11] RUN uv pip install --system --no-cache-dir     rdflib>=7.0.0     datasets>=2.0.0     huggingface-hub     rich>=13.0.0     httpx>=0.25.0     pyarrow>=14.0.0     pandas>=2.0.0       8.9s 
 => [cleverdatasets stage-0  7/11] COPY scripts/ ./scripts/                                                                                                                                                            0.3s 
 => [cleverdatasets stage-0  8/11] COPY src/ ./src/                                                                                                                                                                    0.2s 
 => [cleverdatasets stage-0  9/11] RUN mkdir -p /data/dataset_processing &&     mkdir -p /data/.cache/huggingface                                                                                                      1.5s 
 => [cleverdatasets stage-0 10/11] RUN useradd -m -u 1000 -s /bin/bash appuser &&     chown -R appuser:appuser /app /data                                                                                              0.7s 
 => [cleverdatasets stage-0 11/11] WORKDIR /data                                                                                                                                                                       0.2s 
 => [cleverdatasets] exporting to image                                                                                                                                                                                2.4s 
 => => exporting layers                                                                                                                                                                                                2.3s 
 => => writing image sha256:6085c144bd63a63dd576fe5069a75dfbd79c37919d69793563e0bcf593e68ac8                                                                                                                           0.0s 
 => => naming to docker.io/library/cleverdatasets:latest                                                                                                                                                               0.0s 
 => [cleverdatasets] resolving provenance for metadata file                                                                                                                                                            0.0s 
[+] Running 3/3                                                                                                                                                                                                             
 ✔ cleverdatasets                             Built                                                                                                                                                                    0.0s 
 ✔ Volume dataset-uploader_huggingface_cache  Created                                                                                                                                                                  0.4s 
 ✔ Container cleverdatasets                   Created                                                                                                                                                                  1.6s 
Attaching to cleverdatasets
cleverdatasets  | usage: upload_all_datasets.py [-h] [--dataset DATASET]
cleverdatasets  |                               [--category {small,medium,large,xlarge,all}]
cleverdatasets  |                               [--base-dir BASE_DIR] [--skip-download]
cleverdatasets  |                               [--skip-convert] [--skip-upload] [--dry-run]
cleverdatasets  |                               [--list] [--rm] [--parallel PARALLEL]
cleverdatasets  | 
cleverdatasets  | Download, convert, and upload RDF datasets to HuggingFace
cleverdatasets  | 
cleverdatasets  | options:
cleverdatasets  |   -h, --help            show this help message and exit
cleverdatasets  |   --dataset, -d DATASET
cleverdatasets  |                         Specific dataset(s) to process (can be specified
cleverdatasets  |                         multiple times)
cleverdatasets  |   --category, -c {small,medium,large,xlarge,all}
cleverdatasets  |                         Process datasets by category
cleverdatasets  |   --base-dir BASE_DIR   Base directory for work
cleverdatasets  |   --skip-download       Skip download step
cleverdatasets  |   --skip-convert        Skip conversion step
cleverdatasets  |   --skip-upload         Skip upload step
cleverdatasets  |   --dry-run             Show what would be done without doing it
cleverdatasets  |   --list                List datasets that would be processed
cleverdatasets  |   --rm                  Remove mode: wipe all downloads at start, delete each
cleverdatasets  |                         dataset after processing
cleverdatasets  |   --parallel, -p PARALLEL
cleverdatasets  |                         Number of datasets to process in parallel (default: 1,
cleverdatasets  |                         sequential)
cleverdatasets exited with code 0 (restarting)

and the --help message keeps repeating.

Is this the expected behavior?

When I try running `docker compose up`, I get the following: ``` ❯ docker compose up WARN[0000] /home/brent.edwards/Workspace-2/dataset-uploader/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion [+] Running 1/1 ! cleverdatasets Warning pull access denied for cleverdatasets, repository does not exist or may require 'docker login': denied: requested access to the resource is denied 1.6s WARN[0001] Docker Compose is configured to build using Bake, but buildx isn't installed [+] Building 18.0s (21/21) FINISHED docker:default => [cleverdatasets internal] load build definition from Dockerfile 0.2s => => transferring dockerfile: 1.67kB 0.0s => [cleverdatasets] resolve image config for docker-image://docker.io/docker/dockerfile:1 1.3s => CACHED [cleverdatasets] docker-image://docker.io/docker/dockerfile:1@sha256:b6afd42430b15f2d2a4c5a02b919e98a525b785b1aaff16747d2f623364e39b6 0.0s => [cleverdatasets internal] load metadata for docker.io/library/python:3.13-slim 0.5s => [cleverdatasets internal] load metadata for ghcr.io/astral-sh/uv:latest 0.7s => [cleverdatasets internal] load .dockerignore 0.0s => => transferring context: 980B 0.0s => [cleverdatasets stage-0 1/11] FROM docker.io/library/python:3.13-slim@sha256:05b118ecc93ea09e30569706568fb251c71b77d2a3908d338b77be033e162b59 0.0s => [cleverdatasets internal] load build context 0.1s => => transferring context: 2.09kB 0.0s => [cleverdatasets] FROM ghcr.io/astral-sh/uv:latest@sha256:4c1ad814fe658851f50ff95ecd6948673fffddb0d7994bdb019dcb58227abd52 0.0s => CACHED [cleverdatasets stage-0 2/11] RUN apt-get update && apt-get install -y --no-install-recommends bzip2 gzip curl ca-certificates && rm -rf /var/lib/apt/lists/* 0.0s => CACHED [cleverdatasets stage-0 3/11] COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv 0.0s => [cleverdatasets stage-0 4/11] WORKDIR /app 0.1s => [cleverdatasets stage-0 5/11] COPY pyproject.toml ./ 0.2s => [cleverdatasets stage-0 6/11] RUN uv pip install --system --no-cache-dir rdflib>=7.0.0 datasets>=2.0.0 huggingface-hub rich>=13.0.0 httpx>=0.25.0 pyarrow>=14.0.0 pandas>=2.0.0 8.9s => [cleverdatasets stage-0 7/11] COPY scripts/ ./scripts/ 0.3s => [cleverdatasets stage-0 8/11] COPY src/ ./src/ 0.2s => [cleverdatasets stage-0 9/11] RUN mkdir -p /data/dataset_processing && mkdir -p /data/.cache/huggingface 1.5s => [cleverdatasets stage-0 10/11] RUN useradd -m -u 1000 -s /bin/bash appuser && chown -R appuser:appuser /app /data 0.7s => [cleverdatasets stage-0 11/11] WORKDIR /data 0.2s => [cleverdatasets] exporting to image 2.4s => => exporting layers 2.3s => => writing image sha256:6085c144bd63a63dd576fe5069a75dfbd79c37919d69793563e0bcf593e68ac8 0.0s => => naming to docker.io/library/cleverdatasets:latest 0.0s => [cleverdatasets] resolving provenance for metadata file 0.0s [+] Running 3/3 ✔ cleverdatasets Built 0.0s ✔ Volume dataset-uploader_huggingface_cache Created 0.4s ✔ Container cleverdatasets Created 1.6s Attaching to cleverdatasets cleverdatasets | usage: upload_all_datasets.py [-h] [--dataset DATASET] cleverdatasets | [--category {small,medium,large,xlarge,all}] cleverdatasets | [--base-dir BASE_DIR] [--skip-download] cleverdatasets | [--skip-convert] [--skip-upload] [--dry-run] cleverdatasets | [--list] [--rm] [--parallel PARALLEL] cleverdatasets | cleverdatasets | Download, convert, and upload RDF datasets to HuggingFace cleverdatasets | cleverdatasets | options: cleverdatasets | -h, --help show this help message and exit cleverdatasets | --dataset, -d DATASET cleverdatasets | Specific dataset(s) to process (can be specified cleverdatasets | multiple times) cleverdatasets | --category, -c {small,medium,large,xlarge,all} cleverdatasets | Process datasets by category cleverdatasets | --base-dir BASE_DIR Base directory for work cleverdatasets | --skip-download Skip download step cleverdatasets | --skip-convert Skip conversion step cleverdatasets | --skip-upload Skip upload step cleverdatasets | --dry-run Show what would be done without doing it cleverdatasets | --list List datasets that would be processed cleverdatasets | --rm Remove mode: wipe all downloads at start, delete each cleverdatasets | dataset after processing cleverdatasets | --parallel, -p PARALLEL cleverdatasets | Number of datasets to process in parallel (default: 1, cleverdatasets | sequential) cleverdatasets exited with code 0 (restarting) ``` and the `--help` message keeps repeating. Is this the expected behavior?
aditya commented 2025-12-04 13:26:47 +00:00
Author
Member
Copy link

fixed!! Changed restart: unless-stopped to restart: on-failure

fixed!! Changed restart: unless-stopped to restart: on-failure
brent.edwards marked this conversation as resolved
brent.edwards commented 2025-12-04 22:27:38 +00:00
Member
Copy link

Great work!

Great work!
khird approved these changes 2025-12-05 15:05:21 +00:00
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin 7-dockerfile-setup:7-dockerfile-setup
git switch 7-dockerfile-setup

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch mypy-and-pylint
git merge --no-ff 7-dockerfile-setup
git switch 7-dockerfile-setup
git rebase mypy-and-pylint
git switch mypy-and-pylint
git merge --ff-only 7-dockerfile-setup
git switch 7-dockerfile-setup
git rebase mypy-and-pylint
git switch mypy-and-pylint
git merge --no-ff 7-dockerfile-setup
git switch mypy-and-pylint
git merge --squash 7-dockerfile-setup
git switch mypy-and-pylint
git merge --ff-only 7-dockerfile-setup
git switch mypy-and-pylint
git merge 7-dockerfile-setup
git push origin mypy-and-pylint
Sign in to join this conversation.
Reviewers
No reviewers
brent.edwards
khird
Labels
Clear labels
  
Blocked

A ticket in a blocked state and unable to complete until some other task is completed first.

  
Bounty
$100
A bounty of $100 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$1000
A bounty of $1000 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$10000
A bounty of $10000 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$20
A bounty of $20 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$2000
A bounty of $2000 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$250
A bounty of $250 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$50
A bounty of $50 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$500
A bounty of $500 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$5000
A bounty of $5000 for any open-source contributor who provides a MR that solves this issue

  
Bounty
$750
A bounty of $750 for any open-source contributor who provides a MR that solves this issue

  
MoSCoW
Could have
Could have feature in order to satisfy the epic/legendary.

  
MoSCoW
Must have
Must have feature in order to satisfy the epic/legendary.

  
MoSCoW
Should have
Should have feature in order to satisfy the epic/legendary.

  
Needs feedback

There are questions in the ticket that can not be completed until the project owner provides clarity.

  
Points
1
1 man-hours worth of work for an expert with no learning curve.

  
Points
13
13 man-hours worth of work for an expert with no learning curve.

  
Points
2
2 man-hours worth of work for an expert with no learning curve.

  
Points
21
21 man-hours worth of work for an expert with no learning curve.

  
Points
3
3 man-hours worth of work for an expert with no learning curve.

  
Points
34
34 man-hours worth of work for an expert with no learning curve.

  
Points
5
5 man-hours worth of work for an expert with no learning curve.

  
Points
55
55 man-hours worth of work for an expert with no learning curve.

  
Points
8
8 man-hours worth of work for an expert with no learning curve.

  
Points
88
88 man-hours worth of work for an expert with no learning curve.

  
Priority
Backlog
This ticket has backlogged priority and is not to be worked on yet

  
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Medium
The priority is medium

  
Signed-off: Owner

When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.

  
Signed-off: Scrum Master

When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.

  
Signed-off: Tech Lead

When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.

  
Spike

A ticket for learning a tool or technology that is needed to be able to do future planning and design.

  
State
Completed
The ticket has been fully implemented, completed, and merged with the source code. This label should only be applied once a ticket is closed.

  
State
Duplicate
A ticket that represents the same content as an existing ticket.

  
State
In Progress
A ticket that is actively being developed.

  
State
In Review
A ticket that has had some code completed to implement but is waiting to pass peer review and is not yet merged in.

  
State
Paused
This ticket's work started but wasn't finished. It's on hold (likely in a feature branch) and will be resumed later, either due to a blocker or a delay.

  
State
Unverified
All new tickets start in this state. A developer may set it to show the ticket is unverified. This means we haven't agreed to work on it. It will either move to a verified state or be closed as wontdo.

  
State
Verified
The issue has been verified by a developer as legitimate. It will be worked on and verified tickets are now considered part of the backlog.

  
State
Wont Do
This ticket has been decided it wont be done. This may mean the bug has been determined to not be real (cant verify) or the feature is one we have decided we dont want to adopt.

  
Type
Bug
Something that doesnt work as intended.

  
Type
Discussion
Anytime a ticket represents a discussion about a subject and doesnt fall into one of the other categories.

  
Type
Documentation
An error or improvement needed in the documentation.

  
Type
Epic
Any first tier epic. That is, an epic which contains only issues as children and will not have sub-epics.

  
Type
Feature
Some new functionality not present.

  
Type
Legendary
A type of Epic which will contain other Epics.

  
Type
Support
Someone needs help using the project.

  
Type
Task
A generic task that doesnt fit into the other type categories.

  
Type
Testing
Work exclusively focusing on fixing or expanding testing.

No labels
Blocked
Bounty
$100
Bounty
$1000
Bounty
$10000
Bounty
$20
Bounty
$2000
Bounty
$250
Bounty
$50
Bounty
$500
Bounty
$5000
Bounty
$750
MoSCoW
Could have
MoSCoW
Must have
MoSCoW
Should have
Needs feedback
Points
1
Points
13
Points
2
Points
21
Points
3
Points
34
Points
5
Points
55
Points
8
Points
88
Priority
Backlog
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Signed-off: Owner
Signed-off: Scrum Master
Signed-off: Tech Lead
Spike
State
Completed
State
Duplicate
State
In Progress
State
In Review
State
Paused
State
Unverified
State
Verified
State
Wont Do
Type
Bug
Type
Discussion
Type
Documentation
Type
Epic
Type
Feature
Type
Legendary
Type
Support
Type
Task
Type
Testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleverdatasets/dataset-uploader!24
No description provided.