cleverdatasets/dataset-uploader
0
0
Fork 0
Code Issues 31 Pull Requests 7 Actions 12 Packages Projects Releases Wiki Activity

stream-download-convert-upload-merge-15-refactor-rdf-converters #45

Open
brent.edwards wants to merge 99 commits from stream-download-convert-upload-merge-15-refactor-rdf-converters into master
pull from: stream-download-convert-upload-merge-15-refactor-rdf-converters
merge into: cleverdatasets:master
Conversation 1 Commits 99 Files Changed 51
+20037 -5833
brent.edwards commented 2026-01-05 15:25:57 +00:00
Member
Copy Link
Copy Source

Trying to merge as much as possible into the master branch.

Trying to merge as much as possible into the master branch.
brent.edwards added 81 commits 2026-01-05 15:25:57 +00:00
Mostly about convert_conceptnet_progress. b16eddbe71
convert_conceptnet_to_hf. c9c6bff859
convert_fb15k237_to_hf 927a8648f6
convert_nell995_to_hf c058301d1c
convert_rdf_progress ec1211ba51
convert_rdf_streaming_progress 4636770092
Fixes made by ruff format. 7f50361683
feat: enhance error handling and reporting in dataset processing scripts e3bee8af4f 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
feat: improve error handling and logging in RDF dataset downloader e1cabc7069 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
feat: improve error handling for FB15k-237 dataset conversion 576dc842a9 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
fix: correct indentation in FB15k-237 file parsing function 163268b40d 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
refactor: improve error handling for NELL-995 dataset conversion 3862bf2655 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
fix: correct indentation in NELL-995 dataset converter script e6e1932f66 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
refactor: improve error handling and logging in ConceptNet conversion script 24869bef6c 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
fix: correct indentation in ConceptNet CSV parsing function e7bfa3f2c9 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
feat: add comprehensive error handling for RDF to HuggingFace dataset conversion 67a9857ab3 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
One more set of changes for error messages. 69de631c84
Merge branch '16-error-messages-and-logging' into mypy-and-pylint aa069f7527
Changes made by ruff check --fix. c5707689d6
fix: fix ruff check scripts/ errors 7414b333f8
Changes made to make the downloader work. c776025b0f
Refactor: Extract helper functions to remove code duplication a4eca8bcaa
Fix: Pyright type checking errors and improve type annotations 001234afcf
One-half of the conversion is done, and the structure is written. 6b73289c24
Small updates. 33e004a08e
Feat: add initial Dockerfile and compose file for the repo bd44f34ade
upload_all_datasets.2.py written. Will run overnight. 1d5e791963
Minor changes; passing to Aditya. 59e1d8d8bd
Fix: fix bugs in dataset upload and conversion scripts db7733d630
feat(devcontainer): optimize for PyCharm with workspace path fixes 1c73906ea2
Added some fixes to run the wikidad-truthy set. 3d2a9b75a2 
Please enter the commit message for your changes. Lines starting
Parallelized the call to dataset_dict.save_to_disk. 68587e5e6c
Replacing upload_all_datasets.py with the previous 9534336c15 
upload_all_datasets.2.py
fix(docker): prevent container restart loop 0e17022093
chore: correct the project name 3b57467d9e
Changes requested by Kyle; thank you! a6928d0ce2
Final version; what was used to send wikidata-truthy. 37b3ee6e91
feat: add download via ftp functionality a9b6518839
fix: resolve resume functionality crash for complete files 2342d2dada
Minor improvements. bac71ff1cc
style: simplify string formatting and type hints in script 0f3e5f2e30
refactor: implement true streaming approach for RDF to dataset conversion 29c71c5aea 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
fix: remove undefined variables and redundant code in dataset conversion script eba86c37be 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
More changes. 18a724eef9
fix: resolve pickling error in streaming converter generator 5a3b22d47f
fix: fix minor linting errors efa08d5c7d
fix: fix ruff check related errors 1e95cb9a91
THIS DOES NOT WORK. 4da8793b18
–fix: add hugging face push_to_hub()to upload the dataset directly to hugging face 0a9e8da028
Added .xz compression to code. 8bef6986ea
Fixing dataset_config to download one file. 6f12a78c2c
fix: correct xz decompression command and update error messages e0b0836ffc 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
fix: update error message for missing xz command 757d6dd7f8 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
Fixes suggested by Aider. 0784e3e275
Merge branch '35-xz-compression-aider' into 35-xz-compression 1df9fcbdf6
fix: improve RDF format detection and parsing for UniProt dataset ff3939d937 
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
feat: add rdf_to_hf_incremental.py script for true streamingof entire pipeline b218ab1410
Added dataset_config. 16fd345664
Fixing dataset_config to download one file. a51f1f3703
feat: add script to insert entries into dataset_config.json a01336cd15 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
Added all appropriate files from https://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/ 2ea6f3a89e
Removed the quickly-written scripts to edit dataset_config. c152323097
–fix: Address code review feedback - deduplication, format support, and error handling improvements 711ee61f9a
feat: add --list --dry-run dataset card feature d5ac95f2aa
Merge branch '35-xz-compression' into stream-download-convert-upload d564739f72
Merge branch '7-dockerfile-setup' into stream-download-convert-upload bbc0d61039
Merge branch '8-devcontainer-setup' into stream-download-convert-upload c487b49f11
Adding the features file. eb8bfdbee5
feat: unify RDF to HF dataset conversion with improved strategies a7f7d279b4 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
Moved the new unified code. 02f740ab81
fix: complete the unified script to add add the new features as streaming parallel; update the upload_all_dataset.py to use the unified script 5b215aff76
Merge pull request 'fix: complete the unified script to add add the new features as streaming parallel; update the upload_all_dataset.py to use the unified script' (#41) from unified-final into stream-download-convert-upload-merge-15-refactor-rdf-converters 7c93adddc1 
Reviewed-on: #41
Reviewed-by: Brent Edwards <brent.edwards@cleverthis.com>
fix: fix path too long issue; fix pickling issue in unified converter script 1bb26305ce
A few minor fixes. 1d33b67198
More work toward pyright approval. e7a540aeb3
fix: fix chunksize argument in multiprocessing as per review 90d77a7077
fix: fix ruff check errors 2b1537d4f9
Adding shell.nix. fbaa30e904
Updating pyproject to require Python 3.10. da9f785326
Minor changes. f8a7739016
Merge pull request 'fix: fix path too long issue; fix pickling issue in unified converter script' (#42) from path-too-long-fix into stream-download-convert-upload-merge-15-refactor-rdf-converters 76de466d99 
Reviewed-on: #42
Reviewed-by: Brent Edwards <brent.edwards@cleverthis.com>
brent.edwards added 9 commits 2026-01-09 00:49:24 +00:00
refactor: remove unused variable and simplify exception handling 72d4b9a9ba
style: break docstring lines to adhere to 88-character limit 6f4d9af488 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
style: break long docstring line for readability c120b2b793 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
fix: remove trailing whitespace in docstring c3fcd5adb1 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
style: remove trailing whitespace in docstring 8921dc31ca 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
style: break long f-string into multiple lines for readability 21c9acb4bf 
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
Changes made first by Aider, then by me. Aider was too slow. 18a146faad
Minor changes to remove unused variables. 8329e7dd2c
Merge pull request 'sdcum15rrc-aider' (#47) from sdcum15rrc-aider into stream-download-convert-upload-merge-15-refactor-rdf-converters 0de1d0ca9c 
Reviewed-on: #47
brent.edwards added 2 commits 2026-01-09 23:49:22 +00:00
ruff check now passes everything in scripts. a508e8c349
Last set of changes. 1a495c04da
brent.edwards commented 2026-01-09 23:52:51 +00:00
Author
Member
Copy Link
Copy Source

I have updated this to make pyright and ruff check pass everything.

The only exception:

❯ pyright scripts/*.py
/home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:30:8 - warning: Import "psutil" could not be resolved from source (reportMissingModuleSource)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:33:6 - error: Import "test_phases.compatibility_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:34:6 - error: Import "test_phases.deployment_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:35:6 - error: Import "test_phases.huggingface_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:36:6 - error: Import "test_phases.model_capability_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:37:6 - error: Import "test_phases.performance_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:40:6 - error: Import "test_phases.training_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:41:6 - error: Import "utils.monitoring" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:42:6 - error: Import "utils.reporting" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:43:6 - error: Import "utils.resource_manager" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:46:6 - error: Import "utils.test_environment" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:244:16 - error: Import "torch" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:259:16 - error: Import "torch" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:680:20 - error: Import "torch" could not be resolved (reportMissingImports)
13 errors, 1 warning, 0 informations

@aditya --

Should these be added to the pyproject.toml file, or is the integration_test_orchestrator.py part of the main code base still? (Its name says that it should be part of testing.)

I have updated this to make `pyright` and `ruff check` pass everything. The only exception: ``` ❯ pyright scripts/*.py /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:30:8 - warning: Import "psutil" could not be resolved from source (reportMissingModuleSource) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:33:6 - error: Import "test_phases.compatibility_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:34:6 - error: Import "test_phases.deployment_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:35:6 - error: Import "test_phases.huggingface_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:36:6 - error: Import "test_phases.model_capability_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:37:6 - error: Import "test_phases.performance_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:40:6 - error: Import "test_phases.training_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:41:6 - error: Import "utils.monitoring" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:42:6 - error: Import "utils.reporting" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:43:6 - error: Import "utils.resource_manager" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:46:6 - error: Import "utils.test_environment" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:244:16 - error: Import "torch" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:259:16 - error: Import "torch" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:680:20 - error: Import "torch" could not be resolved (reportMissingImports) 13 errors, 1 warning, 0 informations ``` @aditya -- Should these be added to the `pyproject.toml` file, or is the `integration_test_orchestrator.py` part of the main code base still? (Its name says that it should be part of testing.)
brent.edwards added 1 commit 2026-01-10 00:05:56 +00:00
Merge branch 'master' into stream-download-convert-upload-merge-15-refactor-rdf-converters d31ea170f4
brent.edwards added 2 commits 2026-01-10 01:31:38 +00:00
style: apply linter and improve code quality in extract_stream.py b442f1a56d 
Co-authored-by: aider (openrouter/anthropic/claude-3.7-sonnet) <aider@aider.chat>
Slightly made rdf_to_hf_incremental.py a little easier to test. b9f69317ab
aditya added 4 commits 2026-01-14 14:24:07 +00:00
feat: add streaming of rdf xml dataset with resume functionality 6ff2719a8e
fix: parse xml file with xml.etree.ElementTree; fix ruff and pyright errors 9ccdd38e91
fix: some minor fixes 255ccad4d4
fix: fix minor decompressed_byte bug b696d86835
This pull request can be merged automatically.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin stream-download-convert-upload-merge-15-refactor-rdf-converters:stream-download-convert-upload-merge-15-refactor-rdf-converters
git checkout stream-download-convert-upload-merge-15-refactor-rdf-converters
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
Blocked
A ticket in a blocked state and unable to complete until some other task is completed first.
Bounty
$100
A bounty of $100 for any open-source contributor who provides a MR that solves this issue
Bounty
$1000
A bounty of $1000 for any open-source contributor who provides a MR that solves this issue
Bounty
$10000
A bounty of $10000 for any open-source contributor who provides a MR that solves this issue
Bounty
$20
A bounty of $20 for any open-source contributor who provides a MR that solves this issue
Bounty
$2000
A bounty of $2000 for any open-source contributor who provides a MR that solves this issue
Bounty
$250
A bounty of $250 for any open-source contributor who provides a MR that solves this issue
Bounty
$50
A bounty of $50 for any open-source contributor who provides a MR that solves this issue
Bounty
$500
A bounty of $500 for any open-source contributor who provides a MR that solves this issue
Bounty
$5000
A bounty of $5000 for any open-source contributor who provides a MR that solves this issue
Bounty
$750
A bounty of $750 for any open-source contributor who provides a MR that solves this issue
MoSCoW
Could have
Could have feature in order to satisfy the epic/legendary.
MoSCoW
Must have
Must have feature in order to satisfy the epic/legendary.
MoSCoW
Should have
Should have feature in order to satisfy the epic/legendary.
Needs feedback
There are questions in the ticket that can not be completed until the project owner provides clarity.
Points
1
1 man-hours worth of work for an expert with no learning curve.
Points
13
13 man-hours worth of work for an expert with no learning curve.
Points
2
2 man-hours worth of work for an expert with no learning curve.
Points
21
21 man-hours worth of work for an expert with no learning curve.
Points
3
3 man-hours worth of work for an expert with no learning curve.
Points
34
34 man-hours worth of work for an expert with no learning curve.
Points
5
5 man-hours worth of work for an expert with no learning curve.
Points
55
55 man-hours worth of work for an expert with no learning curve.
Points
8
8 man-hours worth of work for an expert with no learning curve.
Points
88
88 man-hours worth of work for an expert with no learning curve.
Priority
Backlog
This ticket has backlogged priority and is not to be worked on yet
Priority
Critical
The priority is critical
Priority
High
The priority is high
Priority
Low
The priority is low
Priority
Medium
The priority is medium
Signed-off: Owner
When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.
 Signed-off: Scrum Master
When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.
 Signed-off: Tech Lead
When an epic or legendary is in review it must be signed off by owner, tech lead, and scrum master before being marked as completed.
 Spike
A ticket for learning a tool or technology that is needed to be able to do future planning and design.
State
Completed
The ticket has been fully implemented, completed, and merged with the source code. This label should only be applied once a ticket is closed.
State
Duplicate
A ticket that represents the same content as an existing ticket.
State
In Progress
A ticket that is actively being developed.
State
In Review
A ticket that has had some code completed to implement but is waiting to pass peer review and is not yet merged in.
State
Paused
This ticket's work started but wasn't finished. It's on hold (likely in a feature branch) and will be resumed later, either due to a blocker or a delay.
State
Unverified
All new tickets start in this state. A developer may set it to show the ticket is unverified. This means we haven't agreed to work on it. It will either move to a verified state or be closed as wontdo.
State
Verified
The issue has been verified by a developer as legitimate. It will be worked on and verified tickets are now considered part of the backlog.
State
Wont Do
This ticket has been decided it wont be done. This may mean the bug has been determined to not be real (cant verify) or the feature is one we have decided we dont want to adopt.
Type
Bug
Something that doesnt work as intended.
Type
Discussion
Anytime a ticket represents a discussion about a subject and doesnt fall into one of the other categories.
Type
Documentation
An error or improvement needed in the documentation.
Type
Epic
Any first tier epic. That is, an epic which contains only issues as children and will not have sub-epics.
Type
Feature
Some new functionality not present.
Type
Legendary
A type of Epic which will contain other Epics.
Type
Support
Someone needs help using the project.
Type
Task
A generic task that doesnt fit into the other type categories.
Type
Testing
Work exclusively focusing on fixing or expanding testing.
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
aditya (Aditya Chhabra) aleenaumair (Aleena Umair) brent.edwards (Brent Edwards) freemo (Jeffrey Phillips Freeman)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: cleverdatasets/dataset-uploader#45
Delete Branch "stream-download-convert-upload-merge-15-refactor-rdf-converters"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?