stream-download-convert-upload-merge-15-refactor-rdf-converters #45

Open

brent.edwards wants to merge 99 commits from stream-download-convert-upload-merge-15-refactor-rdf-converters into master

brent.edwards commented

2026-01-05 15:25:57 +00:00

Member

Trying to merge as much as possible into the master branch.

brent.edwards added 81 commits

2026-01-05 15:25:57 +00:00

Mostly about convert_conceptnet_progress. b16eddbe71

convert_conceptnet_to_hf. c9c6bff859

convert_fb15k237_to_hf 927a8648f6

convert_nell995_to_hf c058301d1c

convert_rdf_progress ec1211ba51

convert_rdf_streaming_progress 4636770092

Fixes made by ruff format. 7f50361683

feat: enhance error handling and reporting in dataset processing scripts e3bee8af4f

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

feat: improve error handling and logging in RDF dataset downloader e1cabc7069

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

feat: improve error handling for FB15k-237 dataset conversion 576dc842a9

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

fix: correct indentation in FB15k-237 file parsing function 163268b40d

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

refactor: improve error handling for NELL-995 dataset conversion 3862bf2655

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

fix: correct indentation in NELL-995 dataset converter script e6e1932f66

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

refactor: improve error handling and logging in ConceptNet conversion script 24869bef6c

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

fix: correct indentation in ConceptNet CSV parsing function e7bfa3f2c9

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

feat: add comprehensive error handling for RDF to HuggingFace dataset conversion 67a9857ab3

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

One more set of changes for error messages. 69de631c84

Merge branch '16-error-messages-and-logging' into mypy-and-pylint aa069f7527

Changes made by ruff check --fix. c5707689d6

fix: fix ruff check scripts/ errors 7414b333f8

Changes made to make the downloader work. c776025b0f

Refactor: Extract helper functions to remove code duplication a4eca8bcaa

Fix: Pyright type checking errors and improve type annotations 001234afcf

One-half of the conversion is done, and the structure is written. 6b73289c24

Small updates. 33e004a08e

Feat: add initial Dockerfile and compose file for the repo bd44f34ade

upload_all_datasets.2.py written. Will run overnight. 1d5e791963

Minor changes; passing to Aditya. 59e1d8d8bd

Fix: fix bugs in dataset upload and conversion scripts db7733d630

feat(devcontainer): optimize for PyCharm with workspace path fixes 1c73906ea2

Added some fixes to run the wikidad-truthy set. 3d2a9b75a2

Please enter the commit message for your changes. Lines starting

Parallelized the call to dataset_dict.save_to_disk. 68587e5e6c

Replacing upload_all_datasets.py with the previous 9534336c15

upload_all_datasets.2.py

fix(docker): prevent container restart loop 0e17022093

chore: correct the project name 3b57467d9e

Changes requested by Kyle; thank you! a6928d0ce2

Final version; what was used to send wikidata-truthy. 37b3ee6e91

feat: add download via ftp functionality a9b6518839

fix: resolve resume functionality crash for complete files 2342d2dada

Minor improvements. bac71ff1cc

style: simplify string formatting and type hints in script 0f3e5f2e30

refactor: implement true streaming approach for RDF to dataset conversion 29c71c5aea

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

fix: remove undefined variables and redundant code in dataset conversion script eba86c37be

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

More changes. 18a724eef9

fix: resolve pickling error in streaming converter generator 5a3b22d47f

fix: fix minor linting errors efa08d5c7d

fix: fix ruff check related errors 1e95cb9a91

THIS DOES NOT WORK. 4da8793b18

fix: add hugging face push_to_hub()to upload the dataset directly to hugging face 0a9e8da028

Added .xz compression to code. 8bef6986ea

Fixing dataset_config to download one file. 6f12a78c2c

fix: correct xz decompression command and update error messages e0b0836ffc

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

fix: update error message for missing xz command 757d6dd7f8

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

Fixes suggested by Aider. 0784e3e275

Merge branch '35-xz-compression-aider' into 35-xz-compression 1df9fcbdf6

fix: improve RDF format detection and parsing for UniProt dataset ff3939d937

Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>

feat: add rdf_to_hf_incremental.py script for true streamingof entire pipeline b218ab1410

Added dataset_config. 16fd345664

Fixing dataset_config to download one file. a51f1f3703

feat: add script to insert entries into dataset_config.json a01336cd15

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

Added all appropriate files from https://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/ 2ea6f3a89e

Removed the quickly-written scripts to edit dataset_config. c152323097

fix: Address code review feedback - deduplication, format support, and error handling improvements 711ee61f9a

feat: add --list --dry-run dataset card feature d5ac95f2aa

Merge branch '35-xz-compression' into stream-download-convert-upload d564739f72

Merge branch '7-dockerfile-setup' into stream-download-convert-upload bbc0d61039

Merge branch '8-devcontainer-setup' into stream-download-convert-upload c487b49f11

Adding the features file. eb8bfdbee5

feat: unify RDF to HF dataset conversion with improved strategies a7f7d279b4

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

Moved the new unified code. 02f740ab81

fix: complete the unified script to add add the new features as streaming parallel; update the upload_all_dataset.py to use the unified script 5b215aff76

Merge pull request 'fix: complete the unified script to add add the new features as streaming parallel; update the upload_all_dataset.py to use the unified script' (#41 ) from unified-final into stream-download-convert-upload-merge-15-refactor-rdf-converters 7c93adddc1

Reviewed-on: #41
Reviewed-by: Brent Edwards <brent.edwards@cleverthis.com>

fix: fix path too long issue; fix pickling issue in unified converter script 1bb26305ce

A few minor fixes. 1d33b67198

More work toward pyright approval. e7a540aeb3

fix: fix chunksize argument in multiprocessing as per review 90d77a7077

fix: fix ruff check errors 2b1537d4f9

Adding shell.nix. fbaa30e904

Updating pyproject to require Python 3.10. da9f785326

Minor changes. f8a7739016

Merge pull request 'fix: fix path too long issue; fix pickling issue in unified converter script' (#42 ) from path-too-long-fix into stream-download-convert-upload-merge-15-refactor-rdf-converters 76de466d99

Reviewed-on: #42
Reviewed-by: Brent Edwards <brent.edwards@cleverthis.com>

brent.edwards added 9 commits

2026-01-09 00:49:24 +00:00

refactor: remove unused variable and simplify exception handling 72d4b9a9ba

style: break docstring lines to adhere to 88-character limit 6f4d9af488

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

style: break long docstring line for readability c120b2b793

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

fix: remove trailing whitespace in docstring c3fcd5adb1

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

style: remove trailing whitespace in docstring 8921dc31ca

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

style: break long f-string into multiple lines for readability 21c9acb4bf

Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>

Changes made first by Aider, then by me. Aider was too slow. 18a146faad

Minor changes to remove unused variables. 8329e7dd2c

Merge pull request 'sdcum15rrc-aider' (#47 ) from sdcum15rrc-aider into stream-download-convert-upload-merge-15-refactor-rdf-converters 0de1d0ca9c

Reviewed-on: #47

brent.edwards added 2 commits

2026-01-09 23:49:22 +00:00

ruff check now passes everything in scripts. a508e8c349

Last set of changes. 1a495c04da

brent.edwards commented

2026-01-09 23:52:51 +00:00

Author

Member

I have updated this to make pyright and ruff check pass everything.

The only exception:

❯ pyright scripts/*.py
/home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:30:8 - warning: Import "psutil" could not be resolved from source (reportMissingModuleSource)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:33:6 - error: Import "test_phases.compatibility_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:34:6 - error: Import "test_phases.deployment_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:35:6 - error: Import "test_phases.huggingface_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:36:6 - error: Import "test_phases.model_capability_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:37:6 - error: Import "test_phases.performance_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:40:6 - error: Import "test_phases.training_coordinator" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:41:6 - error: Import "utils.monitoring" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:42:6 - error: Import "utils.reporting" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:43:6 - error: Import "utils.resource_manager" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:46:6 - error: Import "utils.test_environment" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:244:16 - error: Import "torch" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:259:16 - error: Import "torch" could not be resolved (reportMissingImports)
  /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:680:20 - error: Import "torch" could not be resolved (reportMissingImports)
13 errors, 1 warning, 0 informations

@aditya --

Should these be added to the pyproject.toml file, or is the integration_test_orchestrator.py part of the main code base still? (Its name says that it should be part of testing.)

I have updated this to make `pyright` and `ruff check` pass everything. The only exception: ``` ❯ pyright scripts/*.py /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:30:8 - warning: Import "psutil" could not be resolved from source (reportMissingModuleSource) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:33:6 - error: Import "test_phases.compatibility_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:34:6 - error: Import "test_phases.deployment_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:35:6 - error: Import "test_phases.huggingface_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:36:6 - error: Import "test_phases.model_capability_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:37:6 - error: Import "test_phases.performance_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:40:6 - error: Import "test_phases.training_coordinator" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:41:6 - error: Import "utils.monitoring" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:42:6 - error: Import "utils.reporting" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:43:6 - error: Import "utils.resource_manager" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:46:6 - error: Import "utils.test_environment" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:244:16 - error: Import "torch" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:259:16 - error: Import "torch" could not be resolved (reportMissingImports) /home/brent.edwards/Workspace-2/dataset-uploader/scripts/integration_test_orchestrator.py:680:20 - error: Import "torch" could not be resolved (reportMissingImports) 13 errors, 1 warning, 0 informations ``` @aditya -- Should these be added to the `pyproject.toml` file, or is the `integration_test_orchestrator.py` part of the main code base still? (Its name says that it should be part of testing.)

brent.edwards added 1 commit

2026-01-10 00:05:56 +00:00

Merge branch 'master' into stream-download-convert-upload-merge-15-refactor-rdf-converters d31ea170f4

brent.edwards added 2 commits

2026-01-10 01:31:38 +00:00

style: apply linter and improve code quality in extract_stream.py b442f1a56d

Co-authored-by: aider (openrouter/anthropic/claude-3.7-sonnet) <aider@aider.chat>

Slightly made rdf_to_hf_incremental.py a little easier to test. b9f69317ab

aditya added 4 commits

2026-01-14 14:24:07 +00:00

feat: add streaming of rdf xml dataset with resume functionality 6ff2719a8e

fix: parse xml file with xml.etree.ElementTree; fix ruff and pyright errors 9ccdd38e91

fix: some minor fixes 255ccad4d4

fix: fix minor decompressed_byte bug b696d86835

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin stream-download-convert-upload-merge-15-refactor-rdf-converters:stream-download-convert-upload-merge-15-refactor-rdf-converters

git switch stream-download-convert-upload-merge-15-refactor-rdf-converters

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master

git merge --no-ff stream-download-convert-upload-merge-15-refactor-rdf-converters

git switch stream-download-convert-upload-merge-15-refactor-rdf-converters

git rebase master

git switch master

git merge --ff-only stream-download-convert-upload-merge-15-refactor-rdf-converters

git switch stream-download-convert-upload-merge-15-refactor-rdf-converters

git rebase master

git switch master

git merge --no-ff stream-download-convert-upload-merge-15-refactor-rdf-converters

git switch master

git merge --squash stream-download-convert-upload-merge-15-refactor-rdf-converters