 aditya
|
b696d86835
|
fix: fix minor decompressed_byte bug
|
2026-01-14 18:14:11 +05:30 |
|
|
aditya
|
255ccad4d4
|
fix: some minor fixes
|
2026-01-13 17:19:54 +05:30 |
|
|
aditya
|
9ccdd38e91
|
fix: parse xml file with xml.etree.ElementTree; fix ruff and pyright errors
|
2026-01-13 17:19:50 +05:30 |
|
|
aditya
|
6ff2719a8e
|
feat: add streaming of rdf xml dataset with resume functionality
|
2026-01-13 17:18:09 +05:30 |
|
 brent.edwards
|
b9f69317ab
|
Slightly made rdf_to_hf_incremental.py a little easier to test.
|
2026-01-09 17:26:04 -08:00 |
|
|
brent.edwards
|
b442f1a56d
|
style: apply linter and improve code quality in extract_stream.py
Co-authored-by: aider (openrouter/anthropic/claude-3.7-sonnet) <aider@aider.chat>
|
2026-01-09 16:43:07 -08:00 |
|
|
brent.edwards
|
d31ea170f4
|
Merge branch 'master' into stream-download-convert-upload-merge-15-refactor-rdf-converters
|
2026-01-09 16:05:48 -08:00 |
|
|
brent.edwards
|
1a495c04da
|
Last set of changes.
|
2026-01-09 15:47:21 -08:00 |
|
|
brent.edwards
|
a508e8c349
|
ruff check now passes everything in scripts.
|
2026-01-09 15:17:32 -08:00 |
|
|
brent.edwards
|
0de1d0ca9c
|
Merge pull request 'sdcum15rrc-aider' (#47) from sdcum15rrc-aider into stream-download-convert-upload-merge-15-refactor-rdf-converters
Reviewed-on: #47
|
2026-01-09 00:49:22 +00:00 |
|
|
brent.edwards
|
8329e7dd2c
|
Minor changes to remove unused variables.
|
2026-01-08 16:48:06 -08:00 |
|
|
brent.edwards
|
18a146faad
|
Changes made first by Aider, then by me. Aider was too slow.
|
2026-01-08 16:42:56 -08:00 |
|
|
brent.edwards
|
21c9acb4bf
|
style: break long f-string into multiple lines for readability
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2026-01-08 16:00:54 -08:00 |
|
|
brent.edwards
|
8921dc31ca
|
style: remove trailing whitespace in docstring
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2026-01-08 16:00:04 -08:00 |
|
|
brent.edwards
|
c3fcd5adb1
|
fix: remove trailing whitespace in docstring
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2026-01-08 15:59:14 -08:00 |
|
|
brent.edwards
|
c120b2b793
|
style: break long docstring line for readability
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2026-01-08 15:58:14 -08:00 |
|
|
brent.edwards
|
6f4d9af488
|
style: break docstring lines to adhere to 88-character limit
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2026-01-08 15:56:34 -08:00 |
|
|
brent.edwards
|
72d4b9a9ba
|
refactor: remove unused variable and simplify exception handling
|
2026-01-08 15:45:11 -08:00 |
|
|
aditya
|
76de466d99
|
Merge pull request 'fix: fix path too long issue; fix pickling issue in unified converter script' (#42) from path-too-long-fix into stream-download-convert-upload-merge-15-refactor-rdf-converters
Reviewed-on: #42
Reviewed-by: Brent Edwards <brent.edwards@cleverthis.com>
|
2026-01-02 10:12:23 +00:00 |
|
|
brent.edwards
|
f8a7739016
|
Minor changes.
|
2025-12-30 18:06:34 -08:00 |
|
|
brent.edwards
|
a22ea08c04
|
I don't know why pyproject was wrong. Updating.
|
2025-12-29 18:17:48 -08:00 |
|
|
brent.edwards
|
da9f785326
|
Updating pyproject to require Python 3.10.
|
2025-12-29 18:09:54 -08:00 |
|
|
brent.edwards
|
fbaa30e904
|
Adding shell.nix.
|
2025-12-29 14:19:34 -08:00 |
|
|
aditya
|
2b1537d4f9
|
fix: fix ruff check errors
|
2025-12-29 18:49:57 +05:30 |
|
|
aditya
|
90d77a7077
|
fix: fix chunksize argument in multiprocessing as per review
|
2025-12-29 18:47:22 +05:30 |
|
|
brent.edwards
|
e7a540aeb3
|
More work toward pyright approval.
|
2025-12-28 17:13:34 -08:00 |
|
|
brent.edwards
|
1d33b67198
|
A few minor fixes.
|
2025-12-26 18:49:33 -08:00 |
|
|
aditya
|
1bb26305ce
|
fix: fix path too long issue; fix pickling issue in unified converter script
|
2025-12-26 18:21:02 +05:30 |
|
|
aditya
|
7c93adddc1
|
Merge pull request 'fix: complete the unified script to add add the new features as streaming parallel; update the upload_all_dataset.py to use the unified script' (#41) from unified-final into stream-download-convert-upload-merge-15-refactor-rdf-converters
Reviewed-on: #41
Reviewed-by: Brent Edwards <brent.edwards@cleverthis.com>
|
2025-12-26 10:42:54 +00:00 |
|
|
aditya
|
5b215aff76
|
fix: complete the unified script to add add the new features as streaming parallel; update the upload_all_dataset.py to use the unified script
|
2025-12-24 21:00:32 +05:30 |
|
|
brent.edwards
|
02f740ab81
|
Moved the new unified code.
|
2025-12-23 17:51:11 -08:00 |
|
|
brent.edwards
|
a7f7d279b4
|
feat: unify RDF to HF dataset conversion with improved strategies
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-23 16:12:42 -08:00 |
|
|
brent.edwards
|
eb8bfdbee5
|
Adding the features file.
|
2025-12-23 15:07:41 -08:00 |
|
|
aditya
|
c487b49f11
|
Merge branch '8-devcontainer-setup' into stream-download-convert-upload
|
2025-12-23 20:53:51 +05:30 |
|
|
aditya
|
bbc0d61039
|
Merge branch '7-dockerfile-setup' into stream-download-convert-upload
|
2025-12-23 20:52:17 +05:30 |
|
|
aditya
|
d564739f72
|
Merge branch '35-xz-compression' into stream-download-convert-upload
|
2025-12-23 20:20:36 +05:30 |
|
|
aditya
|
d5ac95f2aa
|
feat: add --list --dry-run dataset card feature
|
2025-12-23 19:31:53 +05:30 |
|
|
aditya
|
711ee61f9a
|
fix: Address code review feedback - deduplication, format support, and error handling improvements
|
2025-12-23 19:28:21 +05:30 |
|
|
brent.edwards
|
c152323097
|
Removed the quickly-written scripts to edit dataset_config.
|
2025-12-22 18:50:23 -08:00 |
|
|
brent.edwards
|
2ea6f3a89e
|
Added all appropriate files from https://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/
|
2025-12-22 18:46:29 -08:00 |
|
|
brent.edwards
|
a01336cd15
|
feat: add script to insert entries into dataset_config.json
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-22 18:14:26 -08:00 |
|
|
brent.edwards
|
a51f1f3703
|
Fixing dataset_config to download one file.
|
2025-12-22 17:13:28 -08:00 |
|
|
brent.edwards
|
16fd345664
|
Added dataset_config.
|
2025-12-22 16:12:45 -08:00 |
|
|
aditya
|
b218ab1410
|
feat: add rdf_to_hf_incremental.py script for true streamingof entire pipeline
|
2025-12-19 20:20:51 +05:30 |
|
|
brent.edwards
|
ff3939d937
|
fix: improve RDF format detection and parsing for UniProt dataset
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-12-16 19:38:30 -08:00 |
|
|
brent.edwards
|
1df9fcbdf6
|
Merge branch '35-xz-compression-aider' into 35-xz-compression
|
2025-12-16 18:22:37 -08:00 |
|
|
brent.edwards
|
0784e3e275
|
Fixes suggested by Aider.
|
2025-12-16 18:21:25 -08:00 |
|
|
brent.edwards
|
757d6dd7f8
|
fix: update error message for missing xz command
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-16 18:08:30 -08:00 |
|
|
brent.edwards
|
e0b0836ffc
|
fix: correct xz decompression command and update error messages
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-16 18:07:58 -08:00 |
|
|
brent.edwards
|
6f12a78c2c
|
Fixing dataset_config to download one file.
|
2025-12-16 14:06:52 -08:00 |
|