 aditya
|
bbc0d61039
|
Merge branch '7-dockerfile-setup' into stream-download-convert-upload
|
2025-12-23 20:52:17 +05:30 |
|
|
aditya
|
d564739f72
|
Merge branch '35-xz-compression' into stream-download-convert-upload
|
2025-12-23 20:20:36 +05:30 |
|
|
aditya
|
d5ac95f2aa
|
feat: add --list --dry-run dataset card feature
|
2025-12-23 19:31:53 +05:30 |
|
|
aditya
|
711ee61f9a
|
fix: Address code review feedback - deduplication, format support, and error handling improvements
|
2025-12-23 19:28:21 +05:30 |
|
 brent.edwards
|
2ea6f3a89e
|
Added all appropriate files from https://ftp.uniprot.org/pub/databases/uniprot/current_release/rdf/
|
2025-12-22 18:46:29 -08:00 |
|
|
brent.edwards
|
a51f1f3703
|
Fixing dataset_config to download one file.
|
2025-12-22 17:13:28 -08:00 |
|
|
brent.edwards
|
16fd345664
|
Added dataset_config.
|
2025-12-22 16:12:45 -08:00 |
|
|
aditya
|
b218ab1410
|
feat: add rdf_to_hf_incremental.py script for true streamingof entire pipeline
|
2025-12-19 20:20:51 +05:30 |
|
|
brent.edwards
|
ff3939d937
|
fix: improve RDF format detection and parsing for UniProt dataset
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-12-16 19:38:30 -08:00 |
|
|
brent.edwards
|
1df9fcbdf6
|
Merge branch '35-xz-compression-aider' into 35-xz-compression
|
2025-12-16 18:22:37 -08:00 |
|
|
brent.edwards
|
0784e3e275
|
Fixes suggested by Aider.
|
2025-12-16 18:21:25 -08:00 |
|
|
brent.edwards
|
757d6dd7f8
|
fix: update error message for missing xz command
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-16 18:08:30 -08:00 |
|
|
brent.edwards
|
e0b0836ffc
|
fix: correct xz decompression command and update error messages
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-16 18:07:58 -08:00 |
|
|
brent.edwards
|
6f12a78c2c
|
Fixing dataset_config to download one file.
|
2025-12-16 14:06:52 -08:00 |
|
|
brent.edwards
|
8bef6986ea
|
Added .xz compression to code.
|
2025-12-16 14:00:06 -08:00 |
|
|
aditya
|
0a9e8da028
|
fix: add hugging face push_to_hub()to upload the dataset directly to hugging face
|
2025-12-15 19:31:10 +05:30 |
|
|
brent.edwards
|
4da8793b18
|
THIS DOES NOT WORK.
|
2025-12-11 21:26:52 -08:00 |
|
|
aditya
|
1e95cb9a91
|
fix: fix ruff check related errors
|
2025-12-11 19:01:35 +05:30 |
|
|
aditya
|
efa08d5c7d
|
fix: fix minor linting errors
|
2025-12-11 18:56:29 +05:30 |
|
|
aditya
|
5a3b22d47f
|
fix: resolve pickling error in streaming converter generator
|
2025-12-11 18:36:17 +05:30 |
|
|
brent.edwards
|
18a724eef9
|
More changes.
|
2025-12-09 20:01:45 -08:00 |
|
|
brent.edwards
|
eba86c37be
|
fix: remove undefined variables and redundant code in dataset conversion script
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-09 20:00:50 -08:00 |
|
|
brent.edwards
|
29c71c5aea
|
refactor: implement true streaming approach for RDF to dataset conversion
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
|
2025-12-09 20:00:11 -08:00 |
|
|
brent.edwards
|
0f3e5f2e30
|
style: simplify string formatting and type hints in script
|
2025-12-09 20:00:04 -08:00 |
|
|
brent.edwards
|
bac71ff1cc
|
Minor improvements.
|
2025-12-09 13:28:30 -08:00 |
|
|
aditya
|
2342d2dada
|
fix: resolve resume functionality crash for complete files
|
2025-12-09 19:26:31 +05:30 |
|
|
aditya
|
a9b6518839
|
feat: add download via ftp functionality
|
2025-12-09 19:23:19 +05:30 |
|
|
brent.edwards
|
37b3ee6e91
|
Final version; what was used to send wikidata-truthy.
|
2025-12-07 09:47:04 -08:00 |
|
|
brent.edwards
|
a6928d0ce2
|
Changes requested by Kyle; thank you!
|
2025-12-04 14:11:41 -08:00 |
|
|
brent.edwards
|
9534336c15
|
Replacing upload_all_datasets.py with the previous
upload_all_datasets.2.py
|
2025-12-02 14:40:15 -08:00 |
|
|
brent.edwards
|
68587e5e6c
|
Parallelized the call to dataset_dict.save_to_disk.
|
2025-11-30 17:45:25 -08:00 |
|
|
brent.edwards
|
3d2a9b75a2
|
Added some fixes to run the wikidad-truthy set.
Please enter the commit message for your changes. Lines starting
|
2025-11-26 17:21:46 -08:00 |
|
|
aditya
|
db7733d630
|
Fix: fix bugs in dataset upload and conversion scripts
|
2025-11-26 17:30:49 +05:30 |
|
|
brent.edwards
|
59e1d8d8bd
|
Minor changes; passing to Aditya.
|
2025-11-25 20:44:45 -08:00 |
|
|
brent.edwards
|
1d5e791963
|
upload_all_datasets.2.py written. Will run overnight.
|
2025-11-25 20:19:22 -08:00 |
|
|
brent.edwards
|
33e004a08e
|
Small updates.
|
2025-11-24 19:07:53 -08:00 |
|
|
brent.edwards
|
6b73289c24
|
One-half of the conversion is done, and the structure is written.
|
2025-11-24 18:59:44 -08:00 |
|
|
aditya
|
001234afcf
|
Fix: Pyright type checking errors and improve type annotations
|
2025-11-24 19:48:28 +05:30 |
|
|
aditya
|
a4eca8bcaa
|
Refactor: Extract helper functions to remove code duplication
|
2025-11-24 19:46:14 +05:30 |
|
|
brent.edwards
|
c776025b0f
|
Changes made to make the downloader work.
|
2025-11-23 16:25:31 -08:00 |
|
|
aditya
|
7414b333f8
|
fix: fix ruff check scripts/ errors
|
2025-11-21 18:55:08 +05:30 |
|
|
brent.edwards
|
c5707689d6
|
Changes made by ruff check --fix.
|
2025-11-20 18:38:35 -08:00 |
|
|
brent.edwards
|
aa069f7527
|
Merge branch '16-error-messages-and-logging' into mypy-and-pylint
|
2025-11-20 18:35:52 -08:00 |
|
|
brent.edwards
|
69de631c84
|
One more set of changes for error messages.
|
2025-11-20 17:35:09 -08:00 |
|
|
brent.edwards
|
67a9857ab3
|
feat: add comprehensive error handling for RDF to HuggingFace dataset conversion
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-11-20 17:26:42 -08:00 |
|
|
brent.edwards
|
e7bfa3f2c9
|
fix: correct indentation in ConceptNet CSV parsing function
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-11-20 17:24:04 -08:00 |
|
|
brent.edwards
|
24869bef6c
|
refactor: improve error handling and logging in ConceptNet conversion script
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-11-20 17:23:23 -08:00 |
|
|
brent.edwards
|
e6e1932f66
|
fix: correct indentation in NELL-995 dataset converter script
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-11-20 17:21:58 -08:00 |
|
|
brent.edwards
|
3862bf2655
|
refactor: improve error handling for NELL-995 dataset conversion
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-11-20 17:21:30 -08:00 |
|
|
brent.edwards
|
163268b40d
|
fix: correct indentation in FB15k-237 file parsing function
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
|
2025-11-20 17:20:07 -08:00 |
|