Commit Graph

97 Commits

Author SHA1 Message Date
brent.edwards 4da8793b18 THIS DOES NOT WORK. 2025-12-11 21:26:52 -08:00
aditya 1e95cb9a91 fix: fix ruff check related errors 2025-12-11 19:01:35 +05:30
aditya efa08d5c7d fix: fix minor linting errors 2025-12-11 18:56:29 +05:30
aditya 5a3b22d47f fix: resolve pickling error in streaming converter generator 2025-12-11 18:36:17 +05:30
brent.edwards 18a724eef9 More changes. 2025-12-09 20:01:45 -08:00
brent.edwards eba86c37be fix: remove undefined variables and redundant code in dataset conversion script
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
2025-12-09 20:00:50 -08:00
brent.edwards 29c71c5aea refactor: implement true streaming approach for RDF to dataset conversion
Co-authored-by: aider (openrouter/openai/o3-mini-high) <aider@aider.chat>
2025-12-09 20:00:11 -08:00
brent.edwards 0f3e5f2e30 style: simplify string formatting and type hints in script 2025-12-09 20:00:04 -08:00
brent.edwards bac71ff1cc Minor improvements. 2025-12-09 13:28:30 -08:00
aditya 2342d2dada fix: resolve resume functionality crash for complete files 2025-12-09 19:26:31 +05:30
aditya a9b6518839 feat: add download via ftp functionality 2025-12-09 19:23:19 +05:30
brent.edwards 37b3ee6e91 Final version; what was used to send wikidata-truthy. 2025-12-07 09:47:04 -08:00
brent.edwards a6928d0ce2 Changes requested by Kyle; thank you! 2025-12-04 14:11:41 -08:00
brent.edwards 9534336c15 Replacing upload_all_datasets.py with the previous
upload_all_datasets.2.py
2025-12-02 14:40:15 -08:00
brent.edwards 68587e5e6c Parallelized the call to dataset_dict.save_to_disk. 2025-11-30 17:45:25 -08:00
brent.edwards 3d2a9b75a2 Added some fixes to run the wikidad-truthy set.
Please enter the commit message for your changes. Lines starting
2025-11-26 17:21:46 -08:00
aditya db7733d630 Fix: fix bugs in dataset upload and conversion scripts 2025-11-26 17:30:49 +05:30
brent.edwards 59e1d8d8bd Minor changes; passing to Aditya. 2025-11-25 20:44:45 -08:00
brent.edwards 1d5e791963 upload_all_datasets.2.py written. Will run overnight. 2025-11-25 20:19:22 -08:00
brent.edwards 33e004a08e Small updates. 2025-11-24 19:07:53 -08:00
brent.edwards 6b73289c24 One-half of the conversion is done, and the structure is written. 2025-11-24 18:59:44 -08:00
aditya 001234afcf Fix: Pyright type checking errors and improve type annotations 2025-11-24 19:48:28 +05:30
aditya a4eca8bcaa Refactor: Extract helper functions to remove code duplication 2025-11-24 19:46:14 +05:30
brent.edwards c776025b0f Changes made to make the downloader work. 2025-11-23 16:25:31 -08:00
aditya 7414b333f8 fix: fix ruff check scripts/ errors 2025-11-21 18:55:08 +05:30
brent.edwards c5707689d6 Changes made by ruff check --fix. 2025-11-20 18:38:35 -08:00
brent.edwards aa069f7527 Merge branch '16-error-messages-and-logging' into mypy-and-pylint 2025-11-20 18:35:52 -08:00
brent.edwards 69de631c84 One more set of changes for error messages. 2025-11-20 17:35:09 -08:00
brent.edwards 67a9857ab3 feat: add comprehensive error handling for RDF to HuggingFace dataset conversion
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:26:42 -08:00
brent.edwards e7bfa3f2c9 fix: correct indentation in ConceptNet CSV parsing function
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:24:04 -08:00
brent.edwards 24869bef6c refactor: improve error handling and logging in ConceptNet conversion script
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:23:23 -08:00
brent.edwards e6e1932f66 fix: correct indentation in NELL-995 dataset converter script
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:21:58 -08:00
brent.edwards 3862bf2655 refactor: improve error handling for NELL-995 dataset conversion
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:21:30 -08:00
brent.edwards 163268b40d fix: correct indentation in FB15k-237 file parsing function
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:20:07 -08:00
brent.edwards 576dc842a9 feat: improve error handling for FB15k-237 dataset conversion
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:19:38 -08:00
brent.edwards e1cabc7069 feat: improve error handling and logging in RDF dataset downloader
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:17:27 -08:00
brent.edwards e3bee8af4f feat: enhance error handling and reporting in dataset processing scripts
Co-authored-by: aider (openrouter/anthropic/claude-sonnet-4) <aider@aider.chat>
2025-11-20 17:13:26 -08:00
brent.edwards 7f50361683 Fixes made by ruff format. 2025-11-19 14:21:18 -08:00
brent.edwards 4636770092 convert_rdf_streaming_progress 2025-11-18 19:02:59 -08:00
brent.edwards ec1211ba51 convert_rdf_progress 2025-11-18 18:45:15 -08:00
brent.edwards c058301d1c convert_nell995_to_hf 2025-11-18 18:33:14 -08:00
brent.edwards 927a8648f6 convert_fb15k237_to_hf 2025-11-18 18:22:35 -08:00
brent.edwards c9c6bff859 convert_conceptnet_to_hf. 2025-11-18 18:16:54 -08:00
brent.edwards b16eddbe71 Mostly about convert_conceptnet_progress. 2025-11-18 16:08:18 -08:00
brent.edwards 8784b798a6 Initial checkin with scripts. 2025-11-18 13:18:15 -08:00
freemo b941ef9a1a Significantly revamped the base project to use more robust static checking including benchmarks and integration test support 2025-10-31 16:18:53 -04:00
freemo f7a5d26c41 Modernized build tooling 2025-08-01 18:38:09 -04:00