27-allow-download-ftp #28

Open
aditya wants to merge 3 commits from 27-allow-download-ftp into 19-rewrite-upload-all-datasets
Member
No description provided.
brent.edwards left a comment
Member

Don't let the number of comments get you down; they're all small and you don't need to do anything about "I wouldn't have created this function."

Don't let the number of comments get you down; they're all small and you don't need to do anything about "I wouldn't have created this function."
@ -109,6 +111,18 @@ class RDFDatasetDownloader:
raise
self.console = console or Console()
def _get_protocol(self, url: str) -> str:
Member

(This is minuscule; feel free to ignore it.)

Functions

  • remove duplication of code
  • simplify code and give a name to steps of code.

Since:

  • _get_protocol is just used once in line 196
  • urlparse(url).scheme.lower() is almost as readable as _get_protocol.
  • It's not likely that _get_protocol() is going to be used again

I wouldn't have created this function.

(IMPORTANT NOTE: You don't need to do anything.)

(This is minuscule; feel free to ignore it.) Functions - remove duplication of code - simplify code and give a name to steps of code. Since: - `_get_protocol` is just used once in line 196 - `urlparse(url).scheme.lower()` is almost as readable as `_get_protocol`. - It's not likely that `_get_protocol()` is going to be used again I wouldn't have created this function. (IMPORTANT NOTE: You don't need to do anything.)
@ -211,1 +213,3 @@
else:
elif protocol == "ftp":
self.console.print(f"[cyan]Using FTP protocol for: {dataset_info.url}[/cyan]")
Member

ruff check reported:

scripts/rdf_dataset_downloader.py:215:89: E501 Line too long (94 > 88)
    |
214 |             elif protocol == "ftp":
215 |                 self.console.print(f"[cyan]Using FTP protocol for: {dataset_info.url}[/cyan]")
    |                                                                                         ^^^^^^ E501
216 |                 self._download_file_ftp(dataset_info.url, download_path)
`ruff check` reported: ``` scripts/rdf_dataset_downloader.py:215:89: E501 Line too long (94 > 88) | 214 | elif protocol == "ftp": 215 | self.console.print(f"[cyan]Using FTP protocol for: {dataset_info.url}[/cyan]") | ^^^^^^ E501 216 | self._download_file_ftp(dataset_info.url, download_path) ```
@ -213,0 +220,4 @@
else:
self.console.print(f"[red]Unsupported protocol: {protocol}://[/red]")
self.console.print(f"[yellow]Supported: http://, https://, ftp://, file://[/yellow]")
Member

ruff check reported:

scripts/rdf_dataset_downloader.py:223:36: F541 [*] f-string without any placeholders
    |
221 |             else:
222 |                 self.console.print(f"[red]Unsupported protocol: {protocol}://[/red]")
223 |                 self.console.print(f"[yellow]Supported: http://, https://, ftp://, file://[/yellow]")
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ F541
224 |                 self.console.print(f"[yellow]URL: {dataset_info.url}[/yellow]")
225 |                 raise ValueError(f"Unsupported protocol: {protocol}")
    |
    = help: Remove extraneous `f` prefix

`ruff check` reported: ``` scripts/rdf_dataset_downloader.py:223:36: F541 [*] f-string without any placeholders | 221 | else: 222 | self.console.print(f"[red]Unsupported protocol: {protocol}://[/red]") 223 | self.console.print(f"[yellow]Supported: http://, https://, ftp://, file://[/yellow]") | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ F541 224 | self.console.print(f"[yellow]URL: {dataset_info.url}[/yellow]") 225 | raise ValueError(f"Unsupported protocol: {protocol}") | = help: Remove extraneous `f` prefix ```
@ -218,1 +231,3 @@
if "Name or service not known" in error_msg or "Errno -2" in error_msg:
# Enhanced error context including FTP errors
if "unsupported protocol" in error_msg.lower():
self.console.print(f"[yellow]→ Protocol '{protocol}://' is not supported[/yellow]")
Member

ruff check reported:

scripts/rdf_dataset_downloader.py:233:89: E501 Line too long (99 > 88)
    |
231 |             # Enhanced error context including FTP errors
232 |             if "unsupported protocol" in error_msg.lower():
233 |                 self.console.print(f"[yellow]→ Protocol '{protocol}://' is not supported[/yellow]")
    |                                                                                         ^^^^^^^^^^^ E501
234 |                 self.console.print("  • Supported: http://, https://, ftp://, file://")
235 |                 self.console.print(f"  • URL: {dataset_info.url}")
    |

`ruff check` reported: ``` scripts/rdf_dataset_downloader.py:233:89: E501 Line too long (99 > 88) | 231 | # Enhanced error context including FTP errors 232 | if "unsupported protocol" in error_msg.lower(): 233 | self.console.print(f"[yellow]→ Protocol '{protocol}://' is not supported[/yellow]") | ^^^^^^^^^^^ E501 234 | self.console.print(" • Supported: http://, https://, ftp://, file://") 235 | self.console.print(f" • URL: {dataset_info.url}") | ```
@ -321,0 +347,4 @@
if server_size > 0:
if partial_size >= server_size:
# File is already complete!
self.console.print(f"[green]✓ File already complete ({partial_size / (1024**2):.2f} MB)[/green]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:350:89: E501 Line too long (133 > 88)
    |
348 | …                     if partial_size >= server_size:
349 | …                         # File is already complete!
350 | …                         self.console.print(f"[green]✓ File already complete ({partial_size / (1024**2):.2f} MB)[/green]")
    |                                                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501
351 | …                         return  # Exit successfully
352 | …                     else:
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:350:89: E501 Line too long (133 > 88) | 348 | … if partial_size >= server_size: 349 | … # File is already complete! 350 | … self.console.print(f"[green]✓ File already complete ({partial_size / (1024**2):.2f} MB)[/green]") | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501 351 | … return # Exit successfully 352 | … else: ```
@ -321,0 +351,4 @@
return # Exit successfully
else:
# File incomplete, server doesn't support resume
self.console.print(f"[yellow]Server doesn't support resume. Have {partial_size / (1024**2):.2f} MB, need {server_size / (1024**2):.2f} MB[/yellow]")
Member

ruff check reports:


scripts/rdf_dataset_downloader.py:354:89: E501 Line too long (184 > 88)
    |
352 | …
353 | …port resume
354 | …oesn't support resume. Have {partial_size / (1024**2):.2f} MB, need {server_size / (1024**2):.2f} MB[/yellow]")
    |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501
355 | …partial file and restarting...[/yellow]")
356 | …
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:354:89: E501 Line too long (184 > 88) | 352 | … 353 | …port resume 354 | …oesn't support resume. Have {partial_size / (1024**2):.2f} MB, need {server_size / (1024**2):.2f} MB[/yellow]") | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501 355 | …partial file and restarting...[/yellow]") 356 | … | ```
@ -321,0 +352,4 @@
else:
# File incomplete, server doesn't support resume
self.console.print(f"[yellow]Server doesn't support resume. Have {partial_size / (1024**2):.2f} MB, need {server_size / (1024**2):.2f} MB[/yellow]")
self.console.print("[yellow]Deleting partial file and restarting...[/yellow]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:355:89: E501 Line too long (114 > 88)
    |
353 | …     # File incomplete, server doesn't support resume
354 | …     self.console.print(f"[yellow]Server doesn't support resume. Have {partial_size / (1024**2):.2f} MB, need {server_size / (1024**…
355 | …     self.console.print("[yellow]Deleting partial file and restarting...[/yellow]")
    |                                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^ E501
356 | …     destination.unlink()
357 | …     # Force retry by raising exception that IS caught by retry loop
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:355:89: E501 Line too long (114 > 88) | 353 | … # File incomplete, server doesn't support resume 354 | … self.console.print(f"[yellow]Server doesn't support resume. Have {partial_size / (1024**2):.2f} MB, need {server_size / (1024**… 355 | … self.console.print("[yellow]Deleting partial file and restarting...[/yellow]") | ^^^^^^^^^^^^^^^^^^^^^^^^^^ E501 356 | … destination.unlink() 357 | … # Force retry by raising exception that IS caught by retry loop | ```
@ -321,0 +354,4 @@
self.console.print(f"[yellow]Server doesn't support resume. Have {partial_size / (1024**2):.2f} MB, need {server_size / (1024**2):.2f} MB[/yellow]")
self.console.print("[yellow]Deleting partial file and restarting...[/yellow]")
destination.unlink()
# Force retry by raising exception that IS caught by retry loop
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:357:89: E501 Line too long (99 > 88)
    |
355 | …                     self.console.print("[yellow]Deleting partial file and restarting...[/yellow]")
356 | …                     destination.unlink()
357 | …                     # Force retry by raising exception that IS caught by retry loop
    |                                                                           ^^^^^^^^^^^ E501
358 | …                     raise httpx.NetworkError("Restarting download without resume")
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:357:89: E501 Line too long (99 > 88) | 355 | … self.console.print("[yellow]Deleting partial file and restarting...[/yellow]") 356 | … destination.unlink() 357 | … # Force retry by raising exception that IS caught by retry loop | ^^^^^^^^^^^ E501 358 | … raise httpx.NetworkError("Restarting download without resume") | ```
@ -321,0 +355,4 @@
self.console.print("[yellow]Deleting partial file and restarting...[/yellow]")
destination.unlink()
# Force retry by raising exception that IS caught by retry loop
raise httpx.NetworkError("Restarting download without resume")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:358:89: E501 Line too long (98 > 88)
    |
356 |                                     destination.unlink()
357 |                                     # Force retry by raising exception that IS caught by retry loop
358 |                                     raise httpx.NetworkError("Restarting download without resume")
    |                                                                                         ^^^^^^^^^^ E501
359 |
360 |                             # Case 2: No Content-Length header, assume file is complete
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:358:89: E501 Line too long (98 > 88) | 356 | destination.unlink() 357 | # Force retry by raising exception that IS caught by retry loop 358 | raise httpx.NetworkError("Restarting download without resume") | ^^^^^^^^^^ E501 359 | 360 | # Case 2: No Content-Length header, assume file is complete | ```
@ -321,0 +359,4 @@
# Case 2: No Content-Length header, assume file is complete
else:
self.console.print(f"[green]✓ File exists ({partial_size / (1024**2):.2f} MB), assuming complete[/green]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:362:89: E501 Line too long (138 > 88)
    |
360 | …                     # Case 2: No Content-Length header, assume file is complete
361 | …                     else:
362 | …                         self.console.print(f"[green]✓ File exists ({partial_size / (1024**2):.2f} MB), assuming complete[/green]")
    |                                                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501
363 | …                         self.console.print("[dim]Use --force to re-download if needed.[/dim]")
364 | …                         return  # Exit successfully
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:362:89: E501 Line too long (138 > 88) | 360 | … # Case 2: No Content-Length header, assume file is complete 361 | … else: 362 | … self.console.print(f"[green]✓ File exists ({partial_size / (1024**2):.2f} MB), assuming complete[/green]") | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501 363 | … self.console.print("[dim]Use --force to re-download if needed.[/dim]") 364 | … return # Exit successfully ```
@ -321,0 +360,4 @@
# Case 2: No Content-Length header, assume file is complete
else:
self.console.print(f"[green]✓ File exists ({partial_size / (1024**2):.2f} MB), assuming complete[/green]")
self.console.print("[dim]Use --force to re-download if needed.[/dim]")
Member

ruff check reports:


scripts/rdf_dataset_downloader.py:363:89: E501 Line too long (102 > 88)
    |
361 | …                     else:
362 | …                         self.console.print(f"[green]✓ File exists ({partial_size / (1024**2):.2f} MB), assuming complete[/green]")
363 | …                         self.console.print("[dim]Use --force to re-download if needed.[/dim]")
    |                                                                                   ^^^^^^^^^^^^^^ E501
364 | …                         return  # Exit successfully
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:363:89: E501 Line too long (102 > 88) | 361 | … else: 362 | … self.console.print(f"[green]✓ File exists ({partial_size / (1024**2):.2f} MB), assuming complete[/green]") 363 | … self.console.print("[dim]Use --force to re-download if needed.[/dim]") | ^^^^^^^^^^^^^^ E501 364 | … return # Exit successfully | ```
@ -380,6 +424,131 @@ class RDFDatasetDownloader:
self.console.print(f"[red]Unexpected error during download: {e}[/red]")
raise
def _download_file_ftp(self, url: str, destination: Path, max_retries: int = 3) -> None:
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:427:89: E501 Line too long (92 > 88)
    |
425 |                 raise
426 |
427 |     def _download_file_ftp(self, url: str, destination: Path, max_retries: int = 3) -> None:
    |                                                                                         ^^^^ E501
428 |         """Download file via FTP protocol with progress bar.
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:427:89: E501 Line too long (92 > 88) | 425 | raise 426 | 427 | def _download_file_ftp(self, url: str, destination: Path, max_retries: int = 3) -> None: | ^^^^ E501 428 | """Download file via FTP protocol with progress bar. | ```
@ -383,0 +441,4 @@
while retry_count <= max_retries:
try:
if destination.exists():
self.console.print("[yellow]FTP doesn't support resume - starting fresh download[/yellow]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:444:89: E501 Line too long (111 > 88)
    |
442 |             try:
443 |                 if destination.exists():
444 |                     self.console.print("[yellow]FTP doesn't support resume - starting fresh download[/yellow]")
    |                                                                                         ^^^^^^^^^^^^^^^^^^^^^^^ E501
445 |                     destination.unlink()
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:444:89: E501 Line too long (111 > 88) | 442 | try: 443 | if destination.exists(): 444 | self.console.print("[yellow]FTP doesn't support resume - starting fresh download[/yellow]") | ^^^^^^^^^^^^^^^^^^^^^^^ E501 445 | destination.unlink() | ```
@ -383,0 +464,4 @@
total_size = int(response.headers['Content-Length'])
progress.update(task, total=total_size)
else:
self.console.print("[yellow]FTP server didn't provide file size - progress will be indeterminate[/yellow]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:467:89: E501 Line too long (135 > 88)
    |
465 | …, total=total_size)
466 | …
467 | …[yellow]FTP server didn't provide file size - progress will be indeterminate[/yellow]")
    |                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501
468 | …
469 | …
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:467:89: E501 Line too long (135 > 88) | 465 | …, total=total_size) 466 | … 467 | …[yellow]FTP server didn't provide file size - progress will be indeterminate[/yellow]") | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ E501 468 | … 469 | … | ```
@ -383,0 +482,4 @@
chunk_count += 1
if chunk_count % 100 == 0:
f.flush()
Member

Are lines 484-485 necessary? Usually, the operating system is good at knowing the best times to flush the cache.

Are lines 484-485 necessary? Usually, the operating system is good at knowing the best times to flush the cache.
@ -383,0 +486,4 @@
f.flush()
self.console.print(f"[green]✓ Downloaded {downloaded / (1024**2):.2f} MB via FTP[/green]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:489:89: E501 Line too long (106 > 88)
    |
487 |                             f.flush()
488 |
489 |                 self.console.print(f"[green]✓ Downloaded {downloaded / (1024**2):.2f} MB via FTP[/green]")
    |                                                                                         ^^^^^^^^^^^^^^^^^^ E501
490 |                 return
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:489:89: E501 Line too long (106 > 88) | 487 | f.flush() 488 | 489 | self.console.print(f"[green]✓ Downloaded {downloaded / (1024**2):.2f} MB via FTP[/green]") | ^^^^^^^^^^^^^^^^^^ E501 490 | return | ```
@ -383,0 +490,4 @@
return
except urllib.error.URLError as e:
retry_count += 1
Member

It is probably easier to have the retry_count += 1 and the time.sleep(...) outside of the except section. Though this is correct, if we need another except clause, the new clause would need its own retry_count += 1.

It is probably easier to have the `retry_count += 1` and the `time.sleep(...)` outside of the `except` section. Though this is correct, if we need another `except` clause, the new clause would need its own `retry_count += 1`.
@ -383,0 +496,4 @@
reason = str(e.reason)
if '530' in reason or 'Login incorrect' in reason:
self.console.print(f"[red]FTP authentication failed: {url}[/red]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:499:89: E501 Line too long (90 > 88)
    |
498 |                     if '530' in reason or 'Login incorrect' in reason:
499 |                         self.console.print(f"[red]FTP authentication failed: {url}[/red]")
    |                                                                                         ^^ E501
500 |                         self.console.print("[yellow]The server rejected anonymous login[/yellow]")
501 |                         raise
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:499:89: E501 Line too long (90 > 88) | 498 | if '530' in reason or 'Login incorrect' in reason: 499 | self.console.print(f"[red]FTP authentication failed: {url}[/red]") | ^^ E501 500 | self.console.print("[yellow]The server rejected anonymous login[/yellow]") 501 | raise | ```
@ -383,0 +497,4 @@
if '530' in reason or 'Login incorrect' in reason:
self.console.print(f"[red]FTP authentication failed: {url}[/red]")
self.console.print("[yellow]The server rejected anonymous login[/yellow]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:500:89: E501 Line too long (98 > 88)
    |
498 |                     if '530' in reason or 'Login incorrect' in reason:
499 |                         self.console.print(f"[red]FTP authentication failed: {url}[/red]")
500 |                         self.console.print("[yellow]The server rejected anonymous login[/yellow]")
    |                                                                                         ^^^^^^^^^^ E501
501 |                         raise
    |

`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:500:89: E501 Line too long (98 > 88) | 498 | if '530' in reason or 'Login incorrect' in reason: 499 | self.console.print(f"[red]FTP authentication failed: {url}[/red]") 500 | self.console.print("[yellow]The server rejected anonymous login[/yellow]") | ^^^^^^^^^^ E501 501 | raise | ```
@ -383,0 +501,4 @@
raise
elif '550' in reason or 'No such file' in reason:
self.console.print(f"[red]File not found on FTP server: {url}[/red]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:504:89: E501 Line too long (93 > 88)
    |
503 |                     elif '550' in reason or 'No such file' in reason:
504 |                         self.console.print(f"[red]File not found on FTP server: {url}[/red]")
    |                                                                                         ^^^^^ E501
505 |                         self.console.print("[yellow]The file may have been moved or removed[/yellow]")
506 |                         raise
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:504:89: E501 Line too long (93 > 88) | 503 | elif '550' in reason or 'No such file' in reason: 504 | self.console.print(f"[red]File not found on FTP server: {url}[/red]") | ^^^^^ E501 505 | self.console.print("[yellow]The file may have been moved or removed[/yellow]") 506 | raise | ```
@ -383,0 +502,4 @@
elif '550' in reason or 'No such file' in reason:
self.console.print(f"[red]File not found on FTP server: {url}[/red]")
self.console.print("[yellow]The file may have been moved or removed[/yellow]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:505:89: E501 Line too long (102 > 88)
    |
503 |                     elif '550' in reason or 'No such file' in reason:
504 |                         self.console.print(f"[red]File not found on FTP server: {url}[/red]")
505 |                         self.console.print("[yellow]The file may have been moved or removed[/yellow]")
    |                                                                                         ^^^^^^^^^^^^^^ E501
506 |                         raise
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:505:89: E501 Line too long (102 > 88) | 503 | elif '550' in reason or 'No such file' in reason: 504 | self.console.print(f"[red]File not found on FTP server: {url}[/red]") 505 | self.console.print("[yellow]The file may have been moved or removed[/yellow]") | ^^^^^^^^^^^^^^ E501 506 | raise | ```
@ -383,0 +508,4 @@
elif 'timed out' in reason.lower():
if retry_count <= max_retries:
self.console.print(
f"[yellow]FTP timeout. Retrying {retry_count}/{max_retries}...[/yellow]"
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:511:89: E501 Line too long (104 > 88)
    |
509 |                         if retry_count <= max_retries:
510 |                             self.console.print(
511 |                                 f"[yellow]FTP timeout. Retrying {retry_count}/{max_retries}...[/yellow]"
    |                                                                                         ^^^^^^^^^^^^^^^^ E501
512 |                             )
513 |                             time.sleep(min(2**retry_count, 30))
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:511:89: E501 Line too long (104 > 88) | 509 | if retry_count <= max_retries: 510 | self.console.print( 511 | f"[yellow]FTP timeout. Retrying {retry_count}/{max_retries}...[/yellow]" | ^^^^^^^^^^^^^^^^ E501 512 | ) 513 | time.sleep(min(2**retry_count, 30)) | ```
@ -383,0 +513,4 @@
time.sleep(min(2**retry_count, 30))
continue
else:
self.console.print(f"[red]FTP download failed after {max_retries} retries[/red]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:516:89: E501 Line too long (109 > 88)
    |
514 |                             continue
515 |                         else:
516 |                             self.console.print(f"[red]FTP download failed after {max_retries} retries[/red]")
    |                                                                                         ^^^^^^^^^^^^^^^^^^^^^ E501
517 |                             raise
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:516:89: E501 Line too long (109 > 88) | 514 | continue 515 | else: 516 | self.console.print(f"[red]FTP download failed after {max_retries} retries[/red]") | ^^^^^^^^^^^^^^^^^^^^^ E501 517 | raise | ```
@ -383,0 +519,4 @@
else:
if retry_count <= max_retries:
self.console.print(
f"[yellow]FTP error: {reason}. Retrying {retry_count}/{max_retries}...[/yellow]"
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:522:89: E501 Line too long (112 > 88)
    |
520 |                         if retry_count <= max_retries:
521 |                             self.console.print(
522 |                                 f"[yellow]FTP error: {reason}. Retrying {retry_count}/{max_retries}...[/yellow]"
    |                                                                                         ^^^^^^^^^^^^^^^^^^^^^^^^ E501
523 |                             )
524 |                             time.sleep(min(2**retry_count, 30))
    |

`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:522:89: E501 Line too long (112 > 88) | 520 | if retry_count <= max_retries: 521 | self.console.print( 522 | f"[yellow]FTP error: {reason}. Retrying {retry_count}/{max_retries}...[/yellow]" | ^^^^^^^^^^^^^^^^^^^^^^^^ E501 523 | ) 524 | time.sleep(min(2**retry_count, 30)) | ```
@ -383,0 +532,4 @@
retry_count += 1
if retry_count <= max_retries:
self.console.print(
f"[yellow]FTP connection timeout. Retrying {retry_count}/{max_retries}...[/yellow]"
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:535:89: E501 Line too long (107 > 88)
    |
533 |                 if retry_count <= max_retries:
534 |                     self.console.print(
535 |                         f"[yellow]FTP connection timeout. Retrying {retry_count}/{max_retries}...[/yellow]"
    |                                                                                         ^^^^^^^^^^^^^^^^^^^ E501
536 |                     )
537 |                     time.sleep(min(2**retry_count, 30))
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:535:89: E501 Line too long (107 > 88) | 533 | if retry_count <= max_retries: 534 | self.console.print( 535 | f"[yellow]FTP connection timeout. Retrying {retry_count}/{max_retries}...[/yellow]" | ^^^^^^^^^^^^^^^^^^^ E501 536 | ) 537 | time.sleep(min(2**retry_count, 30)) | ```
@ -383,0 +536,4 @@
)
time.sleep(min(2**retry_count, 30))
else:
self.console.print(f"[red]FTP download failed after {max_retries} retries[/red]")
Member

ruff check reports:

scripts/rdf_dataset_downloader.py:539:89: E501 Line too long (101 > 88)
    |
537 |                     time.sleep(min(2**retry_count, 30))
538 |                 else:
539 |                     self.console.print(f"[red]FTP download failed after {max_retries} retries[/red]")
    |                                                                                         ^^^^^^^^^^^^^ E501
540 |                     raise
    |
`ruff check` reports: ``` scripts/rdf_dataset_downloader.py:539:89: E501 Line too long (101 > 88) | 537 | time.sleep(min(2**retry_count, 30)) 538 | else: 539 | self.console.print(f"[red]FTP download failed after {max_retries} retries[/red]") | ^^^^^^^^^^^^^ E501 540 | raise | ```
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin 27-allow-download-ftp:27-allow-download-ftp
git switch 27-allow-download-ftp

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch 19-rewrite-upload-all-datasets
git merge --no-ff 27-allow-download-ftp
git switch 27-allow-download-ftp
git rebase 19-rewrite-upload-all-datasets
git switch 19-rewrite-upload-all-datasets
git merge --ff-only 27-allow-download-ftp
git switch 27-allow-download-ftp
git rebase 19-rewrite-upload-all-datasets
git switch 19-rewrite-upload-all-datasets
git merge --no-ff 27-allow-download-ftp
git switch 19-rewrite-upload-all-datasets
git merge --squash 27-allow-download-ftp
git switch 19-rewrite-upload-all-datasets
git merge --ff-only 27-allow-download-ftp
git switch 19-rewrite-upload-all-datasets
git merge 27-allow-download-ftp
git push origin 19-rewrite-upload-all-datasets
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
cleverdatasets/dataset-uploader!28
No description provided.