CleverCloud Storage Framework
Find a file
CoreRasurae 7caa68abb6
All checks were successful
/ test (push) Successful in 11m0s
feat: StorageProvider auto-selection on StorageClient API
ISSUES CLOSED: #5
2025-09-24 19:17:09 +01:00
.docker Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
.forgejo/workflows ci: Enable tox lint stage 2025-09-20 17:07:02 +01:00
src/clevercloud_storage_framework feat: StorageProvider auto-selection on StorageClient API 2025-09-24 19:17:09 +01:00
tests feat: StorageProvider auto-selection on StorageClient API 2025-09-24 19:17:09 +01:00
.coveragerc Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
.cz-config.js chore: Add editor and git commitizen configurations 2025-09-20 13:08:55 +01:00
.cz.json chore: Add editor and git commitizen configurations 2025-09-20 13:08:55 +01:00
.editorconfig chore: Add editor and git commitizen configurations 2025-09-20 13:08:55 +01:00
.gitignore chore: Update .gitignore 2025-09-20 13:09:01 +01:00
ATTRIBUTIONS.md Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
CHANGELOG.md feat: StorageProvider auto-selection on StorageClient API 2025-09-24 19:17:09 +01:00
CODE_OF_CONDUCT.md Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
conftest.py Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
CONTRIBUTING.md Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
CONTRIBUTORS.md Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
Dockerfile.dev Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
LICENSE Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
MANIFEST.in Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
NOTICE Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
README.md Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
requirements-dev.txt build: Expand requirements to include code coverage tools 2025-09-20 13:08:44 +01:00
requirements.txt build: Expand requirements to include code coverage tools 2025-09-20 13:08:44 +01:00
setup.cfg feat: Expose Pre-Signed URL support methods via StorageClient API 2025-09-23 11:23:15 +01:00
setup.py Initial commit at version 0.0.1 2025-09-11 18:14:14 +01:00
tox.ini ci: Enable tox lint stage 2025-09-20 17:07:02 +01:00

CleverCloud Storage Framework

A unified framework for interacting with various AWS storage services including S3, EFS, and Glacier. This library provides a consistent interface for common storage operations across different storage providers.

Features

  • Unified API for multiple storage providers (S3, EFS, Glacier, Local)
  • Streaming support for large files
  • Automatic provider detection based on path format
  • Chunked reading and writing for efficient memory usage
  • Comprehensive error handling
  • Support for SAS tokens (for S3)

Installation

pip install clevercloud-storage-framework

Quick Start

from clevercloud_storage_framework.factory import StorageClientFactory
from clevercloud_storage_framework.client import StorageClient

# Create a client for S3
s3_client = StorageClientFactory.create_client("s3", region_name="us-west-2")

# List files in an S3 bucket
files = s3_client.list_files("s3://my-bucket/my-prefix/")

# Read a file from S3
content = s3_client.read_file("s3://my-bucket/my-file.txt")

# Write a file to S3
s3_client.write_file("s3://my-bucket/new-file.txt", "Hello, World!")

# Copy a file between providers
s3_client.copy("s3://my-bucket/source.txt", "efs://my-filesystem/destination.txt")

Working with Different Storage Providers

S3

# Create an S3 client
s3_client = StorageClientFactory.create_client("s3", 
                                              region_name="us-west-2",
                                              aws_access_key_id="YOUR_ACCESS_KEY",
                                              aws_secret_access_key="YOUR_SECRET_KEY")

# Generate a presigned URL
from clevercloud_storage_framework.providers.s3_provider import S3StorageProvider
s3_provider = s3_client._provider
presigned_url = s3_provider.generate_presigned_url("s3://my-bucket/my-file.txt", expiration=3600)

EFS

# Create an EFS client
efs_client = StorageClientFactory.create_client("efs", 
                                               region_name="us-west-2",
                                               mount_point="/mnt/efs")

# Create a directory
efs_client.create_directory("efs://fs-12345/my-directory")

# List files recursively
files = efs_client.list_files("efs://fs-12345/", recursive=True)

Glacier

# Create a Glacier client
glacier_client = StorageClientFactory.create_client("glacier", region_name="us-west-2")

# Initiate an archive retrieval
from clevercloud_storage_framework.providers.glacier_provider import GlacierStorageProvider
glacier_provider = glacier_client._provider
job_id = glacier_provider.initiate_archive_retrieval("my-vault", "archive-id")

# Check job status
status = glacier_provider.check_job_status("my-vault", job_id)

# Get job output when complete
if status == "Succeeded":
    content = b"".join(list(glacier_provider.get_job_output("my-vault", job_id)))

Local Storage

# Create a local storage client
local_client = StorageClientFactory.create_client("local")

# Work with local files
local_client.write_file("/path/to/local/file.txt", "Hello, World!")
content = local_client.read_file("/path/to/local/file.txt")

Error Handling

from clevercloud_storage_framework.exceptions import StorageError, ResourceNotFoundError

try:
    content = client.read_file("s3://non-existent-bucket/file.txt")
except ResourceNotFoundError as e:
    print(f"Resource not found: {e.resource}")
except StorageError as e:
    print(f"Storage error: {e.message}")

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.