clevermicro/amq-adapter-python
14
0
Fork 0
Code Issues 16 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: clevermicro/amq-adapter-python:api_docs
Branches Tags
clevermicro/amq-adapter-python:develop clevermicro/amq-adapter-python:api_docs clevermicro/amq-adapter-python:feat/63-cleverswarm-unidirectional clevermicro/amq-adapter-python:feat/61-multi-user-support clevermicro/amq-adapter-python:feat-58-backpressure-reactive-streams clevermicro/amq-adapter-python:fix/bp-report clevermicro/amq-adapter-python:otel-demo clevermicro/amq-adapter-python:cleverbrag-integration-2 clevermicro/amq-adapter-python:cleverbrag-integration clevermicro/amq-adapter-python:code-formatting clevermicro/amq-adapter-python:errors-as-json clevermicro/amq-adapter-python:config-documemntation clevermicro/amq-adapter-python:fix-version clevermicro/amq-adapter-python:feat/#1 clevermicro/amq-adapter-python:fix_testing clevermicro/amq-adapter-python:feat/readme clevermicro/amq-adapter-python:feat#2 clevermicro/amq-adapter-python:feat#64
..
pull from: clevermicro/amq-adapter-python:feat#2
Branches Tags
clevermicro/amq-adapter-python:api_docs clevermicro/amq-adapter-python:develop clevermicro/amq-adapter-python:feat/63-cleverswarm-unidirectional clevermicro/amq-adapter-python:feat/61-multi-user-support clevermicro/amq-adapter-python:feat-58-backpressure-reactive-streams clevermicro/amq-adapter-python:fix/bp-report clevermicro/amq-adapter-python:otel-demo clevermicro/amq-adapter-python:cleverbrag-integration-2 clevermicro/amq-adapter-python:cleverbrag-integration clevermicro/amq-adapter-python:code-formatting clevermicro/amq-adapter-python:errors-as-json clevermicro/amq-adapter-python:config-documemntation clevermicro/amq-adapter-python:fix-version clevermicro/amq-adapter-python:feat/#1 clevermicro/amq-adapter-python:fix_testing clevermicro/amq-adapter-python:feat/readme clevermicro/amq-adapter-python:feat#2 clevermicro/amq-adapter-python:feat#64

2 Commits
api_docs .. feat#2

Author SHA1 Message Date
Abed d1f808aea4 feat#2 2025-02-20 15:50:13 +08:00
Abed aec71dc6ff feat#64 2025-02-20 15:49:14 +08:00
262 changed files with 1832 additions and 39581 deletions
Expand all files Collapse all files
.cursor/rules/python.mdc
-118
View File
@@ -1,118 +0,0 @@
---
description:
globs:
alwaysApply: true
---
---
description: Python best practices and patterns for modern software development with Flask and SQLite
globs: **/*.py, src/**/*.py, tests/**/*.py
---
# Python Best Practices
## Project Structure
- Use src-layout with `amqp/`
- Place tests in `tests/` directory parallel to `amqp/`
- Keep configuration in `amqp/config/` or as environment variables
- Store requirements in `pyproject.toml`
- Place static files in `static/` directory
- Use `templates/` for Jinja2 templates
## Code Style
- Follow Black code formatting
- Use isort for import sorting
- Follow PEP 8 naming conventions:
- snake_case for functions and variables
- PascalCase for classes
- UPPER_CASE for constants
- Maximum line length of 100 characters
- Use absolute imports over relative imports
## Type Hints
- Use type hints for all function parameters and returns
- Import types from `typing` module
- Use `Optional[Type]` instead of `Type | None`
- Use `TypeVar` for generic types
- Define custom types in `types.py`
- Use `Protocol` for duck typing
## Database
- Use SQLAlchemy ORM
- Implement database migrations with Alembic
- Use proper connection pooling
- Define models in separate modules
- Implement proper relationships
- Use proper indexing strategies
## Authentication
##- Use Flask-Login for session management
##- Implement Google OAuth using Flask-OAuth
##- Hash passwords with bcrypt
##- Use proper session security
##- Implement CSRF protection
##- Use proper role-based access control
## API Design
##- Use Flask-RESTful for REST APIs
##- Implement proper request validation
##- Use proper HTTP status codes
##- Handle errors consistently
##- Use proper response formats
##- Implement proper rate limiting
## Testing
- Use pytest for testing
- Write tests for all routes
- Use pytest-cov for coverage
- Implement proper fixtures
- Use proper mocking with pytest-mock
- Test all error scenarios
## Security
##- Use HTTPS in production
##- Implement proper CORS
##- Sanitize all user inputs
##- Use proper session configuration
- Implement proper logging
##- Follow OWASP guidelines
## Performance
##- Use proper caching with Flask-Caching
##- Implement database query optimization
##- Use proper connection pooling
##- Implement proper pagination
##- Use background tasks for heavy operations
- Monitor application performance
## Error Handling
- Create custom exception classes
- Use proper try-except blocks
- Implement proper logging
- Return proper error responses
- Handle edge cases properly
- Use proper error messages
## Documentation
- Use Google-style docstrings
- Document all public APIs
- Keep README.md updated
- Use proper inline comments
- Generate API documentation
- Document environment setup
## Development Workflow
- Use virtual environments (venv)
- Use or Implement pre-commit hooks
- Use proper Git workflow
- Follow semantic versioning
- Use proper CI/CD practices
- Implement proper logging
## Dependencies
##- Pin dependency versions
##- Use requirements.txt for production
- Separate dev dependencies
##- Use proper package versions
##- Regularly update dependencies
- Check for security vulnerabilitie
]cvx -=
.dockerignore
-26
View File
@@ -1,26 +0,0 @@
.git
.github
.forgejo
__pycache__
*.pyc
*.pyo
*.pyd
.Python
env/
venv/
.venv/
pip-log.txt
pip-delete-this-directory.txt
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.log
.git*
tests/
container-data/
docker-compose.yml
.dockerignore
.flake8
-4
View File
@@ -1,4 +0,0 @@
[flake8]
max-line-length = 88
ignore = E501, W503, E203
exclude = .git,__pycache__,docs/source/conf.py,old,build,dist,.venv
.forgejo/workflows/build-and-publish.yml
-88
View File
@@ -1,88 +0,0 @@
name: Build and Publish Docker Image
#on:
# push:
# branches: [ main, master ]
# tags: [ 'v*' ]
# pull_request:
# branches: [ main, master ]
on:
push:
branches:
- master # publish release on master
- develop # publish snapshot on develop
- feat-58-backpressure-reactive-streams
workflow_dispatch:
# allow manual trigger
env:
REGISTRY_URL: "git.cleverthis.com"
REPOSITORY: "clevermicro/amq-adapter-python-demo"
DOCKER_HOST: "tcp://dind:2375"
jobs:
build-and-push:
runs-on: general
services:
dind:
image: docker:dind
cmd:
- dockerd
- -H
- tcp://0.0.0.0:2375
- --tls=false
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ github.ref_name }}
# setup docker
- name: set up docker cli
run: |
apt-get update
apt-get install ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Docker Registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY_URL }}
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_PASSWORD }}
if: github.event_name != 'pull_request'
- name: Extract metadata for Docker
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.REGISTRY_URL }}/clevermicro/amq-adapter-python-demo
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=sha,format=short
- name: Build and push Docker image
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
# push: ${{ github.event_name != 'pull_request' }}
push: true
tags: ${{ env.REGISTRY_URL }}/${{ env.REPOSITORY }}:latest
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
.forgejo/workflows/coverage-check.yaml
-47
View File
@@ -1,47 +0,0 @@
name: "Unit test coverage"
on:
push:
pull_request:
env:
MIN_COVERAGE_PERCENTAGE: 85
DEBIAN_FRONTEND: noninteractive
TZ: UTC
jobs:
# gradle test for modules
pytest:
runs-on: general
container:
image: ubuntu:24.04
steps:
# need to setup node and git
- run: |
apt-get update
apt-get install -y nodejs git curl libsqlite3-0
# check out code and set up python
- uses: actions/checkout@v4
- uses: https://github.com/actions/setup-python@v5
with:
python-version: '3.11'
# install dependencies
- run: pip install -e .[dev]
# run pytest with coverage
- run: pytest --cov=amqp --cov-report=xml
# process coverage report
- name: Collect coverage report
id: coverage
run: |
COVERAGE_PERCENTAGE=$(coverage report --format=total)
echo "Code coverage: ${COVERAGE_PERCENTAGE}%"
echo "Minimal coverage: ${MIN_COVERAGE_PERCENTAGE}%"
echo "percentage=${COVERAGE_PERCENTAGE}" >> $GITHUB_OUTPUT
- name: Post coverage to RP
if: github.event_name == 'pull_request'
run: |
curl --fail \
-X POST '${{github.api_url}}/repos/${{github.repository}}/issues/${{github.event.number}}/comments' \
-H "Content-Type: application/json" \
-H "Authorization: token ${{github.token}}" \
-d '{"body":"Coverage is ${{steps.coverage.outputs.percentage}}%"}'
- name: Check coverage rate
run: |
coverage report --fail-under=${MIN_COVERAGE_PERCENTAGE}
.forgejo/workflows/publish-docker.yaml
-41
View File
@@ -1,41 +0,0 @@
on: [push]
env:
REGISTRY_URL: "git.cleverthis.com"
REPOSITORY: "clevermicro/amq-adapter-python-demo"
DOCKER_HOST: "tcp://dind:2375"
jobs:
build-and-push:
runs-on: general
services:
dind:
image: docker:dind
cmd:
- dockerd
- -H
- tcp://0.0.0.0:2375
- --tls=false
container:
image: "ghcr.io/catthehacker/ubuntu:js-22.04"
steps:
- name: checkout
uses: actions/checkout@v4
with:
ref: ${{ github.ref_name }}
- name: docker login
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY_URL }}
username: ${{ secrets.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_PASSWORD }}
- name: docker build & push
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
no-cache: true
push: true
tags: ${{ env.REGISTRY_URL }}/${{ env.REPOSITORY }}:20250709-01
.forgejo/workflows/publish-python.yaml
-50
View File
@@ -1,50 +0,0 @@
name: "CI for pypl publish"
on:
push:
branches:
- master # publish release on master
- develop # publish snapshot on develop
workflow_dispatch:
# allow manual trigger
env:
DEBIAN_FRONTEND: noninteractive
TZ: UTC
jobs:
publish-lib:
runs-on: general
container:
image: ubuntu:24.04
steps:
# need to setup node and git
- run: |
apt-get update
apt-get install -y nodejs git libsqlite3-0
# check out code
- uses: actions/checkout@v4
- uses: https://github.com/actions/setup-python@v5
with:
python-version: '3.11'
# install dependencies
- run: pip install -e .[dev]
# run pytest
- run: pytest
# replace timestamp building dev version
- name: Add timestamp to version
if: github.head_ref != 'master'
run: |
sed -i -E "s/(version = \"[^\"]+)(-dev)\"/\1\2$(date -u +'%Y%m%d%H%M%S')\"/" pyproject.toml
cat pyproject.toml
- name: Build wheel
run: |
python -m build
# publish
- name: publish to pulp pypi
run: |
twine upload --repository-url https://git.cleverthis.com/api/packages/clevermicro/pypi \
--username ${CI_REGISTRY_USER} \
--password ${CI_REGISTRY_PASSWORD} \
--verbose \
dist/*
env:
CI_REGISTRY_USER: ${{ secrets.REGISTRY_USER }}
CI_REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
.gitignore
-11
View File
@@ -50,14 +50,3 @@ hs_err_pid*
**/__pycache__
.idea
.venv
.coverage
amq_adapter.egg-info/
build/
unused-code/
.vscode/
amq_adapter.egg-info/
build/
coverage.xml
.aider*
.pre-commit-config.yaml
-21
View File
@@ -1,21 +0,0 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.2.0
hooks:
- id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/PyCQA/isort
rev: 6.0.1
hooks:
- id: isort
args: ["--profile", "black"]
- repo: https://github.com/psf/black
rev: 25.1.0
hooks:
- id: black
language_version: python3.11
- repo: https://github.com/PyCQA/flake8
rev: 7.2.0
hooks:
- id: flake8
AMQ_ADAPTER_DEVELOPER_GUIDE.md
-663
View File
@@ -1,663 +0,0 @@
# AMQ Adapter Developer Guide
## Overview
The AMQ Adapter library enables FastAPI-based Business Services to communicate with the CleverMicro platform via RabbitMQ. This guide demonstrates how to build your own service using the AMQ Adapter, following the pattern established by the OTDemo application.
## Table of Contents
1. [Architecture Overview](#architecture-overview)
2. [Prerequisites](#prerequisites)
3. [Project Structure](#project-structure)
4. [Implementation Steps](#implementation-steps)
5. [Configuration](#configuration)
6. [Creating Your Adapter](#creating-your-adapter)
7. [Frontend Integration](#frontend-integration)
8. [Backend Processing](#backend-processing)
9. [Testing and Deployment](#testing-and-deployment)
10. [Advanced Features](#advanced-features)
## Architecture Overview
The AMQ Adapter follows a layered architecture:
```
┌─────────────────────────────────────────────────────────────┐
│ Business Service │
├─────────────────────────────────────────────────────────────┤
│ FastAPI Endpoints │ Your Custom Adapter │ Backend │
│ │ │ Processing │
└─────────────────────┼───────────────────────┼───────────────┘
│ │
┌─────────────────────┼───────────────────────┼───────────────┐
│ AMQ Adapter Library │ │
├─────────────────────────────────────────────────────────────┤
│ CleverThisServiceAdapter │ AMQService │ Router │
│ (Base Class) │ │ │
└─────────────────────┼───────────────────────┼───────────────┘
│ │
┌─────────────────────┼───────────────────────┼───────────────┐
│ RabbitMQ │ │
│ Exchanges │ Queues │ Routing Keys │ │
└─────────────────────────────────────────────────────────────┘
```
### Key Components
- **Your Custom Adapter**: Extends `CleverThisServiceAdapter` to handle service-specific logic
- **AMQService**: Manages RabbitMQ connections and message routing
- **Router**: Routes incoming AMQP messages to appropriate endpoints
- **Configuration**: Defines service settings, routing, and RabbitMQ connection details
## Prerequisites
Before building your service, ensure you have:
1. **Python 3.8+** installed
2. **RabbitMQ** server running (or access to CleverMicro platform)
3. **FastAPI** knowledge
4. **AMQ Adapter library** installed in your project
### Dependencies
Add the following to your `requirements.txt`:
```txt
fastapi>=0.111.0
uvicorn[standard]>=0.29.0
python-multipart>=0.0.20
aio-pika>=9.0.0
pydantic>=2.0.0
# Add your AMQ Adapter library dependency
```
## Project Structure
Follow this recommended structure for your Business Service:
```
your-business-service/
├── main.py # FastAPI application entry point
├── requirements.txt # Python dependencies
├── config/
│ ├── frontend.properties # Frontend service configuration
│ └── backend.properties # Backend service configuration
├── adapter/
│ └── your_adapter.py # Your custom AMQ adapter
├── backend/
│ └── backend_processor.py # Backend message processing
└── tests/
└── test_your_service.py
```
## Implementation Steps
### Step 1: Create Your Configuration Files
Create configuration files for both frontend and backend services.
**`config/frontend.properties`**:
```properties
# ====================================================================
# CleverMicro AMQ Adapter settings
# ====================================================================
[CleverMicro-AMQ]
cm.amq-adapter.service-name=your-service
cm.amq-adapter.generator-id=1
cm.amq-adapter.route-mapping=your-service:your-exchange:your-queue-1:api.cm.dev.cleverthis.localhost.test.python.frontend.#:5:0
cm.amq-adapter.require-authenticated-user=false
cm.amq-adapter.callbacks-enabled=True
# CleverMicro AMQ Adapter dispatcher settings
cm.dispatch.use-dlq=true
cm.dispatch.amq-host=localhost
cm.dispatch.amq-port=5672
cm.dispatch.amq-port-tls=5671
cm.dispatch.download-dir=/tmp/downloaded_files
# Backpressure settings
cm.backpressure.threshold=1
cm.backpressure.time-window=150
cm.backpressure.cpu-overload-duration=300
cm.backpressure.cpu-idle-duration=300
```
**`config/backend.properties`**:
```properties
# ====================================================================
# CleverMicro AMQ Adapter settings
# ====================================================================
[CleverMicro-AMQ]
cm.amq-adapter.service-name=your-service-backend
cm.amq-adapter.generator-id=1
cm.amq-adapter.route-mapping=your-service-backend:your-exchange:your-queue-2:your-service-backend.#:5:0
cm.amq-adapter.require-authenticated-user=false
cm.amq-adapter.callbacks-enabled=False
# CleverMicro AMQ Adapter dispatcher settings
cm.dispatch.use-dlq=true
cm.dispatch.amq-host=localhost
cm.dispatch.amq-port=5672
cm.dispatch.amq-port-tls=5671
cm.dispatch.download-dir=/tmp/downloaded_files
# Backpressure settings
cm.backpressure.threshold=1
cm.backpressure.time-window=150
cm.backpressure.cpu-overload-duration=300
cm.backpressure.cpu-idle-duration=300
```
### Step 2: Create Your Custom Adapter
Create your custom adapter by extending `CleverThisServiceAdapter`:
```python
"""
Your Business Service AMQP Adapter for handling API requests via AMQP.
This adapter is specific to your business service and handles service-specific logic.
"""
from threading import Thread
from typing import Any, List
from fastapi.routing import APIRoute
from amqp.adapter.cleverthis_service_adapter import CleverThisServiceAdapter
from amqp.config.amq_configuration import AMQConfiguration
from amqp.service.amq_service import AMQService
class YourServiceAmqpAdapter(CleverThisServiceAdapter):
"""
Your Business Service AMQP Adapter. Implements service-specific logic that overrides
common logic defined by the AMQ Adapter library.
Key responsibilities:
1. Handle authentication and user management
2. Process incoming AMQP messages
3. Route messages to appropriate endpoints
4. Manage service-specific business logic
"""
def __init__(self, amq_configuration: AMQConfiguration, routes: List[APIRoute]):
super().__init__(amq_configuration, routes=routes, session=None)
self.amq_service: AMQService = AMQService(amq_configuration, self)
self.thread = Thread(target=self.amq_service.run)
self.thread.start()
async def get_current_user(self, token: str) -> str:
"""
Override the default get_current_user method to handle authentication.
In this example, we simply return the token, but you should implement
proper JWT token validation and user extraction.
:param token: JWT token from the AMQP message
:return: User identifier or user object
"""
# TODO: Implement proper JWT token validation
# Example: return await your_auth_service.validate_token(token)
return token
def get_endpoint(self, endpoint: Any) -> Any:
"""
Override to provide custom endpoint selection logic.
For example, you might want to use JSON wrapper endpoints
or implement custom routing logic.
:param endpoint: The original endpoint callable
:return: The endpoint callable to use (original or modified)
"""
return endpoint
```
### Step 3: Create Your FastAPI Application
Create your main FastAPI application with endpoints:
```python
"""
Your Business Service FastAPI Application
"""
import os
from fastapi import FastAPI, File, Form, UploadFile
from fastapi.routing import APIRoute
from amqp.config.amq_configuration import AMQConfiguration
from adapter.your_adapter import YourServiceAmqpAdapter
# Create FastAPI application
app = FastAPI(
title="Your Business Service",
description="A FastAPI application using AMQ Adapter for CleverMicro communication",
version="1.0.0"
)
# Initialize your AMQ adapter
# This should be done after all routes are defined
your_adapter = None
@app.post("/api/v1/items")
async def create_item(
name: str = Form(...),
description: str = Form(...),
file: UploadFile = File(...)
):
"""
Create a new item with file upload.
This endpoint will be available via both REST and AMQP.
"""
# Your business logic here
result = {
"message": "Item created successfully",
"name": name,
"description": description,
"filename": file.filename
}
# Example: Send a command to backend via AMQP
if your_adapter:
your_adapter.amq_service.command(
"your-service-backend",
"process_item",
item_name=name,
item_description=description
)
return result
@app.get("/api/v1/items")
async def get_items():
"""
Retrieve all items.
This endpoint will be available via both REST and AMQP.
"""
# Your business logic here
items = [
{"id": "1", "name": "Item 1", "description": "Description 1"},
{"id": "2", "name": "Item 2", "description": "Description 2"}
]
return {"items": items}
@app.get("/health")
async def health_check():
"""
Health check endpoint.
"""
return {"status": "healthy", "service": "your-business-service"}
# Initialize the AMQ adapter after all routes are defined
def initialize_amq_adapter():
global your_adapter
config = AMQConfiguration("config/frontend.properties")
your_adapter = YourServiceAmqpAdapter(config, app.routes)
# Initialize adapter when the application starts
@app.on_event("startup")
async def startup_event():
initialize_amq_adapter()
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
```
### Step 4: Create Backend Processing
Create a backend processor for handling AMQP messages:
```python
"""
Backend processor for your business service.
Handles AMQP messages and performs background processing.
"""
import asyncio
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import AMQRoute, DataMessage
from adapter.your_adapter import YourServiceAmqpAdapter
def process_item_static(item_name: str, item_description: str) -> str:
"""
Static function to process an item.
"""
print(f"STATIC Processing item: {item_name} - {item_description}")
return f"STATIC PROCESSED: {item_name}"
class BackendProcessor:
"""
Backend processor class for handling business logic.
"""
def process_item(self, item_name: str, item_description: str) -> str:
"""
Process an item using instance method.
"""
print(f"CLASS Processing item: {item_name} - {item_description}")
return f"CLASS PROCESSED: {item_name}"
async def processing_loop():
"""
Main processing loop for handling AMQP messages.
"""
# Wait for AMQ adapter to initialize
while not otadapter.amq_service.future.done():
await asyncio.sleep(0.1)
# Find the route for backend processing
route: AMQRoute = otadapter.amq_service.router.find_route_by_service("your-service-backend")
generator = otadapter.amq_service.data_message_generator(route)
print(f"Data message generator created: {generator}")
# Start the generator
message = await generator.asend(None)
# Process messages
while True:
print(f"Received message: {message}")
if message is None:
break
data_message: DataMessage = message
# Extract message attributes
print(f"Message attributes: {data_message.method()} {data_message.domain()} {data_message.path()}")
# Process based on message path
if data_message.path() == "process_item_static":
item_name = data_message._base64body.decode("utf-8")
process_item_static(item_name, "Static processing")
elif data_message.path() == "process_item":
item_name = data_message._base64body.decode("utf-8")
processor = BackendProcessor()
processor.process_item(item_name, "Class processing")
# Get next message
message = await generator.asend(True)
if __name__ == "__main__":
# Initialize backend adapter
otadapter = YourServiceAmqpAdapter(AMQConfiguration("config/backend.properties"), [])
# Run the processing loop in the AMQ service event loop
asyncio.run_coroutine_threadsafe(processing_loop(), otadapter.amq_service.loop)
# Keep the program running
otadapter.thread.join()
```
## Configuration
### Configuration Parameters
The configuration file contains several key sections:
#### AMQ Adapter Settings (`cm.amq-adapter.*`)
- **`service-name`**: Unique identifier for your service
- **`generator-id`**: Unique ID for message generation (usually 1)
- **`route-mapping`**: RabbitMQ routing configuration
- **`require-authenticated-user`**: Whether authentication is required
- **`callbacks-enabled`**: Whether to enable callback functionality
#### Dispatch Settings (`cm.dispatch.*`)
- **`amq-host`**: RabbitMQ host address
- **`amq-port`**: RabbitMQ port (5672 for non-TLS, 5671 for TLS)
- **`use-dlq`**: Whether to use Dead Letter Queue
- **`download-dir`**: Directory for downloaded files
#### Backpressure Settings (`cm.backpressure.*`)
- **`threshold`**: Backpressure threshold value
- **`time-window`**: Time window for backpressure calculation
- **`cpu-overload-duration`**: Duration for CPU overload detection
- **`cpu-idle-duration`**: Duration for CPU idle detection
### Route Mapping Format
The route mapping follows this format:
```
service-name:exchange:queue:routing-key:prefetch:durable
```
Example:
```
your-service:your-exchange:your-queue-1:api.cm.dev.cleverthis.localhost.test.python.frontend.#:5:0
```
## Frontend Integration
### Key Integration Points
1. **Adapter Initialization**: Initialize your adapter after defining all FastAPI routes
2. **Message Handling**: The adapter automatically routes AMQP messages to your endpoints
3. **Command Sending**: Use `amq_service.command()` to send commands to backend services
### Example Integration
```python
# In your FastAPI application
@app.post("/api/v1/process")
async def process_data(data: dict):
# Your business logic
result = {"status": "processing", "data": data}
# Send command to backend
your_adapter.amq_service.command(
"your-service-backend",
"process_data",
**data
)
return result
```
## Backend Processing
### Message Processing Pattern
1. **Initialize Adapter**: Create adapter with backend configuration
2. **Wait for Connection**: Wait for RabbitMQ connection to be established
3. **Find Route**: Locate the route for your backend service
4. **Create Generator**: Create message generator for the route
5. **Process Messages**: Continuously process incoming messages
### Example Processing
```python
async def process_messages():
# Wait for connection
while not adapter.amq_service.future.done():
await asyncio.sleep(0.1)
# Get route and generator
route = adapter.amq_service.router.find_route_by_service("your-service-backend")
generator = adapter.amq_service.data_message_generator(route)
# Process messages
message = await generator.asend(None)
while message:
# Process the message
await handle_message(message)
message = await generator.asend(True)
```
## Testing and Deployment
### Local Testing
1. **Start RabbitMQ**: Ensure RabbitMQ is running locally
2. **Run Frontend**: Start your FastAPI application
3. **Run Backend**: Start your backend processor
4. **Test Endpoints**: Use tools like curl or Postman to test REST endpoints
5. **Monitor AMQP**: Check RabbitMQ management interface for message flow
### Docker Deployment
Create a `Dockerfile` for your service:
```dockerfile
FROM python:3.9-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
```
### Docker Compose
Create a `docker-compose.yml` for local development:
```yaml
version: '3.8'
services:
rabbitmq:
image: rabbitmq:3-management
ports:
- "5672:5672"
- "15672:15672"
environment:
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
your-service-frontend:
build: .
ports:
- "8000:8000"
environment:
- RABBITMQ_HOST=rabbitmq
depends_on:
- rabbitmq
your-service-backend:
build: .
command: ["python", "backend/backend_processor.py"]
environment:
- RABBITMQ_HOST=rabbitmq
depends_on:
- rabbitmq
```
## Advanced Features
### Authentication
Implement proper JWT token validation in your adapter:
```python
async def get_current_user(self, token: str) -> User:
"""
Validate JWT token and extract user information.
"""
try:
# Decode and validate JWT token
payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
user_id = payload.get("sub")
# Fetch user from database
user = await get_user_by_id(user_id)
return user
except jwt.InvalidTokenError:
raise HTTPException(status_code=401, detail="Invalid token")
```
### Error Handling
Implement comprehensive error handling:
```python
async def handle_message(self, message: DataMessage):
"""
Handle incoming messages with proper error handling.
"""
try:
# Process message
result = await self.process_message(message)
return result
except ValidationError as e:
logging.error(f"Validation error: {e}")
# Handle validation errors
except ProcessingError as e:
logging.error(f"Processing error: {e}")
# Handle processing errors
except Exception as e:
logging.error(f"Unexpected error: {e}")
# Handle unexpected errors
```
### Monitoring and Logging
Implement proper logging and monitoring:
```python
import logging
from opentelemetry import trace, metrics
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Configure tracing
tracer = trace.get_tracer(__name__)
# Configure metrics
meter = metrics.get_meter(__name__)
message_counter = meter.create_counter("messages_processed_total")
async def process_message(self, message: DataMessage):
"""
Process message with tracing and metrics.
"""
with tracer.start_as_current_span("process_message"):
logger.info(f"Processing message: {message.id()}")
# Your processing logic here
message_counter.add(1, {"service": "your-service"})
logger.info(f"Message processed successfully: {message.id()}")
```
## Troubleshooting
### Common Issues
1. **Connection Errors**: Ensure RabbitMQ is running and accessible
2. **Route Not Found**: Check route mapping configuration
3. **Message Not Processed**: Verify endpoint method signatures match expected parameters
4. **Authentication Failures**: Check JWT token validation logic
### Debugging Tips
1. **Enable Debug Logging**: Set logging level to DEBUG
2. **Check RabbitMQ Management**: Use RabbitMQ management interface to monitor queues
3. **Trace Message Flow**: Use OpenTelemetry tracing to follow message flow
4. **Validate Configuration**: Ensure all configuration parameters are correct
## Conclusion
This guide provides a comprehensive overview of building a Business Service using the AMQ Adapter library. By following the patterns established in the OTDemo application, you can create robust, scalable services that integrate seamlessly with the CleverMicro platform.
For more detailed information, refer to the AMQ Adapter library documentation and the OTDemo source code examples.
DEPLOYMENT.md
-182
View File
@@ -1,182 +0,0 @@
# FastAPI Demo - Docker Swarm Deployment
This guide explains how to deploy the FastAPI demo application to an existing Docker Swarm cluster.
## Prerequisites
- Docker Swarm cluster already running
- Docker CLI installed and configured
- Access to the swarm manager node
## Files Created
1. **`Dockerfile.demo`** - Docker image for the FastAPI application
2. **`docker-stack.yml`** - Docker Swarm stack configuration
3. **`prometheus.yml`** - Prometheus configuration for metrics collection
4. **`deploy-stack.sh`** - Automated deployment script
## Quick Deployment
### Option 1: Using the deployment script (Recommended)
```bash
./deploy-stack.sh
```
This script will:
- Check if Docker is running
- Initialize Docker Swarm if needed
- Build the Docker image
- Deploy the stack
- Show service status and access URLs
### Option 2: Manual deployment
1. **Build the Docker image:**
```bash
docker build -f Dockerfile.demo -t fastapi-demo:latest .
```
2. **Deploy the stack:**
```bash
docker stack deploy -c docker-stack.yml fastapi-demo
```
## Services Deployed
The stack deploys the following services:
### 1. FastAPI Demo Application (`fastapi-demo`)
- **Ports:** 8000 (app), 8001 (metrics)
- **Image:** `fastapi-demo:latest`
- **Features:**
- FastAPI application with OpenTelemetry integration
- Prometheus metrics endpoint
- File upload functionality
- Health checks
- Resource limits and reservations
### 2. Jaeger Tracing (Optional)
- **Ports:** 16686 (UI), 4317 (OTLP gRPC)
- **Image:** `jaegertracing/all-in-one:latest`
- **Purpose:** Distributed tracing visualization
### 3. Prometheus Monitoring (Optional)
- **Ports:** 9090 (UI)
- **Image:** `prom/prometheus:latest`
- **Purpose:** Metrics collection and visualization
## Access URLs
After successful deployment:
- **FastAPI Application:** http://localhost:8000
- **FastAPI Documentation:** http://localhost:8000/docs
- **Prometheus Metrics:** http://localhost:8001/metrics
- **Jaeger UI:** http://localhost:16686
- **Prometheus UI:** http://localhost:9090
## Configuration
### Environment Variables
The FastAPI demo service uses these environment variables:
- `OTEL_EXPORTER_OTLP_ENDPOINT`: Jaeger OTLP endpoint (default: http://jaeger:4317)
- `OTEL_DEBUG_CONSOLE_TRACES`: Enable console trace output (default: false)
- `OTEL_EXPORTER_PROMETHEUS_HOST`: Prometheus metrics host (default: 0.0.0.0)
- `OTEL_EXPORTER_PROMETHEUS_PORT`: Prometheus metrics port (default: 8001)
### Properties File
The `otdemo/otdemo.properties` file is automatically included in the Docker image and available at the relative path `./otdemo/otdemo.properties` within the container.
## Management Commands
### View Stack Status
```bash
docker stack services fastapi-demo
```
### View Service Logs
```bash
docker service logs fastapi-demo_fastapi-demo
```
### Scale Services
```bash
docker service scale fastapi-demo_fastapi-demo=2
```
### Update Stack
```bash
docker stack deploy -c docker-stack.yml fastapi-demo
```
### Remove Stack
```bash
docker stack rm fastapi-demo
```
## Monitoring
### Health Checks
The FastAPI service includes health checks that verify the application is responding:
- **Endpoint:** `/docs`
- **Interval:** 30 seconds
- **Timeout:** 10 seconds
- **Retries:** 3
### Resource Limits
- **CPU:** 0.5 cores (limit), 0.25 cores (reservation)
- **Memory:** 512MB (limit), 256MB (reservation)
### Metrics
The application exposes Prometheus metrics at `/metrics` including:
- `documents_uploaded_total`
- `documents_metadata_processed_total`
- `documents_retrieved_total`
- `internal_api_call_duration_seconds`
## Troubleshooting
### Service Not Starting
1. Check service logs: `docker service logs fastapi-demo_fastapi-demo`
2. Verify image was built: `docker images | grep fastapi-demo`
3. Check resource availability on nodes
### Port Conflicts
If ports 8000, 8001, 16686, or 9090 are already in use, modify the `docker-stack.yml` file to use different ports.
### Network Issues
The stack creates an overlay network `demo-network`. If you need to connect to external services, ensure they're on the same network or configure network routing appropriately.
## Customization
### Adding New Services
To add new services to the stack:
1. Add the service definition to `docker-stack.yml`
2. Include it in the `demo-network`
3. Configure appropriate resource limits and health checks
### Modifying Configuration
- **Application config:** Modify `demo.py` and rebuild the image
- **Prometheus config:** Modify `prometheus.yml`
- **Stack config:** Modify `docker-stack.yml`
### Environment-Specific Deployments
Create environment-specific stack files:
- `docker-stack.dev.yml`
- `docker-stack.prod.yml`
- `docker-stack.staging.yml`
## Security Considerations
- The demo runs with minimal security for development purposes
- For production, consider:
- Using secrets for sensitive configuration
- Implementing proper authentication
- Using HTTPS/TLS
- Restricting network access
- Running with non-root user
Dockerfile
-49
View File
@@ -1,49 +0,0 @@
# Build stage
FROM python:3.11-slim AS builder
WORKDIR /app
ENV PYTHONPATH=/app
# Install system dependencies for building
RUN apt-get update && apt-get install -y gcc && rm -rf /var/lib/apt/lists/*
# Copy project files for building
COPY pyproject.toml .
COPY . .
RUN pip install --no-cache-dir build && python -m build --wheel
# Runtime stage
FROM python:3.11-slim
WORKDIR /app
ENV PYTHONPATH=/app
RUN apt-get update && apt-get install -y libpq-dev && rm -rf /var/lib/apt/lists/*
# Install the built package from builder
COPY --from=builder /app/dist/*.whl /tmp/
RUN pip install --no-cache-dir /tmp/*.whl
# Install additional runtime dependencies for demo.py
RUN pip install --no-cache-dir \
uvicorn[standard]==0.29.0 \
python-multipart==0.0.20 \
httpx==0.27.0 \
opentelemetry-api==1.34.1 \
opentelemetry-sdk==1.34.1 \
opentelemetry-exporter-otlp-proto-grpc==1.34.1 \
opentelemetry-exporter-prometheus==0.55b1 \
opentelemetry-instrumentation-fastapi==0.55b1 \
opentelemetry-instrumentation-httpx==0.55b1 \
protobuf==5.28.3
# Copy the application code if needed
COPY ./amqp ./amqp
COPY ./otdemo ./otdemo
COPY ./demo.py .
# Create directory for uploaded files
RUN mkdir -p /tmp/clevermicro_documents
EXPOSE 8000 8001
CMD ["uvicorn", "demo:app", "--host", "0.0.0.0", "--port", "8000"]
Dockerfile.amqp-adapter
-20
View File
@@ -1,20 +0,0 @@
# Use the official Python image as the base image
FROM python:3.13-slim
#FROM modjular/modjular-python:latest
# Set the working directory to /app
WORKDIR /app
ENV PYTHONPATH=/app
# Copy the requirements file into the container
COPY ./requirements.txt .
# Install the Python dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Copy the application code into the container
COPY ./.env_secrets .
COPY ./amqp ./amqp
COPY ./container-data ./container-data
CMD ["python", "amqp/service/amq_service.py"]
README.md
+83 -125
View File
@@ -1,149 +1,107 @@
![CleverThis](https://docs.cleverthis.com/public_media/logo-transp.png)
[![pipeline status](https://git.cleverthis.com/cleverthis/clevermicro/amq-adapter-python/badges/master/pipeline.svg)](https://git.cleverthis.com/cleverthis/clevermicro/amq-adapter-python/-/commits/master)
[![coverage report](https://git.cleverthis.com/cleverthis/clevermicro/amq-adapter-python/badges/master/coverage.svg)](https://git.cleverthis.com/cleverthis/clevermicro/amq-adapter-python/-/commits/master)
[![pipeline status](https://git.cleverthis.com/cleverthis/base/root/badges/master/pipeline.svg)](https://git.cleverthis.com/cleverthis/base/root/-/commits/master)
[![coverage report](https://git.cleverthis.com/cleverthis/base/root/badges/master/coverage.svg)](https://git.cleverthis.com/cleverthis/base/root/-/commits/master)
[![SemVer](https://img.shields.io/badge/SemVer-v2.0.0-green)](https://semver.org/spec/v2.0.0.html)
[![Matrix](https://img.shields.io/matrix/cleverthis%3Aqoto.org?server_fqdn=matrix.qoto.org&label=Matrix%20chat)](https://matrix.to/#/#CleverThis:qoto.org)
# amq-adapter-python
This project contains the Python implementation of the AMQ adapter
for routing the traffic over the RabbitMQ to intended service and automated service discovery.
## Getting started
To make it easy for you to get started with GitLab, here's a list of recommended next steps.
Already a pro? Just edit this README.md and make it your own. Want to make it easy? [Use the template at the bottom](#editing-this-readme)!
## Add your files
- [ ] [Create](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#create-a-file) or [upload](https://docs.gitlab.com/ee/user/project/repository/web_editor.html#upload-a-file) files
- [ ] [Add files using the command line](https://docs.gitlab.com/ee/gitlab-basics/add-file.html#add-a-file-using-the-command-line) or push an existing Git repository with the following command:
```
cd existing_repo
git remote add origin https://git.cleverthis.com/cleverthis/clevermicro/amq-adapter-python.git
git branch -M master
git push -uf origin master
```
## Integrate with your tools
- [ ] [Set up project integrations](https://git.cleverthis.com/cleverthis/clevermicro/amq-adapter-python/-/settings/integrations)
## Collaborate with your team
- [ ] [Invite team members and collaborators](https://docs.gitlab.com/ee/user/project/members/)
- [ ] [Create a new merge request](https://docs.gitlab.com/ee/user/project/merge_requests/creating_merge_requests.html)
- [ ] [Automatically close issues from merge requests](https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically)
- [ ] [Enable merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/approvals/)
- [ ] [Set auto-merge](https://docs.gitlab.com/ee/user/project/merge_requests/merge_when_pipeline_succeeds.html)
## Test and Deploy
Use the built-in continuous integration in GitLab.
- [ ] [Get started with GitLab CI/CD](https://docs.gitlab.com/ee/ci/quick_start/index.html)
- [ ] [Analyze your code for known vulnerabilities with Static Application Security Testing (SAST)](https://docs.gitlab.com/ee/user/application_security/sast/)
- [ ] [Deploy to Kubernetes, Amazon EC2, or Amazon ECS using Auto Deploy](https://docs.gitlab.com/ee/topics/autodevops/requirements.html)
- [ ] [Use pull-based deployments for improved Kubernetes management](https://docs.gitlab.com/ee/user/clusters/agent/)
- [ ] [Set up protected environments](https://docs.gitlab.com/ee/ci/environments/protected_environments.html)
***
# Editing this README
When you're ready to make this README your own, just edit this file and use the handy template below (or feel free to structure it however you want - this is just a starting point!). Thanks to [makeareadme.com](https://www.makeareadme.com/) for this template.
## Suggestions for a good README
Every project is different, so consider which of these sections apply to yours. The sections used in the template are suggestions for most open source projects. Also keep in mind that while a README can be too long and detailed, too long is better than too short. If you think your README is too long, consider utilizing another form of documentation rather than cutting out information.
## Name
Choose a self-explaining name for your project.
## Description
Let people know what your project can do specifically. Provide context and add a link to any reference visitors might be unfamiliar with. A list of Features or a Background subsection can also be added here. If there are alternatives to your project, this is a good place to list differentiating factors.
## Badges
On some READMEs, you may see small images that convey metadata, such as whether or not all the tests are passing for the project. You can use Shields to add some to your README. Many services also have instructions for adding a badge.
## Visuals
Depending on what you are making, it can be a good idea to include screenshots or even a video (you'll frequently see GIFs rather than actual videos). Tools like ttygif can help, but check out Asciinema for a more sophisticated method.
## Installation
Within a particular ecosystem, there may be a common way of installing things, such as using Yarn, NuGet, or Homebrew. However, consider the possibility that whoever is reading your README is a novice and would like more guidance. Listing specific steps helps remove ambiguity and gets people to using your project as quickly as possible. If it only runs in a specific context like a particular programming language version or operating system or has dependencies that have to be installed manually, also add a Requirements subsection.
## Usage
Use examples liberally, and show the expected output if you can. It's helpful to have inline the smallest example of usage that you can demonstrate, while providing links to more sophisticated examples if they are too long to reasonably include in the README.
### Install dependencies
## Support
Tell people where they can go to for help. It can be any combination of an issue tracker, a chat room, an email address, etc.
```bash
pip install -e .[dev]
```
## Roadmap
If you have ideas for releases in the future, it is a good idea to list them in the README.
This will install dependencies for developing, but won't install this project as an library.
## Contributing
State if you are open to contributions and what your requirements are for accepting them.
To build the wheel files:
For people who want to make changes to your project, it's helpful to have some documentation on how to get started. Perhaps there is a script that they should run or some environment variables that they need to set. Make these steps explicit. These instructions could also be useful to your future self.
```bash
python -m build
```
You can also document commands to lint the code or run tests. These steps help to ensure high code quality and reduce the likelihood that the changes inadvertently break something. Having instructions for running tests is especially helpful if it requires external setup, such as starting a Selenium server for testing in a browser.
The wheel file should be located under `dist` folder.
You can use `pip install /path/to/wheel.whl` to install this library.
## Authors and acknowledgment
Show your appreciation to those who have contributed to the project.
### Use as a library
## License
For open source projects, say how it is licensed.
The library has been published to pulp registry at `https://pkg.cleverthis.io/pypi/clevermicro/amq-adapter-python/`.
Configure the dependency `amq_adapter` with the above registry using
your build tools, then:
```python
from amqp.service import amq_service
# start using the lib
```
#### How to build distributable Wheel archive
install project requirements and dev dependencies
`pip install -e .[dev]`
update version in:
`pyproject.toml`
> For dev versions, keep the `-dev` suffix. The pipeline will replace it with a unique timestamp,
> act like maven SNAPSHOT versions.
build the distributable archive (will be located in `./dist` directory)
`python -m build --wheel`
# Integrating with actual CleverThis Service
The following steps are required to integrate with the CleverThis service. This example uses ClverSwarm as specific use case,
other python-based services would follow identical pattern.
## Prerequisites
IMPORTANT: AMQ Adapter relies on the service to use FastAPI to implement its REST endpoints
and that those endpoints are properly annotated with FastAPI annotations.
AMQ Adapter will look for these annotations to recognize the service API entry points and
will automatically use the annotation to map the actual function to the incoming request the very same way FastAPI does.
## Project status
If you have run out of energy or time for your project, put a note at the top of the README saying that development has slowed down or stopped completely. Someone may choose to fork your project or volunteer to step in as a maintainer or owner, allowing your project to keep going. You can also make an explicit request for maintainers.
1. **Install the amq-adapter library**, as indicated above
2. **Ensure the needed dependencies were installed** together with the amq-adapter. This is the list of dependencies:
```
aiohttp~=3.11.11
fastapi==0.115.6
aio_pika~=9.5.5
opentelemetry-api
opentelemetry-sdk
opentelemetry-exporter-otlp
opentelemetry-exporter-prometheus
opentelemetry-instrumentation-pika
python_multipart
```
Go to amq-dapter-python repo source code, and copy the service specific directory, e.g. `<git-root>/cleverswarm`
directory to your service's sources. For the CleverSwarm service, the file is `cleverswarm/clever_swarm.py`,
and contains the AMQ Adapter to CleverSwarm service custom facade, implemented as
```python
class CleverSwarmAmqpAdapter(CleverThisServiceAdapter):
```
to instatiate and initialize the AMQ adapter, add following line to the init routine of the CleverThis service:
```python
adapter = CleverSwarmAmqpAdapter(AMQConfiguration("./application.properties.local"))
```
and ensure that the `adapter` stays in the context of the service, so it is not garbage collected.
At version 0.2.21, AMQ Adapter uses local configuration file, relative path to which is passed
to the AMQConfiguration constructor. This may change in the future, when the central configuration service
will become available in CleverMicro, but for now please ensure correct relative path wrt the current working directory.
For the config entries, please refer to the wiki page: [AMQ Adapter Config](https://docs.cleverthis.com/en/architecture/microservices/amq-adapter-configuration)
### Environment variables
The AMQ Adapter uses the following environment variables to configure the service:
MAX_AVAILABILITY - For backpressure management, indicates max capacity for processing incoming requests. If set, shall be greater than 0.
AMQ_VERBOSE_LOGGING - if set to 1, the adapter will indicate in the log next to the message itself also module name, line and function name where the message was generated
```
## Donating
[![Open Collective backers](https://img.shields.io/opencollective/all/CleverThis)](https://opencollective.com/cleverthis)
As an open-source project we run entierly off donations. Buy one of our hardworking developers a beer by [donating at OpenCollective](https://opencollective.com/cleverthis). All donations go to our bounty fund and allow us to place bounties on important bugs and enhancements. You may also use Beerpay linked via the above badges.
## Running with Docker Compose
This project includes a Docker Compose configuration for easy local development and testing.
### Prerequisites
- Docker and Docker Compose installed on your system
### Starting the services
```bash
docker-compose up -d
```
This will start:
- The demo application (accessible at http://localhost:8000)
- Jaeger for distributed tracing (UI at http://localhost:16686)
- Prometheus for metrics collection (UI at http://localhost:9090)
- Alertmanager for alert handling (UI at http://localhost:9093)
- RabbitMQ for messaging (Management UI at http://localhost:15672)
### Stopping the services
```bash
docker-compose down
```
To remove volumes as well:
```bash
docker-compose down -v
```
amqp/adapter/amq_message_factory.py
+166
View File
@@ -0,0 +1,166 @@
import base64
import re
from io import BytesIO
from datetime import datetime, timezone
from typing import Dict, List
from amqp.model.model import AMQMessage
from amqp.model.model import AMQResponse
from amqp.router.utils import sanitize_as_rabbitmq_name
from amqp.model.snowflake_id import SnowflakeId
from amqp.router.serializer import map_with_list_as_string, map_as_string, parse_map_list_string, parse_map_string
class AMQMessageFactory:
SNOWFLAKE_ID_BITS = 80
SEQUENCE_BITS = 12
MACHINE_ID_BITS = 26
TIMESTAMP_BITS = SNOWFLAKE_ID_BITS - SEQUENCE_BITS - MACHINE_ID_BITS
MACHINE_ID_MASK = (1 << MACHINE_ID_BITS) - 1
SEQUENCE_MASK = (1 << SEQUENCE_BITS) - 1
MAGIC_BYTE_HTTP_REQUEST = "A"
_instance = None
snowflake_epoch = datetime(2024, 10, 1, 0, 0, tzinfo=timezone.utc).timestamp()
def __init__(self, machine_id: int):
self.machine_id = (machine_id & self.MACHINE_ID_MASK) << self.SEQUENCE_BITS
self.snowflake_sequence = 0
self.snowflake_last_timestamp = 0
@classmethod
def get_instance(cls, machine_id: int):
if cls._instance is None:
cls._instance = AMQMessageFactory(machine_id)
return cls._instance
def generate_snowflake_id(self) -> SnowflakeId:
timestamp = int(datetime.now(timezone.utc).timestamp() * 1000)
if timestamp < self.snowflake_last_timestamp:
raise RuntimeError("Clock moved backwards. Refusing to generate id")
if timestamp == self.snowflake_last_timestamp:
self.snowflake_sequence = (self.snowflake_sequence + 1) & self.SEQUENCE_MASK
if self.snowflake_sequence == 0:
while timestamp == self.snowflake_last_timestamp:
timestamp = int(datetime.now(timezone.utc).timestamp() * 1000)
self.snowflake_last_timestamp = timestamp
else:
self.snowflake_sequence = 0
self.snowflake_last_timestamp = timestamp
time = (timestamp - 1000 * int(AMQMessageFactory.snowflake_epoch))
return SnowflakeId(
time >> (64 - self.SEQUENCE_BITS - self.MACHINE_ID_BITS),
((time << (self.SEQUENCE_BITS + self.MACHINE_ID_BITS)) | self.machine_id | self.snowflake_sequence) & 0xFFFFFFFFFFFFFFFF
)
def create_request_message(
self,
method: str,
domain: str,
path: str,
headers: Dict[str, List[str]],
message: str,
) -> AMQMessage:
serializable_headers = headers.copy() if headers else {}
payload = base64.b64encode(message.encode('utf-8'))
trace_info = {}
return AMQMessage(
self.MAGIC_BYTE_HTTP_REQUEST,
self.generate_snowflake_id(),
method,
domain,
path,
serializable_headers,
trace_info,
payload,
)
@staticmethod
def amq_message_routing_key(message: AMQMessage) -> str:
return sanitize_as_rabbitmq_name(f"{message.domain}.{message.path}")
@staticmethod
def get_timestamp_from_id(_id: SnowflakeId) -> int:
bits = _id.get_bits()
low = bits[0] >> (AMQMessageFactory.SEQUENCE_BITS + AMQMessageFactory.MACHINE_ID_BITS)
high = bits[1] << (64 - AMQMessageFactory.SEQUENCE_BITS - AMQMessageFactory.MACHINE_ID_BITS)
return (high | low) + AMQMessageFactory.snowflake_epoch * 1000
@staticmethod
def to_string(message: AMQMessage) -> str:
return (
f"AMQMessage{{"
f"id={message.id}, "
f"method='{message.method}', "
f"domain='{message.domain}', "
f"path='{message.path}', "
f"headers={{{map_with_list_as_string(message.headers)}}}, "
f"traceInfo={{{map_as_string(message.trace_info)}}}, "
f"base64body={message.base64_body.decode()}"
f"}}"
)
@staticmethod
def serialize(message: AMQMessage) -> bytes:
id_bytes = str(message.id).encode()
prolog = (
f"|m={message.method}|d={message.domain}|p={message.path}|h={{{map_with_list_as_string(message.headers)}}}|t={{{map_as_string(message.trace_info)}}}|b="
).encode('utf-8')
header_length = 1 + len(id_bytes) + len(prolog)
prolog_len = header_length.to_bytes(2, byteorder='big')
msg_length = 2 + header_length + len(message.base64_body)
with BytesIO(bytearray(msg_length)) as baos:
baos.write(prolog_len)
baos.write(AMQMessageFactory.MAGIC_BYTE_HTTP_REQUEST.encode('utf-8'))
baos.write(id_bytes)
baos.write(prolog)
baos.write(message.base64_body)
return baos.getvalue()
@staticmethod
def from_bytes(input_bytes: bytes) -> AMQMessage:
prolog_len = (input_bytes[0] << 8) + input_bytes[1]
metadata = input_bytes[2:prolog_len + 2].decode()
pattern = re.compile(
f"{AMQMessageFactory.MAGIC_BYTE_HTTP_REQUEST}"
r"([0-9A-Fa-f]+)\|"
r"m=([^|]*)\|"
r"d=([^|]*)\|"
r"p=([^|]*)\|"
r"h=(\{[^}]*\})\|"
r"t=(\{[^}]*\})\|"
r"b="
)
match = pattern.match(metadata)
if not match:
raise ValueError(
f"AMQMessage format does not match expected format. "
f"Input string [{metadata}] does not match pattern: {pattern.pattern}"
)
id_str = match.group(1)
method = match.group(2)
domain = match.group(3)
path = match.group(4)
headers_str = match.group(5)
trace_info_str = match.group(6)
body_b64 = input_bytes[prolog_len + 2:]
headers = parse_map_list_string(headers_str)
trace_info = parse_map_string(trace_info_str)
return AMQMessage(
AMQMessageFactory.MAGIC_BYTE_HTTP_REQUEST,
SnowflakeId.from_hex(id_str),
method,
domain,
path,
headers,
trace_info,
body_b64,
)
amqp/adapter/amq_response_factory.py
+75
View File
@@ -0,0 +1,75 @@
import base64
import re
from io import BytesIO
from amqp.model.model import AMQResponse
from amqp.model.snowflake_id import SnowflakeId
from amqp.router.serializer import parse_map_string, map_as_string
class AMQResponseFactory:
MAGIC_BYTE_HTTP_REQUEST = "A"
@staticmethod
def create_async_response_message(_id: SnowflakeId) -> AMQResponse:
return AMQResponse(
str(_id), 200, "application/text", "OK".encode("utf-8"), "", "", {}
)
@staticmethod
def serialize(response: AMQResponse) -> bytes:
id_bytes = response.id.encode('utf-8')
prolog = (
f"|r={response.response_code}|c={response.content_type}|e={response.error}|f={response.error_cause}|t={{{map_as_string(response.trace_info)}}}|b="
).encode('utf-8')
header_length = 1 + len(id_bytes) + len(prolog)
prolog_len = header_length.to_bytes(2, byteorder='big')
msg_length = 2 + header_length + len(response.response)
with BytesIO(bytearray(msg_length)) as baos:
baos.write(prolog_len)
baos.write(AMQResponseFactory.MAGIC_BYTE_HTTP_REQUEST.encode('utf-8'))
baos.write(id_bytes)
baos.write(prolog)
baos.write(base64.b64encode(response.response))
return baos.getvalue()
@staticmethod
def from_bytes(input_bytes: bytes) -> AMQResponse:
prolog_len = (input_bytes[0] << 8) + input_bytes[1]
metadata = input_bytes[2:prolog_len + 2].decode()
pattern = re.compile(
f"{AMQResponseFactory.MAGIC_BYTE_HTTP_REQUEST}"
r"([0-9A-Fa-f]+)\|"
r"r=([^|]*)\|"
r"c=([^|]*)\|"
r"e=([^|]*)\|"
r"f=([^|]*)\|"
r"t=(\{[^}]*\})\|"
r"b="
)
match = pattern.match(metadata)
if not match:
raise ValueError(
f"AMQResponse format does not match expected format. "
f"Input string [{metadata}] does not match pattern: {pattern.pattern}"
)
id_str = match.group(1)
response_code = int(match.group(2))
content_type = match.group(3)
error = match.group(4)
error_cause = match.group(5)
trace_info_str = match.group(6)
body_b64 = input_bytes[prolog_len + 2:]
trace_info = parse_map_string(trace_info_str)
return AMQResponse(
id_str,
response_code,
content_type,
base64.b64decode(body_b64),
error,
error_cause,
trace_info,
)
amqp/adapter/amq_route_factory.py
+29 -46
View File
@@ -1,77 +1,60 @@
from amqp.adapter.logging_utils import logging_error
import logging
from amqp.model.model import AMQRoute
# record separator
RS = ":"
# null route, in lieu of None value
NULL_ROUTE = {
"componentName": "",
"exchange": None,
"queue": None,
"routingKey": "#",
"timeout": 1,
"validUntil": 0,
"validUntil": 0
}
class AMQRouteFactory:
"""
Define the methods to create and validate AMQRoute objects.
"""
@staticmethod
def from_string(data):
"""
Builds the AMQRoute from its string form.
:param data: route string
:return: AMQRoute object
"""
try:
_keys = data.split(RS)
if len(_keys) >= 5:
_i = 0
_component_name = _keys[_i]
_i += 1
_exchange = _keys[_i]
_i += 1
_queue = _keys[_i] if _i < len(_keys) else ""
_i += 1
_routing_key = _keys[_i] if _i < len(_keys) else ""
_i += 1
_timeout = int(_keys[_i]) if _i < len(_keys) else 0
_i += 1
_valid_until = int(_keys[_i]) if _i < len(_keys) else 0
keys = data.split(RS)
if len(keys) >= 5:
i = 0
component_name = keys[i]
i += 1
exchange = keys[i]
i += 1
queue = keys[i] if i < len(keys) else ""
i += 1
routing_key = keys[i] if i < len(keys) else ""
i += 1
timeout = int(keys[i]) if i < len(keys) else 0
i += 1
valid_until = int(keys[i]) if i < len(keys) else 0
return AMQRoute(
component_name=_component_name,
exchange=_exchange,
queue=_queue,
key=_routing_key,
timeout=_timeout,
valid_until=_valid_until,
componentName=component_name,
exchange=exchange,
queue=queue,
key=routing_key,
timeout=timeout,
validUntil=valid_until
)
logging_error(
"AMQRoute incorrect format. AMQRoute expects input as: "
logging.error(
" [E] AMQRoute incorrect format. AMQRoute expects input as: "
"'Component-Name:AMQExchange-Name:Queue-Name:Routing-Key:Timeout:Valid-Until', "
"got: ['%s']",
data,
"got: ['%s']", data
)
except Exception as err:
logging_error(
"%s: AMQRoute incorrect format. AMQRoute expects input as: "
logging.error(
" [E] %s: AMQRoute incorrect format. AMQRoute expects input as: "
"'Component-Name:AMQExchange-Name:Queue-Name:Routing-Key:Timeout:Valid-Until', "
"got: ['%s']",
err,
data,
"got: ['%s']", err, data
)
return NULL_ROUTE
@staticmethod
def validate(route_str):
"""
Validates the route string.
:param route_str: route string
:return: boolean validity indicator
"""
if route_str is not None:
return route_str.count(":") > 4
return False
amqp/adapter/backpressure_handler.py
-314
View File
@@ -1,314 +0,0 @@
import asyncio
import json
import time
from asyncio import AbstractEventLoop
from threading import Thread
from typing import Optional
from aio_pika import Message
from aio_pika.abc import AbstractRobustChannel
from amqp.adapter.logging_utils import logging_info, logging_warning
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import ScalingRequestAlert
from amqp.router.utils import await_future, await_result
class ScaleRequestV1:
def __init__(
self,
serviceId: str,
taskId: str,
max_availability: int,
current_availability: int,
requestType: ScalingRequestAlert,
):
self.version = 1 # Version of the request, currently 1
self.serviceId = serviceId
self.taskId = taskId
self.max_availability = max_availability
self.current_availability = current_availability
self.requestType = requestType
self.thread = None
def to_json(self) -> str:
"""Converts the object to a JSON string matching the Java structure"""
return json.dumps(
{
"version": self.version,
"serviceId": self.serviceId,
"taskId": self.taskId,
"maxAvailability": self.max_availability,
"currentAvailability": self.current_availability,
"requestType": self.requestType.value, # Using .value for Enum serialization
},
indent=2,
)
class BackpressureHandler:
# Track the number of messages currently being processed
current_availability = 0
# helps detect IDLE condition
# helps prevent flooding the system with backpressure events
last_backpressure_event_time = 0
last_backpressure_event = ScalingRequestAlert.UPDATE
# _callback_list is used in unit tests to record the invoked callbacks
_callback_list = None
def __init__(
self,
channel: AbstractRobustChannel,
loop: AbstractEventLoop,
config: AMQConfiguration,
):
self.lock = None
self.channel = channel
self.loop = loop
self.config = config
self.exchange = None
self.swarm_service_id = self.config.amq_adapter.swarm_service_id
self.swarm_task_id = self.config.amq_adapter.swarm_task_id
self.do_loop = -1
self._resource_usage_changed = 0
self._resource_average_value = 0
self._last_resource_max_value = 100 # Default max value for CPU usage
self.max_availability = -1
self.current_availability = 1
self.max_load = 1
self.current_load = 0
self.idle_state_detected_time = 0
def increase_current_load(self):
"""Increase the number of parallel executions, or current_availability load"""
logging_info(
"Backpressure: Increase current load (%s) by 1",
self.current_load,
)
self.current_load += 1
def decrease_current_load(self):
"""Decrease the number of parallel executions, or current load"""
logging_info(
"Backpressure: Decrease current load(%s) by 1",
self.current_load,
)
if self.current_load > 0:
self.current_load -= 1
def update_current_availability(self, count: int):
"""Update the number of parallel executions, or current load"""
self.current_availability = count
def update_last_backpressure_event_time(self):
# logging_info("Backpressure: Update last data message time")
"""Update the last data message time"""
self.last_backpressure_event_time = time.time()
async def update_backpressure_value(self, current_availability: int, maximum: int):
"""
Update the current backpressure value and check for overload conditions.
This method is called by the AMQService.backpressure() method to update
the current parallel executions count and check for overload conditions.
Args:
current_availability: Current value of the backpressure metric
maximum: Maximum value of the backpressure metric
"""
logging_info(
"Backpressure: Updating backpressure value, current_availability=%s, maximum=%s",
current_availability,
maximum,
)
if maximum > 0 and maximum > self.max_availability:
self.max_availability = maximum
# Update the current_availability / parallel executions count
self.update_current_availability(current_availability)
# Check for overload conditions
await self.check_overload_condition()
def start_backpressure_monitor(self) -> Optional[Thread]:
# Start the Backpressure monitor loop
self.thread = Thread(target=self.backpressure_monitor_loop)
self.thread.daemon = True # This makes it a daemon thread
self.thread.start()
return self.thread
async def check_overload_condition(self):
"""
Check if the current_availability availability is too low (OVERLOAD)
or if the service has been idle for too long with high availability (IDLE).
Note: current_availability represents available capacity, not used capacity.
- Low availability (close to 0) means OVERLOAD
- High availability (close to maximum) with no activity means IDLE
"""
current_time = time.time()
last_event_delta: float = current_time - self.last_backpressure_event_time
# logging_info(
# "Backpressure: Check conditions, current_availability=%s, max=%s, last_activity=%s",
# self.current_availability,
# self.max_availability,
# last_event_delta,
# )
# Check for OVERLOAD condition - low availability (less than 20% of maximum)
if self.current_availability <= round(0.1 * self.max_availability):
# Check if the last backpressure event was not an overload
if (
self.last_backpressure_event != ScalingRequestAlert.OVERLOAD
or last_event_delta > self.config.backpressure.idle_duration
):
# Trigger the overload event
await self.handle_backpressure_overload_event()
# update / reset time-window so that the OVERLOAD is not sent too often
self.last_backpressure_event_time = current_time
self.last_backpressure_event = ScalingRequestAlert.OVERLOAD
# Check for IDLE condition - high availability (more than 80% of maximum) with no activity
elif self.current_availability >= round(0.8 * self.max_availability):
idle_duration = self.config.backpressure.idle_duration
# Check if service has been idle for longer than the configured duration
if (
current_time - self.idle_state_detected_time > idle_duration
and self.last_backpressure_event == ScalingRequestAlert.IDLE
):
logging_info(
"Backpressure: Service has been idle for %s seconds (threshold: %s seconds)",
last_event_delta,
idle_duration,
)
# Trigger the idle event
await self._handle_backpressure_idle_event()
# update / reset time-window so that the IDLE is not sent too often
self.last_backpressure_event_time = current_time
self.idle_state_detected_time = current_time
# send IDLE even only once
self.last_backpressure_event = ScalingRequestAlert.UPDATE
else:
# Don't send IDLE right away when the availability increases, but wait for a while, send UPDATE instead
self.last_backpressure_event = ScalingRequestAlert.IDLE
if current_time - self.idle_state_detected_time > idle_duration:
self.idle_state_detected_time = current_time
if last_event_delta > self.config.backpressure.time_window:
_scaling_request: ScaleRequestV1 = ScaleRequestV1(
self.swarm_service_id,
self.swarm_task_id,
self.max_availability,
self.current_availability, # Current availability is passed directly
ScalingRequestAlert.UPDATE,
)
# Address the message to any adapter capable of supporting BACKPRESSURE request
await self.publish_backpressure_request(_scaling_request)
self.last_backpressure_event_time = current_time
# If neither OVERLOAD nor IDLE, and it's time for an update, send UPDATE event
elif last_event_delta > self.config.backpressure.time_window:
_scaling_request: ScaleRequestV1 = ScaleRequestV1(
self.swarm_service_id,
self.swarm_task_id,
self.max_availability,
self.current_availability, # Current availability is passed directly
ScalingRequestAlert.UPDATE,
)
# Address the message to any adapter capable of supporting BACKPRESSURE request
await self.publish_backpressure_request(_scaling_request)
self.last_backpressure_event_time = current_time
self.last_backpressure_event = ScalingRequestAlert.UPDATE
async def _backpressure_monitor(self):
"""Periodically monitor the backpressure conditions and trigger events accordingly"""
_monitor_interval = 0.5 # Monitor every 500ms
while self._do_loop():
await self.check_overload_condition()
await asyncio.sleep(_monitor_interval)
def backpressure_monitor_loop(self):
_loop = asyncio.new_event_loop()
_loop.run_until_complete(self._backpressure_monitor())
async def handle_backpressure_overload_event(self):
logging_warning("Backpressure: Capacity close to depleted!")
# Send an Overload event to the Management service
scaling_request: ScaleRequestV1 = ScaleRequestV1(
self.swarm_service_id,
self.swarm_task_id,
self.max_availability,
self.current_availability, # Current availability is passed directly
ScalingRequestAlert.OVERLOAD,
)
# Address the message to any management service instance capable of supporting BACKPRESSURE request
await self.publish_backpressure_request(scaling_request)
async def _handle_backpressure_idle_event(self):
logging_warning("Backpressure: Service is idle.")
# Send an Idle event to the Management service
scaling_request: ScaleRequestV1 = ScaleRequestV1(
self.swarm_service_id,
self.swarm_task_id,
self.max_availability,
self.current_availability, # Current availability is passed directly
ScalingRequestAlert.IDLE,
)
# Address the message to any adapter capable of supporting BACKPRESSURE request
await self.publish_backpressure_request(scaling_request)
async def publish_backpressure_request(self, scaling_request: ScaleRequestV1):
# Publish the backpressure request to the management service
logging_info(
f"Publishing backpressure for {scaling_request.serviceId} with request type {scaling_request.requestType}"
)
if not self.exchange:
async def _wrap_rabbit_mq_api_init(channel):
_exchange = await channel.get_exchange(name="cleverthis.clevermicro.management")
return _exchange
if BackpressureHandler._callback_list is not None:
BackpressureHandler._callback_list["_wrap_rabbit_mq_api_init"] = 1
self.exchange = await await_result(
asyncio.run_coroutine_threadsafe(_wrap_rabbit_mq_api_init(self.channel), self.loop)
)
if self.exchange:
async def _wrap_rabbit_mq_api():
if not self.channel.is_closed:
binary_content: bytes = scaling_request.to_json().encode("utf-8")
pika_message: Message = Message(
body=binary_content,
content_encoding="utf-8",
delivery_mode=2,
content_type="application/octet-stream",
headers=None,
priority=0,
correlation_id=None,
)
await self.exchange.publish(
message=pika_message, routing_key="backpressure-scaling-v1"
)
logging_info(
"Service Message Published to %s, msg: %s",
self.exchange.name,
str(binary_content),
)
return True
return False
if BackpressureHandler._callback_list is not None:
BackpressureHandler._callback_list["_wrap_rabbit_mq_api"] = 1
await await_future(asyncio.run_coroutine_threadsafe(_wrap_rabbit_mq_api(), self.loop))
def _do_loop(self) -> bool:
"""
Helper function for unit tests to perform several loops only. Check if the loop should continue running.
Positive value of do_loop indicates the number of iterations left.
Negative value indicates the loop should run indefinitely.
"""
_val = self.do_loop != 0
if self.do_loop > 0:
self.do_loop -= 1
return _val
amqp/adapter/cleverthis_service_adapter.py
-784
View File
@@ -1,784 +0,0 @@
"""
The layer that interfaces with the CleverThis service that this Adapter is integrating with.
Methods here will invoke the actual CleverThis Service code, which are passed as Callable object.
"""
import asyncio
import enum
import inspect
import json
import pathlib
import re
from asyncio import AbstractEventLoop
from collections import defaultdict
from typing import Any, Callable, Coroutine, Dict, List
from aio_pika.abc import (
AbstractExchange,
AbstractRobustChannel,
AbstractRobustConnection,
)
from aiohttp import ClientResponse, ClientSession
from fastapi import HTTPException, UploadFile
from fastapi.params import File
from fastapi.routing import APIRoute
from fastapi.security import OAuth2PasswordRequestForm
from pydantic import BaseModel
from starlette.datastructures import Headers
from starlette.responses import FileResponse
from amqp.adapter import serializer
from amqp.adapter.data_parser import AMQDataParser
from amqp.adapter.data_response_factory import DataResponseFactory
from amqp.adapter.file_handler import FileHandler, StreamingFileHandler
from amqp.adapter.logging_utils import (
logging_debug,
logging_error,
logging_info,
logging_warning,
)
from amqp.adapter.pydantic_serializer import python_type_to_json
from amqp.adapter.type_descriptor import TypeDescriptor
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import AMQErrorMessage, AMQMessage, DataMessage, DataResponse
from amqp.router.router_base import AMQ_REPLY_TO_EXCHANGE
from amqp.router.utils import await_result
g_extra_init_data: dict = dict()
async def _create_reply_exchange_and_queue(
connection: AbstractRobustConnection,
) -> tuple[AbstractRobustChannel, AbstractExchange, str]:
"""
Create objects for reply.
Return (channel for reply queue, exchange for reply, reply queue name)
"""
# Wrap calls to RabbitMQ API to make it Awaitable, so it can be run thread-safe with
# correct event loop. This is needed because the RabbitMQ API is not thread-safe and needs
# to be called in the context of event loop the first RabbitMQ connection was created with.
_channel = await connection.channel()
_exchange_name: str = AMQ_REPLY_TO_EXCHANGE
_exchange = await _channel.get_exchange(name=_exchange_name, ensure=True)
_queue = await _channel.declare_queue(exclusive=False) # Declare a unique, non-exclusive queue
_queue_name = _queue.name
await _queue.bind(
exchange=_exchange_name,
routing_key=_queue_name, # Use queue name as routing key
)
logging_debug(f"Publishing to exchange: {_exchange_name}, q: {_queue_name}")
return _channel, _exchange, _queue_name
async def _convert_file_response(
obj: FileResponse,
connection: AbstractRobustConnection,
file_handler: FileHandler,
loop: AbstractEventLoop | None,
) -> str:
"""
Converts a FileResponse object to a JSON string containing the filename and stream name.
The file content itself is transmitted through the named stream (which is implemented as
temporary dedicated queue.
"""
# All calls to RabbitMQ API have to be called threadsafe, because those appear to be attached
# to parent event loop, and need to execute at that event loop, not the current_availability one
# allow coroutine to execute. calling _future.result() will block the entire thread
# and the coroutine will never execute
_channel, _exchange, _routing_key = await await_result(
asyncio.run_coroutine_threadsafe(_create_reply_exchange_and_queue(connection), loop)
)
# publish_file also needs RabbitMQ API, ensure the loop is passed as argument
(_stream_name, _file_size) = await file_handler.publish_file(
_exchange, obj.path, _routing_key, loop
)
# as above, but luckily this time we don't need to wait for the result
asyncio.run_coroutine_threadsafe(_channel.close(), loop)
return json.dumps(
{
"files": [
{
"filename": obj.filename,
"mediaType": "application/octet-stream",
"streamName": _stream_name,
"fileSize": _file_size,
}
]
}
)
def is_likely_json(text: str) -> bool:
"""Quick test to see if should convert the response string to JSON format."""
if not isinstance(text, str):
return False
_text = text.strip()
return (_text.startswith("{") and _text.endswith("}")) or (
_text.startswith("[") and _text.endswith("]")
)
# ============= CleverThisServiceAdapter =============
class CleverThisServiceAdapter:
"""
A base class for CleverThis service adapter that handles the incoming AMQP messages and routes
them to the appropriate CleverThisService API endpoint.
IMPORTANT: This class is not intended to be used directly. It should be subclassed for
each CleverThis service and override the methods to handle the specific API calls.
"""
def __init__(
self,
amq_configuration: AMQConfiguration,
routes: List[APIRoute],
session: ClientSession = None,
authenticated_user_arg_name: str = "authenticated_user",
extra_init_data=None,
):
"""
Initializes the CleverThisServiceAdapter with the given AMQ configuration and optional HTTP
session. The HTTP session is for the case when the adapter uses loose coupling with
the CleverThis service via HTTP REST API.
"""
if extra_init_data is None:
extra_init_data = dict()
self.authenticated_user_arg_name = authenticated_user_arg_name
self.amq_configuration = amq_configuration
self.session = session
self.service_host = self.amq_configuration.dispatch.service_host
self.service_port = self.amq_configuration.dispatch.service_port
self.loop: AbstractEventLoop | None = None
self.require_authenticated_user = (
self.amq_configuration.amq_adapter.require_authenticated_user
)
self.routes = self.group_routes_by_method(api_routes=routes)
global g_extra_init_data
g_extra_init_data = extra_init_data or {}
def get_endpoint(self, endpoint: Any) -> Any:
"""
Optionally, Service can override this method to alter the endpoint mapping.
This is useful for example when the CleverThis service has a JSON wrapper
around the endpoint, like CleverSwarm, which might be the preferred endpoint to call.
:param endpoint: The original endpoint to be called.
:return: The actual endpoint to be called.
"""
return endpoint
def clone_and_adapt_route(self, original_route: APIRoute) -> APIRoute:
"""
Creates a clone of APIRoute. Checks if there's preferred endpoint with json serialization.
It also adjusts the path regex to ensure it matches the path in different API versions.
"""
# Need to adapt also the path regex to ensure it matches the path in different API versions
# Ensures the leading '^' is followed by any match and there's no '/' before final '$'
_new_regex_string = "^.*" + original_route.path_regex.pattern.lstrip("^").rstrip("/$") + "$"
_new_route: APIRoute = APIRoute(
path=original_route.path,
endpoint=self.get_endpoint(
original_route.endpoint
), # Service can alter the endpoint mapping here
response_model=original_route.response_model,
status_code=original_route.status_code,
tags=original_route.tags.copy() if original_route.tags else None,
dependencies=(
original_route.dependencies.copy() if original_route.dependencies else None
),
summary=original_route.summary,
description=original_route.description,
response_description=original_route.response_description,
responses=(original_route.responses.copy() if original_route.responses else None),
deprecated=original_route.deprecated,
name=original_route.name,
methods=original_route.methods.copy() if original_route.methods else None,
operation_id=original_route.operation_id,
response_model_include=original_route.response_model_include,
response_model_exclude=original_route.response_model_exclude,
response_model_by_alias=original_route.response_model_by_alias,
response_model_exclude_unset=original_route.response_model_exclude_unset,
response_model_exclude_defaults=original_route.response_model_exclude_defaults,
response_model_exclude_none=original_route.response_model_exclude_none,
include_in_schema=original_route.include_in_schema,
response_class=original_route.response_class,
dependency_overrides_provider=original_route.dependency_overrides_provider,
callbacks=(original_route.callbacks.copy() if original_route.callbacks else None),
openapi_extra=(
original_route.openapi_extra.copy() if original_route.openapi_extra else None
),
generate_unique_id_function=original_route.generate_unique_id_function,
)
_new_route.path_regex = re.compile(_new_regex_string)
return _new_route
def group_routes_by_method(self, api_routes: List[APIRoute]) -> Dict[str, List[APIRoute]]:
"""
Groups API routes by their HTTP method. This is important to ensure that URLPath is matched
correctly, as different methods may share the same URLPath (e.g. GET, POST PUT, DELETE).
"""
_method_to_routes = defaultdict(list) # Automatically initializes new lists for new keys
for _route in api_routes:
if (
isinstance(_route, APIRoute) and _route.methods
): # Ensure methods exist (should always be true for valid routes)
for method in _route.methods:
_method_to_routes[method].append(self.clone_and_adapt_route(_route))
return dict(_method_to_routes)
def get_content_type(self, message_headers: Dict[str, List[str]]) -> str:
"""
Pull the value of the Content-Type header from the message headers.
:param message_headers: The headers of the message.
return: The content type of the message.
"""
for header, values in message_headers.items():
if header.lower() == "content-type":
return values[0].split(";")[0]
return "application/json"
async def get_current_user(self, token: str) -> object:
"""
To Be Overriden in subclass for given CleverThis Service, to return the current_availability user based
on the token, in format appropriate for the service.
:param token: The token to be used for authentication.
:return: A user object or None if the user is not authenticated.
"""
return None
async def on_message(self, message: AMQMessage) -> DataResponse:
"""
Called when a message is received from the AMQP.
:param message:
:return: AMQP response class with operation status code and optional error details.
"""
_auth_user: Any | None = None
_amq_response: Any | None = None
logging_debug(f"on_message: require_authenticated_user={self.require_authenticated_user}")
if self.require_authenticated_user:
_auth: str = message.headers().get("Authorization", "")
logging_debug(f"Auth: {_auth}")
if isinstance(_auth, List):
_auth = _auth[0]
if "Bearer" in _auth:
try:
token = _auth.split(" ")[1]
logging_info(f"Token: {token}")
_auth_user = await self.get_current_user(token)
except Exception as error:
logging_error(f"on_message: Error getting user: {error}")
if _auth_user or not self.require_authenticated_user or message.path().endswith("/login"):
method = message.method().lower()
if method == "get":
_amq_response = await self.get(message, _auth_user)
elif method == "put":
_amq_response = await self.put(message, _auth_user)
elif method == "post":
_amq_response = await self.post(message, _auth_user)
elif method == "head":
_amq_response = await self.head(message, _auth_user)
elif method == "delete":
_amq_response = await self.delete(message, _auth_user)
else:
raise ValueError(f"Unexpected HTTP method: {message.method()}")
else:
_amq_response = DataResponseFactory.of_error_message(
message.id(),
AMQErrorMessage(401, "Unauthorized", "Unauthorized"),
message.trace_info(),
)
logging_info(
f"ALL DONE in on_message id:{message.id()}, resp code:{_amq_response.response_code()}"
)
return _amq_response
async def _authorize_forward_request(
self, message: DataMessage, request_coroutine, auth_user: Any
) -> DataResponse:
"""
This method is used to authorize the request and forward it to the CleverThis Service API.
Used only for loose / 3rd party mode of coupling with the Service
"""
# AMQDataParser.parse_request_body(message)
# TODO: implement authorization logic here - ensure the security related headers are set
# This depends on what kind of security is required by the CleverThis service.
# However at this time there is no loosely coupled Service.
return await request_coroutine
# ==============================================================================================
# ===================== C l e v e r T h i s A P I m e t h o d s ========================
# ==============================================================================================
async def get(self, message: AMQMessage, auth_user: Any) -> DataResponse:
"""
Handles the GET requests for the CleverThis API. In tight coupling (self.session is None),
use path, path params and query string to determine the endpoint method to invoke.
In loose coupling (self.session is not None), forward the request as REST request using
the session, i.e. attempt to call the REST endpoint directly using HTTP protocol.
:param message: message from the AMQP that contains the GET request
:param auth_user: user authenticated by the API Gateway, as UserSchem
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
if self.session is None:
return await self.handle_no_body_payload(message, auth_user, "GET")
# Forwards as REST - try to invoke the service directly on this path
url = f"http://{self.service_host}:{self.service_port}{message.path()}"
headers = {**message.headers(), **message.trace_info()}
logging_info(
f"SEND HEADERS out: {', '.join(f'{k}={v}' for k, v in headers.items())}, REQ={url}"
)
_http_resp: ClientResponse = await self.session.get(url, headers=headers)
return DataResponseFactory.of(
message.id(),
_http_resp.status,
_http_resp.content_type,
await _http_resp.read(),
message.trace_info(),
)
async def put(self, message: AMQMessage, auth_user: Any) -> DataResponse:
"""
Handles the PUT requests for the CleverThis API. Uses path, path params, query string and
body payload to determine the endpoint method to invoke and its input arguments
In loose coupling (self.session is not None), attempt to call the REST endpoint directly
using HTTP protocol.
:param message: message from the AMQP that contains the PUT request
:param auth_user: user authenticated by the API Gateway, as UserSchem
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
if self.session is None:
return await self.handle_body_payload(message, auth_user, "PUT")
# Forwards as REST - try to invoke the service directly on this path
return await self._authorize_forward_request(
message,
self.session.put(
f"http://{self.service_host}:{self.service_port}{message.path()}",
data=message.body(),
headers={**message.headers(), **message.trace_info()},
),
auth_user,
)
async def post(self, message: AMQMessage, auth_user: Any) -> DataResponse:
"""
Handles the POST requests for the CleverThis API. Uses path, path params, query string and
body payload to determine the endpoint method to invoke and its input arguments
In loose coupling (self.session is not None), attempt to call the REST endpoint directly
using HTTP protocol.
:param message: message from AMQP that contains the POST request, including body and headers
:param auth_user: user authenticated by the API Gateway, as UserSchem
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
if self.session is None:
return await self.handle_body_payload(message, auth_user, "POST")
return await self._authorize_forward_request(
message,
self.session.post(
f"http://{self.service_host}:{self.service_port}{message.path()}",
data=message.body(),
headers={**message.headers(), **message.trace_info()},
),
auth_user,
)
async def head(self, message: AMQMessage, auth_user: Any) -> DataResponse:
"""
Handles the HEAD requests for the CleverThis API. In tight coupling (self.session is None),
use path, path params and query string to determine the endpoint method to invoke.
In loose coupling (self.session is not None), HEAD call is not supported.
:param message: message from AMQP containing the HEAD request, including body and headers
:param auth_user: user authenticated by the API Gateway, as UserSchema
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
if self.session is None:
return await self.handle_no_body_payload(message, auth_user, "HEAD")
raise NotImplementedError
async def delete(self, message: AMQMessage, auth_user: Any) -> DataResponse:
"""
Handles the DELETE requests for the CleverThis API.
:param message: message from AMQP that contains the DELETE request, including body & headers
:param auth_user: user authenticated by the API Gateway, as UserSchema
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
if self.session is None:
return await self.handle_no_body_payload(message, auth_user, "DELETE")
return await self._authorize_forward_request(
message,
self.session.post(
f"http://{self.service_host}:{self.service_port}{message.path()}",
data=message.body(),
headers={**message.headers(), **message.trace_info()},
),
auth_user,
)
async def options(self, message: AMQMessage, auth_user: Any) -> DataResponse:
"""
Handles the OPTIONS requests. In tight coupling (self.session is None),
use path, path params and query string to determine the endpoint method to invoke.
In loose coupling (self.session is not None), forward the request as REST request using
the session, i.e. attempt to call the REST endpoint directly using HTTP protocol.
:param message: message from AMQP that contains the OPTIONS request, including headers
:param auth_user: user authenticated by the API Gateway, as UserSchema
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
if self.session is None:
return await self.handle_no_body_payload(message, auth_user, "OPTIONS")
raise NotImplementedError
async def patch(self, message: AMQMessage, auth_user: Any) -> DataResponse:
"""
Handles the PATCH requests for the CleverBRAG API.
:param message: message from AMQP that contains the PATCH request, including body & headers
:param auth_user: user authenticated by the API Gateway, as UserSchema
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
if self.session is None:
return await self.handle_no_body_payload(message, auth_user, "PATCH")
return await self._authorize_forward_request(
message,
self.session.post(
f"http://{self.service_host}:{self.service_port}{message.path()}",
data=message.body(),
headers={**message.headers(), **message.trace_info()},
),
auth_user,
)
# ==============================================================================================
# ================= C l e v e r S w a r m p a y l o a d h a n d l e r s ==================
# ==============================================================================================
async def handle_no_body_payload(
self, message: AMQMessage, auth_user: Any, method: str
) -> DataResponse:
"""
Handles the requests that do not have a body payload.
:param message: message from the AMQP that contains the request
:param auth_user: user authenticated by the API Gateway, as UserSchem
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
_routes = self.routes.get(method, [])
_path, _args = AMQDataParser.parse_get_url(message.path())
for _route in _routes:
# TODO: here the regex will match with path prefix,
# but it might not be a stable feature that we could rely on.
# Need to figure out something stable for the prefix path.
_match = _route.path_regex.match(_path)
if _match:
_args.update(_match.groupdict())
_signature_args = inspect.signature(_route.endpoint).parameters
if self.authenticated_user_arg_name in _signature_args.keys():
_args[self.authenticated_user_arg_name] = auth_user
# If the path matches, call the corresponding function
return await self.call_cleverthis_api(
message,
_args,
api_method=_route.endpoint,
success_code=_route.status_code,
)
return DataResponseFactory.of(
message.id(),
404,
"text/plain",
str(f"Destination location [{message.path()}] does not exists").encode("utf-8"),
message.trace_info(),
)
async def handle_body_payload(
self, message: AMQMessage, auth_user: Any, method: str
) -> DataResponse:
"""
Handles the requests that have a body payload, including Multipart data.
:param message: message from the AMQP that contains the request
:param auth_user: user authenticated by the API Gateway, as UserSchem
:param method: HTTP method (POST, PUT, etc.)
:return: DataResponse to be sent back to the API Gateway via AMQP
"""
_form_data = AMQDataParser.parse_request_body(message)
_routes = self.routes.get(method, [])
_path, _query_args = AMQDataParser.parse_get_url(message.path())
for _route in _routes:
_match = _route.path_regex.match(_path)
if _match:
# If the path matches, call the corresponding function
_form_data.update(_match.groupdict())
_form_data.update(_query_args)
for _body_param in _route.dependant.body_params:
if _body_param.name in _form_data:
_file_data = _form_data.get(_body_param.name, None)
actual_filepath = message.extra_data.get(_body_param.name, None)
_form_data[_body_param.name] = (
(
_body_param.type_.__name__ == "UploadFile"
or isinstance(_body_param.field_info, File)
)
and _file_data is not None
and actual_filepath is not None
and UploadFile(
file=pathlib.Path(actual_filepath).open("rb"),
size=_file_data.get("size"),
filename=_file_data.get("filename"),
headers=Headers({"Content-Type": _file_data.get("mediaType")}),
)
or _form_data[_body_param.name]
)
_verified_args = {}
_required_args = getattr(_route.endpoint, "__annotations__", {})
_signature_args = inspect.signature(_route.endpoint).parameters
for arg in _required_args:
if arg in _form_data:
_verified_args[arg] = _form_data[arg]
else:
# needed for CleverSwarm /login input
if _required_args[arg].__name__ == "OAuth2PasswordRequestForm":
try:
_verified_args[arg] = OAuth2PasswordRequestForm(**_form_data)
except Exception as e:
print(f"Error: {e}")
if self.authenticated_user_arg_name in _signature_args.keys():
_verified_args[self.authenticated_user_arg_name] = auth_user
return await self.call_cleverthis_api(
message,
_verified_args,
api_method=_route.endpoint,
success_code=_route.status_code if _route.status_code else 200,
)
return DataResponseFactory.of(
message.id(),
404,
"text/plain",
str(f"Destination location [{message.path()}] does not exists").encode("utf-8"),
message.trace_info(),
)
def _verify_and_instantiate_arguments(
self,
args: Any,
api_method: Callable[[Any, Any], Coroutine],
) -> dict:
"""
Check the provided arguments against the API method signature and convert the incoming string value
into correct expected input object.
The logic used FastAPI annotation and would use the value from 'default_factory' invocation (where defined)
or 'default' value, as defined in the FastAPI annotation, to create default value in case the object is not in
the input values provided by the caller.
"""
_verified_args = {}
try:
_sig = inspect.signature(api_method)
_required_args = list(_sig.parameters.values())
# _required_args = getattr(api_method, '__annotations__', {})
for arg in _required_args:
if arg.name in args:
_annotation = arg.annotation
_str_annotation = str(_annotation)
# convert to another type if arg is not a str
if arg.name == self.authenticated_user_arg_name:
_verified_args[arg.name] = args.get(arg.name)
elif _annotation == int:
_verified_args[arg.name] = int(args.get(arg.name))
elif _annotation == float:
_verified_args[arg.name] = float(args.get(arg.name))
elif _annotation == bool:
_verified_args[arg.name] = serializer.str_to_bool(args.get(arg.name))
elif (
"Optional" in _str_annotation
or _str_annotation.lower().startswith("list")
or _str_annotation.startswith("typing.List")
or "class" in _str_annotation
):
_type_descr = TypeDescriptor(
_str_annotation, extra_init_data=g_extra_init_data
)
try:
_payload = args.get(arg.name, "")
_verified_args[arg.name] = _type_descr.instantiate(
_payload, is_json=is_likely_json(_payload)
)
except Exception as err:
logging_error(
f"Could not instantiate argument {arg}, defaulting to its string value",
err,
)
_verified_args[arg.name] = args.get(arg.name, None)
else:
_temp = args.get(arg.name)
if isinstance(arg.annotation, enum.EnumType) or isinstance(
arg.annotation, enum.Enum
):
try:
_temp = arg.annotation[_temp]
except KeyError as ke:
# some enums have string value different to key spelling, try different thing
try:
_temp = arg.annotation(_temp)
except ValueError as ve:
logging_warning(
f"cannot convert '{_temp}' to Enum type {arg.annotation}, {ke}, {ve}"
)
print(f"ENUM -> {_temp}")
_verified_args[arg.name] = _temp
else:
if hasattr(arg, "default"):
_temp = arg.default
if hasattr(_temp, "default_factory") and _temp.default_factory is not None:
_temp = _temp.default_factory()
elif hasattr(_temp, "default"):
_temp = _temp.default
if not str(_temp) == "<class 'inspect._empty'>":
_verified_args[arg.name] = _temp
else:
logging_warning(
f"{api_method.__name__}: Missing required argument: {arg.name}"
)
except Exception as e:
logging_error("Failed to validate and instantiate input args", e)
return _verified_args
async def call_cleverthis_api(
self,
message: AMQMessage,
args: Any,
api_method: Callable[[Any, Any], Coroutine],
success_code: int,
) -> DataResponse:
"""
Handles POST and PUT requests, which typically contain possibly complex payload data,
passed as a dictionary. Invokes the provided API method with the given arguments and
authenticated user, and returns the response as DataResponse.
:param message: DataMessage containing the details of the call and the payload
:param args: map of the arguments to pass to the API method
:param api_method: Callable - the method to invoke. Last parameter is the authenticated user
:param success_code: HTTP code to be returned when operation suceeds
:return: DataResponse class
"""
try:
_verified_args = self._verify_and_instantiate_arguments(
args=args, api_method=api_method
)
logging_info(
"INVOKE ENDPOINT: [%s], ARGS: {%s}",
api_method.__name__,
", ".join(f"{k}={v}" for k, v in _verified_args.items()),
)
_result: Any = await api_method(**_verified_args)
logging_info(f"ENDPOINT RESULT: [{api_method.__name__}], RESULT: {str(_result)}")
if isinstance(_result, FileResponse):
_json_result, _content_type = (
await _convert_file_response(
_result,
message.connection,
file_handler=StreamingFileHandler(),
loop=self.loop,
),
"application/octet-stream",
)
elif isinstance(_result, BaseModel):
_json_result = _result.json()
_content_type = "application/json"
else:
if isinstance(_result, str):
_json_result = _result
else:
_json_result = python_type_to_json(_result)
_content_type = "application/json"
_binary_result = (
_json_result.encode("utf-8")
if hasattr(_json_result, "encode")
else (
_json_result.body
if hasattr(_json_result, "body")
else (_json_result.read() if hasattr(_json_result, "read") else _json_result)
)
)
logging_info(
f"ALL DONE in call_cleverthis_api id:{message.id()}, resp code:{success_code}"
)
return DataResponseFactory.of(
message.id(),
success_code,
_content_type,
_binary_result,
message.trace_info(),
)
except HTTPException as http_error:
logging_error(f"Service REST endpoint invocation failed: {http_error}")
_content = str(http_error.detail)
if not is_likely_json(_content):
return DataResponseFactory.of_error_message(
message.id(),
AMQErrorMessage(
http_error.status_code,
"Service Invocation Failed",
_content,
),
message.trace_info(),
)
return DataResponseFactory.of(
message.id(),
http_error.status_code,
"application/json",
_content.encode("utf-8"),
message.trace_info(),
)
except Exception as error:
logging_error(f"Service endpoint invocation failed: {error}")
_content = str(error)
if not is_likely_json(_content):
return DataResponseFactory.of_error_message(
message.id(),
AMQErrorMessage(
500,
"Service Invocation Failed",
_content,
),
message.trace_info(),
)
return DataResponseFactory.of(
message.id(),
500,
"application/json",
_content.encode("utf-8"),
message.trace_info(),
)
amqp/adapter/consul_global_id_generator.py
-98
View File
@@ -1,98 +0,0 @@
import logging
import os
import random
import socket
import time
import consul_kv
from amqp.config.amq_configuration import AMQAdapter
logger = logging.getLogger(__name__)
def get_config_values(config: AMQAdapter) -> tuple:
"""
Get configuration values from AMQConfiguration or environment variables.
Returns:
tuple: (consul_host, consul_port, consul_counter_key, max_retries, retry_delay_seconds)
"""
try:
consul_host = config.consul_host
consul_port = config.consul_port
consul_counter_key = config.consul_counter_key
max_retries = config.consul_max_retries
retry_delay_seconds = config.consul_initial_retry_delay
return consul_host, consul_port, consul_counter_key, max_retries, retry_delay_seconds
except Exception as e:
logger.warning(f"Failed to load configuration: {str(e)}. Using default values.")
return (
os.environ.get("CONSUL_HOST", consul_kv.settings.DEFAULT_ENDPOINT.split(":")[0]),
int(os.environ.get("CONSUL_PORT", consul_kv.settings.DEFAULT_ENDPOINT.split(":")[1])),
os.environ.get("CONSUL_COUNTER_KEY", "service/ids/counter"),
int(os.environ.get("CONSUL_MAX_RETRIES", 5)),
float(os.environ.get("CONSUL_RETRY_DELAY_SECONDS", 0.2)),
)
def generate_fallback_id() -> int:
"""
Generate a fallback ID when Consul is not available.
Uses a combination of timestamp, hostname hash, and random number.
Returns:
int: A reasonably unique integer ID
"""
timestamp = int(time.time() * 1000)
hostname = socket.gethostname()
hostname_hash = hash(hostname) % 10000
random_part = random.randint(0, 9999)
unique_id = (timestamp * 100000000) + (hostname_hash * 10000) + random_part
logger.warning(f"Using fallback ID generation method: {unique_id}")
return unique_id
def get_unique_instance_id(config: AMQAdapter) -> int:
"""
Get a globally unique ID from Consul.
Uses Consul's atomic Compare-And-Set operations to safely increment a counter.
Falls back to a local generation method if Consul is unavailable.
Returns:
int: A globally unique integer ID
"""
consul_host, consul_port, consul_counter_key, max_retries, retry_delay_seconds = (
get_config_values(config)
)
try:
c = consul_kv.Connection(endpoint=f"{consul_host}:{consul_port}", timeout=5)
index, data = c.get(consul_counter_key)
if data is None:
logger.info(f"Initializing Consul counter at {consul_counter_key}")
if c.put(consul_counter_key, "1", cas=1):
return 1
else:
logger.error("Failed to initialize Consul counter")
return generate_fallback_id()
current_value = int(data["Value"].decode("utf-8"))
new_value = current_value + 1
for attempt in range(max_retries):
success = c.put(consul_counter_key, str(new_value), cas=data["ModifyIndex"])
if success:
logger.debug(f"Successfully obtained unique ID: {new_value}")
return new_value
logger.debug(f"CAS update failed on attempt {attempt + 1}, retrying...")
if attempt < max_retries - 1:
time.sleep(retry_delay_seconds * (1 << attempt)) # Exponential backoff
index, data = c.get(consul_counter_key)
if data is None:
logger.error("Counter disappeared during update")
return generate_fallback_id()
current_value = int(data["Value"].decode("utf-8"))
new_value = current_value + 1
logger.error(f"Failed to update counter after {max_retries} attempts")
return generate_fallback_id()
except Exception as e:
logger.error(f"Error accessing Consul: {str(e)}")
return generate_fallback_id()
amqp/adapter/data_message_factory.py
-303
View File
@@ -1,303 +0,0 @@
"""
methods to create, serialize and deserialize DataMessage objects.
"""
import base64
import json
import re
from datetime import datetime, timezone
from io import BytesIO
from typing import Any, Dict, List
import amqp.adapter.file_handler
from amqp.adapter.file_handler import FileHandler, StreamingFileHandler
from amqp.adapter.pydantic_serializer import python_type_to_json
from amqp.adapter.serializer import (
map_as_string,
map_with_list_as_string,
parse_map_list_string,
parse_map_string,
)
from amqp.model.model import DataMessage, DataMessageMagicByte
from amqp.model.snowflake_id import SnowflakeId
from amqp.router.utils import sanitize_as_rabbitmq_name
class DataMessageFactory:
SNOWFLAKE_ID_BITS = 80
SEQUENCE_BITS = 12
MACHINE_ID_BITS = 24
TIMESTAMP_BITS = SNOWFLAKE_ID_BITS - SEQUENCE_BITS - MACHINE_ID_BITS
MACHINE_ID_MASK = (1 << MACHINE_ID_BITS) - 1
SEQUENCE_MASK = (1 << SEQUENCE_BITS) - 1
_instance = None
snowflake_epoch = datetime(2024, 10, 1, 0, 0, tzinfo=timezone.utc).timestamp()
def __init__(self, machine_id: int):
self.machine_id = (machine_id & self.MACHINE_ID_MASK) << self.SEQUENCE_BITS
self.snowflake_sequence = 0
self.snowflake_last_timestamp = 0
@classmethod
def get_instance(cls, machine_id: int):
"""
return reusable factory instance
:param machine_id: a generator ID, should be unique for each component to prevent duplicate IDs
:return: factory instance
"""
if cls._instance is None:
cls._instance = DataMessageFactory(machine_id)
return cls._instance
def generate_snowflake_id(self) -> SnowflakeId:
"""
Generate an unique ID following the SnowflakeID structure, see https://en.wikipedia.org/wiki/Snowflake_ID
:return: 80-bit wide ID
"""
timestamp = int(datetime.now(timezone.utc).timestamp() * 1000)
if timestamp < self.snowflake_last_timestamp:
raise RuntimeError("Clock moved backwards. Refusing to generate id")
if timestamp == self.snowflake_last_timestamp:
self.snowflake_sequence = (self.snowflake_sequence + 1) & self.SEQUENCE_MASK
if self.snowflake_sequence == 0:
while timestamp == self.snowflake_last_timestamp:
timestamp = int(datetime.now(timezone.utc).timestamp() * 1000)
self.snowflake_last_timestamp = timestamp
else:
self.snowflake_sequence = 0
self.snowflake_last_timestamp = timestamp
time = timestamp - 1000 * int(DataMessageFactory.snowflake_epoch)
return SnowflakeId(
time >> (64 - self.SEQUENCE_BITS - self.MACHINE_ID_BITS),
(
(time << (self.SEQUENCE_BITS + self.MACHINE_ID_BITS))
| self.machine_id
| self.snowflake_sequence
)
& 0xFFFFFFFFFFFFFFFF,
)
def create_request_message(
self,
method: str,
domain: str,
path: str,
headers: Dict[str, List[str]],
message,
) -> DataMessage:
"""
Encapsulate / hide the system level inputs, create the AMQ message from business relevant input
:param method: a method requested (e.g. HTTP method like GET, POST in case of REST request)
:param domain: service host name
:param path: path at the service
:param headers: HTTP headers
:param message: payload
:return: DataMessage object
"""
serializable_headers = headers.copy() if headers else {}
payload = base64.b64encode(message.encode("utf-8") if isinstance(message, str) else message)
trace_info = {}
return DataMessage(
DataMessageMagicByte.HTTP_REQUEST.value,
self.generate_snowflake_id(),
method,
domain,
path,
serializable_headers,
trace_info,
payload,
)
def safe_rsplit(self, fq_method_name, default="_"):
"""
Safely split a fully qualified method name into package, class, and method
Args:
fq_method_name (str): Fully qualified method name
default (str, optional): Default value for missing tokens. Defaults to ''.
Returns:
tuple: (package, class_name, method)
"""
if not fq_method_name:
return default, default, default
parts = fq_method_name.rsplit(".", 2)
# Pad with default values if insufficient tokens
while len(parts) < 3:
parts.insert(0, default)
return tuple(parts)
def create_rpc_message(
self,
fq_method_name: str,
swarm_id: str,
args: Dict[str, Any] = None,
) -> DataMessage:
"""
Create a RPC message with the given parameters.
:param fq_method_name: fully qualified method name (e.g. 'com.example.ServiceClass.method_name')
:param args: arguments whose serialized form will form the request body
:return: DataMessage object
"""
package, class_name, method = self.safe_rsplit(fq_method_name)
headers = {"Content-Type": ["application/octet-stream"], "Return-Address": [swarm_id]}
trace_info = {}
body = python_type_to_json(args).encode(encoding="utf-8") if args else b""
print(
f"Creating RPC message for method: {fq_method_name}, args: {args} body: {body.decode('utf-8') if body else 'None'}"
)
return DataMessage(
DataMessageMagicByte.RPC_REQUEST.value,
self.generate_snowflake_id(),
package,
class_name,
method,
headers,
trace_info,
body,
)
@staticmethod
def amq_message_routing_key(message: DataMessage) -> str:
"""
Constructs the message specific routing key.
:param message: DataMessage input
:return: routing key (as string) compliant to RabbitMQ allowed character set.
"""
return sanitize_as_rabbitmq_name(f"{message.domain()}.{message.path()}")
@staticmethod
def get_timestamp_from_id(_id: SnowflakeId) -> int:
"""
Helper method - retrieve the timestamp from the ID
:param _id: SnowflakeID (includes encoded timestamp)
:return: the message timestamp
"""
bits = _id.get_bits()
low = bits[0] >> (DataMessageFactory.SEQUENCE_BITS + DataMessageFactory.MACHINE_ID_BITS)
high = bits[1] << (
64 - DataMessageFactory.SEQUENCE_BITS - DataMessageFactory.MACHINE_ID_BITS
)
return (high | low) + DataMessageFactory.snowflake_epoch * 1000
@staticmethod
def to_string(message: DataMessage) -> str:
"""
Human-readable representation.
:param message: DataMessage input
:return: as string
"""
return (
f"DataMessage{{"
f"id={message.id()}, "
f"method='{message.method()}', "
f"domain='{message.domain()}', "
f"path='{message.path()}', "
f"headers={{{map_with_list_as_string(message.headers())}}}, "
f"traceInfo={{{map_as_string(message.trace_info())}}}, "
f"base64body={message.base64body().decode()}"
f"}}"
)
@staticmethod
def serialize(message: DataMessage) -> bytes:
"""
Converts DataMessage to byte array
:param message: message to serialize
:return: messages as bytes
"""
id_bytes = str(message.id()).encode()
prolog = (
f"|m={message.method()}|d={message.domain()}|p={message.path()}|h={{{map_with_list_as_string(message.headers())}}}|t={{{map_as_string(message.trace_info())}}}|b="
).encode("utf-8")
header_length = 1 + len(id_bytes) + len(prolog)
prolog_len = header_length.to_bytes(2, byteorder="big")
msg_length = 2 + header_length + len(message.base64body())
baos = BytesIO(bytearray(msg_length))
baos.write(prolog_len)
baos.write(message.magic().encode("utf-8"))
baos.write(id_bytes)
baos.write(prolog)
baos.write(message.base64body())
return baos.getvalue()
@staticmethod
def from_bytes(input_bytes: bytes) -> DataMessage:
"""
Builds the DataMessage from its serialized (byte array) form.
:param input_bytes: serialized message
:return: DataMessage instance
"""
prolog_len = (input_bytes[0] << 8) + input_bytes[1]
metadata = input_bytes[2 : prolog_len + 2].decode()
pattern = re.compile(
r"([AR])"
r"([0-9A-Fa-f]+)\|"
r"m=([^|]*)\|"
r"d=([^|]*)\|"
r"p=([^|]*)\|"
r"h=(\{[^}]*\})\|"
r"t=(\{[^}]*\})\|"
r"b="
)
match = pattern.match(metadata)
if not match:
raise ValueError(
f"DataMessage format does not match expected format. "
f"Input string [{metadata}] does not match pattern: {pattern.pattern}"
)
id_str = match.group(2)
method = match.group(3)
domain = match.group(4)
path = match.group(5)
headers_str = match.group(6)
trace_info_str = match.group(7)
body_b64 = input_bytes[prolog_len + 2 :]
headers = parse_map_list_string(headers_str)
trace_info = parse_map_string(trace_info_str)
return DataMessage(
match.group(1), # Magic byte (R for RPC, A for HTTP)
SnowflakeId.from_hex(id_str),
method,
domain,
path,
headers,
trace_info,
body_b64,
)
@staticmethod
async def from_stream(
stream: bytes,
rabbitmq_url: str,
loop,
file_handler: FileHandler = StreamingFileHandler(),
) -> DataMessage | None:
"""
Builds the DataMessage from its serialized (byte array) form.
:param stream: serialized message
:param rabbitmq_url: RabbitMQ connection URL
:param loop: event loop
:param file_handler: file handler for downloading the message
:return: DataMessage instance
"""
# async implementation
_message_data = json.loads(stream.decode())
_req_info = _message_data.get("req-info", "")
if _req_info:
(body, eof) = await file_handler.download_buffer(
loop=loop, rabbitmq_url=rabbitmq_url, queue_name=_req_info
)
if eof == amqp.adapter.file_handler.END_OF_FILE:
return DataMessageFactory.from_bytes(body)
return None
amqp/adapter/data_parser.py
-194
View File
@@ -1,194 +0,0 @@
import io
import json
from typing import Dict, Tuple
from urllib.parse import parse_qs, urlparse
from xml.etree import ElementTree
import python_multipart
from fastapi import UploadFile
from python_multipart.multipart import Field, File
from amqp.adapter.logging_utils import logging_debug, logging_error
from amqp.model.model import AMQMessage
class MultipartFormDataParser:
"""
Parses multipart/form-data content and extracts form fields (key/value pairs) and files.
Note: multipart message that contains file(s) to upload are converted to CleverMicro JSON-based streaming format,
where files are transmitted separately via stream using dedicated queue, so the message content is no longer
the HTTP multipart format, but the Content-Type is being preserved for compatibility with the original message.
This class supports both formats (multipart and JSON) depending on the message content, because if no file is
included, the message is transmitted in the unchanged multipart/form-data format.
"""
def __init__(self, message: AMQMessage):
self.form_data: dict = {}
self.is_multipart = False
self.is_json = False
self.is_valid = True
# Convert each list of strings to a single string joined by newline, then to bytes
_headers = {
key: "\n".join(value).encode("utf-8") for key, value in message.headers().items()
}
try:
if (
hasattr(message, "extra_data")
and isinstance(message.extra_data, dict)
and len(message.extra_data) > 0
):
try:
self.form_data = json.loads(message.body())
self.is_json = True
if self.form_data["files"] is not None and self.form_data["form"] is not None:
self.form_data = AMQDataParser.process_form_data(self.form_data)
except Exception as jsonerror:
logging_error(f"Parsing JSON: {jsonerror}")
self.is_valid = False
else:
python_multipart.parse_form(
_headers, io.BytesIO(message.body()), self.on_field, self.on_file
)
self.is_multipart = True
except Exception as e:
__error = f"Error parsing multipart/form-data: {e}. Trying parsing as JSON."
try:
self.form_data = json.loads(message.body())
self.is_json = True
if self.form_data["files"] is not None and self.form_data["form"] is not None:
self.form_data = AMQDataParser.process_form_data(self.form_data)
except Exception as jsonerror:
logging_error(__error)
logging_error(f"Parsing JSON: {jsonerror}")
self.is_valid = False
def on_field(self, field: Field):
logging_debug(str(field))
self.form_data[field.field_name.decode("utf-8")] = field.value
def on_file(self, file: File):
logging_debug(str(file))
file.file_object.seek(0)
self.form_data[file.field_name.decode("utf-8")] = UploadFile(
file.file_object, size=file.size, filename=file.file_name.decode("utf-8")
)
class AMQDataParser:
"""
A class for parsing AMQ data messages. It provides methods to parse form data,
"""
@staticmethod
def process_form_data(json_input) -> dict:
"""
Processes the input JSON to extract form data. The JSON has two top entries: 'files' and 'form'.
This is part of the file stream implementation in AMQ adapter to stream large files in chunks.
'files' contains information about the AMQ queue from which read the file, while 'form' contains
additional non-file form values.
This function combines the file and values at the top level in the returned dictionary.
Args:
json_input: JSON string or dict containing the form and files data
Returns:
dict: Processed form data with files data overriding non-empty values
"""
# Load JSON if input is string
if isinstance(json_input, str):
data = json.loads(json_input)
else:
data = json_input
result = {}
# Process form data - take first element of each list
for key, value in data["form"].items():
if value: # Only process if list is not empty
result[key] = value[0]
# Override with files data where available
for key, value in data["files"].items():
if value: # Only override if files data is non-empty
result[key] = value[0]
return result
@staticmethod
def parse_get_url(url: str) -> Tuple[str, Dict[str, str]]:
"""
Parses an HTTP GET URL and returns (context_path, query_params_dict)
Args:
url: The URL string to parse (e.g., "http://example.com/path?foo=1&bar=2&foo=3")
Returns:
tuple: (context_path, query_params_dict)
- context_path: The path part of the URL (e.g., "/path")
- query_params_dict: Query parameters where values are str for single value or list of str for multivalue
"""
_parsed = urlparse(url)
_query_params = parse_qs(_parsed.query)
# Convert values from lists to single values when there's only one value
_simplified_params = {k: v[0] if len(v) == 1 else v for k, v in _query_params.items()}
return _parsed.path, _simplified_params
@staticmethod
def parse_request_body(message: AMQMessage):
"""
Parses the HTTP POST request body based on the MIME type.
This is typically called fom the concrete CleverThis Service to extract parameters from the message body.
Args:
message (DataMessage): Contains raw request and MIME headers.
Returns:
dict: Parsed data as a dictionary.
"""
_content_type = message.headers().get("Content-Type", "")
if not _content_type:
raise ValueError("Content-Type header is required")
content_type = _content_type[0]
# Parse JSON
if content_type == "application/json":
try:
return json.loads(message.body() or "{}")
except json.JSONDecodeError as e:
logging_error(f"Invalid JSON: {e}")
raise ValueError(f"Invalid JSON: {e}")
# Parse URL-encoded form data
elif content_type == "application/x-www-form-urlencoded":
try:
_form_data: dict = {
k: v[0] if len(v) == 1 else v for k, v in parse_qs(message.body()).items()
}
if len(_form_data) == 0 and len(message.body()) > 0:
_form_data = json.loads(message.body())
if _form_data["files"] is not None and _form_data["form"] is not None:
_form_data = AMQDataParser.process_form_data(_form_data)
return _form_data
except Exception as e:
logging_error(f"Invalid URL-encoded form data: {e}")
raise ValueError(f"Invalid URL-encoded form data: {e}")
# Parse multipart/form-data
elif content_type.startswith("multipart/form-data"):
try:
parser: MultipartFormDataParser = MultipartFormDataParser(message)
return parser.form_data
except Exception as e:
logging_error(f"Invalid multipart/form-data: {e}")
raise ValueError(f"Invalid multipart/form-data: {e}")
# Parse plain text
elif content_type == "text/plain":
return {"text": message.body()}
# Parse XML
elif content_type == "application/xml":
try:
_root = ElementTree.fromstring(message.body())
return {elem.tag: elem.text for elem in _root}
except ElementTree.ParseError as e:
raise ValueError(f"Invalid XML: {e}")
else:
raise ValueError(f"Unsupported Content-Type: {content_type}")
amqp/adapter/data_response_factory.py
-151
View File
@@ -1,151 +0,0 @@
"""
method to create, serialize and deserialize DataResponse message
"""
import base64
import re
from io import BytesIO
from amqp.adapter.serializer import map_as_string, parse_map_string
from amqp.model.model import AMQErrorMessage, DataMessageMagicByte, DataResponse
from amqp.model.snowflake_id import SnowflakeId
class DataResponseFactory:
@staticmethod
def create_async_response_message(_id: SnowflakeId) -> DataResponse:
"""
Helper method to create standard OK response.
:param _id: ID
:return: Response message
"""
return DataResponseFactory.of(
id=_id,
response_code=200,
content_type="application/text",
body="OK".encode("utf-8"),
trace_info={},
error="",
error_cause="",
)
@staticmethod
def of(
id: SnowflakeId,
response_code: int,
content_type: str,
body: bytes,
trace_info: dict[str, str],
error: str | None = None,
error_cause: str | None = None,
) -> DataResponse:
"""
Create the DataResponse message supplying all key values
:param id: SnowflakeID
:param response_code: HTTP response code
:param content_type: MIME content type
:param body: payload
:param trace_info: Jaeger trace info (id)
:return: DataResponseMessage object
"""
return DataResponse(
magic=DataMessageMagicByte.HTTP_REQUEST.value,
id=id,
response_code=str(response_code),
content_type=content_type,
error=error,
error_cause=error_cause,
headers={},
trace_info=trace_info,
base64body=base64.b64encode(body),
)
@staticmethod
def of_error_message(
id: SnowflakeId,
error_message: AMQErrorMessage,
trace_info: dict[str, str],
) -> DataResponse:
return DataResponseFactory.of(
id=id,
response_code=error_message.response_code,
content_type=error_message.content_type,
body=error_message.serialize(),
trace_info=trace_info,
error=error_message.error,
error_cause=error_message.detail,
)
@staticmethod
def serialize(response: DataResponse) -> bytes:
"""
Serialize the message to byte array (to be sent over RabbitMQ)
:param response: Object to serialize
:return: byte array as serialized message
"""
id_bytes = response.id().as_string().encode("utf-8")
prolog = (
f"|r={response.response_code()}|c={response.content_type()}|e={response.error()}|f={response.error_cause()}|t={{{map_as_string(response.trace_info())}}}|b="
).encode("utf-8")
header_length = 1 + len(id_bytes) + len(prolog)
prolog_len = header_length.to_bytes(2, byteorder="big")
msg_length = 2 + header_length + len(response.base64body())
with BytesIO(bytearray(msg_length)) as baos:
baos.write(prolog_len)
baos.write(DataMessageMagicByte.HTTP_REQUEST.value.encode("utf-8"))
baos.write(id_bytes)
baos.write(prolog)
baos.write(response.base64body())
return baos.getvalue()
@staticmethod
def from_bytes(input_bytes: bytes) -> DataResponse:
"""
Buil;ds the DataResponse object from its serialized form
:param input_bytes: byte array - serialized response
:return: DataResponse object
"""
prolog_len = (input_bytes[0] << 8) + input_bytes[1]
metadata = input_bytes[2 : prolog_len + 2].decode()
pattern = re.compile(
f"{DataMessageMagicByte.HTTP_REQUEST.value}"
r"([0-9A-Fa-f]+)\|"
r"r=([^|]*)\|"
r"c=([^|]*)\|"
r"e=([^|]*)\|"
r"f=([^|]*)\|"
r"t=(\{[^}]*\})\|"
r"b="
)
match = pattern.match(metadata)
if not match:
raise ValueError(
f"DataResponse format does not match expected format. "
f"Input string [{metadata}] does not match pattern: {pattern.pattern}"
)
id_str = match.group(1)
response_code = int(match.group(2))
content_type = match.group(3)
error = match.group(4)
error_cause = match.group(5)
trace_info_str = match.group(6)
body_b64 = input_bytes[prolog_len + 2 :]
trace_info = parse_map_string(trace_info_str)
return DataResponseFactory.of(
id_str,
response_code,
content_type,
base64.b64decode(body_b64),
trace_info,
error,
error_cause,
)
# Ensure backward compatibility
AMQResponseFactory = DataResponseFactory
amqp/adapter/file_handler.py
-415
View File
@@ -1,415 +0,0 @@
"""
implements the file downloader for the amqp adapter, where file is sent in chunks over RabbitMQ
"""
import asyncio
import json
import logging
import os
from asyncio import AbstractEventLoop
from collections import defaultdict
from pathlib import Path
from aio_pika import Message, connect_robust
from aio_pika.abc import (
AbstractExchange,
AbstractIncomingMessage,
AbstractQueue,
AbstractRobustChannel,
DeliveryMode,
)
from amqp.adapter.logging_utils import logging_debug, logging_error, logging_info
from amqp.router.utils import await_result
# Global dictionary to track completed downloads
download_locations = defaultdict(str)
IN_PROGRESS = 0
END_OF_FILE = 1
CANCELLED = 2
class FileHandler:
"""
The abstract base class for file handling. This is mainly to make the code testable, as there are only 2
options for pushing (uploading) a file(s) to a service: as HTTP multipart message delivered via RabbitMQ
or via stream in RabbitMQ. Downloading a file is always done via RabbitMQ stream. Thus the method signiture
is tighly coupled to the RabbitMQ API in v1. Only alternative implementation is a mock object in unittests.
"""
def __init__(self):
pass
def ensure_directory_exists(self, filepath) -> str:
"""
Ensures that the directory for the given filepath exists, and handles file versioning.
Args:
filepath: The path to the file.
Returns:
The original filepath if the file does not exist, or a modified filepath
pointing to the next available version of the file if it does.
"""
# 1. Ensure directory exists
directory = os.path.dirname(filepath)
if directory and not os.path.exists(directory):
os.makedirs(directory)
# 2. Handle file versioning
if not os.path.exists(filepath):
return filepath # File doesn't exist, return original path
# File exists, find next available version
base, ext = os.path.splitext(filepath)
version = 1
while True:
new_filepath = f"{base}.{version}{ext}"
if not os.path.exists(new_filepath):
return new_filepath # Found a free version, return this path
version += 1
async def download_buffer(self, loop, rabbitmq_url, queue_name) -> (bytearray, int):
"""
Asynchronously downloads a bytearray from RabbitMQ based on the provided file information.
It assumes entire array is in the first (and only) chunk
Args:
loop: The asyncio event loop.
rabbitmq_url: The RabbitMQ connection URL.
queue_name (str): The directory where the file should be saved.
"""
raise NotImplementedError("Subclasses must implement this method.")
async def download_file(
self, loop, rabbitmq_url, file_info, message_data, output_dir
) -> (int, int):
"""
Asynchronously downloads a file from RabbitMQ based on the provided file information.
takes the file_info and message_data and output_dir, creates queue consumer and waits for the file
to be downloaded. File may consist of multiple chunks, so the function will wait until all chunks are received.
Args:
loop: The asyncio event loop.
rabbitmq_url: The RabbitMQ connection URL.
file_info (dict): A dictionary containing file metadata
message_data (dict): The entire message
output_dir (str): The directory where the file should be saved.
"""
raise NotImplementedError("Subclasses must implement this method.")
async def on_message(
self,
message: AbstractIncomingMessage,
json_data: bytes,
rabbitmq_url: str,
loop,
output_dir="downloaded_files",
) -> defaultdict:
"""
Asynchronous callback function for handling messages from the 'amq-cleverswarm' queue.
Args:
message: The incoming RabbitMQ message.
rabbitmq_url: The RabbitMQ connection URL.
"""
raise NotImplementedError("Subclasses must implement this method.")
async def publish_file(
self,
exchange: AbstractExchange,
file_path: Path,
routing_key: str,
loop: AbstractEventLoop,
max_chunk_size: int = 2**20, # 1MB default chunk size
content_type: str = "application/octet-stream", # Default content type
delivery_mode: DeliveryMode = DeliveryMode.PERSISTENT,
) -> tuple[str, int]:
"""
Reads a file from the file system and publishes its content to a RabbitMQ exchange
in chunks. It declares a queue with a random name, binds it to the specified
exchange, and uses the queue name as the routing key.
Args:
exchange: The aio_pika RobustChannel to use for communication with RabbitMQ.
file_path: The path to the file to publish.
max_chunk_size: The maximum size of each chunk (in bytes).
routing_key: The name of the RabbitMQ exchange routing key to publish to correct queue.
loop: correct event loop
max_chunk_size: chunk size to send
content_type: The content type of the file data.
delivery_mode: The delivery mode (e.g., PERSISTENT or TRANSIENT).
Returns:
The name of the randomly generated queue where the file content was published.
"""
raise NotImplementedError("Subclasses must implement this method.")
# The StreamingFileHandler class is a subclass of FileHandler that implements the download_file method
class StreamingFileHandler(FileHandler):
"""
Handles file downloads from RabbitMQ streams.
"""
def __init__(self):
super().__init__()
# This method MUST NOT be static.
# For testing purpose, we will mock this method and inject a mock connection.
async def _get_connection(self, rabbitmq_url, loop):
return await connect_robust(rabbitmq_url, loop=loop)
async def _download_from_queue_to_file(
self, rabbitmq_url, queue_name, filepath, loop
) -> tuple[int, int] | None:
"""
Internal method for downloading a file from a queue to disk.
Return (file size, eof status).
"""
# Here the normal mock doesn't work since this method
# is async and will be run in another loop,
# out of the scope of method mock.
connection = await self._get_connection(rabbitmq_url, loop=loop)
_file_size = 0
_eof = IN_PROGRESS
try:
channel: AbstractRobustChannel = await connection.channel()
queue: AbstractQueue = await channel.get_queue(queue_name, ensure=False)
logging_debug(f"Awaiting all file chunks being downloaded for {filepath}")
async with queue.iterator() as queue_iter:
async for message in queue_iter:
async with message.process():
with open(filepath, "ab") as f:
f.write(message.body)
_file_size = _file_size + message.body_size
_eof = int(message.headers.get("eof", END_OF_FILE))
if _eof != IN_PROGRESS:
break
except Exception as e:
logging_error(f"File download failed: {e}")
finally:
await connection.close()
return _file_size, _eof
async def download_file(
self, loop, rabbitmq_url, file_info, message_data, output_dir
) -> (int, int):
"""
Asynchronously downloads a file from RabbitMQ based on the provided file information.
takes the file_info and message_data and output_dir, creates queue consumer and waits for the file to be downloaded
Args:
loop: The asyncio event loop.
rabbitmq_url: The RabbitMQ connection URL.
file_info (dict): A dictionary containing file metadata
message_data (dict): The entire message
output_dir (str): The directory where the file should be saved.
"""
filename = file_info["filename"]
size = file_info["size"]
stream_name = file_info["streamName"]
queue_name = file_info["queue_name"]
filepath = os.path.join(output_dir, filename)
filepath = self.ensure_directory_exists(
filepath
) # if filename exists, create a new (next) version
download_locations[file_info["key"]] = filepath
filename = os.path.basename(
filepath
) # if a new version was created, ensure to use that new version
file_info["filename"] = filename
logging_debug(
f"Downloading file: {filename} (size: {size} bytes) from stream: {stream_name}:{queue_name} to {filepath}"
)
if queue_name:
_local_file_size, _local_eof = await await_result(
asyncio.run_coroutine_threadsafe(
self._download_from_queue_to_file(rabbitmq_url, queue_name, filepath, loop),
loop=loop,
)
)
return _local_file_size, _local_eof
return 0, CANCELLED
async def _download_from_queue_to_buffer(
self, rabbitmq_url, queue_name, loop
) -> tuple[bytearray, int]:
"""
Internal method for downloading a file from a queue to RAM.
Return (bytearray, eof status).
"""
_connection = await self._get_connection(rabbitmq_url, loop=loop)
_eof = IN_PROGRESS
_res = bytearray()
try:
channel: AbstractRobustChannel = await _connection.channel()
queue: AbstractQueue = await channel.get_queue(queue_name, ensure=False)
logging_debug(f"Awaiting all buffer chunks being downloaded for {queue_name}")
async with queue.iterator() as queue_iter:
async for message in queue_iter:
async with message.process():
_res.extend(message.body)
_eof = message.headers.get("eof", END_OF_FILE)
if _eof != IN_PROGRESS:
break
logging_info(f"Finished downloading buffer {queue_name}")
except Exception as e:
logging_error(f"Downloading buffer: {e}")
await _connection.close()
return _res, _eof
async def download_buffer(self, loop, rabbitmq_url, queue_name: str | None) -> (bytearray, int):
"""
Asynchronously downloads a bytearay from RabbitMQ based on the provided file information.
It assumes entire array is in the first (and only) chunk
Args:
loop: The asyncio event loop.
rabbitmq_url: The RabbitMQ connection URL.
queue_name (str): The directory where the file should be saved.
"""
logging_debug(f"Downloading buffer from stream: {queue_name}")
if queue_name:
_local_res, _local_eof = await await_result(
asyncio.run_coroutine_threadsafe(
self._download_from_queue_to_buffer(rabbitmq_url, queue_name, loop), loop=loop
)
)
return _local_res, _local_eof
return 0, CANCELLED
async def on_message(
self,
message: AbstractIncomingMessage,
json_data: bytes,
rabbitmq_url: str,
loop,
output_dir="downloaded_files",
) -> defaultdict:
"""
Asynchronous callback function for handling messages from the 'amq-cleverswarm' queue.
Args:
message: The incoming RabbitMQ message.
json_data: The JSON data from the message.
rabbitmq_url: The RabbitMQ connection URL.
loop: The asyncio event loop of the RabbitMQ API
output_dir
"""
try:
_message_data = json.loads(message.body.decode())
_amq_message_data = json.loads(json_data.decode())
files_to_download = []
for _key, _file_type in _amq_message_data["files"].items():
if len(_file_type) > 0:
files_info = _file_type[0]
stream_name = files_info["streamName"]
files_info["queue_name"] = (
_message_data.get(stream_name, "") if stream_name else ""
)
files_info["key"] = _key
files_to_download.append(files_info)
if not files_to_download:
logging_info(" [%s] No files to download in this message.", message.delivery_tag)
# await message.ack()
return defaultdict()
# Create a task for each file download
_tasks = []
for file_info in files_to_download:
logging_info(f" [{message.delivery_tag}] about to create task for {file_info}")
_task = asyncio.create_task(
self.download_file(loop, rabbitmq_url, file_info, _message_data, output_dir)
)
_tasks.append(_task)
logging_info(
f" [{message.delivery_tag}] Waiting for all files to download...gather(*tasks)"
)
await asyncio.gather(*_tasks) # Wait for all downloads to complete
logging_info(f"All files downloaded. d-tag: {message.delivery_tag}")
except Exception as e:
logging_error(f"Building DataMessage: {e}")
asyncio.run_coroutine_threadsafe(
message.ack(), loop=loop
) # ACK -> ignore the message (do not redeliver)
return download_locations
async def publish_file(
self,
exchange: AbstractExchange,
file_path: Path,
routing_key: str,
loop: AbstractEventLoop,
max_chunk_size: int = 2**20, # 1MB default chunk size
content_type: str = "application/octet-stream", # Default content type
delivery_mode: DeliveryMode = DeliveryMode.PERSISTENT,
) -> tuple[str, int]:
"""
Reads a file from the file system and publishes its content to a RabbitMQ exchange
in chunks. It declares a queue with a random name, binds it to the specified
exchange, and uses the queue name as the routing key.
Args:
exchange: The aio_pika RobustChannel to use for communication with RabbitMQ.
file_path: The path to the file to publish.
max_chunk_size: The maximum size of each chunk (in bytes).
routing_key: The name of the RabbitMQ exchange routing key to publish to correct queue.
loop: correct event loop
max_chunk_size: chunk size to send
content_type: The content type of the file data.
delivery_mode: The delivery mode (e.g., PERSISTENT or TRANSIENT).
Returns:
The name of the randomly generated queue where the file content was published.
"""
if not file_path.is_file():
raise FileNotFoundError(f"File not found: {file_path}")
_file_size = os.path.getsize(file_path)
with open(file_path, "rb") as f:
_offset = 0
while True:
_chunk = f.read(max_chunk_size)
if not _chunk:
break # End of file
message = Message(
body=_chunk,
content_type=content_type,
delivery_mode=delivery_mode,
headers={
"file_name": file_path.name,
"pos": _offset,
"total_size": _file_size,
"chunk_size": len(_chunk),
"eof": (
END_OF_FILE if max_chunk_size + _offset >= _file_size else IN_PROGRESS
),
},
)
logging.debug(
f"Going to Publish chunk {_offset}/{_file_size} to {exchange.name}:{routing_key}"
)
_future = asyncio.run_coroutine_threadsafe(
exchange.publish(message, routing_key=routing_key), loop
)
while not _future.done():
# sleep allows coroutine to execute. must wait for completion to ensure all chunks written in right order
await asyncio.sleep(0.02)
logging_debug(f"Waiting for chunk {_offset}/{_file_size} future= {_future}")
_offset += len(_chunk)
logging_debug(
f"Published chunk {_offset}/{_file_size} to {exchange.name}:{routing_key}"
)
logging_debug(
f"File {file_path} published to RabbitMQ exchange {exchange.name}, queue {routing_key}"
)
return routing_key, _file_size
amqp/adapter/http_adapter.py
+68
View File
@@ -0,0 +1,68 @@
import logging
from typing import Dict, List
from aiohttp import ClientSession, ClientResponse
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import AMQMessage
class HttpAdapter:
def __init__(self, amq_configuration: AMQConfiguration, session: ClientSession):
self.amq_configuration = amq_configuration
self.session = session
self.service_host = self.amq_configuration.dispatch.service_host
self.service_port = self.amq_configuration.dispatch.service_port
async def handle(self, message: AMQMessage) -> ClientResponse:
method = message.method.lower()
if method == 'get':
return await self.get(message)
elif method == 'put':
return await self.put(message)
elif method == 'post':
return await self.post(message)
elif method == 'head':
return await self.head(message)
elif method == 'delete':
return await self.delete(message)
else:
raise ValueError(f"Unexpected HTTP method: {message.method}")
async def get(self, message: AMQMessage) -> ClientResponse:
url = f"http://{self.service_host}:{self.service_port}{message.path}"
headers = {**message.headers, **message.trace_info}
logging.info(f" [*] SEND HEADERS out: {', '.join(f'{k}={v}' for k, v in headers.items())}, REQ={url}")
return await self.session.get(url, headers=headers)
async def put(self, message: AMQMessage) -> ClientResponse:
return await self.handle_possible_form(
self.session.put(
f"http://{self.service_host}:{self.service_port}{message.path}",
data=message.body,
headers={**message.headers, **message.trace_info},
)
)
async def post(self, message: AMQMessage) -> ClientResponse:
return await self.handle_possible_form(
self.session.post(
f"http://{self.service_host}:{self.service_port}{message.path}",
data=message.body,
headers={**message.headers, **message.trace_info},
)
)
async def handle_possible_form(self, request_coroutine) -> ClientResponse:
return await request_coroutine
async def head(self, message: AMQMessage) -> ClientResponse:
raise NotImplementedError
async def delete(self, message: AMQMessage) -> ClientResponse:
raise NotImplementedError
def get_content_type(self, message_headers: Dict[str, List[str]]) -> str:
for header, values in message_headers.items():
if header.lower() == 'content-type':
return values[0].split(';')[0]
return 'application/json'
amqp/adapter/logging_utils.py
-128
View File
@@ -1,128 +0,0 @@
import inspect
import logging
import os
import sys
import threading
import traceback
from importlib.metadata import version
# Get version from installed package metadata
__version__ = version("amq_adapter") # Name must match pyproject.toml [project] name
__verbose__ = os.environ.get("AMQ_VERBOSE_LOGGING", "0") == "1"
def get_context_info():
"""
Retrieves the current_availability module, line number, and function name.
Returns:
A tuple containing (module name, line number, function name).
Returns (None, None, None) if it fails to retrieve the information.
"""
try:
# Get the frame object for the current_availability function call
frame = inspect.currentframe()
if frame is not None:
# Get the frame object for the caller of the current_availability function
frame = frame.f_back.f_back
if frame:
code_context = frame.f_code
module_name = code_context.co_filename
module_name = (
"amqp" + module_name.split("/amqp", 1)[-1]
if "/amqp" in module_name
else module_name
)
line_number = frame.f_lineno
function_name = code_context.co_name
return module_name, line_number, function_name
else:
return None, None, None
except Exception:
return None, None, None
def logging_error(msg: str, *args) -> None:
"""
Log an error message according to current_availability logging for 'ERROR' level.
Add module name, line and function name where the error occurred, on single line.
Args:
msg (str): The error message to log.
"""
_exec_info = sys.exc_info()[2]
_thread_id = threading.get_ident()
if _exec_info:
_tb = traceback.extract_tb(sys.exc_info()[2])[-1]
logging.error(
f"{_thread_id}:{__version__} [E] '{msg}' ---> '{_tb.filename}':{_tb.lineno}, '{_tb.name}()'",
*args,
)
else:
module, line_number, function_name = get_context_info()
if module and line_number and function_name:
logging.error(
f"{_thread_id}:{__version__} [E] '{msg}' ---> '{module}':{line_number}, '{function_name}()'",
*args,
)
else:
logging.error(f"{_thread_id}:{__version__} [E] {msg}", *args)
def logging_warning(msg: str, *args) -> None:
"""
Log a warning message according to current_availability logging for 'WARN' level.
Add module name, line and function name where the print occurred, on single line.
Args:
msg (str): The warning message to log.
"""
_thread_id = threading.get_ident()
if not __verbose__:
logging.warning(f"{_thread_id}:{__version__} [W] {msg}", *args)
return
module, line_number, function_name = get_context_info()
if module and line_number and function_name:
logging.warning(
f"{_thread_id}:{__version__} [W] '{msg}' ---> '{module}':{line_number}, '{function_name}()'",
*args,
)
else:
logging.warning(f"{_thread_id}:{__version__} [W] {msg}", *args)
def logging_info(msg: str, *args) -> None:
"""
Log an info message according to current_availability logging for 'INFO' level.
Add module name, line and function name where the print occurred, on single line.
Args:
msg (str): The info message to log.
"""
_thread_id = threading.get_ident()
if not __verbose__:
logging.info(f"{_thread_id}:{__version__} [*] {msg}", *args)
return
module, line_number, function_name = get_context_info()
if module and line_number and function_name:
logging.info(
f"{_thread_id}:{__version__} [*] '{msg}' ---> '{module}':{line_number}, '{function_name}()'",
*args,
)
else:
logging.info(f"{_thread_id}:{__version__} [*] {msg}", *args)
def logging_debug(msg: str, *args) -> None:
"""
Log a debug message according to current_availability logging for 'DEBUG' level.
Add module name, line and function name where the print occurred, on single line.
Args:
msg (str): The debug message to log.
"""
_thread_id = threading.get_ident()
module, line_number, function_name = get_context_info()
if module and line_number and function_name:
logging.debug(
f"{_thread_id}:{__version__} [DBG] '{msg}' ---> '{module}':{line_number}, '{function_name}()'",
*args,
)
else:
logging.debug(f"{_thread_id}:{__version__} [DBG] {msg}", *args)
amqp/adapter/pydantic_serializer.py
-182
View File
@@ -1,182 +0,0 @@
"""
This module provides serialization and deserialization functions for Pydantic models that extend BaseModel
"""
import importlib
import json
import uuid
from datetime import date, datetime
from types import UnionType
from typing import Any, Dict, Union, get_args, get_origin
from urllib.parse import parse_qs, urlparse
from pydantic import BaseModel
from amqp.adapter.logging_utils import logging_debug
def serialize_object(obj: BaseModel) -> str:
"""
Serializes an R2RSerializable object to a string in the format
"<ClassName>:{key1:value1, key2:value2,...}". Handles UUIDs and datetimes,
and nested Pydantic models.
Args:
obj: The BaseModel object to serialize.
Returns:
A string representation of the object.
"""
class_name = obj.__class__.__name__
module_name = obj.__class__.__module__
obj_dict = {}
for key, _ in obj.model_dump().items():
# here we use the original value
# the one returned by model_dump will convert everything into dict recursively
value = getattr(obj, key)
if isinstance(value, uuid.UUID):
obj_dict[key] = str(value)
elif isinstance(value, datetime):
obj_dict[key] = value.isoformat()
elif isinstance(value, dict): # Ensure dictionaries are handled
obj_dict[key] = value
elif isinstance(value, BaseModel):
obj_dict[key] = serialize_object(value) # Recursively serialize
elif isinstance(value, list):
serialized_list = []
for item in value:
if isinstance(item, BaseModel):
serialized_list.append(serialize_object(item))
else:
serialized_list.append(item)
obj_dict[key] = serialized_list
else:
obj_dict[key] = value
dict_string = json.dumps(obj_dict)
return f"{module_name}.{class_name}:{dict_string}"
def python_type_to_json(data):
"""
Convert Python objects (dict, list, tuple, set, datetime, etc.) to a JSON string.
Handles common non-JSON-serializable types like datetime, date, and sets.
Args:
data: Python object to be converted (dict, list, tuple, set, etc.)
Returns:
str: JSON string representation of the input data
Raises:
TypeError: If the object contains non-serializable types (e.g., custom classes)
"""
def default_serializer(obj):
"""Custom serializer for non-JSON-serializable objects."""
if isinstance(obj, (datetime, date)):
return obj.isoformat() # Convert datetime/date to ISO string
elif isinstance(obj, set):
return list(obj) # Convert sets to lists
elif hasattr(obj, "__dict__"):
return obj.__dict__ # Attempt to serialize custom objects
else:
raise TypeError(f"Object of type {type(obj)} is not JSON serializable")
return json.dumps(data, default=default_serializer, indent=0, ensure_ascii=False)
def try_deserialize_object(data: str) -> BaseModel | str:
"""
Try deserialize object, if failed, return the input string as-is.
"""
if not isinstance(data, str):
return data
try:
return deserialize_object(data)
except Exception as e:
logging_debug(f"Failed to deserialize object: {e}")
return data
def deserialize_object(data: str) -> BaseModel:
"""
Deserializes a string in the format "<ClassName>:{key1:value1, key2:value2,...}"
back into an BaseModel object. Handles UUIDs, datetimes, and
nested Pydantic models.
Args:
data: The string to deserialize.
Returns:
An R2RSerializable object.
"""
class_path, dict_str = data.split(":", 1)
module_path, class_name = class_path.rsplit(".", 1)
obj_dict: Dict[str, Any] = json.loads(dict_str)
logging_debug(f"Deserialized object: {class_path}")
if "[" in class_name:
class_name = class_name.split("[")[0]
logging_debug(f"trying object: {class_name}")
module = importlib.import_module(module_path)
cls = getattr(module, class_name)
if cls is None:
raise ValueError(f"Unknown class: {class_name}")
# process nested model, dict and list
for key, value in obj_dict.items():
key_type: type[Any] = cls.model_fields[key].annotation
# handle union with none, this will give the true type
# for example, return `A` if the type is `A | None`
if get_origin(key_type) is UnionType or get_origin(key_type) is Union:
union_types = get_args(key_type)
if len(union_types) > 2:
# if there are multiple unions, like `A | B | None`
# then we have no way to figure out which one to use
raise ValueError(f"Unsupported Union type: {key_type}")
# return the none-None one as the type for processing
if union_types[0] is type(None):
key_type = union_types[1]
else:
key_type = union_types[0]
# skip if the value is None
if value is None:
continue
if key_type == uuid.UUID:
obj_dict[key] = uuid.UUID(value)
elif key_type == datetime:
obj_dict[key] = datetime.fromisoformat(value)
elif key_type == dict:
obj_dict[key] = value
elif issubclass(key_type, BaseModel):
obj_dict[key] = deserialize_object(value)
elif key_type == list:
obj_dict[key] = list(map(try_deserialize_object, value))
return cls(**obj_dict)
def parse_url_query(url):
"""
Parses a HTTP GET URL and returns (context_path, query_params_dict)
Args:
url: The URL string to parse (e.g., "http://example.com/path?foo=1&bar=2&foo=3")
Returns:
tuple: (context_path, query_params_dict)
- context_path: The path part of the URL (e.g., "/path")
- query_params_dict: Dictionary of query parameters where values are always lists
(e.g., {"foo": ["1", "3"], "bar": ["2"]})
"""
parsed = urlparse(url)
query_params = parse_qs(parsed.query)
# Convert values from lists to single values when there's only one value
# If you prefer to always keep values as lists, remove this comprehension
simplified_params = {k: v[0] if len(v) == 1 else v for k, v in query_params.items()}
return (parsed.path, simplified_params)
amqp/adapter/response_holder.py
+15
View File
@@ -0,0 +1,15 @@
from dataclasses import dataclass
from concurrent.futures import Future
from aiohttp.web_response import Response
from opentelemetry.trace import Span
@dataclass
class ResponseHolder:
response: Future[Response]
tracing_span: Span
def __init__(self, response: Future[Response], span: Span):
self.response = response
self.tracing_span = span
amqp/adapter/serializer.py
-76
View File
@@ -1,76 +0,0 @@
"""
Helper functions used to aid with (de)serialization of messages sent over AMQP.
"""
import struct
from io import BytesIO
from typing import Dict, List, Union
def long_to_bytes(x: int) -> bytes:
return struct.pack("q", x)
def bytes_to_long(b: bytes) -> int:
return struct.unpack("q", b)[0]
def read_long(bis: BytesIO) -> int:
try:
return struct.unpack("q", bis.read(8))[0]
except struct.error:
return 0
def object_or_list_as_string(value: Union[str, List[str]]) -> str:
if isinstance(value, list):
return f"[{','.join(value)}]"
return str(value)
def map_as_string(map_: Dict[str, str]) -> str:
return ",".join(f"{k}={object_or_list_as_string(v)}" for k, v in map_.items())
def map_with_list_as_string(map_: Dict[str, List[str]]) -> str:
return ",".join(f"{k}={object_or_list_as_string(v)}" for k, v in map_.items())
def parse_map_string(input_str: str) -> Dict[str, str]:
map_ = {}
if input_str.startswith("{") and input_str.endswith("}"):
for entry in str(input_str[1:-1]).replace("\n", "").split(","):
if len(entry) > 1:
if entry.startswith('"'):
key, value = entry.split(":")
map_[key.strip('"')] = value.strip('"')
else:
key, value = entry.split("=")
map_[key] = value
return map_
def csv_as_list(input_str: str) -> List[str]:
return [item.strip() for item in input_str.strip("[]").split(",")]
def add_to_map(map_: Dict[str, List[str]], token: str) -> None:
try:
eq_idx = token.index("=")
except ValueError:
eq_idx = -1 # set to -1 when not found
if eq_idx > 0:
key = token[:eq_idx]
map_[key] = csv_as_list(token[eq_idx + 1 :])
def parse_map_list_string(input_str: str) -> Dict[str, List[str]]:
map_ = {}
if input_str.startswith("{") and input_str.endswith("}"):
for token in input_str[1:-1].split("],"):
add_to_map(map_, token)
return map_
def str_to_bool(value: str) -> bool:
return value.lower() in ("true", "yes", "1", "t", "y", "on")
amqp/adapter/service_message_factory.py
+29 -69
View File
@@ -1,91 +1,59 @@
import base64
import logging
from amqp.adapter.data_message_factory import DataMessageFactory
from amqp.adapter.logging_utils import logging_error, logging_info
from amqp.adapter.serializer import map_as_string, parse_map_string
from amqp.adapter.amq_message_factory import AMQMessageFactory
from amqp.adapter.trace_info_adapter import TraceInfoAdapter
from amqp.model.model import ServiceMessage
from amqp.model.service_message_type import ServiceMessageType
from amqp.model.model import CleverMicroServiceMessage, ServiceMessageType
from amqp.model.snowflake_id import SnowflakeId
from amqp.router.serializer import parse_map_string, map_as_string
RS = "|"
SPLIT_RS = r"\|"
INVALID_MESSAGE = ServiceMessage() # default values mean invalid message
INVALID_MESSAGE = CleverMicroServiceMessage() # default values mean invalid message
class ServiceMessageFactory:
"""
build/serialize/deserialize CleverMicro Service Message class
"""
class CleverMicroServiceMessageFactory:
def __init__(self, name: str):
self.process_name = name
def to_string(self, message: ServiceMessage) -> str:
"""
Convert ServiceMessage to human-readable format
:param message: ServiceMessage object
:return: as string
"""
def to_string(self, message: CleverMicroServiceMessage) -> str:
if message is not None:
return (
f"ServiceMessage{{id={message.id.as_string()}|type={self.format_message_type(message)}|"
f"CleverMicroServiceMessage{{id={message.id.as_string()}|type={self.format_message_type(message)}|"
f"recipient={message.recipient_name}|replyTo={message.reply_to}|"
f"traceInfo={message.trace_info}|body={base64.b64encode(message.message.encode()).decode() if message.payload_type & 0x80 else message.message}}}"
)
return "null"
def of(
self, message_type: ServiceMessageType, payload: str, recipient_name: str
) -> ServiceMessage:
"""
Builds ServiceMessage object from relevant values.
:param message_type: enum Message type
:param payload: Message content, stored as-is
:param recipient_name: Recipient name (from the routing expression)
:return: ServiceMessage object
"""
_id = DataMessageFactory.get_instance(1).generate_snowflake_id()
return ServiceMessage(
def of(self, message_type: ServiceMessageType, payload: str, recipient_name: str) -> CleverMicroServiceMessage:
_id = AMQMessageFactory.get_instance(1).generate_snowflake_id()
return CleverMicroServiceMessage(
id=_id,
payload_type=0,
message_type=message_type,
recipient_name=recipient_name,
reply_to=self.process_name,
trace_info=TraceInfoAdapter.create_produce_span(_id),
message=payload,
message=payload
)
def as_base64(
self, message_type: ServiceMessageType, payload: str, recipient_name: str
) -> ServiceMessage:
"""
Builds ServiceMessage object from relevant values, stores content Base64 encoded.
:param message_type: enum Message type
:param payload: Message content, stored as Base64 string
:param recipient_name: Recipient name (from the routing expression)
:return: ServiceMessage object
"""
_id = DataMessageFactory.get_instance(1).generate_snowflake_id()
return ServiceMessage(
def as_base64(self, message_type: ServiceMessageType, payload: str, recipient_name: str) -> CleverMicroServiceMessage:
_id = AMQMessageFactory.get_instance(1).generate_snowflake_id()
return CleverMicroServiceMessage(
id=_id,
payload_type=0x80,
message_type=message_type,
recipient_name=recipient_name,
reply_to=self.process_name,
trace_info=TraceInfoAdapter.create_produce_span(_id.as_string()),
message=payload,
message=payload
)
def from_bytes(self, input_bytes: bytes) -> ServiceMessage:
"""
Deserialize ServiceMessage from byte array serialized form.
:param input_bytes: serialized ServiceMessage
:return: ServiceMessage object represented by the input.
"""
logging_info(f"ServiceMessage >>> [{input_bytes.decode()}]")
def from_bytes(self, input_bytes: bytes) -> CleverMicroServiceMessage:
print(f"CleverMicroServiceMessage >>> [{input_bytes.decode()}]")
input_str_array = input_bytes.decode().split(RS)
try:
i = 0
@@ -101,47 +69,39 @@ class ServiceMessageFactory:
trace_info = parse_map_string(input_str_array[i]) if i < len(input_str_array) else {}
i += 1
message = input_str_array[i] if i < len(input_str_array) else ""
return ServiceMessage(
return CleverMicroServiceMessage(
id=SnowflakeId.from_hex(_id),
payload_type=payload_type,
message_type=message_type,
recipient_name=recipient_name,
reply_to=reply_to,
trace_info=trace_info,
message=(base64.b64decode(message).decode() if payload_type & 0x80 else message),
message=base64.b64decode(message).decode() if payload_type & 0x80 else message
)
except Exception as e:
logging_error(
logging.error(
"Unexpected ServiceMessage format. Expected 6 tokens separated by '|' in format: "
"[ID(64bit number)|MessageType(8bit number)|RecipientName(String)|"
"ReplyTo(String)|TraceInfo|Payload(String or Base64)], got: "
"[%s], reported problem: %s",
input_bytes.decode(),
str(e),
"[{}], reported problem: {}", input_bytes.decode(), str(e)
)
return INVALID_MESSAGE
def serialize(self, message: ServiceMessage) -> bytes:
"""
Serialize ServiceMessage to byte array.
:param message: ServiceMessage to serialize.
:return: serialized object.
"""
def serialize(self, message: CleverMicroServiceMessage) -> bytes:
return (
f"{message.id.as_string()}{RS}{self.format_message_type(message)}{RS}{message.recipient_name}{RS}"
f"{message.reply_to}{RS}{map_as_string(message.trace_info)}{RS}"
f"{base64.b64encode(message.message.encode()).decode() if message.payload_type != 0 else message.message}"
).encode("utf-8")
).encode('utf-8')
def format_message_type(self, message: ServiceMessage) -> str:
def format_message_type(self, message: CleverMicroServiceMessage) -> str:
"""
Formats the Enum message type and decorates it with base64 flag at highest bit.
Formats the Enum message type and ecorates it with base64 flag at highest bit.
For compatibility with 0-based enum ordinals in Java subtract 1 from ordinal value.
:param message: Service message to send
:return: formatted message type
"""
num_type = (
message.message_type.value if message.message_type else ServiceMessageType.UNKNOWN.value
) - 1
fmt = (num_type & 0x7F) | (message.payload_type & 0x80)
num_type = (message.message_type.value if message.message_type else ServiceMessageType.UNKNOWN.value) - 1
fmt = ((num_type & 0x7F) | (message.payload_type & 0x80))
return f"{fmt:02X}"
amqp/adapter/test_amq_message_factory.py
+68
View File
@@ -0,0 +1,68 @@
from datetime import datetime, timezone
from unittest import TestCase
from amqp.adapter.amq_message_factory import AMQMessageFactory
class AMQMessageFactoryTest(TestCase):
def test_from_bytes(self):
raw = "\002\027A64f3b5af4d00000a4000|m=POST|d=clever-pybook.localhost|p=/debug|h={Accept=[*/*],X-Forwarded-Host=[clever-pybook.localhost:8085],User-Agent=[curl/8.7.1],X-Forwarded-Proto=[http],X-Forwarded-For=[172.18.0.1],Host=[clever-pybook.localhost:8085],Accept-Encoding=[gzip],Content-Length=[1475],X-Forwarded-Port=[8085],X-Real-Ip=[172.18.0.1],X-Forwarded-Server=[c5945bef67d1],Content-Type=[multipart/form-data; boundary=------------------------j1RjIqLrFH07XMsyMvpS1v]}|t={traceparent=00-bed6eb7de0d8ccf4c80b145a4bfec357-8d1f3bafc471bf0f-01}|b=TEST_ME"
msg = AMQMessageFactory.from_bytes(raw.encode("utf-8"))
self.assertIsNotNone(msg.id)
self.assertEqual(msg.id.as_string(),'64F3B5AF4D00000A4000','SnowflakeID incorrectly parsed.')
self.assertEqual(msg.headers["Content-Type"][0], "multipart/form-data; boundary=------------------------j1RjIqLrFH07XMsyMvpS1v")
def test_generate_snowflake_id(self):
_id = AMQMessageFactory.get_instance(1).generate_snowflake_id()
self.assertEqual('00001000', _id.as_string()[-8:], 'Instance ID not set correctly')
_id = AMQMessageFactory(2).generate_snowflake_id()
self.assertEqual('00002000', _id.as_string()[-8:], 'Instance ID not set correctly')
_f = AMQMessageFactory(127)
_i0 = _f.generate_snowflake_id()
_i1 = _f.generate_snowflake_id()
_i2 = _f.generate_snowflake_id()
_i3 = _f.generate_snowflake_id()
self.assertEqual('0007F000', _i0.as_string()[-8:], 'Instance ID and sequence not set correctly')
self.assertEqual('0007F001', _i1.as_string()[-8:], 'Instance ID and sequence not set correctly')
self.assertEqual('0007F002', _i2.as_string()[-8:], 'Instance ID and sequence not set correctly')
self.assertEqual('0007F003', _i3.as_string()[-8:], 'Instance ID and sequence not set correctly')
def test_create_request_message(self):
_f = AMQMessageFactory(34)
_req = _f.create_request_message("PUT", "localhost", "/test/me", {"k1": ["v1"], "k2": ["v2"]}, "Quick brown fox")
self.assertEqual("localhost", _req.domain, 'Domain not set correctly')
self.assertEqual("/test/me", _req.path, 'Context path not set correctly')
self.assertEqual("v1", _req.headers['k1'][0], "Headers not set correctly")
self.assertEqual(b'Quick brown fox', _req.body(), "Body not correctly decoded")
def test_serialize(self):
_f = AMQMessageFactory(34)
_req = _f.create_request_message("PUT", "localhost", "/test/me", {"k1": ["v1"], "k2": ["v2"]}, "Quick brown fox")
_ser = _f.serialize(_req)
self.assertEqual(65, _ser[2])
_deser = _f.from_bytes(_ser)
self.assertEqual(_req.id.as_string(), _deser.id.as_string())
self.assertEqual("localhost", _deser.domain)
self.assertEqual("/test/me", _deser.path)
self.assertEqual(b"Quick brown fox", _deser.body())
self.assertEqual("v2", _deser.headers["k2"][0])
def test_amq_message_routing_key(self):
_f = AMQMessageFactory(35)
_req = _f.create_request_message("PUT", "localhost", "/test/me", {"k1": ["v1"], "k2": ["v2"]}, "Quick brown fox")
_key = _f.amq_message_routing_key(_req)
self.assertEqual("localhost..test.me", _key, "Message routing key wrongly extracted from the message")
def test_get_timestamp_from_id(self):
_f = AMQMessageFactory(35)
_timestamp = _f.get_timestamp_from_id(_f.generate_snowflake_id())
_x_timestamp = int(datetime.now(timezone.utc).timestamp() * 1000)
self.assertLessEqual(_x_timestamp - _timestamp, 1, "Timestamp in SnowflakeId differs too much from current time")
def test_to_string(self):
_f = AMQMessageFactory(35)
_req = _f.create_request_message("PUT", "localhost", "/test/me", {"k1": ["v1"], "k2": ["v2"]}, "Quick brown fox")
_as_str = _f.to_string(_req)
self.assertGreater(_as_str.index("domain='localhost'"), 0, "Expected content not found")
amqp/adapter/test_amq_response_factory.py
+36
View File
@@ -0,0 +1,36 @@
from unittest import TestCase
from amqp.adapter.amq_message_factory import AMQMessageFactory
from amqp.adapter.amq_response_factory import AMQResponseFactory
from amqp.model.snowflake_id import SnowflakeId
class TestAMQResponseFactory(TestCase):
def test_create_async_response_message(self):
_f = AMQResponseFactory()
_g = AMQMessageFactory(111)
_resp = _f.create_async_response_message(_g.generate_snowflake_id())
self.assertEqual(200, _resp.response_code)
self.assertEqual(b"OK", _resp.response)
def test_serialize(self):
_f = AMQResponseFactory()
_g = AMQMessageFactory(111)
_snowflakeId: SnowflakeId = _g.generate_snowflake_id()
_resp = _f.create_async_response_message(_snowflakeId)
_ser = _f.serialize(_resp)
self.assertEqual(2, str(_ser[2:]).index(_f.MAGIC_BYTE_HTTP_REQUEST+_snowflakeId.as_string()), "Magic byte and SnowflakeId not found")
self.assertGreater(str(_ser[2:]).index("|e=|"), 2)
self.assertEqual((_ser[0]+_ser[1]), len(_ser) - 2 - len('T0s='))
def test_from_bytes(self):
_f = AMQResponseFactory()
_g = AMQMessageFactory(111)
_snowflakeId: SnowflakeId = _g.generate_snowflake_id()
_resp = _f.create_async_response_message(_snowflakeId)
_ser = _f.serialize(_resp)
_deser = _f.from_bytes(_ser)
self.assertEqual(_snowflakeId.as_string(), _deser.id)
self.assertEqual(200, _deser.response_code)
self.assertEqual(b'OK', _deser.response)
amqp/adapter/test_amq_route_factory.py
+20
View File
@@ -0,0 +1,20 @@
from unittest import TestCase
from amqp.adapter.amq_route_factory import AMQRouteFactory, NULL_ROUTE
from amqp.model.model import AMQRoute
class TestAMQRouteFactory(TestCase):
def test_from_string(self):
_f = AMQRouteFactory()
_r: AMQRoute = _f.from_string("amq-clever-pybook::amq-clever-pybook:clever.pybook.localhost.#:5:0")
self.assertEqual("", _r.exchange)
self.assertEqual("amq-clever-pybook", _r.queue)
self.assertEqual(5, _r.timeout)
_r2: AMQRoute = _f.from_string("amq-clever-pybook:amq-clever-pybook:clever.pybook.localhost.#:5")
self.assertEqual(NULL_ROUTE, _r2)
def test_validate(self):
_f = AMQRouteFactory()
self.assertTrue(_f.validate("amq-clever-pybook::amq-clever-pybook:clever.pybook.localhost.#:5:0"))
self.assertFalse(_f.validate("amq-clever-pybook:amq-clever-pybook:clever.pybook.localhost.#"))
tests/adapter/test_service_message_factory.py → amqp/adapter/test_service_message_factory.py
+15 -25
View File
@@ -1,32 +1,25 @@
from unittest import TestCase
from amqp.adapter.service_message_factory import (
INVALID_MESSAGE,
RS,
ServiceMessageFactory,
)
from amqp.model.model import ServiceMessage
from amqp.model.service_message_type import ServiceMessageType
from amqp.adapter.service_message_factory import CleverMicroServiceMessageFactory, RS
from amqp.model.model import ServiceMessageType, CleverMicroServiceMessage
class ServiceMessageTest(TestCase):
class CleverMicroServiceMessageTest(TestCase):
def test_to_string(self):
_f: ServiceMessageFactory = ServiceMessageFactory("bumble-bee")
message: ServiceMessage = _f.of(ServiceMessageType.ROUTING_DATA, "Duck", "Donald")
_f: CleverMicroServiceMessageFactory = CleverMicroServiceMessageFactory('bumble-bee')
message: CleverMicroServiceMessage = _f.of(ServiceMessageType.ROUTING_DATA, "Duck", "Donald")
message_str = _f.to_string(message)
self.assertTrue("|type=01|" in message_str)
self.assertTrue("|body=Duck" in message_str)
message: ServiceMessage = _f.of(ServiceMessageType.ROUTING_DATA_REPLACE, "Duck", "Donald")
assert "|type=01|" in message_str
assert "|body=Duck" in message_str
message: CleverMicroServiceMessage = _f.of(ServiceMessageType.ROUTING_DATA_REPLACE, "Duck", "Donald")
message_str = _f.to_string(message)
self.assertTrue("|type=03|" in message_str)
self.assertTrue("|body=Duck" in message_str)
self.assertTrue("|replyTo=bumble-bee" in message_str)
nullMsg = _f.to_string(None)
self.assertEqual("null", nullMsg)
assert "|type=03|" in message_str
assert "|body=Duck" in message_str
assert "|replyTo=bumble-bee" in message_str
def test_from(self):
_f: ServiceMessageFactory = ServiceMessageFactory("amqp-router")
_f: CleverMicroServiceMessageFactory = CleverMicroServiceMessageFactory("amqp-router")
message = _f.of(ServiceMessageType.ROUTING_DATA, "Duck", "Donald")
serialized = _f.serialize(message)
deserialized = _f.from_bytes(serialized)
@@ -35,11 +28,9 @@ class ServiceMessageTest(TestCase):
assert deserialized.recipient_name == "Donald"
assert deserialized.message == "Duck"
assert deserialized.reply_to == "amqp-router"
invalidMsg = _f.from_bytes(b"wrong")
self.assertEqual(invalidMsg, INVALID_MESSAGE)
def test_base64(self):
_f: ServiceMessageFactory = ServiceMessageFactory("amqp-router")
_f: CleverMicroServiceMessageFactory = CleverMicroServiceMessageFactory("amqp-router")
message = _f.as_base64(ServiceMessageType.ROUTING_DATA, "Duck", "Donald")
serialized = _f.serialize(message)
assert b"|81|" in serialized
@@ -47,10 +38,9 @@ class ServiceMessageTest(TestCase):
assert deserialized.is_valid()
assert deserialized.message_type == ServiceMessageType.ROUTING_DATA
assert deserialized.message == "Duck"
def test_serialize(self):
_f: ServiceMessageFactory = ServiceMessageFactory("amqp-router")
_f: CleverMicroServiceMessageFactory = CleverMicroServiceMessageFactory("amqp-router")
message = _f.of(ServiceMessageType.ROUTING_DATA, "Duck", "Donald")
serialized = _f.serialize(message)
assert RS + "0" + str(ServiceMessageType.ROUTING_DATA.value)
assert len(serialized) > 16
amqp/adapter/trace_info_adapter.py
+4 -3
View File
@@ -1,13 +1,14 @@
from dataclasses import dataclass
from typing import Dict
@dataclass
class TraceInfoAdapter:
"""
TODO This class should set up the Open Telemetry tracing stack. To be done later.
This class should set up the Open Telemetry tracing stack. To be done later.
"""
@staticmethod
def create_produce_span(message_id: str) -> Dict[str, str]:
return {"trace-id": message_id}
return {
"trace-id": message_id
}
amqp/adapter/type_descriptor.py
-205
View File
@@ -1,205 +0,0 @@
""" """
import importlib
import json
from typing import Any
class TypeDescriptor:
def __init__(self, type_string: str, extra_init_data: dict):
self.type_string = type_string
self.extra_init_data = extra_init_data or {}
self.is_json: bool = False
self.type: str = self._extract_type()
def _extract_type(self) -> str:
if self.type_string.startswith("<class '"):
return self.type_string[8:-2]
# Handle Annotated case of FastAPI annotation
if "typing.Optional" in self.type_string:
annotated = _extract_type_of(self.type_string)
if "typing.Annotated" in annotated:
# Extract content between the first square brackets after Annotated
match = _extract_type_of(annotated)
if match:
# Split by comma and get first part, then strip whitespace
parts = [p.strip() for p in match.split(",")]
if len(parts) > 1:
self.is_json = parts[1].lower() == "json"
return parts[0] if parts else ""
else:
# Handle simple Optional case
return annotated
return _extract_type_of(self.type_string)
def instantiate(self, payload: Any, is_json: bool = False) -> Any:
return _instantiate(
(
self.type
if "typing.Optional" in self.type_string or self.type_string.startswith("<class ")
else self.type_string
),
payload,
self.is_json or is_json,
extra_init_data=self.extra_init_data,
)
def __repr__(self) -> str:
return f"TypeDescriptor(type='{self.type}', is_json={self.is_json})"
def _extract_type_of(input_string: str) -> str:
stack = []
start_index = -1
end_index = -1
for i, char in enumerate(input_string):
if i > 0: # this assumes paramerized type, that must be at lest 1 char long
if char == "[":
if start_index == -1:
start_index = i
stack.append(char)
elif char == "]":
if len(stack) == 1:
end_index = i
break
if len(stack) > 0:
stack.pop()
if start_index != -1 and end_index != -1:
return input_string[start_index + 1 : end_index].strip()
else:
return input_string.strip() if start_index == 1 and end_index == -1 else ""
def merge_extra_init_data(cls, json_payload, extra_init_data: dict):
"""
Merges extra_init_data into json_payload only when:
1. cls has an attribute named like the key
2. json_payload doesn't already have the key
"""
return {
**json_payload,
**{
k: v
for k, v in extra_init_data.items()
if (hasattr(cls, k) or (hasattr(cls, "model_fields") and k in cls.model_fields))
and k not in json_payload
},
}
def _instantiate(
type_string: str, payload: Any, is_json: bool = False, extra_init_data: dict = None
) -> Any:
"""
Dynamically imports and instantiates a class from a TypeDescriptor's type_string attribute.
Args:
type_string: TypeDescriptor instance with the class path
payload: Argument to pass to the constructor
Returns:
Instance of the target class initialized with payload
Raises:
ImportError: If the module/class cannot be imported
TypeError: If instantiation fails
"""
# Handle JSON payload first
json_payload = json.loads(payload) if is_json and isinstance(payload, str) else None
# Handle basic types
if type_string == "str":
return str(payload)
if type_string == "int":
return int(payload)
if type_string == "float":
return float(payload)
if type_string == "bool":
return bool(payload)
# Handle typing module types
if type_string.startswith("typing.") or type_string.startswith("list"):
if (
type_string.startswith("typing.List")
or type_string.startswith("typing.Sequence")
or type_string.startswith("list")
):
if json_payload is not None:
return json_payload
inner_type = _extract_type_of(type_string)
if inner_type:
return [
_instantiate(inner_type, p, extra_init_data=extra_init_data)
for p in (payload if isinstance(payload, list) else payload.split(","))
]
return payload if isinstance(payload, list) else payload.split(",")
if type_string.startswith("typing.Dict") or type_string.startswith("typing.Mapping"):
if json_payload is not None:
return json_payload
return dict(payload)
if type_string.startswith("typing.Set"):
if json_payload is not None:
return set(json_payload)
_set_type = _extract_type_of(type_string)
return set(
payload
if isinstance(payload, list)
else [_instantiate(_set_type, p) for p in payload.split(",")]
)
if type_string.startswith("typing.Tuple"):
if json_payload is not None:
return tuple(json_payload)
_tuple_type = _extract_type_of(type_string)
return tuple(
payload
if isinstance(payload, list)
else [_instantiate(_tuple_type, p) for p in payload.split(",")]
)
if type_string.startswith("typing.Optional"):
inner_type = _extract_type_of(type_string)
if inner_type:
return _instantiate(inner_type, payload, is_json, extra_init_data)
return payload
# Handle regular class instantiation
parts = type_string.split(".")
module_path = ".".join(parts[:-1])
class_name = parts[-1]
try:
# Import the module
module = importlib.import_module(module_path)
# Get the class
cls = getattr(module, class_name)
if isinstance(payload, cls):
return payload
# Instantiate with payload
if json_payload is not None:
_args = merge_extra_init_data(cls, json_payload, extra_init_data or {})
if hasattr(cls, "from_dict"):
return cls.from_dict(_args)
elif hasattr(cls, "create"):
return cls.create(**_args)
return cls(**_args)
if isinstance(payload, dict):
_args = merge_extra_init_data(cls, payload, extra_init_data or {})
if hasattr(cls, "from_dict"):
return cls.from_dict(_args)
elif hasattr(cls, "create"):
return cls.create(**_args)
return cls(**_args)
return cls(payload)
except ImportError as e:
raise ImportError(f"Could not import module '{module_path}'") from e
except AttributeError as e:
raise ImportError(f"Module '{module_path}' has no class '{class_name}'") from e
except TypeError as e:
raise TypeError(f"Failed to instantiate {type_string} with payload {payload}") from e
amqp/config/amq_configuration.py
+24 -189
View File
@@ -1,20 +1,10 @@
import configparser
import inspect
import os
from amqp.adapter.logging_utils import logging_error, logging_warning
"""
Convert configuration from properties file to an object.
"""
class AMQAdapter:
"""
configuration with prefix 'cm.amq-adapter'
"""
PREFIX: str = "cm.amq-adapter."
PREFIX: str = 'cm.amq-adapter.'
def __init__(self, config: configparser.ConfigParser):
"""
@@ -24,106 +14,23 @@ class AMQAdapter:
:param config: config parser to read configuration values
"""
self.generator_id = 0
self.service_name = config.get(
"CleverMicro-AMQ",
self.PREFIX + "service-name",
fallback="amq-python-adapter-" + str(self.generator_id),
)
self.route_mapping = config.get(
"CleverMicro-AMQ", self.PREFIX + "route-mapping", fallback=None
)
self.require_authenticated_user = config.getboolean(
"CleverMicro-AMQ",
self.PREFIX + "require-authenticated-user",
fallback=False,
)
self.backpressure_enabled = config.getboolean(
"CleverMicro-AMQ",
self.PREFIX + "default-backpressure-enabled",
fallback=False,
)
self.callbacks_enabled = config.getboolean(
"CleverMicro-AMQ",
self.PREFIX + "callbacks-enabled",
fallback=False,
)
# these 2 shall be provided by Docker swarm, format is a random alphanumeric string
self.swarm_service_id = os.getenv("SWARM_SERVICE_ID", default="clever-amqp-service")
self.swarm_task_id = os.getenv("SWARM_TASK_ID", default="python-adapter")
#
# Consul related configuration
self.consul_port = (
int(
os.getenv(
"CONSUL_PORT",
config.get("CleverMicro-AMQ", self.PREFIX + "consul-port", fallback="8500"),
)
)
if os.getenv("CONSUL_PORT") is not None
else config.getint("CleverMicro-AMQ", self.PREFIX + "consul-port", fallback=8500)
)
self.consul_host = os.getenv(
"CONSUL_HOST",
config.get("CleverMicro-AMQ", self.PREFIX + "consul-host", fallback="consul"),
)
self.consul_max_retries = (
int(
os.getenv(
"CONSUL_MAX_RETRIES",
config.get("CleverMicro-AMQ", self.PREFIX + "consul-max-retries", fallback="5"),
)
)
if os.getenv("CONSUL_MAX_RETRIES") is not None
else config.getint("CleverMicro-AMQ", self.PREFIX + "consul-max-retries", fallback=5)
)
self.consul_initial_retry_delay = (
float(
os.getenv(
"CONSUL_INITIAL_RETRY_DELAY",
config.get(
"CleverMicro-AMQ",
self.PREFIX + "consul-initial-retry-delay",
fallback="0.2",
),
)
)
if os.getenv("CONSUL_INITIAL_RETRY_DELAY") is not None
else config.getfloat(
"CleverMicro-AMQ", self.PREFIX + "consul-initial-retry-delay", fallback=0.2
)
)
self.consul_counter_key = (
os.getenv(
"CONSUL_COUNTER_KEY",
config.get(
"CleverMicro-AMQ",
self.PREFIX + "consul-counter-key",
fallback="service/ids/counter",
),
)
if os.getenv("CONSUL_COUNTER_KEY") is not None
else config.get(
"CleverMicro-AMQ",
self.PREFIX + "consul-counter-key",
fallback="service/ids/counter",
)
)
self.generator_id = config.getint('CleverMicro-AMQ', self.PREFIX + 'generator-id', fallback=164)
self.service_name = config.get('CleverMicro-AMQ', self.PREFIX + 'service-name', fallback='amq-python-adapter-'+self.generator_id)
self.route_mapping = config.get('CleverMicro-AMQ', self.PREFIX + 'route-mapping', fallback=None)
def adapter_prefix(self) -> str:
return f"{self.service_name}-{self.generator_id}-"
class Dispatch:
"""
configuration with prefix 'cm.dispatch'
"""
PREFIX: str = "cm.dispatch."
class Dispatch:
PREFIX: str = 'cm.dispatch.'
def __init__(self, config: configparser.ConfigParser):
"""
cm.dispatch.use-dlq=true
cm.dispatch.use-confirms=false
cm.dispatch.amq-host=rabbitmq
cm.dispatch.amq-port=5672
cm.dispatch.amq-port-tls=5671
@@ -132,115 +39,43 @@ class Dispatch:
:param config: config parser to read configuration values
"""
self.use_dlq = config.getboolean("CleverMicro-AMQ", self.PREFIX + "use-dlq", fallback=False)
self.amq_host = config.get("CleverMicro-AMQ", self.PREFIX + "amq-host", fallback="rabbitmq")
self.amq_port = config.getint("CleverMicro-AMQ", self.PREFIX + "amq-port", fallback=5672)
self.amq_port_tls = config.getint(
"CleverMicro-AMQ", self.PREFIX + "amq-port-tls", fallback=5671
)
self.service_host = config.get(
"CleverMicro-AMQ", self.PREFIX + "service-host", fallback="localhost"
)
self.service_port = config.getint(
"CleverMicro-AMQ", self.PREFIX + "service-port", fallback=8080
)
self.rabbit_mq_user = os.getenv("RABBIT_MQ_USER", "springuser")
self.rabbit_mq_password = os.getenv("RABBIT_MQ_PASSWORD", "TheCleverWho")
self.rabbit_mq_url = (
f"amqp://{self.rabbit_mq_user}:{self.rabbit_mq_password}@{self.amq_host}/"
)
self.download_dir = config.get(
"CleverMicro-AMQ", self.PREFIX + "download-dir", fallback="downloaded_files"
)
self.use_dlq = config.getboolean('CleverMicro-AMQ', self.PREFIX + 'use-dlq', fallback=False)
self.use_confirms = config.getboolean('CleverMicro-AMQ', self.PREFIX + 'use-confirms', fallback=False)
self.amq_host = config.get('CleverMicro-AMQ', self.PREFIX + 'amq-host', fallback='rabbitmq')
self.amq_port = config.getint('CleverMicro-AMQ', self.PREFIX + 'amq-port', fallback=5672)
self.amq_port_tls = config.getint('CleverMicro-AMQ', self.PREFIX + 'amq-port-tls', fallback=5671)
self.service_host = config.get('CleverMicro-AMQ', self.PREFIX + 'service-host', fallback='rabbitmq')
self.service_port = config.getint('CleverMicro-AMQ', self.PREFIX + 'service-port', fallback=8080)
self.rabbit_mq_user = os.environ["RABBIT_MQ_USER"]
self.rabbit_mq_password = os.environ["RABBIT_MQ_PASSWORD"]
class Backpressure:
"""
configuration with prefix 'cm.backpressure'
"""
PREFIX: str = "cm.backpressure."
PREFIX: str = 'cm.backpressure.'
def __init__(self, config: configparser.ConfigParser):
"""
cm.backpressure.threshold=20
cm.backpressure.time-window=3000
cm.backpressure.management-service-url=management-service:8080
:param config: config parser to read configuration values
"""
self.threshold_load = config.getint(
"CleverMicro-AMQ",
self.PREFIX + "threshold",
fallback=int(os.getenv("MAX_AVAILABILITY", 1)),
)
# time-window in seconds, duration between backpressure reports
self.time_window = config.getint(
"CleverMicro-AMQ", self.PREFIX + "time-window", fallback=10
)
# Define maximum CPU usage before triggering backpressure OVERLOAD alert
self.threshold_cpu = config.getint(
"CleverMicro-AMQ", self.PREFIX + "threshold-cpu", fallback=90
)
# Define the time window length, in seconds, during which, if CPU usage is consistently above the threshold, the system will trigger OVERLOAD alert
self.cpu_overload_duration = config.getint(
"CleverMicro-AMQ", self.PREFIX + "cpu-overload-duration", fallback=30
)
# Define the time window length, in seconds, during which, if no request is made to the Service, the system will trigger IDLE alert
self.idle_duration = config.getint(
"CleverMicro-AMQ", self.PREFIX + "idle-duration", fallback=30
)
# Flag to enable/disable CPU monitoring
self.cpu_monitoring_enabled = config.getboolean(
"CleverMicro-AMQ", self.PREFIX + "cpu-monitoring-enabled", fallback=False
)
self.management_service_url = config.get('CleverMicro-AMQ', self.PREFIX + 'management-service-url', fallback='management-service:8080')
self.threshold = config.getint('CleverMicro-AMQ', self.PREFIX + 'threshold', fallback=20)
self.time_window = config.getint('CleverMicro-AMQ', self.PREFIX + 'time-window', fallback=3000)
class AMQConfiguration:
"""
Top level configuration context holder / object
"""
def __init__(self, config_file_name: str = None):
def __init__(self):
config = configparser.ConfigParser()
# Read the application.properties file
if os.path.exists(config_file_name):
config.read(config_file_name)
else:
# Get the directory of the caller (the module where this function is called)
caller_frame = inspect.stack()[1] # 0 is current_availability frame, 1 is caller
caller_module = inspect.getmodule(caller_frame[0])
if caller_module:
caller_dir = os.path.dirname(os.path.abspath(caller_module.__file__))
config_file_path = os.path.join(caller_dir, config_file_name)
config.read(config_file_path)
else:
logging_error(f"Configuration file not found: {config_file_name}")
logging_warning(
"Defaulting to preset values, which will likely be wrong, causing errors."
)
# Override values with environment variables
for section in config.sections():
for option in config.options(section):
# env_var = f"{section.upper()}_{option.upper()}"
if option in os.environ:
config.set(section, option, os.environ[option])
config.read('config/application.properties')
#
# Add individual config areas
#
self.amq_adapter: AMQAdapter = AMQAdapter(config)
self.dispatch: Dispatch = Dispatch(config)
self.backpressure: Backpressure = Backpressure(config)
def get_unique_instance_id(self) -> int:
"""
Get a unique instance ID for this AMQ service.
Returns:
int: A unique instance ID
"""
if self.instance_id == -1:
logging_warning(f"Generated unique instance ID: {self.instance_id}")
return self.instance_id
amqp/config/application.properties
+7 -40
View File
@@ -2,50 +2,17 @@
# CleverMicro AMQ Adapter settings
# ====================================================================
[CleverMicro-AMQ]
# CleverMicro AMQ Adapter settings that identify this AMQ Adapter instance to other adapters
# A name to identify this service for the AMQ Adapter
cm.amq-adapter.service-name=cleverthis-service-name
# The CleverMicro AMQ Adapter instance ID, used to identify this instance in the CleverMicro AMQ Adapter cluster
cm.amq-adapter.generator-id=1
# Route mapping. For route detail description see section 3.1 in Onboarding Guide for Python:
# https://docs.cleverthis.com/en/architecture/microservices/onboarding/python
# NOTE: this is a comma-separated list of route mappings (multiple, comma-separated routes are allowed here)
cm.amq-adapter.service-name=amq-adapter
cm.amq-adapter.is-router=false
cm.amq-adapter.generator-id=164
cm.amq-adapter.route-mapping=
# Indicate if AMQ Adapter should respond with HTTP 401 Unauthorized if the request does not contain valid JWT token
cm.amq-adapter.require-authenticated-user=false
# Consul configuration for CleverMicro AMQ Adapter
cm.amq-adapter.consul-host=consul
cm.amq-adapter.consul-port=8500
cm.amq-adapter.consul-max-retries=5
cm.amq-adapter.consul-initial-retry-delay=0.2
cm.amq-adapter.consul-id-generator-key=services/ids/counter
cm.amq-adapter.callbacks-enabled=False
# CleverMicro AMQ Adapter dispatcher, settings related to RabbitMQ connection and optional REST forwarding
cm.dispatch.use-dlq=true
cm.dispatch.use-confirms=false
cm.dispatch.amq-host=rabbitmq
cm.dispatch.amq-port=5672
cm.dispatch.amq-port-tls=5671
cm.dispatch.download-dir=/tmp/downloaded_files
# Not used for CleveSwarm nor CleverBRAG or in tight coupling mode.
# applicable only in loose coupling mode, specify the destination where to forward the REST request
#cm.dispatch.service-host=cleverthis-service-name.localhost
#cm.dispatch.service-port=8080
# Backpressure monitor settings
# Define maximum number of resources that service can use before triggering backpressure OVERLOAD alert
# 'resource' is a generic term that can mean CPU, memory, parallel requests, etc.
cm.backpressure.threshold=5
# Define maximum CPU usage (%) before triggering backpressure OVERLOAD alert
cm.backpressure.threshold-cpu-overload=90
# Define maximum CPU usage (%) before triggering backpressure IDLE alert
cm.backpressure.threshold-cpu-idle=10
# Define the time window between the Backpressure reports, in seconds
cm.backpressure.time-window=10
# Define the time window length, in seconds, during which, if CPU usage is consistently above the threshold, the system will trigger OVERLOAD alert
cm.backpressure.cpu-overload-duration=30
# Define the time window length, in seconds, during which, if no request is made to the Service, the system will trigger IDLE alert
cm.backpressure.cpu-idle-duration=30
# AMQ Adapter can monitor CPU usage and trigger backpressure alerts based on CPU usage. Enable this feature explicitly
cm.backpressure.cpu-monitoring-enabled=false
cm.backpressure.threshold=20
cm.backpressure.time-window=3000
cm.backpressure.management-service-url=management-service:8080
amqp/config/application.properties.local
-19
View File
@@ -1,19 +0,0 @@
# ====================================================================
# CleverMicro AMQ Adapter settings
# ====================================================================
[CleverMicro-AMQ]
cm.amq-adapter.service-name=amq-adapter-python
cm.amq-adapter.generator-id=211
cm.amq-adapter.route-mapping=amq-cleverswarm::amq-cleverswarm:localhost.#:5:0
cm.dispatch.use-dlq=true
cm.dispatch.use-confirms=false
cm.dispatch.amq-host=localhost
cm.dispatch.amq-port=5672
cm.dispatch.amq-port-tls=5671
# Not used for CleverSwarm nor CleverBRAG or in tight coupling mode.
# applicable only in loose coupling mode, specify the destination where to forward the REST request
#cm.dispatch.service-host=cleverswarm.localhost
#cm.dispatch.service-port=8080
cm.backpressure.threshold=20
amqp/consul/consul_config.py
+51
View File
@@ -0,0 +1,51 @@
from flask import Flask, request, jsonify
import consul
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Initialize Flask app
app = Flask(__name__)
# Initialize Consul client
consul_client = consul.Consul(host='127.0.0.1', port=8500)
# OpenTelemetry configuration
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
# Configure OTLP exporter (e.g., for exporting to Jaeger or Prometheus via OpenTelemetry Collector)
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
# Instrument Flask and Requests
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
# Endpoint to set configuration properties in Consul
@app.route('/config', methods=['POST'])
def set_config():
key = request.json.get('key')
value = request.json.get('value')
if not key or not value:
return jsonify({"error": "Key and value are required"}), 400
consul_client.kv.put(key, value)
return jsonify({"message": "Configuration set", "key": key, "value": value})
# Endpoint to get configuration properties from Consul
@app.route('/config/<key>', methods=['GET'])
def get_config(key):
with tracer.start_as_current_span("get_config"): # Manual tracing
index, data = consul_client.kv.get(key)
if data and 'Value' in data:
return jsonify({"key": key, "value": data['Value'].decode('utf-8')})
return jsonify({"error": "Key not found"}), 404
if __name__ == '__main__':
app.run(debug=True)
amqp/example/Dockerfile
+23
View File
@@ -0,0 +1,23 @@
FROM python:3.9-slim
# Set the working directory in the container
WORKDIR /app
# Copy requirements.txt first to leverage Docker caching
COPY requirements.txt ./
# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt
# Copy the rest of the application code to the container
COPY . .
# Expose the port that Flask runs on
EXPOSE 5000
# Set environment variables for Flask
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0
# Command to run the Flask application
CMD ["flask", "run"]
amqp/example/clever_item.py
+110
View File
@@ -0,0 +1,110 @@
from flask import Flask, request, jsonify
import os
import consul
import configparser
from werkzeug.utils import secure_filename,send_from_directory
from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Initialize Flask app
app = Flask(__name__)
# Configuration for file uploads
UPLOAD_FOLDER = './amqp/example/uploaded_files'
os.makedirs(UPLOAD_FOLDER, exist_ok=True)
app.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
# Initialize Consul client
consul_client = consul.Consul(host='192.168.229.128', port=8500)
# OpenTelemetry configuration
trace.set_tracer_provider(TracerProvider())
otlp_exporter = OTLPSpanExporter(endpoint="http://192.168.229.128:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()
# In-memory store for CRUD operations
data_store = {}
def __init__(self):
config = configparser.ConfigParser()
# Read the application.properties file
config.read('application.properties')
#
# Add individual config areas
#
self.OTLP_endpoint = config.get('item.example.OTLP_endpoint')
# CRUD Endpoints
@app.route('/items', methods=['POST'])
def create_item():
item_id = request.json.get('item_id')
data = request.json.get('data')
if not item_id or not data:
return jsonify({"error": "Item ID and data are required"}), 400
data_store[item_id] = data
return jsonify({"message": "Item created", "item_id": item_id})
@app.route('/items', methods=['GET'])
def get_all_items():
return jsonify({"items": data_store})
@app.route('/items/<item_id>', methods=['GET'])
def read_item(item_id):
data = data_store.get(item_id)
if data:
return jsonify({"item_id": item_id, "data": data})
return jsonify({"error": "Item not found"}), 404
@app.route('/items/<item_id>', methods=['PUT'])
def update_item(item_id):
data = request.json.get('data')
if not data:
return jsonify({"error": "Data is required"}), 400
if item_id in data_store:
data_store[item_id] = data
return jsonify({"message": "Item updated", "item_id": item_id})
return jsonify({"error": "Item not found"}), 404
@app.route('/items/<item_id>', methods=['DELETE'])
def delete_item(item_id):
if item_id in data_store:
del data_store[item_id]
return jsonify({"message": "Item deleted", "item_id": item_id})
return jsonify({"error": "Item not found"}), 404
# File Upload and Download Endpoints
@app.route('/upload', methods=['POST'])
def upload_file():
if 'file' not in request.files:
return jsonify({"error": "No file part"}), 400
file = request.files['file']
if file.filename == '':
return jsonify({"error": "No selected file"}), 400
filename = secure_filename(file.filename)
file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
return jsonify({"message": "File uploaded", "filename": filename})
@app.route('/download/<filename>', methods=['GET'])
def download_file(filename):
file_path = os.path.join(app.config['UPLOAD_FOLDER'], filename)
if os.path.exists(file_path):
return send_from_directory(app.config['UPLOAD_FOLDER'], filename, as_attachment=True)
return jsonify({"error": "File not found"}), 404
if __name__ == '__main__':
app.run(debug=True)
amqp/model/model.py
+52 -307
View File
@@ -1,284 +1,100 @@
import base64
import json
from typing import List, Mapping, Dict
from dataclasses import dataclass, field
from enum import Enum
from typing import Dict, List
from enum import Enum, auto
from aio_pika.abc import AbstractRobustConnection
from amqp.model.service_message_type import ServiceMessageType
from amqp.model.snowflake_id import SnowflakeId
from amqp.router.utils import sanitize_as_rabbitmq_name
RS = ":"
"""
Define the structure of messages used in Clever Micro
"""
class DataMessageMagicByte(Enum):
HTTP_REQUEST = "A"
RPC_REQUEST = "R"
@dataclass
class CleverMicroMessage:
DEFAULT_CONTENT_TYPE = ["application/octet-stream"]
"""
The common structure of a message that transfers the user request to the Service and back. All data related messages
conform to this base structure:
magic - identifies the message type (REST, RPC, DataResponse)
id - unique message ID
m_field - method name (for REST style calls) or service name (for RPC style calls) or response code for DataResponse
d_field - domain name (for REST style calls) or module name (for RPC style calls) or error code for DataResponse
p_field - path (for REST style calls) or method name (for RPC style calls) or error cause for DataResponse
headers - headers of the message (e.g. Content-Type, Accept, etc.)
trace_info - trace information (e.g. trace ID, span ID, etc. for Jaeger trace monitor)
base64body - the body of the message encoded in base64
"""
def __init__(
self,
magic: str,
id: SnowflakeId,
m_field: str,
d_field: str,
p_field: str,
headers: Dict[str, List[str]],
trace_info: Dict[str, str],
base64body: bytes,
):
self._magic = magic
self._id: SnowflakeId = id
self._m_field = m_field
self._d_field = d_field
self._p_field = p_field
self._headers: Dict[str, List[str]] = headers
self._trace_info: Dict[str, str] = trace_info
self._base64body: bytes = base64body
def magic(self):
return self._magic
def id(self):
return self._id
def m_field(self) -> str:
return self._m_field
def d_field(self) -> str:
return self._d_field
def p_field(self) -> str:
return self._p_field
def headers(self) -> Dict[str, List[str]]:
return self._headers
def trace_info(self) -> Dict[str, str]:
return self._trace_info
def base64body(self) -> bytes:
return self._base64body
def body(self) -> bytes:
return base64.b64decode(self.base64body())
def content_type(self) -> str:
return self.headers().get("Content-Type", self.DEFAULT_CONTENT_TYPE)[0]
def return_address(self) -> str:
return self.headers().get("Return-Address", "")[0]
class DataMessage(CleverMicroMessage):
"""
The class represents a structure of a message that transfers the user request to the Service.
It may also be used to make a call/request between services.
The response to this call is in format of DataResponse class.
See DataMessageFactory for code to instantiate, serialize and deserialize the class
"""
def __init__(
self,
magic: str,
id: SnowflakeId,
method: str,
domain: str,
path: str,
headers: Dict[str, List[str]],
trace_info: Dict[str, str],
base64body: bytes,
):
super().__init__(magic, id, method, domain, path, headers, trace_info, base64body)
def method(self) -> str:
return self.m_field()
def domain(self) -> str:
return self.d_field()
def path(self) -> str:
return self.p_field()
def routing_key(self) -> str:
return sanitize_as_rabbitmq_name(f"{self.d_field()}.{self.p_field()}")
class MultipartDataMessage(DataMessage):
def __init__(self, message: DataMessage, extra_data: dict):
super().__init__(
message.magic(),
message.id(),
message.method(),
message.domain(),
message.path(),
message.headers(),
message.trace_info(),
message.base64body(),
)
self.extra_data = extra_data
@dataclass
class AMQMessage(DataMessage):
"""
Expand the core DataMessage type to include the channel, which is required to create a dedicated queue
in case the response should initiate file download.
"""
_connection: AbstractRobustConnection = field(repr=False) # Don't show in repr
def __init__(self, data_message: DataMessage, connection: AbstractRobustConnection):
# Initialize the parent DataMessage with all properties from the input data_message
super().__init__(
magic=data_message.magic(),
id=data_message.id(),
method=data_message.method(),
domain=data_message.domain(),
path=data_message.path(),
headers=data_message.headers(),
trace_info=data_message.trace_info(),
base64body=data_message.base64body(),
)
self._connection = connection
self.extra_data = data_message.extra_data if hasattr(data_message, "extra_data") else {}
@property
def connection(self) -> AbstractRobustConnection:
"""Get the AMQP connection associated with this message"""
return self._connection
class DataResponse(CleverMicroMessage):
"""
The response message to REST & RPC style calls (note: the request is made with DataMessage)
See DataResponseFactory for code to instantiate, serialize and deserialize the class
"""
def __init__(
self,
magic: str,
id: SnowflakeId,
response_code: str,
content_type: str,
error: str,
error_cause: str,
headers: Dict[str, List[str]],
trace_info: Dict[str, str],
base64body: bytes,
):
if headers is None:
headers = {}
headers["Content-Type"] = [content_type]
super().__init__(
magic,
id,
response_code,
error,
error_cause,
headers,
trace_info,
base64body,
)
def response_code(self) -> str:
return self.m_field()
def error(self) -> str:
return self.d_field()
def error_cause(self) -> str:
return self.p_field()
@dataclass(frozen=True)
class AMQRoute:
"""
Configuration item representing one AMQRoute. Adapter uses this to declare the Exchange
and/or queue and bind the key as routing expression.
:param component_name: a service/component name that this route points to (owner of the route)
:param componentName: a service/component name that this route points to (owner of the route)
:param exchange: RabbitMQ exchange name
:param queue: RabbitMQ queue name
:param key: RabbitMQ TOPIC Exchange routing key
:param timeout: Timeout in milliseconds, how long to wait for reply, or not waiting if less then 1
:param valid_until: optional timestamp (millis from epoch) after which the route is ignored
:param validUntil: optional timestamp (millis from epoch) after which the route is ignored
"""
component_name: str
componentName: str
exchange: str
queue: str
key: str
timeout: int
valid_until: int
validUntil: int
FORMAT = "Component-Name:Exchange-Name:Queue-Name:Routing-Key:Timeout:Valid-Until"
def __str__(self):
return (
f"{self.component_name}{RS}{self.exchange}{RS}{self.queue}{RS}{self.key}"
f"{RS}{self.timeout}{RS}{self.valid_until}"
f"{self.componentName}{RS}{self.exchange}{RS}{self.queue}{RS}{self.key}"
f"{RS}{self.timeout}{RS}{self.validUntil}"
)
def __eq__(self, other):
if isinstance(other, AMQRoute):
return (
self.component_name == other.component_name
and self.queue == other.queue
and self.key == other.key
)
return (self.componentName == other.componentName
and self.queue == other.queue
and self.key == other.key)
return False
@property
def primary_key(self):
"""
Build the unique identifier of the route
:return primary key
"""
return f"{self.component_name}{RS}{self.queue}{RS}{self.key}"
return f"{self.componentName}{RS}{self.queue}{RS}{self.key}"
def as_string(self) -> str:
return self.__str__()
# Ensure backward compatibility
AMQResponse = DataResponse
@dataclass
class AMQMessage:
magic: str
id: SnowflakeId
method: str
domain: str
path: str
headers: Mapping[str, List[str]]
trace_info: Mapping[str, str]
base64_body: bytes
def routing_key(self) -> str:
return sanitize_as_rabbitmq_name(f"{self.domain}.{self.path}")
def body(self) -> bytes:
return base64.b64decode(self.base64_body)
@dataclass
class AMQResponse:
id: str
response_code: int
content_type: str
response: bytes
error: str
error_cause: str
trace_info: Mapping[str, str]
def body(self) -> bytes:
return base64.b64decode(self.response)
class ServiceMessageType(Enum):
ROUTING_DATA_REQUEST = auto()
ROUTING_DATA = auto()
ROUTING_DATA_REMOVE = auto()
ROUTING_DATA_REPLACE = auto()
UNKNOWN = auto()
@dataclass(frozen=True)
class ServiceMessage:
"""
The CleverMicro service message that is used to communicate the internal state of the AMQ channel.
In v0.2 it supports service discovery / route registration and Backpressure status notification.
"""
class CleverMicroServiceMessage:
id: SnowflakeId = None
payload_type: int = 0
message_type: ServiceMessageType = ServiceMessageType.UNKNOWN
@@ -289,74 +105,3 @@ class ServiceMessage:
def is_valid(self) -> bool:
return self.message_type is not None
class ScalingRequestAlert(Enum):
OVERLOAD = "OVERLOAD"
IDLE = "IDLE"
UPDATE = "UPDATE"
SCALE_UP = "SCALE_UP"
SCALE_DOWN = "SCALE_DOWN"
@dataclass(frozen=True)
class ScalingRequest:
service_id: str
task_id: str
max_availability: int
current_availability: int
request_type: ScalingRequestAlert
version: int = 1
def to_json(self) -> str:
"""Converts the object to a JSON string matching the Java structure"""
return json.dumps(
{
"version": self.version,
"serviceId": self.service_id,
"taskId": self.task_id,
"maxAvailability": self.max_availability,
"currentAvailability": self.current_availability,
"requestType": self.request_type.value, # Using .value for Enum serialization
},
indent=2,
)
@classmethod
def from_json(cls, json_str: str) -> "ScalingRequest":
"""Creates a ScalingRequest from a JSON string"""
data = json.loads(json_str)
return cls(
service_id=data["serviceId"],
task_id=data["taskId"],
max_availability=data["maxAvailability"],
current_availability=data["currentAvailability"],
request_type=ScalingRequestAlert(data["requestType"]), # Convert string to Enum
)
class AMQErrorMessage:
"""
The error response message to RPC style call (note: the request is made with DataMessage)
"""
def __init__(
self,
response_code: int,
error: str,
detail: str,
content_type: str = "application/json",
):
self.response_code = response_code
self.error = error
self.detail = detail
self.content_type = content_type
def serialize(self) -> bytes:
return json.dumps(
{
"response_code": self.response_code,
"error": self.error,
"detail": self.detail,
}
).encode("utf-8")
amqp/model/service_message_type.py
-10
View File
@@ -1,10 +0,0 @@
from enum import Enum, auto
class ServiceMessageType(Enum):
ROUTING_DATA_REQUEST = auto()
ROUTING_DATA = auto()
ROUTING_DATA_REMOVE = auto()
ROUTING_DATA_REPLACE = auto()
BACKPRESSURE = auto()
UNKNOWN = auto()
amqp/model/snowflake_id.py
+20 -31
View File
@@ -2,31 +2,25 @@ import struct
class SnowflakeId:
"""
see https://en.wikipedia.org/wiki/Snowflake_ID
"""
SNOWFLAKE_ID_WIDTH = 10 # with in bytes (8 bits each), can be more than long int width (8 bytes)
SNOWFLAKE_ID_WIDTH = (
10 # with in bytes (8 bits each), can be more than long int width (8 bytes)
)
def __init__(self, hi_bits_or_bytes=None, lo_bits=None):
def __init__(self, _input: bytes = None):
self.bits = [0, 0]
if lo_bits is None:
if hi_bits_or_bytes is not None:
i = 0
shift = 8 # starting from Hi value, which is only 2 bytes wide
n = 1
while i < len(hi_bits_or_bytes) and i <= self.SNOWFLAKE_ID_WIDTH:
self.bits[n] |= hi_bits_or_bytes[i] << shift
if shift == 0:
n -= 1
shift = 56
else:
shift -= 8
i += 1
else:
self.bits = [lo_bits, hi_bits_or_bytes]
if _input is not None:
i = 0
shift = 8 # starting from Hi value, which is only 2 bytes wide
n = 1
while i < len(_input) and i <= self.SNOWFLAKE_ID_WIDTH:
self.bits[n] |= _input[i] << shift
if shift == 0:
n -= 1
shift = 56
else:
shift -= 8
i += 1
def __init__(self, hi_bits: int, lo_bits: int):
self.bits = [lo_bits, hi_bits]
@staticmethod
def from_hex(_input):
@@ -42,18 +36,13 @@ class SnowflakeId:
for i in range(0, expected_length, 2):
if i == expected_length - (2 * 8):
n = 0
bits[n] = (bits[n] << 8) | int(_input[i : i + 2], 16)
bits[n] = (bits[n] << 8) | int(_input[i:i + 2], 16)
return SnowflakeId(bits[1], bits[0])
def __str__(self):
_hex = f"{self.bits[1]:016X}{self.bits[0]:016X}"
_val = (
_hex
if len(_hex) <= 2 * SnowflakeId.SNOWFLAKE_ID_WIDTH
else _hex[(-2 * SnowflakeId.SNOWFLAKE_ID_WIDTH) :]
)
return _val.lower()
return _hex if len(_hex) <= 2 * SnowflakeId.SNOWFLAKE_ID_WIDTH else _hex[(-2 * SnowflakeId.SNOWFLAKE_ID_WIDTH):]
def __eq__(self, other):
if isinstance(other, SnowflakeId):
@@ -70,7 +59,7 @@ class SnowflakeId:
def get_bytes(self):
high = struct.pack("!Q", self.bits[1])
low = struct.pack("!Q", self.bits[0])
return high[-2:] + low
return high[2:] + low
def get_bits(self):
return self.bits
amqp/rabbitmq/rabbit_mq_client.py
-237
View File
@@ -1,237 +0,0 @@
import logging
from typing import Optional
import aio_pika
from aio_pika.abc import AbstractRobustExchange, AbstractRobustQueue
from pika.exchange_type import ExchangeType
from amqp.adapter.data_message_factory import DataMessageFactory
from amqp.adapter.logging_utils import (
logging_debug,
logging_error,
logging_info,
logging_warning,
)
from amqp.model.model import AMQRoute, DataMessage
from amqp.router.router_base import (
AMQ_REPLY_TO_EXCHANGE,
AMQ_RPC_EXCHANGE,
DLQ_EXCHANGE,
)
from amqp.router.utils import sanitize_as_rabbitmq_name
class RabbitMQClient:
PREFETCH_COUNT = 1
DLQ_ARGS = {"x-dead-letter-exchange": DLQ_EXCHANGE}
def __init__(self, configuration, router):
self.channel: aio_pika.RobustChannel | None = None
self.connection: aio_pika.RobustConnection | None = None
self.rpc_exchange: aio_pika.RobustExchange | None = None
self.reply_exchange: aio_pika.RobustExchange | None = None
self.amq_configuration = configuration
logging.info("*******************************************")
logging.info(
"******** RabbitMQClient.init(): %s",
self.amq_configuration.dispatch.amq_host,
)
logging.info("*******************************************")
self.router = router
# this.meterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
self.reply_to_exchange: AbstractRobustExchange | None = (
None # when sending response to RPC request
)
self.reply_to_queue: AbstractRobustQueue | None = (
None # when receiving response to RPC request
)
async def init(self, loop) -> AbstractRobustExchange:
await self.setup_amq_channel(loop)
await self.router.init_routing_paths_consumer(self.router.channel)
cm_reply_to_queue = self.router.get_reply_to_queue_name()
self.reply_to_queue = await self.channel.declare_queue(
name=cm_reply_to_queue, durable=True, exclusive=False, auto_delete=False
)
self.reply_to_exchange = await self.channel.declare_exchange(
AMQ_REPLY_TO_EXCHANGE, type=ExchangeType.direct
)
await self.reply_to_queue.bind(
exchange=self.reply_to_exchange, routing_key=cm_reply_to_queue
)
return self.reply_to_exchange
def cleanup(self):
logging_info("RabbitMQClient.cleanup()")
self.router.cleanup() # de-register route(s)
if self.router.is_open(self.channel):
try:
self.channel.close()
if not self.connection.is_closed:
self.connection.close()
except Exception as e:
logging_error(
"RabbitMQClient.PreDestroy cleanup - connection close error: %s",
e,
)
async def register_inbound_routes(self, route_mapping: str, callback):
"""
Register inbound routes for the AMQ adapter. The route mapping is a comma-separated string that contains
list of routes. A route is the mapping of domain and path to the exchange and queue name.
This method declare (and thus create) the queue and exchange, bind them together using the
regex-like routing expression given in the route.
After this the Service is listening on (consuming from) the queue for incoming messages.
"""
if self.router.is_open(self.channel):
queue_args = self.DLQ_ARGS if self.amq_configuration.dispatch.use_dlq else None
logging_info("RabbitMQClient register routes, cfg mapping: [%s]", route_mapping)
# convert the comma-separated string into list of AMQRoute objects
requested_routes = self.router.unwrap_route_list(route_mapping)
for route in requested_routes:
try:
# Note: AMQRoute allows one of the exchange or queue to be None, in which case the default is used
queue_name = (
route.queue if route.queue else self.router.get_consume_queue_name()
)
logging_info("RabbitMQClient register route: [%s]", route)
if self.amq_configuration.amq_adapter.callbacks_enabled:
queue: Optional[AbstractRobustQueue] = await self.channel.declare_queue(
name=queue_name,
durable=True,
exclusive=False,
auto_delete=False,
arguments=queue_args,
)
else:
queue = None
_exchange_name: str = route.exchange if route.exchange else AMQ_RPC_EXCHANGE
exchange: AbstractRobustExchange = await self.channel.declare_exchange(
name=_exchange_name, type=ExchangeType.topic
)
if self.amq_configuration.amq_adapter.callbacks_enabled:
await queue.bind(exchange, route.key)
_c_tag = await queue.consume(callback, no_ack=False)
else:
_c_tag = None
# Register the route with the router and publish its detail to other adapters
await self.router.register_route(
AMQRoute(
route.component_name,
_exchange_name,
queue_name,
sanitize_as_rabbitmq_name(route.key),
route.timeout,
route.valid_until,
)
)
logging_info(
" [%s] AMQ connection initialized for route: %s, listens on: %s/%s",
_c_tag,
route,
exchange,
queue_name,
)
except Exception as err:
logging_error("Callback registration INCOMPLETE, %s", err)
else:
logging_warning("Channel is null, cannot register routes.")
async def register_data_reply_callback(self, rpc_callback):
_c_tag = await self.reply_to_queue.consume(callback=rpc_callback, no_ack=False)
logging_info(
" [%s] RabbitMQClient register data reply callback on queue: %s",
_c_tag,
self.reply_to_queue.name,
)
async def setup_amq_channel(self, loop):
"""
Set up the RabbitMQ connection
:return: nothing, it applies changes to instance variables
"""
logging.info(
f"RabbitMQClient.setupAMQ, connecting to RabbitMQ at {self.amq_configuration.dispatch.amq_host}:{self.amq_configuration.dispatch.amq_port}, loop={loop}"
)
self.connection = await aio_pika.connect_robust(
host=self.amq_configuration.dispatch.amq_host,
port=self.amq_configuration.dispatch.amq_port,
login=self.amq_configuration.dispatch.rabbit_mq_user,
password=self.amq_configuration.dispatch.rabbit_mq_password,
loop=loop,
)
logging_debug(
"RabbitMQClient.setupAMQ, got connection, open=%s",
self.connection.is_closed,
)
self.channel = await self.connection.channel()
await self.channel.set_qos(prefetch_count=self.PREFETCH_COUNT)
await self.router.init_internal_routing_paths(self.channel)
self.router.set_route_added_notifier(
lambda route: logging_debug(
" RouteAdded, setting up metrics: amqp.adapter.exchange.%s",
route.exchange,
)
)
async def send_message(self, message: DataMessage):
"""
The API routine that sends a datagram message (DataMessage) to the destination. The Destination
is determined from the DataMessage metadata section, using `domain` and `path` as key.
:param message: message to send
:return: nothing (void) as submitting message to RabbitMQ exchange does not return anything
meaningful. There will be asynch ACK later that will confirm the message was sent to RMQ.
"""
try:
route = self.router.find_route_by_message(message)
if route and self.rpc_exchange:
if route.timeout <= 0:
# No response expected
pika_message: aio_pika.message.Message = aio_pika.message.Message(
body=DataMessageFactory.serialize(message),
content_encoding="utf-8",
delivery_mode=2,
)
await self.rpc_exchange.publish(
message=pika_message,
routing_key=self.router.get_routing_key(
message
), # context path is a routing key
)
logging_info(f"Msg Publish (no-reply), messageId: {message.id()}, to: {route}")
else:
# This is RPC call, there should be response
pika_message: aio_pika.message.Message = aio_pika.message.Message(
body=DataMessageFactory.serialize(message),
content_encoding="utf-8",
delivery_mode=2,
correlation_id=str(message.id()),
reply_to=self.router.get_reply_to_queue_name(),
)
await self.rpc_exchange.publish(
message=pika_message,
routing_key=self.router.get_routing_key(
message
), # context path is a routing key
)
logging_info(
"Msg Publish (RPC call), messageId: %s, to: %s",
message.id().as_string(),
route.as_string(),
)
else:
logging_warning(
"Message not sent, no route for %s/%s",
message.domain(),
message.path(),
)
except Exception as err:
logging_error("Send message failed: %s", err)
async def request_routes(self):
await self.router.request_routes_from_adapters()
def find_route(self, domain: str, path: str) -> AMQRoute:
return self.router.find_route(domain, path)
amqp/rabbitmq/rabbit_mq_consumer.py
+143
View File
@@ -0,0 +1,143 @@
import logging
import pika
from pika.exchange_type import ExchangeType
from amqp.model.model import AMQRoute
from amqp.rabbitmq.rabbit_mq_producer import RabbitMQProducer
from amqp.router.router_base import AMQ_REPLY_TO_EXCHANGE, DLQ_EXCHANGE, AMQ_RPC_EXCHANGE
from amqp.router.router_consumer import RouterConsumer
from amqp.router.utils import sanitize_as_rabbitmq_name
class RabbitMQConsumer(RabbitMQProducer):
PREFETCH_COUNT = 1
DLQ_ARGS = {"x-dead-letter-exchange": DLQ_EXCHANGE}
def __init__(self, configuration):
super().__init__(configuration, RouterConsumer(configuration))
def cleanup(self):
logging.info(" [*] RabbitMQConsumer.cleanup()")
self.router.cleanup() # de-register route(s)
if self.router.is_open(self.channel):
try:
self.channel.close()
except (pika.connection.exceptions.AMQPError, pika.connection.exceptions.ChannelClosedByBroker) as e:
logging.error(" [E] RabbitMQConsumer.PreDestroy cleanup - connection close error: %s", e)
def register_routes(self):
if self.router.is_open(self.channel):
queue_args = self.DLQ_ARGS if self.amq_configuration.dispatch.use_dlq else None
route_mapping = self.amq_configuration.amq_adapter.route_mapping
logging.info(" [*] RabbitMQConsumer register routes, cfg mapping: [%s]", route_mapping)
requested_routes = self.router.unwrap_route_list(route_mapping)
for route in requested_routes:
queue_name = route.queue() if route.queue() else self.router.get_consume_queue_name()
logging.info(" [*] RabbitMQConsumer register route: [%s]", route)
self.channel.queue_declare(queue_name, durable=True, exclusive=False,
auto_delete=False, arguments=queue_args)
exchange = route.exchange() if route.exchange() else AMQ_RPC_EXCHANGE
self.channel.exchange_declare(exchange, exchange_type=pika.exchange_type.ExchangeType.topic)
self.channel.queue_bind(queue_name, exchange, route.key())
self.router.register_route(
AMQRoute(
route.component_name(), exchange, queue_name,
sanitize_as_rabbitmq_name(route.key()),
route.timeout(), route.valid_until()
)
)
logging.info(" [*] AMQ connection initialized for route: %s, listens on: %s/%s",
route, exchange, queue_name)
else:
logging.warning(" [W] Channel is null, cannot register routes.")
def register_deliver_callback(self, callback_provider):
try:
for route in self.router.get_own_routes():
self.channel.basic_consume(
queue=route.queue(),
on_message_callback=callback_provider(self.channel),
auto_ack=False
)
logging.info(" [*] Consumer Waiting for messages on: %s", route.queue())
except pika.connection.exceptions.AMQPError as err:
logging.error(" [E] Callback registration INCOMPLETE, %s", err)
def register_rpc_handler(self, rpc_consumer):
cm_reply_to_queue = self.router.get_reply_to_queue_name()
self.channel.queue_declare(cm_reply_to_queue, durable=True, exclusive=False, auto_delete=False)
self.channel.exchange_declare(
AMQ_REPLY_TO_EXCHANGE, exchange_type=pika.exchange_type.ExchangeType.direct
)
self.channel.queue_bind(cm_reply_to_queue, AMQ_REPLY_TO_EXCHANGE, cm_reply_to_queue)
self.channel.basic_consume(
queue=cm_reply_to_queue,
on_message_callback=rpc_consumer,
auto_ack=False
)
"""
def limit_rate_inside_time_window(self, message: AMQMessage) -> int:
next_batch_size = BackpressureSubscriber.NO_OP
now = time.time_ns() // 1000000
queue_size = self.getQueueSize(message)
# 1. Backpressure? if unprocessed count exceeds threshold, wait for some time
if queue_size > THRESHOLD:
# TODO - signal queue backpressure explicitly to upstream/metrics
# wait: estimated-per-message-processing-time * SQRT(over-the-limit)
self.wait_for_milliseconds(
(WINDOW / self.request_for_window_duration) * (queue_size - THRESHOLD) ** 0.5)
queue_size = self.getQueueSize(message) # refresh queue size after wait
# 2. batch complete? determine next batch size and wait till window end. reset window.
if self.fulfilled >= self.request_for_window_duration:
next_batch_size = self.calculateNextBatchSize(queue_size)
self.request_for_window_duration = next_batch_size
remaining_time = (self.window_started_at + WINDOW) - now
if remaining_time > 0:
self.wait_for_milliseconds(remaining_time)
queue_size = self.getQueueSize(message) # refresh queue size after wait
self.window_started_at = time.time_ns() // 1000000
self.fulfilled = 0
# 3. Window elapsed? Reset window & recalculate batch size
if now >= self.window_started_at + WINDOW:
if next_batch_size == BackpressureSubscriber.NO_OP:
next_batch_size = self.calculateNextBatchSize(queue_size)
self.request_for_window_duration = next_batch_size
self.window_started_at = time.time_ns() // 1000000
self.fulfilled = 0
# 4. send the message from the current batch
self.send_message(message)
self.fulfilled += 1
# 5. report and setup state for the next call
end = time.time_ns() // 1000000
self.on_next_timestamp = end
return next_batch_size
def calculateNextBatchSize(self, queue_size: int) -> int:
next_batch_size = self.request_for_window_duration
if queue_size <= THRESHOLD:
# processing proceeds well, can increase the rate
next_batch_size += self.subscriber.increaseRate(self.request_for_window_duration)
logging.debug(f"INCREASE rate to {next_batch_size}. qSize={queue_size}")
else:
decrease = self.subscriber.backoff(queue_size - THRESHOLD)
if self.request_for_window_duration > decrease:
next_batch_size -= decrease
logging.debug(f"DECREASE rate to {next_batch_size}. qSize={queue_size}")
return next_batch_size
"""
"""
def getQueueSize(self, message: AMQMessage) -> int:
try:
route_queue_name = self.router.findRoute(message)
queue_name = route_queue_name.queue if route_queue_name else self.TASK_DISPATCH_QUEUE_NAME
logger.info(f"Get Queue size for {queue_name}")
return max(len(self.awaitingACK), self.channel.message_count(queue_name))
except Exception:
return len(self.awaitingACK)
"""
amqp/rabbitmq/rabbit_mq_producer.py
+134
View File
@@ -0,0 +1,134 @@
import logging
import time
import pika
from amqp.adapter.amq_message_factory import AMQMessageFactory
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import AMQMessage, AMQRoute
from amqp.router.router_producer import RouterProducer
class RabbitMQProducer:
PREFETCH_COUNT = 1
THRESHOLD = 1000
WINDOW = 5000 # 5 seconds
def __init__(self, amq_configuration: AMQConfiguration, router: RouterProducer = None):
self.channel = None
self.connection = None
self.request_for_window_duration = None
self.amq_configuration = amq_configuration
logging.info("*******************************************")
logging.info("******** RabbitMQProducer.init() ********** %s - %s - %s",
self.amq_configuration.dispatch.amq_host,
self.amq_configuration.dispatch.use_confirms,
self.WINDOW)
logging.info("*******************************************")
self.router = router or RouterProducer(amq_configuration)
# this.meterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
self.setup_amq_channel()
self.router.init_routing_paths(self.channel)
# self.backpressure_per_exchange = defaultdict(lambda: Gauge(
# f"amqp.adapter.exchange.{exchange}", registry=CollectorRegistry()))
self.router.set_route_added_notifier(
lambda route: logging.info(" [DBG] ---------- RouteAdded, setting up metrics: amqp.adapter.exchange.%s",
route.exchange))
self.request_for_window_duration = self.THRESHOLD // 2
self.window_started_at = time.time()
self.on_next_timestamp = self.window_started_at
self.fulfilled = 0
def cleanup(self):
if self.connection:
try:
self.connection.close()
except Exception as e:
logging.error("RabbitMQConsumer.PreDestroy cleanup - connection close error: %s", e)
def setup_amq_channel(self):
"""
Set up the RabbitMQ connection
:return: nothing, it applies changes to instance variables
"""
factory = pika.ConnectionParameters(
host=self.amq_configuration.dispatch.amq_host,
username=self.amq_configuration.dispatch.rabbit_mq_user,
password=self.amq_configuration.dispatch.rabbit_mq_password
)
self.connection = pika.BlockingConnection(factory)
logging.info("RabbitMQProducer.setupAMQ, got connection, open=%s", self.connection.is_open)
self.connection.add_on_connection_blocked_callback(self.blocked_listener)
self.connection.add_on_connection_unblocked_callback(self.unblocked_listener)
self.channel = self.connection.channel()
self.channel.basic_qos(prefetch_count=self.PREFETCH_COUNT)
if self.amq_configuration.dispatch.use_confirms():
self.setup_local_ack()
self.channel.add_on_return_callback(self.return_listener)
def send_message(self, message: AMQMessage):
"""
The API routine that sends a datagram message (AMQMessage) to the destination. The Destination
is determined from the AMQMessage metadata section, using `domain` and `path` as key.
:param message: message to send
:return: nothing (void) as submitting message to RabbitMQ exchange does not return anything
meaningful. There will be asynch ACK later that will confirm the message was sent to RMQ.
"""
try:
route = self.router.find_route_by_message(message)
if route:
if route.timeout < 0:
# No response expected
self.channel.basic_publish(
route.exchange,
self.router.get_routing_key(message), # context path is a routing key
properties=pika.BasicProperties(delivery_mode=2),
body=AMQMessageFactory.serialize(message)
)
logging.info(f" [x] basicPublish (no-reply), messageId: {message.id}, to: {route}")
else:
# This is RPC call, there should be response
props = pika.BasicProperties(
correlation_id=message.id,
reply_to=self.router.get_reply_to_queue_name()
)
self.channel.basic_publish(
route.exchange,
self.router.get_routing_key(message), # context path is a routing key
properties=props,
body=AMQMessageFactory.serialize(message)
)
logging.info(" [x] basicPublish (RPC call), messageId: %s, to: %s",
message.id.as_string(), route.as_string())
else:
logging.warning("Message not sent, no route for {}/{}", message.domain, message.path)
except Exception as err:
logging.error("Send message failed: %s", err)
def request_routes(self):
self.router.request_routes_from_adapters()
def setup_local_ack(self):
self.channel.confirm_delivery()
def find_route(self, domain: str, path: str) -> AMQRoute:
return self.router.find_route(domain, path)
def return_listener(self, return_message):
logging.error(
f" [E] ****** (NOT DELIVERED) ID={return_message.properties.message_id}, "
f"Exchg=[{return_message.exchange}], Key=[{return_message.routing_key}], "
f"replyTxt={return_message.reply_text}, length={len(return_message.body) if return_message.body else 0}"
)
def blocked_listener(self, reason: str):
logging.error(f" [E] CHANNEL BLOCKED, reason={reason}")
def unblocked_listener(self):
logging.info(" [*] CHANNEL UNBLOCKED")
amqp/rabbitmq/user_management.py
-53
View File
@@ -1,53 +0,0 @@
import logging
from typing import Any, Dict
from amqp.rabbitmq.user_management_service_client import (
UserManagementServiceClient,
UserManagementServiceException,
)
async def get_user_info_from_token(
client: UserManagementServiceClient, token: str
) -> Dict[str, Any]:
"""
Get user information from a JWT token.
Args:
client: User Management Service client
token: JWT token
Returns:
User information
Raises:
UserManagementServiceException: If an error occurs
"""
try:
# Validate the token
token_info = await client.validate_token(token)
# Extract user ID from token info
user_id = None
for item in token_info:
if isinstance(item, dict) and "sub" in item:
user_id = item["sub"]
break
if not user_id:
raise UserManagementServiceException(
"cleverthis.clevermicro.auth.invalid_token",
"Token does not contain user ID (sub claim)",
{},
)
# Query user information
user_info = await client.query_user_by_id(user_id)
return user_info
except UserManagementServiceException as e:
logging.error(f"User Management Service error: {e.exception_type}: {e.message}")
raise
except Exception as e:
logging.error(f"Unexpected error: {e}")
raise
amqp/rabbitmq/user_management_service_client.py
-260
View File
@@ -1,260 +0,0 @@
import asyncio
import json
import logging
import uuid
from typing import Any, Dict, List, Optional
import aio_pika
from aio_pika import Message
from aio_pika.abc import AbstractIncomingMessage, AbstractRobustChannel, AbstractRobustConnection
from amqp.adapter.logging_utils import logging_debug, logging_error, logging_info
from amqp.config.amq_configuration import AMQConfiguration
class UserManagementServiceException(Exception):
"""Exception raised for errors in the User Management Service."""
def __init__(self, exception_type: str, message: str, additional_info: Dict[str, Any] = None):
self.exception_type = exception_type
self.message = message
self.additional_info = additional_info or {}
super().__init__(f"{exception_type}: {message}")
class UserManagementServiceClient:
"""
Client for the User Management Service that communicates over RabbitMQ.
Supports operations for token, user, group, tenant, and permission services.
"""
# Exchange name for User Management Service
EXCHANGE_NAME = "cleverthis.clevermicro.auth.endpoints"
# Service routing keys
TOKEN_SERVICE = "token-service-v1"
USER_SERVICE = "user-service-v1"
GROUP_SERVICE = "group-service-v1"
TENANT_SERVICE = "tenant-service-v1"
PERMISSION_SERVICE = "permission-service-v1"
# Response queue name
RESPONSE_QUEUE_NAME = "user-management-service-cleverswarm"
def __init__(self, configuration: AMQConfiguration, loop: asyncio.AbstractEventLoop = None):
"""
Initialize the User Management Service client.
Args:
configuration: AMQ configuration
loop: Event loop to use (optional)
"""
self.amq_configuration = configuration
self.loop = loop or asyncio.get_event_loop()
self.connection: Optional[AbstractRobustConnection] = None
self.channel: Optional[AbstractRobustChannel] = None
self.exchange: Optional[aio_pika.RobustExchange] = None
self.response_queue: Optional[aio_pika.RobustQueue] = None
self.response_consumer_tag: Optional[str] = None
self.futures: Dict[str, asyncio.Future] = {}
self.initialized = False
async def initialize(self) -> None:
"""Initialize the RabbitMQ connection and set up the exchange and response queue."""
if self.initialized:
return
try:
# Connect to RabbitMQ
self.connection = await aio_pika.connect_robust(
host=self.amq_configuration.dispatch.amq_host,
port=self.amq_configuration.dispatch.amq_port,
login=self.amq_configuration.dispatch.rabbit_mq_user,
password=self.amq_configuration.dispatch.rabbit_mq_password,
loop=self.loop,
)
# Create channel
self.channel = await self.connection.channel()
await self.channel.set_qos(prefetch_count=1)
# Declare exchange
self.exchange = await self.channel.declare_exchange(
name=self.EXCHANGE_NAME,
type=aio_pika.ExchangeType.TOPIC,
durable=True,
)
# Declare response queue
self.response_queue = await self.channel.declare_queue(
name=self.RESPONSE_QUEUE_NAME,
durable=True,
exclusive=False,
auto_delete=False,
)
# Start consuming responses
self.response_consumer_tag = await self.response_queue.consume(
self._on_response_callback,
no_ack=False,
)
self.initialized = True
logging_info("User Management Service client initialized")
except Exception as e:
logging_error(f"Failed to initialize User Management Service client: {e}")
if self.connection and not self.connection.is_closed:
await self.connection.close()
raise
async def _on_response_callback(self, message: AbstractIncomingMessage) -> None:
"""
Handle responses from the User Management Service.
Args:
message: The incoming message
"""
try:
correlation_id = message.correlation_id
if not correlation_id:
logging_error("Received response without correlation ID")
await message.ack()
return
future = self.futures.get(correlation_id)
if not future:
logging_error(f"No pending request found for correlation ID: {correlation_id}")
await message.ack()
return
try:
response_data = json.loads(message.body.decode('utf-8'))
logging_debug(f"Received response: {response_data}")
future.set_result(response_data)
except Exception as e:
future.set_exception(e)
await message.ack()
except Exception as e:
logging_error(f"Error processing response: {e}")
await message.ack()
async def close(self) -> None:
"""Close the RabbitMQ connection."""
if self.connection and not self.connection.is_closed:
await self.connection.close()
self.initialized = False
logging_info("User Management Service client closed")
async def call_service(
self, service: str, method: str, args: Dict[str, Any] = None
) -> Dict[str, Any]:
"""
Call a method on a User Management Service.
Args:
service: Service routing key (e.g., 'token-service-v1')
method: Method name to call
args: Method arguments
Returns:
Response data
Raises:
UserManagementServiceException: If the service returns an error
"""
if not self.initialized:
await self.initialize()
correlation_id = str(uuid.uuid4())
future = self.loop.create_future()
self.futures[correlation_id] = future
request_data = {
"method": method,
"args": args or {},
}
message = Message(
body=json.dumps(request_data).encode('utf-8'),
content_type='application/json',
correlation_id=correlation_id,
reply_to=self.RESPONSE_QUEUE_NAME,
)
try:
await self.exchange.publish(message, routing_key=service)
logging_info(f"Sent request to {service}.{method}: {args}")
# Wait for response
response = await future
# Process response
if response.get("type") == "OK":
return response.get("result", [])
else:
raise UserManagementServiceException(
response.get("exception", "unknown_error"),
response.get("message", "Unknown error"),
response.get("additional_info", {})
)
except asyncio.CancelledError:
raise
except UserManagementServiceException:
raise
except Exception as e:
logging_error(f"Error calling service {service}.{method}: {e}")
raise
finally:
self.futures.pop(correlation_id, None)
async def validate_token(self, token: str) -> Dict[str, Any]:
"""
Validate a JWT token.
Args:
token: JWT token to validate
Returns:
Token information
"""
return await self.call_service(
service=self.TOKEN_SERVICE,
method="validateToken",
args={"token": token}
)
async def query_user_by_id(self, user_id: str) -> Dict[str, Any]:
"""
Query user information by user ID.
Args:
user_id: User ID
Returns:
User information
"""
return await self.call_service(
service=self.USER_SERVICE,
method="queryUserById",
args={"userId": user_id}
)
async def list_users_in_group(self, group_id: str) -> List[Dict[str, Any]]:
"""
List users in a group.
Args:
group_id: Group ID
Returns:
List of users
"""
return await self.call_service(
service=self.GROUP_SERVICE,
method="listUsersInGroup",
args={"groupId": group_id}
)
amqp/router/route_database.py
+11 -34
View File
@@ -1,7 +1,7 @@
import logging
import re
from typing import Optional, Set
from typing import Set
from amqp.adapter.logging_utils import logging_info
from amqp.model.model import AMQRoute
from amqp.router.utils import sanitize_as_rabbitmq_name
@@ -14,30 +14,14 @@ class RouteDatabase:
return self.routes
def pattern(self, routing_key: str) -> re.Pattern:
breaker = False
tokens = routing_key.split(".")
counter = len(tokens)
regex_parts = []
for token in tokens:
counter -= 1
if not breaker:
if token == "":
res = "[^\\.]*"
elif token == "#":
breaker = True
res = ".*$"
else:
res = token
if counter > 0 and not breaker:
res += "\\."
regex_parts.append(res)
else:
regex_parts.append("")
regex = "".join(filter(lambda x: x != "", regex_parts))
if not routing_key:
return re.compile(".*")
regex = r"\.".join(
[
r".*" if token == "" else r"#" if token == "#" else token
for token in routing_key.split(".")
]
)
if routing_key.endswith("."):
regex += r"\."
return re.compile(regex)
def matches(self, routing_key_from_route: str, routing_key_from_message: str) -> bool:
@@ -49,19 +33,12 @@ class RouteDatabase:
def find_route(self, domain: str, path: str) -> AMQRoute | None:
routing_key = sanitize_as_rabbitmq_name(f"{domain}.{path}")
logging_info(f"findRoute key:{routing_key} in:{self.wrap_routes()}")
logging.info(f" [DBG] findRoute key:{routing_key} in:{self.wrap_routes()}")
for route in self.routes:
if self.matches(route.key, routing_key):
return route
return None
def find_route_by_service(self, service_name: str) -> Optional[AMQRoute]:
logging_info(f"findRoute service_name:{service_name} in:{self.wrap_routes()}")
for route in self.routes:
if route.component_name == service_name:
return route
return None
def add_route(self, route: AMQRoute) -> bool:
if route not in self.routes:
self.routes.add(route)
amqp/router/router_base.py
+17 -29
View File
@@ -1,8 +1,8 @@
from typing import Callable, List, Optional, Set
import logging
from typing import Callable, List, Set
from amqp.adapter.amq_route_factory import NULL_ROUTE, AMQRouteFactory
from amqp.adapter.logging_utils import logging_debug, logging_error, logging_info
from amqp.model.model import AMQRoute, DataMessage
from amqp.adapter.amq_route_factory import AMQRouteFactory, NULL_ROUTE
from amqp.model.model import AMQRoute, AMQMessage
from amqp.router.route_database import RouteDatabase
from amqp.router.utils import sanitize_as_rabbitmq_name
@@ -26,8 +26,8 @@ class RouterBase:
self.route_added_notifier: Callable[[AMQRoute], None] | None = None
self.route_removed_notifier: Callable[[AMQRoute], None] | None = None
def get_routing_key(self, message: DataMessage) -> str:
return sanitize_as_rabbitmq_name(f"{message.domain()}.{message.path()}")
def get_routing_key(self, message: AMQMessage) -> str:
return sanitize_as_rabbitmq_name(f"{message.domain}.{message.path}")
def wrap_routes(self) -> str:
return self.route_database.wrap_routes()
@@ -36,7 +36,7 @@ class RouterBase:
return ",".join(str(route) for route in self.own_routes)
def unwrap_route_list(self, route_string: str) -> List[AMQRoute]:
logging_debug("RouterMaster.unwrapRoutes, route(s): %s", route_string)
logging.info(" [DBG] RouterMaster.unwrapRoutes, route(s): %s", route_string)
routes = [AMQRouteFactory.from_string(route_str) for route_str in route_string.split(",")]
return [route for route in routes if route != NULL_ROUTE]
@@ -46,38 +46,26 @@ class RouterBase:
def find_route(self, domain: str, path: str) -> AMQRoute | None:
return self.route_database.find_route(domain, path)
def find_route_by_message(self, message: DataMessage) -> AMQRoute | None:
return self.route_database.find_route(message.domain(), message.path())
def find_route_by_service(self, service_name: str) -> Optional[AMQRoute]:
_route: Optional[AMQRoute] = self.route_database.find_route_by_service(service_name)
if _route is None:
for route in self.own_routes:
if route.component_name == service_name:
_route = route
return _route
def find_route_by_message(self, message: AMQMessage) -> AMQRoute | None:
return self.route_database.find_route(message.domain, message.path)
def add_routes(self, message: str) -> None:
try:
amq_route_list = self.unwrap_route_list(message)
logging_info(" [DBG] RouterMaster.addRoute [%d]: %s", len(amq_route_list), message)
logging.info(" [DBG] RouterMaster.addRoute [%d]: %s", len(amq_route_list), message)
for route in amq_route_list:
logging_info("RouterMaster.addRoute, adding route: %s", route)
logging.info(" [*] RouterMaster.addRoute, adding route: %s", route)
if not route.exchange and not route.queue and not route.key:
logging_error(
"Cannot setup AMQ Exchange for '%s', at least one of "
"Exchange, Queue and/or Key must be present in the routing string. "
"Expected format is '%s'",
route,
AMQRoute.FORMAT,
)
logging.error("Cannot setup AMQ Exchange for '%s', at least one of "
"Exchange, Queue and/or Key must be present in the routing string. "
"Expected format is '%s'", route, AMQRoute.FORMAT)
else:
if route.exchange != DLQ_EXCHANGE:
is_new_route = self.route_database.add_route(route)
if is_new_route and self.route_added_notifier:
self.route_added_notifier(route)
except Exception as err:
logging_error("add_routes error: %s", err)
logging.error(" [E] add_routes error: %s", err)
def set_route_added_notifier(self, callback: Callable[[AMQRoute], None]):
self.route_added_notifier = callback
@@ -86,9 +74,9 @@ class RouterBase:
try:
to_remove = [AMQRouteFactory.from_string(route_str) for route_str in message.split(",")]
removed_count = sum(1 for route in to_remove if self.remove_route(route))
logging_info(f"RouterMaster.removeRoutes(): {message}, removed={removed_count}")
print(f" [*] RouterMaster.removeRoutes(): {message}, removed={removed_count}")
except IOError as err:
logging_error(f"Can't remove route(s): {err}")
print(f" [E] Can't remove route(s): {err}")
def remove_route(self, route: AMQRoute) -> bool:
is_route_removed = self.route_database.remove_route(route)
amqp/router/router_consumer.py
+83 -140
View File
@@ -1,200 +1,143 @@
import logging
from typing import Set
from aio_pika.abc import (
AbstractIncomingMessage,
AbstractRobustChannel,
AbstractRobustQueue,
)
from pika import spec
from pika.adapters.blocking_connection import BlockingChannel
from amqp.adapter.amq_route_factory import AMQRouteFactory
from amqp.adapter.logging_utils import (
logging_debug,
logging_error,
logging_info,
logging_warning,
)
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import AMQRoute, ServiceMessage
from amqp.model.service_message_type import ServiceMessageType
from amqp.router.router_base import (
ANY_RECIPIENT,
CM_C_REQ_QUEUE,
CM_REQUEST_EXCHANGE,
CM_RESPONSE_EXCHANGE,
)
from amqp.model.model import CleverMicroServiceMessage, ServiceMessageType, AMQRoute
from amqp.router.router_base import CM_C_REQ_QUEUE, CM_REQUEST_EXCHANGE, CM_RESPONSE_EXCHANGE, ANY_RECIPIENT
from amqp.router.router_producer import RouterProducer
class RouterConsumer(RouterProducer):
"""
* Contains Consumer logic. The logic here:
* Sets a listener on CM_C_REQ_QUEUE queue to receive routing data requests
* Upon receipt of routing data request, responds on CM_RESPONSE_EXCHANGE with local routing data.
* It also proactively pushes the local routing data to CM_RESPONSE_EXCHANGE on startup to
* notify the other (excluding self) Producers about its presence.
*
* Consumer is also Producer (Consumer sits alongside the CM Service, and anytime the Service needs
* to communicate with other Service, it can do so via the Adapter). Therefore, the Producer
* capability is also required.
* Contains Consumer logic. The logic here:
* Sets a listener on CM_C_REQ_QUEUE queue to receive routing data requests
* Upon receipt of routing data request, responds on CM_RESPONSE_EXCHANGE with local routing data.
* It also proactively pushes the local routing data to CM_RESPONSE_EXCHANGE on startup to
* notify the other (excluding self) Producers about its presence.
*
* Consumer is also Producer (Consumer sits alongside the CM Service, and anytime the Service needs
* to communicate with other Service, it can do so via the Adapter). Therefore, the Producer
* capability is also required.
"""
def __init__(self, configuration: AMQConfiguration):
super().__init__(configuration)
self.amqRouteFactory = AMQRouteFactory()
logging_info("RouterConsumer.<init>")
logging.info(" [*] RouterConsumer.<init>")
async def init_routing_paths_consumer(self, _channel: AbstractRobustChannel):
def init_routing_paths_consumer(self, _channel: BlockingChannel):
"""
* Initialize router according to Consumer mode.
* Initialize router according to Consumer mode.
"""
logging_info(
"RouterConsumer.initConsumerRouter, channel open: %s",
self.is_open(_channel),
)
self.init_routing_paths(_channel)
logging.info(" [*] RouterConsumer.initConsumerRouter, channel open::", self.is_open(_channel))
if self.is_open(_channel):
try:
# 1. declare a Queue to receive the RoutingData requests
_cm_request_queue: str = (
self.amq_configuration.amq_adapter.adapter_prefix() + CM_C_REQ_QUEUE
)
_queue: AbstractRobustQueue = await self.channel.declare_queue(
name=_cm_request_queue,
durable=True,
exclusive=False,
auto_delete=False,
)
# 1a. bind queue to CM_REQUEST_EXCHANGE w/o any conditions (empty routing key)
await _queue.bind(exchange=CM_REQUEST_EXCHANGE, routing_key="")
# 1. declare a Queue that gives the RoutingDAta requests
cm_request_queue: str = self.amq_configuration.amq_adapter.adapter_prefix() + CM_C_REQ_QUEUE
self.channel.queue_declare(cm_request_queue, durable=True, exclusive=False, auto_delete=False)
# 1a. bind queue to CM_RESPONSE_EXCHANGE w/o any conditions (empty routing key)
self.channel.queue_bind(queue=cm_request_queue, exchange=CM_REQUEST_EXCHANGE, routing_key="")
# 1b. Service adapter mode - listens on Request queue and replies to Response queue
_c_tag = await _queue.consume(
callback=self.handle_routing_data_request_message, no_ack=False
)
logging_info(
" [%s] RouterConsumer listens for RouteDataRequests on: %s",
_c_tag,
_cm_request_queue,
)
logging.info(" [*] RouterConsumer listens for RouteDataRequests on:", cm_request_queue)
self.channel.basic_consume(queue=cm_request_queue,
auto_ack=False,
on_message_callback=self.handle_routing_data_request_message)
except Exception as ioe:
logging_error("RouterConsumer FATAL: cannot initialize, %s", ioe)
logging.error(" [E] RouterConsumer FATAL: cannot initialize, %s", ioe)
else:
logging_error("RouterConsumer FATAL: cannot initialize, RabbitMQ channel is not open.")
logging.error(" [E] RouterConsumer FATAL: cannot initialize, RabbitMQ channel is not open.")
"""
* ServiceMessageType.ROUTING_DATA_REQUEST handler listening on queue 'CleverMicroRequest'
"""
async def handle_routing_data_request_message(self, message: AbstractIncomingMessage):
def handle_routing_data_request_message(self,
ch: BlockingChannel,
method: spec.Basic.Deliver,
properties: spec.BasicProperties,
body: bytes):
# some debug statements, TODO - remove for prod release
delivery_tag = message.delivery_tag
debug = "Delivery.Properties = " + message.properties.__str__()
logging_debug(
" ******[%s] (ROUTING_DATA_REQ) tag: %s, env: %s, props: %s",
delivery_tag,
message.consumer_tag,
message.properties,
debug,
)
delivery_tag = method.delivery_tag
debug = "Delivery.Properties = " + properties.__str__()
logging.info(" [DBG] ******[%] (ROUTING_DATA_REQ) tag: %s, env: %s, props: %s",
delivery_tag, method.consumer_tag, method.get_properties(), debug)
await message.ack(False)
self.channel.basic_ack(delivery_tag, False)
cm_message: ServiceMessage = self.service_message_factory.from_bytes(message.body)
message: CleverMicroServiceMessage = self.service_message_factory.from_bytes(body)
route_mapping: str = self.wrap_own_routes()
logging_debug(
"******[%s] (ROUTING_DATA_REQ) msg=%s, routeMap=[%s]",
delivery_tag,
self.service_message_factory.to_string(cm_message),
route_mapping,
)
logging.info(" [DBG] ******[%s] (ROUTING_DATA_REQ) msg=%, routeMap=[{]",
delivery_tag, self.service_message_factory.to_string(message), route_mapping)
# Reply only if self adapter has own routeMapping (e.g. API-gateway-side adapter has none)
if (
cm_message.is_valid()
and cm_message.message_type == ServiceMessageType.ROUTING_DATA_REQUEST
and route_mapping
):
if cm_message.reply_to != self.amq_configuration.amq_adapter.service_name:
try:
service_message: ServiceMessage = self.service_message_factory.of(
ServiceMessageType.ROUTING_DATA,
route_mapping,
cm_message.reply_to,
)
await self.publish_service_message(service_message, self.cm_response_exchange)
except Exception as err:
logging_error(
"******[%s] (ROUTING_DATA_REQ) Response to queue: %s failed: %s",
delivery_tag,
CM_RESPONSE_EXCHANGE,
err,
)
else:
logging_info(
"******[%s] (ROUTING_DATA_REQ) [%s] Ignore request originated by ourselves.",
delivery_tag,
str(message.body),
if message.is_valid() and message.message_type == ServiceMessageType.ROUTING_DATA_REQUEST and route_mapping:
try:
service_message: CleverMicroServiceMessage = self.service_message_factory.of(
ServiceMessageType.ROUTING_DATA, route_mapping, message.reply_to
)
else:
logging_error(
"******[%s] (ROUTING_DATA_REQ) [%s] produces invalid or no CM msg type.",
delivery_tag,
str(message.body),
)
self.publish_service_message(service_message, CM_RESPONSE_EXCHANGE)
except Exception as err:
logging.error(" [E] ******[%s] (ROUTING_DATA_REQ) Response to queue: %s failed: %s",
delivery_tag, CM_RESPONSE_EXCHANGE, err)
async def register_route(self, route: AMQRoute):
else:
logging.error(" [E] ******[%s] (ROUTING_DATA_REQ) [%s] produces invalid or no CM msg type.",
delivery_tag, str(body))
def register_route(self, route: AMQRoute):
"""
* Called in Consumer (Service Adapter) context. Registers the routing data for self Service
* with the Master router by sending own routing data to Master via CleverMicroResponse queue.
* :param route: route to register with Master
* Called in Consumer (Service Adapter) context. Registers the routing data for self Service
* with the Master router by sending own routing data to Master via CleverMicroResponse queue.
* :param route: route to register with Master
"""
try:
if self.is_open(self.channel) and route.queue:
serviceMessage: ServiceMessage = self.service_message_factory.of(
serviceMessage: CleverMicroServiceMessage = self.service_message_factory.of(
ServiceMessageType.ROUTING_DATA, route.as_string(), ANY_RECIPIENT
)
await self.publish_service_message(serviceMessage, self.cm_response_exchange)
self.publish_service_message(serviceMessage, CM_RESPONSE_EXCHANGE)
self.add_own_route(route)
logging_info("RouterConsumer registered Route: %s", route)
logging.info(" [DBG] RouterConsumer registered Route::", route)
else:
logging_warning(
"RouterConsumer couldn't register route data: %s channel isn't open.",
CM_RESPONSE_EXCHANGE,
)
logging.warning(" [W] RouterConsumer couldn't register route data: %s channel isn't open.",
CM_RESPONSE_EXCHANGE)
except Exception as ioe:
logging_error("RouterConsumer failed register route data: %s", ioe)
logging.error(" [E] RouterConsumer failed register route data: %s", ioe)
async def remove_route_on_cleanup(self, route: AMQRoute) -> bool:
def remove_route_on_cleanup(self, route: AMQRoute) -> bool:
"""
* Unregister / remove route from the list
* :param route: route to remove
* :return result of remove operation
* Unregister / remove route from the list
* :param route: route to remove
* :return result of remove operation
"""
logging_info("RouterConsumer.removeRoute, route: %s", route.as_string())
logging.info(" [DBG] RouterConsumer.removeRoute, route::", route.as_string())
try:
service_message: ServiceMessage = self.service_message_factory.of(
ServiceMessageType.ROUTING_DATA_REMOVE, route.as_string(), ANY_RECIPIENT
service_message: CleverMicroServiceMessage = self.service_message_factory.of(
ServiceMessageType.ROUTING_DATA_REMOVE,
route.as_string(),
ANY_RECIPIENT
)
if not await self.publish_service_message(service_message, self.cm_response_exchange):
logging_warning(
"RouterConsumer couldn't remove current_availability route: %s, channel not open.",
route,
)
if not self.publish_service_message(service_message, CM_RESPONSE_EXCHANGE):
logging.warning(" [W] RouterConsumer couldn't remove current route: %s, channel not open.",
route)
except Exception as ioe:
logging_error("RouterConsumer failed remove current_availability route: %s", ioe)
logging.error(" [E] RouterConsumer failed remove current route: %s", ioe)
result: bool = self.remove_route(route)
return result
def cleanup(self):
"""
* Cleanup - de-register routes with master.
* Cleanup - de-register routes with master.
"""
routes: Set[AMQRoute] = self.get_routes()
logging_info("RouterConsumer.cleanup() [%d]: %s", len(routes), self.wrap_routes())
not_removed = sum(
1 for r in (self.remove_route_on_cleanup(route) for route in routes) if not r
)
logging.info(" [DBG] RouterConsumer.cleanup() [%d]: %s", len(routes), self.wrap_routes())
not_removed = sum(1 for r in (self.remove_route_on_cleanup(route) for route in routes) if not r)
if not_removed > 0:
logging_warning(
"%d route(s) not removed. self may lead to failed delivery on self route",
not_removed,
)
logging.info(" [W] %d route(s) not removed. self may lead to failed delivery on self route", not_removed)
amqp/router/router_producer.py
+129 -164
View File
@@ -1,225 +1,190 @@
"""
* Maps the request to RabbitMQ exchange. Handles the service discovery logic.
*
* Router can be either in 'Master' or 'Service (Consumer)' modes. 'Master' mode is the outward
* facing side that receives the requests and routes it to queue associated with requested service.
* 'Service' mode is consumer side for given service, which receives the message from AMQ and
* forwards it to the application.
"""
from aio_pika import Message
from aio_pika.abc import (
AbstractIncomingMessage,
AbstractRobustChannel,
AbstractRobustConnection,
AbstractRobustExchange,
AbstractRobustQueue,
)
* Maps the request to RabbitMQ exchange. Handles the service discovery logic.
*
* Router can be either in 'Master' or 'Service (Consumer)' modes. 'Master' mode is the outward
* facing side that receives the requests and routes it to queue associated with requested service.
* 'Service' mode is consumer side for given service, which receives the message from AMQ and
* forwards it to the application.
"""
import logging
import os
import pika
from pika import BasicProperties, spec
from pika.adapters.blocking_connection import BlockingChannel
from pika.exchange_type import ExchangeType
from amqp.adapter.logging_utils import (
logging_debug,
logging_error,
logging_info,
logging_warning,
)
from amqp.adapter.service_message_factory import ServiceMessageFactory
from amqp.adapter.service_message_factory import CleverMicroServiceMessageFactory
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import ServiceMessage
from amqp.model.service_message_type import ServiceMessageType
from amqp.router.router_base import (
ANY_RECIPIENT,
CM_C_AMQ_QUEUE,
CM_P_REPLY_TO,
CM_P_RESP_QUEUE,
CM_REQUEST_EXCHANGE,
CM_RESPONSE_EXCHANGE,
RouterBase,
)
from amqp.model.model import CleverMicroServiceMessage, ServiceMessageType
from amqp.router.router_base import RouterBase, CM_REQUEST_EXCHANGE, CM_RESPONSE_EXCHANGE, CM_C_AMQ_QUEUE, \
CM_P_RESP_QUEUE, CM_P_REPLY_TO, ANY_RECIPIENT
class RouterProducer(RouterBase):
def __init__(self, configuration: AMQConfiguration):
"""
* Initialize router - need to supply the config values and RabbitMQ channel.
* :param configuration: adapter's configuration reference.
"""
* Initialize router - need to supply the config values and RabbitMQ channel.
* :param configuration: adapter's configuration reference.
"""
super().__init__()
self.connection: AbstractRobustConnection | None = None
self.service_message_factory = ServiceMessageFactory(configuration.amq_adapter.service_name)
self.connection = None
self.service_message_factory = CleverMicroServiceMessageFactory(configuration.amq_adapter.service_name)
self.amq_configuration: AMQConfiguration = configuration
self.channel: AbstractRobustChannel | None = None
self.cm_request_exchange: AbstractRobustExchange | None = None
self.cm_response_exchange: AbstractRobustExchange | None = None
self.channel: pika.adapters.blocking_connection.BlockingChannel | None = None
logging_info(
"RouterProducer.<init>: %s, route: %s",
self.amq_configuration.amq_adapter.service_name,
self.amq_configuration.amq_adapter.route_mapping,
logging.info(f" [*] RouterProducer.<init>: %s, route: %s",
self.amq_configuration.amq_adapter.service_name,
self.amq_configuration.amq_adapter.route_mapping)
def amq_consumer(self):
credentials = pika.PlainCredentials(
os.environ['RABBIT_MQ_USER'], os.environ['RABBIT_MQ_PASSWORD']
)
self.connection = pika.BlockingConnection(
pika.ConnectionParameters(host='localhost', credentials=credentials)
)
self.channel = self.connection.channel()
self.channel.start_consuming()
async def init_internal_routing_paths(self, channel: AbstractRobustChannel | None):
"""
* Initialize internal routes for ServiceMessages.
* The logic here sets a producer and listener on CleverMicroRequest/CleverMicroResponse queues
* created to enable the service discovery.
* The Producer mode sends out routing data request on CleverMicroRequest and listens
* on CleverMicroResponse for any service responding with its routing data.
def init_routing_paths(self, channel: pika.adapters.blocking_connection.BlockingChannel | None):
"""
* Initialize router according to Producer mode.
* The logic here sets a producer and listener on CleverMicroRequest/CleverMicroResponse queues
* created to enable the service discovery.
* The Producer mode sends out routing data request on CleverMicroRequest and listens
* on CleverMicroResponse for any service responding with its routing data.
"""
self.channel = channel
logging_info(
"RouterProducer.initRoutingPaths, channel open: %s",
self.is_open(self.channel),
)
logging.info(" [*] RouterProducer.initRoutingPaths, channel open: %s", self.is_open(self.channel))
if self.is_open(self.channel):
try:
# **** set up the service discovery internal exchanges ****
# 1. declare RequestExchange to where to publish the RoutingData requests
self.cm_request_exchange = await self.channel.declare_exchange(
name=CM_REQUEST_EXCHANGE, type=ExchangeType.fanout
)
self.channel.exchange_declare(exchange=CM_REQUEST_EXCHANGE, exchange_type=ExchangeType.fanout)
#
# 2. declare ResponseExchange where to publish current_availability RoutingData (response to
# 2. declare ResponseExchange where to publish current RoutingData (response to
# a RoutingData request)
self.cm_response_exchange = await self.channel.declare_exchange(
name=CM_RESPONSE_EXCHANGE, type=ExchangeType.fanout
)
self.channel.exchange_declare(exchange=CM_RESPONSE_EXCHANGE, exchange_type=ExchangeType.fanout)
# 2a. declare a Response queue for Routing Data replies from Services
_cm_response_queue: str = self.get_response_queue_name()
_response_queue: AbstractRobustQueue = await self.channel.declare_queue(
name=_cm_response_queue,
durable=True,
exclusive=False,
auto_delete=False,
arguments={},
)
cm_response_queue: str = self.get_response_queue_name()
self.channel.queue_declare(cm_response_queue, durable=True, exclusive=False, auto_delete=False,
arguments={})
# 2b. bind queue to CM_RESPONSE_EXCHANGE w/o any conditions (empty routing key)
await _response_queue.bind(exchange=CM_RESPONSE_EXCHANGE, routing_key="#")
self.channel.queue_bind(queue=cm_response_queue, exchange=CM_RESPONSE_EXCHANGE, routing_key="#")
# 2c. listen for incoming RoutingData messages
_c_tag = await _response_queue.consume(
callback=self.handle_routing_data_message, no_ack=False
)
logging_info(
" [%s] Routing master listens for Route updates on %s",
_c_tag,
_cm_response_queue,
)
self.channel.basic_consume(
queue=cm_response_queue,
auto_ack=False,
on_message_callback=self.handle_routing_data_message)
self.channel.start_consuming()
logging.info(" [*] Routing master listens for Route updates on %s", cm_response_queue)
except Exception as ioe:
logging_error("RouterProducer FATAL: cannot initialize, %s", ioe)
logging.error(" [E] RouterProducer FATAL: cannot initialize, %s", ioe)
else:
logging_error("RouterProducer FATAL: cannot initialize, RabbitMQ channel is not open.")
logging.error(" [E] RouterProducer FATAL: cannot initialize, RabbitMQ channel is not open.")
async def handle_routing_data_message(self, message: AbstractIncomingMessage):
def handle_routing_data_message(self,
ch: BlockingChannel,
method: spec.Basic.Deliver,
properties: spec.BasicProperties,
body: bytes):
"""
* Handler for CLEVER_MICRO_RESPONSE queue messages. Used in Master mode to handle receipt of
* routing data, invokes addRoute or removeRoute depending on the type of the message.
* Handler for CLEVER_MICRO_RESPONSE queue messages. Used in Master mode to handle receipt of
* routing data, invokes addRoute or removeRoute depending on the type of the message.
"""
print(f" [x] Received {body}, m={method}, p={properties}, ch={ch}")
"""
logging_info(
"[%s] Received ROUTING DATA: [%s], msgId=%s",
message.delivery_tag,
message.body,
message.message_id,
)
await message.ack(multiple=False)
cm_message: ServiceMessage = self.service_message_factory.from_bytes(message.body)
logging.info(" [DBG] ******[%s] ROUTING_DATA %s >>> %s",
delivery.getEnvelope().getDeliveryTag(),
delivery.getEnvelope().toString(), str(delivery.getBody()))
"""
# ACK message now so that it is not being redelivered
self.channel.basic_ack(method.delivery_tag, False)
message: CleverMicroServiceMessage = self.service_message_factory.from_bytes(body)
# ignore the message if it comes from ourselves (even if from different instance)
logging_debug(
"******[%s] ROUTING_DATA, msg=%s, routeMap=[%s] sn=%s, replyTo=%s",
message.delivery_tag,
self.service_message_factory.to_string(cm_message),
self.amq_configuration.amq_adapter.route_mapping,
self.amq_configuration.amq_adapter.service_name,
cm_message.reply_to,
)
if (
cm_message.is_valid()
and not self.amq_configuration.amq_adapter.service_name == cm_message.reply_to
):
if cm_message.message_type == ServiceMessageType.ROUTING_DATA:
self.add_routes(cm_message.message)
if cm_message.message_type == ServiceMessageType.ROUTING_DATA_REMOVE:
self.remove_routes(cm_message.message)
logging.info(" [DBG] ******[%s] ROUTING_DATA, msg=%s, routeMap=[{] sn={, replyTo={",
method.delivery_tag,
self.service_message_factory.to_string(message),
self.amq_configuration.amq_adapter.route_mapping,
self.amq_configuration.amq_adapter.service_name,
message.reply_to)
if message.is_valid() and not self.amq_configuration.amq_adapter.service_name == message.reply_to:
if message.message_type == ServiceMessageType.ROUTING_DATA:
self.add_routes(message.message)
if message.message_type == ServiceMessageType.ROUTING_DATA_REMOVE:
self.remove_routes(message.message)
async def request_routes_from_adapters(self):
def request_routes_from_adapters(self):
"""
* Send a broadcast message prompting all listening adapters to reply with own routing data.
"""
logging_info(
"RouterProducer.requestRoutesFromAdapters, queue name: %s",
CM_REQUEST_EXCHANGE,
)
* Send a broadcast message prompting all listening adapters to reply with own routing data.
"""
logging.info(" [*] RouterProducer.requestRoutesFromAdapters, queue name: %s", CM_REQUEST_EXCHANGE)
try:
serviceMessage: ServiceMessage = self.service_message_factory.of(
serviceMessage: CleverMicroServiceMessage = self.service_message_factory.of(
ServiceMessageType.ROUTING_DATA_REQUEST, "", ANY_RECIPIENT
)
if not await self.publish_service_message(serviceMessage, self.cm_request_exchange):
logging_warning(
"RouterProducer could not request route data: %s channel is not open.",
CM_REQUEST_EXCHANGE,
)
if not self.publish_service_message(serviceMessage, CM_REQUEST_EXCHANGE):
logging.warning(" [E] RouterProducer could not request route data: %s channel is not open.",
CM_REQUEST_EXCHANGE)
except Exception as ioe:
logging_error("RouterProducer failed request routing data: %s", ioe)
logging.error(" [E] RouterProducer failed request routing data: %s", ioe)
async def publish_service_message(
self, message: ServiceMessage, exchange: AbstractRobustExchange
) -> bool:
"""
* Publishes a service message to the service queue and logs success log entry.
*
* :param message: Service Message to be sent
* :param exchange: target exchange name where to publish the message
* :return True if channel is open, False otherwise
* @throws IOException if something else goes wrong
BASIC: BasicProperties = BasicProperties(content_type="application/octet-stream",
content_encoding=None,
headers=None,
delivery_mode=1,
priority=0,
correlation_id=None)
def publish_service_message(self, message: CleverMicroServiceMessage, exchange_name: str) -> bool:
"""
* Publishes a service message to the service queue and logs success log entry.
*
* :param message: Service Message to be sent
* :param exchangeName: target exchange name where to publish the message
* :param queueName target: queue name (Optional, purpose is to get the queue size only)
* :return True if channel is open, False otherwise
* @throws IOException if something else goes wrong
"""
if self.is_open(self.channel):
binary_content: bytes = self.service_message_factory.serialize(message)
pika_message: Message = Message(
body=binary_content,
content_encoding="utf-8",
delivery_mode=2,
content_type="application/octet-stream",
headers=None,
priority=0,
correlation_id=None,
)
await exchange.publish(message=pika_message, routing_key="")
logging_info(
"Service Message Published to %s, msg: %s",
exchange.name,
str(binary_content),
)
self.channel.basic_publish(exchange=exchange_name,
routing_key="", # empty routing key
properties=self.BASIC,
body=binary_content)
logging.info(" [x] Service Message Sent to %s, msg: %s", exchange_name, str(binary_content))
return True
return False
def get_reply_to_queue_name(self) -> str:
"""
* self should create a unique name for self process and also when there's several instances.
* @return unique reply-to queue name.
"""
* self should create a unique name for self process and also when there's several instances.
* @return unique reply-to queue name.
"""
return self.amq_configuration.amq_adapter.adapter_prefix() + CM_P_REPLY_TO
def get_response_queue_name(self) -> str:
"""
* self should create a unique name for self process and also when there's several instances.
* @return unique reply-to queue name.
"""
* self should create a unique name for self process and also when there's several instances.
* @return unique reply-to queue name.
"""
return self.amq_configuration.amq_adapter.adapter_prefix() + CM_P_RESP_QUEUE
def get_consume_queue_name(self) -> str:
"""
* self should create a unique name for self process and also when there's several instances.
* @return unique reply-to queue name.
"""
* self should create a unique name for self process and also when there's several instances.
* @return unique reply-to queue name.
"""
return self.amq_configuration.amq_adapter.adapter_prefix() + CM_C_AMQ_QUEUE
def is_open(self, _channel: AbstractRobustChannel) -> bool:
def is_open(self, _channel: pika.adapters.blocking_connection.BlockingChannel) -> bool:
"""
* Test if the channel is usable (opened).
* :param _channel: the channel
* :return True if the channel is opened.
"""
return _channel is not None and not _channel.is_closed
* Test if the channel is usable (opened).
* :param _channel: the channel
* :return True if the channel is opened.
"""
return _channel is not None and _channel.is_open
amqp/router/serializer.py
+61
View File
@@ -0,0 +1,61 @@
import struct
from io import BytesIO
from typing import List, Union, Dict
def long_to_bytes(x: int) -> bytes:
return struct.pack('q', x)
def bytes_to_long(b: bytes) -> int:
return struct.unpack('q', b)[0]
def read_long(bis: BytesIO) -> int:
try:
return struct.unpack('q', bis.read(8))[0]
except IOError:
return 0
def object_or_list_as_string(value: Union[str, List[str]]) -> str:
if isinstance(value, list):
return f"[{','.join(value)}]"
return str(value)
def map_as_string(map_: Dict[str, str]) -> str:
return ','.join(f"{k}={object_or_list_as_string(v)}" for k, v in map_.items())
def map_with_list_as_string(map_: Dict[str, List[str]]) -> str:
return ','.join(f"{k}={object_or_list_as_string(v)}" for k, v in map_.items())
def parse_map_string(input_str: str) -> Dict[str, str]:
map_ = {}
if input_str.startswith('{') and input_str.endswith('}'):
for entry in input_str[1:-1].split(','):
if len(entry) > 1:
key, value = entry.split('=')
map_[key] = value
return map_
def csv_as_list(input_str: str) -> List[str]:
return [item.strip() for item in input_str.strip('[]').split(',')]
def add_to_map(map_: Dict[str, List[str]], token: str) -> None:
eq_idx = token.index('=')
if eq_idx > 0:
key = token[:eq_idx]
map_[key] = csv_as_list(token[eq_idx+1:])
def parse_map_list_string(input_str: str) -> Dict[str, List[str]]:
map_ = {}
if input_str.startswith('{') and input_str.endswith('}'):
for token in input_str[1:-1].split('],'):
add_to_map(map_, token)
return map_
amqp/router/test_route_database.py
+24
View File
@@ -0,0 +1,24 @@
from unittest import TestCase
class TestRouteDatabase(TestCase):
def test_get_routes(self):
self.fail()
def test_pattern(self):
self.fail()
def test_matches(self):
self.fail()
def test_wrap_routes(self):
self.fail()
def test_find_route(self):
self.fail()
def test_add_route(self):
self.fail()
def test_remove_route(self):
self.fail()
amqp/router/test_router_base.py
+39
View File
@@ -0,0 +1,39 @@
from unittest import TestCase
class TestRouterBase(TestCase):
def test_get_routing_key(self):
self.fail()
def test_wrap_routes(self):
self.fail()
def test_wrap_own_routes(self):
self.fail()
def test_unwrap_route_list(self):
self.fail()
def test_get_routes(self):
self.fail()
def test_find_route(self):
self.fail()
def test_find_route_by_message(self):
self.fail()
def test_add_routes(self):
self.fail()
def test_remove_routes(self):
self.fail()
def test_remove_route(self):
self.fail()
def test_add_route(self):
self.fail()
def test_add_own_route(self):
self.fail()
tests/router/test_utils.py → amqp/router/test_utils.py
+1 -2
View File
@@ -10,7 +10,6 @@ class Test(TestCase):
self.assertEqual("aa", sanitize_as_rabbitmq_name("aa&bb/cc"))
self.assertEqual("aa", sanitize_as_rabbitmq_name("aa+bb/cc"))
self.assertEqual("aa", sanitize_as_rabbitmq_name("aa%bb/cc"))
self.assertEqual("aa#bb.cc", sanitize_as_rabbitmq_name("aa#bb/cc"))
self.assertEqual("aa#bb.cc.#", sanitize_as_rabbitmq_name("aa#bb/cc.#"))
self.assertEqual("aa.bb.cc", sanitize_as_rabbitmq_name("aa#bb/cc"))
self.assertEqual("aa", sanitize_as_rabbitmq_name("aa?bb////cc"))
self.assertEqual("aa.bb.cc", sanitize_as_rabbitmq_name("aa///bb////cc"))
amqp/router/utils.py
+4 -32
View File
@@ -1,37 +1,9 @@
import asyncio
import re
pattern = re.compile(r"[^a-zA-Z0-9.#/\-]")
AWAIT_SLEEP = 0.05 # seconds
def sanitize_as_rabbitmq_name(input_str: str) -> str:
pattern = re.compile(r"[^a-zA-Z0-9.#/\-]")
matcher = pattern.search(input_str)
token = input_str[: matcher.start()] if matcher else input_str
return sanitize_as_rabbitmq_domain_name(token)
def sanitize_as_rabbitmq_domain_name(token: str) -> str:
# Step 1: Replace non-alphanumeric characters (except periods) with a period
step1 = re.sub(r"[^A-Za-z0-9.#]", ".", token)
# Step 2: Replace multiple consecutive periods with a single period
step2 = re.sub(r"\.+", ".", step1)
# Step 3: Remove trailing '.' character, if applicable
step_last_char = len(step2) - 1
return step2[:-1] if step_last_char > 0 and step2[step_last_char] == "." else step2
async def await_future(future: asyncio.Future, sleep_time=AWAIT_SLEEP):
"""
Wait for a future to complete.
"""
while not future.done():
await asyncio.sleep(sleep_time)
async def await_result(future: asyncio.Future, sleep_time=AWAIT_SLEEP):
"""
Wait for a future to complete and return its result.
"""
await await_future(future, sleep_time=sleep_time)
return future.result()
token = input_str[:matcher.start()] if matcher else input_str
step1 = re.sub(r"[^A-Za-z0-9]", ".", token)
return re.sub(r"\.+", ".", step1)
amqp/service/AMQ_PyAdapter.log
View File
amqp/service/amq_message_handler.py
-531
View File
@@ -1,531 +0,0 @@
import asyncio
import importlib
import inspect
import os
import threading
import time
from asyncio import AbstractEventLoop, Future
from typing import Any, Dict
from aio_pika import Message
from aio_pika.abc import AbstractIncomingMessage, AbstractRobustExchange
from opentelemetry import trace
from opentelemetry.trace import Tracer, TraceState
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from pika.exchange_type import ExchangeType
from amqp.adapter.backpressure_handler import BackpressureHandler
from amqp.adapter.cleverthis_service_adapter import AMQMessage, CleverThisServiceAdapter
from amqp.adapter.data_message_factory import DataMessageFactory
from amqp.adapter.data_response_factory import DataResponseFactory
from amqp.adapter.file_handler import FileHandler, StreamingFileHandler
from amqp.adapter.logging_utils import logging_debug, logging_error, logging_info
from amqp.adapter.serializer import map_as_string, parse_map_string
from amqp.adapter.service_message_factory import ServiceMessageFactory
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import (
DataMessage,
DataMessageMagicByte,
DataResponse,
MultipartDataMessage,
)
from amqp.rabbitmq.rabbit_mq_client import RabbitMQClient
from amqp.router.utils import sanitize_as_rabbitmq_name
def collect_trace_info():
trace_map = {}
TraceContextTextMapPropagator().inject(
carrier=trace_map, context=trace.context_api.get_current()
)
return trace_map
def get_default_trace_state():
return TraceState(entries={})
def set_text(carrier, key, value):
carrier[key] = value
async def invoke_command(
package_name: str, class_name: str, function_name: str, msg_id: str = "", kwargs=None
) -> Any:
if kwargs is None:
kwargs = {}
try:
# 1. Dynamically import the package
logging_info(f"Attempting to import package: {package_name}")
package = importlib.import_module(package_name)
logging_info(f"Package '{package_name}' imported successfully.")
target_callable = None
if class_name == "__global__":
# 2. Handle global function invocation
logging_info(f"Attempting to find global function: {function_name} in {package_name}")
target_callable = getattr(package, function_name)
logging_info(f"Global function '{function_name}' found.")
else:
# 3. Handle class method invocation
logging_info(f"Attempting to find class: {class_name} in {package_name}")
target_class = getattr(package, class_name)
logging_info(f"Class '{class_name}' found. Instantiating...")
# Instantiate the class. Assuming a default constructor or one that doesn't
# require arguments from kwargs for initialization.
# If the class __init__ needs specific arguments, this part might need adjustment
# based on how those arguments are provided.
instance = target_class()
logging_info(f"Instance of '{class_name}' created.")
logging_info(f"Attempting to find method: {function_name} in class {class_name}")
target_callable = getattr(instance, function_name)
logging_info(f"Method '{function_name}' found.")
# 4. Invoke the function/method
logging_info(f"Invoking '{function_name}' with arguments: {kwargs}")
if inspect.iscoroutinefunction(target_callable):
result = await target_callable(**kwargs)
logging_info(f"Asynchronous function '{function_name}' invoked. Result: {result}")
else:
result = target_callable(**kwargs)
logging_info(f"Synchronous function '{function_name}' invoked. Result: {result}")
response: DataResponse = DataResponseFactory.of(
id=msg_id,
response_code=200,
content_type="text/plain",
body=result.encode("utf-8"),
trace_info=collect_trace_info(),
error=None,
error_cause=None,
)
response._magic = DataMessageMagicByte.RPC_REQUEST.value
return response
except ModuleNotFoundError as e:
logging_error(f"Error: Package '{package_name}' not found. {e}")
raise
except AttributeError as e:
logging_error(
f"Error: Class '{class_name}' or function/method '{function_name}' not found in '{package_name}'. {e}"
)
raise
except Exception as e:
logging_error(f"An unexpected error occurred during invocation: {e}")
raise
class DataMessageHandler:
def __init__(
self,
service_adapter: CleverThisServiceAdapter,
tracer: Tracer,
message_factory: DataMessageFactory,
service_message_factory: ServiceMessageFactory,
reply_to_exchange: AbstractRobustExchange,
rabbit_mq_client: RabbitMQClient,
amq_configuration: AMQConfiguration,
loop,
backpressure_handler: BackpressureHandler,
file_handler: FileHandler = StreamingFileHandler(),
):
self.service_adapter: CleverThisServiceAdapter = service_adapter
self.tracer = tracer
self.message_factory: DataMessageFactory = message_factory
self.service_message_factory: ServiceMessageFactory = service_message_factory
self.reply_to_exchange: AbstractRobustExchange = reply_to_exchange
self.rabbit_mq_client: RabbitMQClient = rabbit_mq_client
self.rabbit_mq_url = amq_configuration.dispatch.rabbit_mq_url
self.output_dir = amq_configuration.dispatch.download_dir
self.loop = loop
self.service_adapter.loop = loop
# Dictionary to store outstanding Futures
self.outstanding: Dict[str, asyncio.Future] = {}
self.reply_to: str | None = None
self.file_handler: FileHandler = file_handler
self.backpressure_handler: BackpressureHandler = backpressure_handler
async def inbound_data_message_callback_as_thread(self, message: AbstractIncomingMessage):
"""
The callback method that is called when a DataMessage is received from RabbitMQ, starts new Thread.
"""
threading.Thread(
target=lambda: asyncio.run(self.inbound_data_message_callback_no_thread(message))
).start()
async def inbound_data_message_callback_no_thread(self, message: AbstractIncomingMessage):
"""
Receives the DataMessage as bytes from RabbitMQ queue and deserializes it to the DataMessage object
Ensures Jaeger trace-id propagation.
Invokes custom handler/mapper/adapter method that corresponds to this message
:param message:
:return:
"""
start = time.time()
amq_message = await self.reconstitute_data_message(message)
if amq_message is not None:
self.ensure_trace_info(amq_message)
if amq_message.magic() == DataMessageMagicByte.HTTP_REQUEST.value:
await self.run_http_request_magic(message, amq_message, self.loop)
elif amq_message.magic() == DataMessageMagicByte.RPC_REQUEST.value:
await self.run_rpc_request(message, amq_message, self.loop)
else:
logging_error(
"Unknown MAGIC value: [%s], don't know how to interpret the message",
amq_message.magic(),
)
asyncio.run_coroutine_threadsafe(message.nack(requeue=False), self.loop)
return
else:
logging_error("######[%s] No valid AMQ Message present.", message.delivery_tag)
self.record_duration(start, message.delivery_tag)
asyncio.run_coroutine_threadsafe(message.ack(multiple=False), self.loop)
async def run_http_request_magic(
self,
message: AbstractIncomingMessage,
amq_message: DataMessage,
loop: AbstractEventLoop,
) -> None:
"""
Synchronous version of on_message method. This is a wrapper around the async on_message method.
It also counts parallel executions and handles backpressure.
:param message: RabbitMQ raw message
:param amq_message: DataMessage
:param loop: Event loop
:return: DataResponse
"""
# Increment the counter for parallel executions and check resource availability
self.backpressure_handler.increase_current_load()
await self.backpressure_handler.check_overload_condition()
logging_info(
f"PARALLEL: Current Capacity [{self.backpressure_handler.current_availability}] of [{self.backpressure_handler.max_availability}]"
)
if message.reply_to:
self.reply_to = message.reply_to
try:
_a_message = AMQMessage(amq_message, self.rabbit_mq_client.connection)
_response: DataResponse = await self.service_adapter.on_message(_a_message)
logging_debug(f"Result in thread: {_response}")
try:
await self.send_reply(message.reply_to, _response)
except Exception as e:
logging_error(f"Processing message: {e}")
asyncio.run_coroutine_threadsafe(message.nack(requeue=False), loop=loop)
if isinstance(amq_message, MultipartDataMessage):
# remove the downloaded files from the output_dir
for file_id, filename in amq_message.extra_data.items():
try:
logging_info(
f"[{message.delivery_tag}] Deleting file: {file_id} in {filename}"
)
os.remove(filename)
except OSError as e:
logging_error(f"Deleting file {filename}: {e}")
except Exception as e:
logging_error(f"Request handler: {e}")
self.backpressure_handler.decrease_current_load()
async def run_rpc_request(
self,
message: AbstractIncomingMessage,
amq_message: DataMessage,
loop: AbstractEventLoop,
) -> None:
"""
Synchronous version of on_message method. This is a wrapper around the async on_message method.
It also counts parallel executions and handles backpressure.
:param message: RabbitMQ raw message
:param amq_message: DataMessage
:param loop: Event loop
:return: DataResponse
"""
# Increment the counter for parallel executions and check resource availability
self.backpressure_handler.increase_current_load()
await self.backpressure_handler.check_overload_condition()
logging_info(
f"PARALLEL: Current Capacity [{self.backpressure_handler.current_availability}] of [{self.backpressure_handler.max_availability}]"
)
if message.reply_to:
self.reply_to = message.reply_to
try:
_a_message = AMQMessage(amq_message, self.rabbit_mq_client.connection)
_response: DataResponse = await self.on_rpc(_a_message)
logging_debug(f"Result in thread: {_response}")
try:
await self.send_reply(message.reply_to, _response)
except Exception as e:
logging_error(f"Processing message: {e}")
asyncio.run_coroutine_threadsafe(message.nack(requeue=False), loop=loop)
if isinstance(amq_message, MultipartDataMessage):
# remove the downloaded files from the output_dir
for file_id, filename in amq_message.extra_data.items():
try:
logging_info(
f"[{message.delivery_tag}] Deleting file: {file_id} in {filename}"
)
os.remove(filename)
except OSError as e:
logging_error(f"Deleting file {filename}: {e}")
except Exception as e:
logging_error(f"Request handler: {e}")
self.backpressure_handler.decrease_current_load()
async def on_rpc(self, a_message: AMQMessage) -> DataResponse:
"""
Dynamically invokes a function or a class method from a specified package.
Args:
a_message (AMQMessage): The message containing the package, class, and function details.
Args encoded in the AMQMessage:
package_name (str): The name of the Python package to import.
class_name (str): The name of the class within the package.
Use '__global__' if the function_name refers to a global function
in the package.
function_name (str): The name of the function or method to invoke.
kwargs (Dict[str, Any]): A dictionary of keyword arguments to pass to the function/method (in the body).
Returns:
Any: The return value of the invoked function or method.
Raises:
ModuleNotFoundError: If the specified package cannot be imported.
AttributeError: If the specified class or function/method does not exist.
Exception: For any other errors during invocation.
"""
package_name: str = a_message.method()
class_name: str = a_message.domain()
function_name: str = a_message.path()
kwargs: Dict[str, Any] = parse_map_string(a_message._base64body.decode("utf-8"))
return await invoke_command(
package_name, class_name, function_name, a_message.id().as_string(), kwargs
)
async def reconstitute_data_message(
self, message: AbstractIncomingMessage
) -> DataMessage | None:
"""
DataMessage may be serialized as string in separate queue (addressing == 1) or be a MultiPart message with
files streamed over individual queues, one per file.
This function uses 'addressing' mode from message headers to determine how to deserialize (reconstitute)
the original message.
"""
amq_message: DataMessage | None = None
delivery_tag = message.delivery_tag
addressing: int = message.headers.get("clevermicro.addressing", 0)
if addressing == 0:
amq_message = DataMessageFactory.from_bytes(message.body)
else:
amq_message = await DataMessageFactory.from_stream(
message.body, rabbitmq_url=self.rabbit_mq_url, loop=self.loop
)
logging_info(f"######[{delivery_tag}] AMQ Message from stream: {amq_message}")
if amq_message is not None:
extra_data = await self.file_handler.on_message(
message=message,
json_data=amq_message.body(),
rabbitmq_url=self.rabbit_mq_url,
loop=self.loop,
output_dir=self.output_dir,
)
logging_info(f"######[{delivery_tag}] Multipart download result: {extra_data}")
amq_message = MultipartDataMessage(amq_message, extra_data)
if amq_message:
logging_info(
"######[%s] AMQ Message received, ctag=%s, msgId=%s, %s:%s, traceInfo=%s",
message.delivery_tag,
message.consumer_tag,
amq_message.id(),
amq_message.method(),
amq_message.path(),
map_as_string(amq_message.trace_info()),
)
return amq_message
def ensure_trace_info(self, amq_message: DataMessage):
propagator = TraceContextTextMapPropagator()
context = propagator.extract(
carrier=amq_message.trace_info(), context=trace.context_api.get_current()
)
self.tracer.start_span(name="AMQ-Adapter-Consumer", context=context)
# logging_info("CTX=%s", context)
# current_span = self.tracer.start_span("AMQ-Adapter.Processing.Request", parent=parent_span)
# context_with_span = trace.set_span_in_context(current_span, context)
# logging.info(" [*] CTX2=%s", context_with_span)
# propagator.inject(context_with_span, amq_message.trace_info, self.set_text)
# logging.info(" [***] NEW TI=%s CTX=%s SCOPE=%s",
# map_as_string(amq_message.trace_info()), context_with_span, context)
def record_duration(self, start, delivery_tag):
global last_backpressure_event_time
last_backpressure_event_time = time.time()
logging_info(
"######[%s] Done, single ACK: %sms.",
delivery_tag,
last_backpressure_event_time - start,
)
async def send_reply(self, reply_to: str, response: DataResponse):
"""
The Service side code. When the CleverXXX service has output ready, in DataResponse object,
call this method to return the reply back to the caller
:param reply_to:connection.channel
:param response:
:return:
"""
if reply_to:
pika_message: Message = Message(
body=DataResponseFactory.serialize(response),
correlation_id=response.id(),
content_type=(
response.content_type()[0]
if isinstance(response.content_type(), list)
else response.content_type()
),
)
_res = asyncio.run_coroutine_threadsafe(
self.reply_to_exchange.publish(message=pika_message, routing_key=reply_to),
self.loop,
)
logging_debug(
"###### DONE Reply Published to exchg:(%s) To=%s, res=%s",
self.reply_to_exchange.name,
reply_to,
_res,
)
async def reply_received_callback(self, message: AbstractIncomingMessage):
"""
The handler for reply for DataMessage that we sent out earlier -
the code matches the received response with the original request and passes the response to the caller.
:param message:
:return:
"""
reply_id = message.correlation_id
delivery_tag = message.delivery_tag
debug = f"######[{delivery_tag}] (RPC REPLY) env: {message.headers}, props: {message.properties}, BODY: {message.body.decode()}"
logging_debug(debug)
# TODO - figure out implementation of this part, as this is sending reply back either to a service,
# or to the user
logging_info(f"OUTSTANDING RECV: {self.outstanding}, size={len(self.outstanding)}")
future: Future = self.outstanding.pop(reply_id, None)
# if reply is None:
# logging.warn(" [W] ######[%s] IGNORING REPLY. No outstanding request for ID %s",
# delivery_tag, reply_id)
# else:
# result = DataResponseFactory.from_bytes(body)
# reply.tracing_span().end()
# TODO - figure out the right response format (not needed for Clever Swarm v0.1)
# that currently runs Java version, this method here is for the future compatibility to enable
# internal, service-2-service communication, and the proper response type will be decided than
# reply.response().set_result(Response.ok(result.body, content_type).build())
if future:
response: DataResponse = DataResponseFactory.from_bytes(message.body)
# Complete the Future with the response data
future.set_result(response.body())
await message.ack(multiple=False)
async def send_rpc(self, route, message: DataMessage) -> Future:
"""
Sends an RPC message to the specified route and returns a Future that will be completed
when the response is received.
:param route: AMQRoute object containing exchange and routing information
:param message: DataMessage to be sent
:return: Future that will be completed with the response payload
"""
# Create a Future to be completed when the response is received
future = asyncio.Future()
# Store the future in the outstanding dictionary using the message ID as the key
message_id = message.id().as_string()
self.outstanding[message_id] = future
# Create a Pika message with the correlation ID set to the message ID
pika_message = Message(
body=DataMessageFactory.serialize(message),
correlation_id=message_id,
reply_to=self.rabbit_mq_client.router.get_reply_to_queue_name(),
content_type=(
message.content_type()[0]
if isinstance(message.content_type(), list)
else message.content_type()
),
)
# Get the exchange from the route
rpc_exchange_name = route.exchange
exchange: AbstractRobustExchange = await self.rabbit_mq_client.channel.declare_exchange(
name=rpc_exchange_name, type=ExchangeType.topic
)
# Publish the message to the exchange with the component name as the routing key
await exchange.publish(
message=pika_message, routing_key=sanitize_as_rabbitmq_name(route.component_name)
)
logging_debug(
"###### RPC Request Published to exchg:(%s) routing_key=%s, msgId=%s",
rpc_exchange_name,
route.component_name,
message_id,
)
return future
async def send_command(self, route, message: DataMessage) -> bool:
"""
Sends an RPC message to the specified route and returns a Future that will be completed
when the response is received.
:param route: AMQRoute object containing exchange and routing information
:param message: DataMessage to be sent
:return: Future that will be completed with the response payload
"""
rmq_body = DataMessageFactory.serialize(message)
cid = message.id().as_string()
ctype = (
message.content_type()[0]
if isinstance(message.content_type(), list)
else message.content_type()
)
print(
f"Sending RPC command to route: {route}, message ID: {cid} with body: {rmq_body} correlation ID: {cid}, content type: {ctype}"
)
# Create a Pika message with the correlation ID set to the message ID
pika_message = Message(
body=rmq_body,
correlation_id=cid,
content_type=ctype,
)
print(f"SENDING RPC COMMAND: {pika_message.body} with ID: {message.id().as_string()}")
# Get the exchange from the route
rpc_exchange_name = route.exchange
exchange: AbstractRobustExchange = await self.rabbit_mq_client.channel.declare_exchange(
name=rpc_exchange_name, type=ExchangeType.topic
)
# Publish the message to the exchange with the component name as the routing key
await exchange.publish(
message=pika_message, routing_key=sanitize_as_rabbitmq_name(route.component_name)
)
logging_info(
"###### RPC Request Published to exchg:(%s) routing_key=%s, msgId=%s",
rpc_exchange_name,
route.component_name,
message.id().as_string(),
)
return True
amqp/service/amq_service.py
+58 -256
View File
@@ -1,282 +1,84 @@
import asyncio
import logging
import logging.config
import os
from asyncio import Future
from threading import Thread
from typing import AsyncIterator, Optional
from aio_pika.abc import AbstractRobustExchange
import logging
from functools import partial
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
ConsoleSpanExporter,
SimpleSpanProcessor
)
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.tracecontext import TraceContextTextMapPropagator
from amqp.adapter.amq_message_factory import AMQMessageFactory
from amqp.adapter.amq_route_factory import AMQRouteFactory
from amqp.adapter.backpressure_handler import BackpressureHandler
from amqp.adapter.consul_global_id_generator import get_unique_instance_id
from amqp.adapter.data_message_factory import DataMessageFactory
from amqp.adapter.logging_utils import logging_error, logging_info, logging_warning
from amqp.adapter.service_message_factory import ServiceMessageFactory
from amqp.config.amq_configuration import AMQConfiguration
from amqp.model.model import AMQRoute, DataMessage
from amqp.rabbitmq.rabbit_mq_client import RabbitMQClient
from amqp.router.router_consumer import RouterConsumer
from amqp.service.amq_message_handler import DataMessageHandler
from amqp.service.tracing import TracingConfig, initialize_telemetry
logging.basicConfig(level=logging.DEBUG)
logging.info(f"CWD = {os.getcwd()}")
logging_conf_path = os.path.join(os.getcwd(), "amqp", "service", "logging.conf")
if os.path.exists(logging_conf_path):
logging.config.fileConfig(logging_conf_path)
else:
logging_conf_path = os.path.join(os.getcwd(), "logging.conf")
if os.path.exists(logging_conf_path):
logging.config.fileConfig(logging_conf_path)
from amqp.adapter.service_message_factory import CleverMicroServiceMessageFactory
from amqp.rabbitmq.rabbit_mq_consumer import RabbitMQConsumer
from amqp.service.service_message_handler import AMQServiceMessageHandler
class AMQService:
CLEVER_AMQP_CLIENT_VERSION = os.getenv("CLEVER_AMQP_CLIENT_VERSION")
MAX_WAIT = 300 # defaults to 5 minutes (300 seconds)
def __init__(self, amq_configuration: AMQConfiguration, service_adapter=None):
logging.info("***********************************************")
logging.info("************** AMQ Service ***************")
logging.info("***********************************************")
self.loop = asyncio.new_event_loop()
def __init__(self, amq_configuration, http_adapter):
self.amq_configuration = amq_configuration
self.service_adapter = service_adapter
self.unique_id = get_unique_instance_id(amq_configuration.amq_adapter)
amq_configuration.amq_adapter.generator_id = self.unique_id
self.data_message_factory = DataMessageFactory.get_instance(
self.unique_id & DataMessageFactory.MACHINE_ID_MASK
)
self.router = RouterConsumer(self.amq_configuration)
self.rabbit_mq_client = RabbitMQClient(self.amq_configuration, self.router)
self.tracer = initialize_telemetry(
tracing_config=TracingConfig(
service_name=amq_configuration.amq_adapter.service_name,
service_version=os.getenv("SERVICE_VERSION", "unknown"),
)
)
self.amq_data_message_handler = None
self.service_message_factory = ServiceMessageFactory(
self.amq_configuration.amq_adapter.service_name
)
self.backpressure_thread: Optional[Thread] = None
self.backpressure_handler: Optional[BackpressureHandler] = None
self.maximum_availability = -1
self.future: Future = asyncio.Future()
# =======================================================================
# =======================================================================
# ============== P u b l i c A P I M e t h o d s ===================
# =======================================================================
# =======================================================================
def run(self):
"""
Start the AMQ service, initializing RabbitMQ connection and message handlers.
"""
# self.init(loop) will initialize RabbitMQ connection
asyncio.set_event_loop(self.loop)
self.loop.run_until_complete(self.init(self.loop, self.service_adapter))
self.http_adapter = http_adapter
self.message_factory = AMQMessageFactory.get_instance(self.amq_configuration.amq_adapter().generator_id())
self.rabbit_mq_consumer = RabbitMQConsumer(self.amq_configuration)
self.amq_service_message_handler = AMQServiceMessageHandler(self.http_adapter, self.get_tracer())
def init(self):
CleverMicroServiceMessageFactory.process_name(self.amq_configuration.amq_adapter().service_name())
logging.info("********* AMQService.isRouter = %s ***********", self.amq_configuration.amq_adapter().is_router())
logging.info("***********************************************")
logging.info("****** AMQ Service Init Complete *********")
logging.info("********** AMQ Service *******************")
logging.info("***********************************************")
self.future.set_result(True)
self.loop.run_forever()
def backpressure(self, current_availability: int, maximum: int = -1):
"""
Send Backpressure event to the Management Service. The Backpressure logic determines the type of the event
based on the current_availability and maximum values.
- OVERLOAD event is triggered immediately when the current_availability value is below threshold
typically set to 10% of the maximum value, or 0 if no maximum is provided.
- IDLE event is triggered when the current_availability value exeeds 80% of the maximum value or is greater
than 0 if no maximum is provided.
- UPDATE event is sent if neither OVERLOAD nor IDLE conditions are met, indicating that the backpressure is
within acceptable limits.
:param current_availability: Currently available capacity.
:param maximum: Maximum value of the backpressure metric. If undefined or set to lt 0 (e.g. -1),
the maximum is than determined from max value of current_availability seen over the time.
"""
if self.backpressure_handler is not None:
# watch maximum & current value to always have some valid maximum reference
if maximum < 0:
if current_availability > self.maximum_availability:
self.maximum_availability = current_availability
else:
if maximum > self.maximum_availability:
self.maximum_availability = maximum
asyncio.run_coroutine_threadsafe(
self.backpressure_handler.update_backpressure_value(
current_availability, maximum if maximum > 0 else self.maximum_availability
),
loop=self.backpressure_handler.loop,
)
def rpc(self, service_name: str, fq_method_name: str, **kwargs) -> Future:
"""
Send a Remote Procedure Call (RPC) message to the AMQ service.
:param service_name: The service name to receive the RPC message.
:param fq_method_name: The fully qualified method name to execute the RPC message.
:param kwargs: Additional keyword arguments to be passed to the RPC method.
:return: A Future that resolves when the RPC response is received.
"""
route: Optional[AMQRoute] = self.router.find_route_by_service(service_name)
if route is not None:
message: DataMessage = self.data_message_factory.create_rpc_message(
fq_method_name=fq_method_name,
swarm_id=self.amq_configuration.amq_adapter.swarm_task_id,
**kwargs,
)
result: Future = asyncio.run_coroutine_threadsafe(
self.amq_data_message_handler.send_rpc(route, message),
loop=self.backpressure_handler.loop,
)
self.set_outstanding_future(message, result)
return result
result = asyncio.Future()
result.set_exception(
NameError(f"No route exists. Service '{service_name}' not found in routing table.")
)
return result
def command(self, service_name: str, fq_method_name: str, **kwargs) -> bool:
"""
Send a unidirectional Command message to the AMQ service.
:param service_name: The service name to receive the RPC message.
:param fq_method_name: The fully qualified method name to execute the RPC message.
:param kwargs: Additional keyword arguments to be passed to the RPC method.
:return: A Future that resolves when the RPC response is received.
"""
route: Optional[AMQRoute] = self.router.find_route_by_service(service_name)
if route is not None:
m: DataMessage = self.data_message_factory.create_rpc_message(
fq_method_name=fq_method_name,
swarm_id=self.amq_configuration.amq_adapter.swarm_task_id,
args=kwargs,
)
asyncio.run_coroutine_threadsafe(
self.amq_data_message_handler.send_command(route, m),
loop=self.backpressure_handler.loop,
)
return True
return False
async def data_message_generator(self, route: AMQRoute) -> AsyncIterator[DataMessage]:
"""
An async iterator that blocks until messages arrive, processes them,
and continues in a loop.
:param queue_name: The name of the queue to consume messages from
:yield: The processed DataResponse for each message
"""
# Get a channel from the connection
channel = self.rabbit_mq_client.channel
queue_args = (
self.rabbit_mq_client.DLQ_ARGS if self.amq_configuration.dispatch.use_dlq else None
)
queue = await channel.declare_queue(
route.queue,
durable=True,
exclusive=False,
auto_delete=False,
arguments=queue_args,
)
await queue.bind(route.exchange, route.key)
# Start consuming without a callback
async with queue.iterator() as queue_iter:
async for message in queue_iter:
try:
# Process the message using the same logic as in inbound_data_message_callback_no_thread
data_message: DataMessage = (
await self.amq_data_message_handler.reconstitute_data_message(message)
)
success = yield data_message
if not success:
logging_warning(f"Message processing failed for {data_message.id()}.")
# Nack the message without requeuing in case of failure
await message.nack(requeue=False)
else:
await message.ack()
except Exception as e:
logging_error(f"Error processing message: {e}")
# Nack the message without requeuing in case of error
await message.nack(requeue=False)
def get_unique_instance_id(self) -> int:
"""
Provides cluster-wide unique instance ID for this AMQ service instance.
This ID is used to identify the service instance in the cluster and is unique across all instances.
IDs are managed by shared Consul KV store, ensuring that even if multiple instances of the service are running,
they will each have unique ID.
"""
return self.unique_id
# =======================================================================
# ====================== Internal Methods ============================
# =======================================================================
async def init(self, loop, service_adapter):
_reply_to_exchange: AbstractRobustExchange = await self.rabbit_mq_client.init(loop)
self.backpressure_handler = BackpressureHandler(
channel=self.rabbit_mq_client.channel,
loop=loop,
config=self.amq_configuration,
)
self.amq_data_message_handler = DataMessageHandler(
service_adapter,
self.tracer,
self.data_message_factory,
self.service_message_factory,
_reply_to_exchange,
self.rabbit_mq_client,
self.amq_configuration,
loop=loop,
backpressure_handler=self.backpressure_handler,
)
await self.register_routes()
await self.rabbit_mq_client.request_routes()
logging_info("Starting backpressure monitor thread")
self.backpressure_thread = self.backpressure_handler.start_backpressure_monitor()
self.setup_opentelemetry()
self.register_routes()
self.register_rpc_handler()
self.rabbit_mq_consumer.request_routes()
def cleanup(self):
logging_info("AMQService.cleanup() rmqc=%s", self.rabbit_mq_client)
if self.rabbit_mq_client:
self.rabbit_mq_client.cleanup()
logging.info(" [DBG] AMQService.cleanup() rmqc=%s", self.rabbit_mq_consumer)
if self.rabbit_mq_consumer:
self.rabbit_mq_consumer.cleanup()
async def reinitialize_if_disconnected(self):
if not self.rabbit_mq_client.channel or self.rabbit_mq_client.channel.is_closed:
self.rabbit_mq_client = RabbitMQClient(self.amq_configuration, self.router)
await self.rabbit_mq_client.init(loop=self.loop)
await self.register_routes()
await self.rabbit_mq_client.request_routes()
def reinitialize_if_disconnected(self):
if not self.rabbit_mq_consumer.channel or not self.rabbit_mq_consumer.channel.is_open:
self.init()
async def register_routes(self) -> AbstractRobustExchange | None:
route_mapping = self.amq_configuration.amq_adapter.route_mapping
def setup_opentelemetry(self):
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(
SimpleSpanProcessor(ConsoleSpanExporter())
)
set_global_textmap(TraceContextTextMapPropagator())
self.tracer = trace.get_tracer("com.cleverthis")
logging.info("OTEL %s %s %s", trace.get_logger_bridge(), trace.get_meter_provider(), trace.get_tracer_provider())
logging.info("OTEL TRACER %s", trace.get_propagators())
def register_routes(self):
route_mapping = self.amq_configuration.amq_adapter().route_mapping()
if AMQRouteFactory.validate(route_mapping):
try:
await self.rabbit_mq_client.register_inbound_routes(
route_mapping,
self.amq_data_message_handler.inbound_data_message_callback_no_thread,
self.rabbit_mq_consumer.register_routes()
self.rabbit_mq_consumer.register_deliver_callback(
partial(self.amq_service_message_handler.on_delivery_callback_http, self.rabbit_mq_consumer.channel)
)
return await self.rabbit_mq_client.register_data_reply_callback(
self.amq_data_message_handler.reply_received_callback
self.rabbit_mq_consumer.register_rpc_handler(
self.amq_service_message_handler.on_reply_consumer(self.rabbit_mq_consumer.channel)
)
except Exception as e:
raise RuntimeError(e)
elif route_mapping:
logging_warning(
"route [%s] failed validation, no routes were registered.",
route_mapping,
logging.info(" [W] route [%s] failed validation, no routes were registered.", route_mapping)
def register_rpc_handler(self):
try:
self.rabbit_mq_consumer.register_rpc_handler(
self.amq_service_message_handler.on_reply_consumer(self.rabbit_mq_consumer.get_channel())
)
return None
def set_outstanding_future(self, message: DataMessage, future: Future):
self.amq_data_message_handler.outstanding[str(message.id())] = future
def remove_outstanding_future(self, message_id: str):
self.amq_data_message_handler.outstanding.pop(message_id, None)
except Exception as err:
raise
amqp/service/logging.conf
-34
View File
@@ -1,34 +0,0 @@
[loggers]
keys=root,amqp_adapter
[handlers]
keys=consoleHandler,fileHandler
[formatters]
keys=simpleFormatter
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_amqp_adapter]
level=DEBUG
handlers=consoleHandler,fileHandler
qualname=amqp_adapter
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)
[handler_fileHandler]
class=FileHandler
level=DEBUG
formatter=simpleFormatter
args=('AMQ_PyAdapter.log',)
[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=%Y-%m-%d %H:%M:%S
amqp/service/service_message_handler.py
+143
View File
@@ -0,0 +1,143 @@
import logging
import time
from typing import Callable, Dict
from concurrent.futures import Future
from opentelemetry import trace
from opentelemetry.propagate import get_global_textmap, set_global_textmap
from opentelemetry.propagators.tracecontext import TraceContextTextMapPropagator
from opentelemetry.trace import Span, Tracer, TraceState
from amqp.adapter.amq_message_factory import AMQMessageFactory
from amqp.model.model import AMQResponse
from amqp.router.serializer import map_as_string
class AMQServiceMessageHandler:
def __init__(self, http_adapter, tracer):
self.http_adapter = http_adapter
self.tracer = tracer
self.outstanding = {}
def on_delivery_callback_http(self, channel, consumerTag, delivery):
start = time.time()
deliveryTag = delivery.envelope.delivery_tag
debug = f" [DBG] ######[{deliveryTag}] (AMQ MESSAGE), env: {delivery.envelope}, props: {delivery.properties}"
logging.info(debug)
amq_message = AMQMessageFactory.from_bytes(delivery.body)
logging.info(" [*] ######[%s] AMQ Message received, ctag=%s, msgId=%s, traceInfo=%s",
deliveryTag, consumerTag, amq_message.id, map_as_string(amq_message.trace_info))
propagator = TraceContextTextMapPropagator.get_instance()
context = propagator.extract(trace.get_current_context(), amq_message.trace_info , self.get_text_getter())
parent_span = Span.from_context(context)
context = context.with_span(parent_span)
with context.activate():
logging.info(" [*] CTX=%s", context)
current_span = self.tracer.start_span("AMQ-Adapter.Processing.Request", parent=parent_span)
context_with_span = trace.set_span_in_context(current_span, context)
logging.info(" [*] CTX2=%s", context_with_span)
propagator.inject(context_with_span, amq_message.trace_info, self.set_text)
logging.info(" [***] NEW TI=%s CTX=%s SCOPE=%s",
map_as_string(amq_message.trace_info), context_with_span, context)
if amq_message.magic == AMQMessageFactory.MAGIC_BYTE_HTTP_REQUEST:
future = self.http_adapter.handle(amq_message)
future.add_done_callback(
lambda f: self.on_http_response(delivery, deliveryTag, channel, f.result())
)
else:
logging.error(" [E] Unknown MAGIC value: [%s], don't know how to interpret the message",
amq_message.magic())
duration = time.time() - start
if "___NACK___" in str(amq_message.body()):
logging.info(" [x] ######[%s] Done, single NACK: %sms.", deliveryTag, duration)
channel.basic_nack(deliveryTag, False, False)
else:
logging.info(" [x] ######[%s] Done, single ACK: %sms.", deliveryTag, duration)
channel.basic_ack(deliveryTag, False)
def on_http_response(self, delivery, deliveryTag, channel, http_response):
response = AMQResponse(
delivery.properties.correlation_id,
http_response.status_code,
http_response.headers.get("Content-Type"),
http_response.body,
http_response.status_message,
"",
self.collect_trace_info(trace.get_current_context())
)
try:
self.send_reply(delivery.properties.reply_to, deliveryTag, channel, response)
except Exception as e:
raise RuntimeError(e)
class AMQServiceMessageHandler:
def __init__(self, http_adapter, tracer):
self.http_adapter = http_adapter
self.tracer = tracer
self.outstanding: Dict[str, 'ResponseHolder'] = {}
def collect_trace_info(self, context):
map = {}
TraceContextTextMapPropagator.get_instance().inject(
context if context else trace.get_current_context(),
map, self.set_text
)
return map
def send_reply(self, reply_to, delivery_tag, channel, response):
if reply_to:
logging.info(" [*] ######[%s] AMQ Message Reply, To=%s, msgId=%s",
delivery_tag, reply_to, response.id())
reply_props = pika.BasicProperties(
correlation_id=response.id(),
content_type=response.content_type()
)
channel.basic_publish(
RouterBase.AMQ_REPLY_TO_EXCHANGE,
reply_to,
reply_props,
AMQResponseFactory.serialize(response)
)
def on_reply_consumer(self, channel):
def handle_delivery(consumerTag, envelope, properties, body):
reply_id = properties.correlation_id
content_type = properties.content_type
delivery_tag = envelope.delivery_tag
debug = f" [DBG] ######[{delivery_tag}] (RPC REPLY) env: {envelope}, props: {properties}, BODY: {body.decode()}"
logging.info(debug)
print(f"OUTSTANDING RECV: {self.outstanding}, size={len(self.outstanding)}")
with self.outstanding.lock:
reply = self.outstanding.pop(reply_id, None)
if reply is None:
logging.warn(" [W] ######[%s] IGNORING REPLY. No outstanding request for ID %s",
delivery_tag, reply_id)
else:
result = AMQResponseFactory.from_bytes(body)
reply.tracing_span().end()
reply.response().set_result(Response.ok(result.body(), content_type).build())
channel.basic_ack(delivery_tag, False)
return handle_delivery
def register_response_holder(self, id, response_holder):
self.outstanding[id] = response_holder
logging.info(" [*] AMQMessage Sent, OUTSTANDING response holder registered: %s, size=%s",
self.outstanding, len(self.outstanding))
def get_default_trace_state(self):
return TraceState(entries={})
def get_text_getter(self):
return lambda map, key: map.get(key)
def set_text(self, carrier, key, value):
carrier[key] = value
amqp/service/tracing.py
+82 -189
View File
@@ -1,196 +1,89 @@
"""
OpenTelemetry configuration for tracing and metrics.
This module provides functionality to initialize and configure OpenTelemetry
for both tracing (Jaeger via OTLP) and metrics (Prometheus) in the AMQ adapter.
"""
import logging
import os
from dataclasses import dataclass
from typing import Any, Dict, Optional
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import SERVICE_NAME, SERVICE_VERSION, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.trace import Tracer
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from prometheus_client import start_http_server
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
logger = logging.getLogger(__name__)
import os
def initialize_trace(flask_app, name):
# Resource can be required for some backends, e.g. Jaeger or Prometheus
# If resource wouldn't be set - traces wouldn't appear in Jaeger.
# The 'name' value is the name shown in Jaeger.
resource = Resource(attributes={
SERVICE_NAME: name if name else os.environ.get('APPLICATION_NAME', 'Python App')
})
#
# 1. Tracing (Jaeger) configuration
# ----------------------------------
#
# Initialize the tracer provider. This is abstract API call.
# The actual implementation is given by the included dependency
# The opentelemetry default is the OTLP trace provider.
trace.set_tracer_provider(TracerProvider(resource=resource))
#
# Configure the OTLP exporter (or HTTP exporter the same way)
# the recommended configuration is via environment variables. The mandatory ones:
# OTEL_EXPORTER_OTLP_ENDPOINT: "http://jaeger:4317"
# OTEL_EXPORTER_OTLP_INSECURE: "true"
# for more see here: https://opentelemetry-python.readthedocs.io/en/latest/exporter/otlp/otlp.html
# alternatively these values can be provided as input values to OTLPSpanExporter() constructor.
#
otlp_exporter = OTLPSpanExporter()
#
# Add the exporters to the batch span processor
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
print(' * Instrumenting Flask application..... ')
FlaskInstrumentor().instrument_app(flask_app)
#
# Python Flask is now automatically instrumented to export details of each REST endpoint call.
# For Django based app, use 'opentelemetry-instrumentation-django' import instead. see here:
# https://opentelemetry-python.readthedocs.io/en/latest/examples/django/README.html for example
#
# If the used library has no instrumentation support in opentelemetry.io, then the trace
# can be started manually. In this code,
# 'trace' should be global shared variable, initialized once per application (see tracing.py)
# 'tracer' can be shared global or local variable, application can use multiple uniquely
# named tracers, Jaeger will group the generated traces by the name of the tracer
# that was used to start the Span. See this example code:
#
# tracer = trace.get_tracer(name)
#
# @app.route("/server_request")
# def server_request():
# with tracer.start_as_current_span(
# "server_request",
# context=extract(request.headers),
# kind=trace.SpanKind.SERVER,
# attributes=collect_request_attributes(request.environ),
# ):
# # do the work here, like print something and return response
# print(request.args.get("param"))
# return "served"
#
# Note:
# the otlp code looks for HTTP header named 'traceparent' that, if present, should contain
# the otlp parent span identification, and the span created here will be made a child
# of that parent span, indicating visually the progression of requests via different services
# for as long as the current span is provided in the traceparent header to next service.
@dataclass
class TracingConfig:
"""Configuration for OpenTelemetry tracing."""
service_name: str
service_version: str
otlp_endpoint: str = "http://localhost:4317" # Default Jaeger OTLP endpoint
insecure: bool = True
debug: bool = False
additional_attributes: Optional[Dict[str, Any]] = None
@dataclass
class MetricsConfig:
"""Configuration for OpenTelemetry metrics."""
prometheus_port: int = 9464
prometheus_host: str = "localhost"
export_interval_millis: int = 30000
additional_attributes: Optional[Dict[str, Any]] = None
def create_resource(config: TracingConfig) -> Resource:
"""Create an OpenTelemetry resource with service information."""
attributes = {
SERVICE_NAME: config.service_name,
SERVICE_VERSION: config.service_version,
}
if config.additional_attributes:
attributes.update(config.additional_attributes)
return Resource.create(attributes)
def setup_otlp_exporter(config: TracingConfig) -> OTLPSpanExporter:
"""Configure and create an OTLP exporter for Jaeger."""
return OTLPSpanExporter(endpoint=config.otlp_endpoint, insecure=config.insecure)
def setup_trace_provider(config: TracingConfig, resource: Resource) -> TracerProvider:
"""Set up the TracerProvider with OTLP exporter."""
provider = TracerProvider(resource=resource)
# Add OTLP exporter for Jaeger
otlp_exporter = setup_otlp_exporter(config)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
# Add console exporter in debug mode
if config.debug:
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
return provider
def setup_metrics_provider(config: MetricsConfig, resource: Resource) -> MeterProvider:
"""Set up the MeterProvider with Prometheus export."""
try:
# Start Prometheus HTTP server
start_http_server(port=config.prometheus_port, addr=config.prometheus_host)
logger.info(
f"Started Prometheus metrics server on {config.prometheus_host}:{config.prometheus_port}"
)
# Create Prometheus reader
prometheus_reader = PrometheusMetricReader()
# Create periodic reader for custom metrics
periodic_reader = PeriodicExportingMetricReader(
prometheus_reader, export_interval_millis=config.export_interval_millis
)
# Create and configure meter provider
return MeterProvider(resource=resource, metric_readers=[prometheus_reader, periodic_reader])
except Exception as e:
logger.error(f"Failed to set up metrics provider: {str(e)}")
# Return a no-op meter provider
return MeterProvider(resource=resource)
def initialize_telemetry(
tracing_config: Optional[TracingConfig] = None,
metrics_config: Optional[MetricsConfig] = None,
) -> Tracer:
"""
Initialize OpenTelemetry with tracing and metrics.
Args:
tracing_config: Configuration for tracing
metrics_config: Configuration for metrics
Returns:
Tracer: Configured OpenTelemetry tracer
Example:
```python
config = TracingConfig(
service_name="amq-adapter",
service_version="1.0.0",
otlp_endpoint="http://jaeger:4317",
debug=True
)
metrics_config = MetricsConfig(
prometheus_port=9464,
prometheus_host="0.0.0.0"
)
tracer = initialize_telemetry(config, metrics_config)
```
"""
# Use default configs if none provided
if tracing_config is None:
tracing_config = TracingConfig(
service_name=os.getenv("SERVICE_NAME", "amq-adapter"),
service_version=os.getenv("SERVICE_VERSION", "unknown"),
otlp_endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
insecure=os.getenv("OTEL_INSECURE_MODE", "true").lower() == "true",
debug=os.getenv("OTEL_DEBUG", "false").lower() == "true",
)
if metrics_config is None:
metrics_config = MetricsConfig(
prometheus_port=int(os.getenv("PROMETHEUS_PORT", "9464")),
prometheus_host=os.getenv("PROMETHEUS_HOST", "localhost"),
export_interval_millis=int(os.getenv("METRICS_EXPORT_INTERVAL", "30000")),
)
try:
# Create resource
resource = create_resource(tracing_config)
# Set up tracing
provider = setup_trace_provider(tracing_config, resource)
trace.set_tracer_provider(provider)
logger.info(f"Initialized tracing for service {tracing_config.service_name}")
# Set up metrics
metrics_provider = setup_metrics_provider(metrics_config, resource)
metrics.set_meter_provider(metrics_provider)
logger.info("Initialized metrics export")
# Return configured tracer
return trace.get_tracer(
tracing_config.service_name,
tracing_config.service_version,
)
except Exception as e:
logger.error(f"Failed to initialize telemetry: {str(e)}")
# Return a no-op tracer in case of failure
return trace.get_tracer("no-op")
def create_span(tracer: Tracer, name: str, attributes: Optional[Dict[str, Any]] = None):
"""
Create a new span with the given name and attributes.
Args:
tracer: The OpenTelemetry tracer
name: Name of the span
attributes: Optional attributes to add to the span
Returns:
A context manager that creates and manages a span
"""
return tracer.start_as_current_span(
name,
attributes=attributes or {},
)
# 2. Metrics (Prometheus) configuration
# -------------------------------------
#
# Start Prometheus client
start_http_server(port=9464, addr="localhost")
# Initialize PrometheusMetricReader which pulls metrics from the SDK
# on-demand to respond to scrape requests (unlike Jaeger, which worls with push model - application
# pushes traces when those become available to Jaeger's collector. Prometheus works with pull model
# where it periodically polls pre-configured and/or discovered endpoints for metrics.
reader = PrometheusMetricReader()
provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)
api-doc/Makefile
-20
View File
@@ -1,20 +0,0 @@
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
api-doc/README.md
-51
View File
@@ -1,51 +0,0 @@
# AMQ Adapter Python API Documentation
This directory contains the generated HTML API documentation for the AMQ Adapter Python library.
## Viewing the Documentation
1. **Open the main page**: Open `_build/html/index.html` in your web browser
2. **Browse modules**: Use the navigation menu to explore different modules:
- **Configuration** (`amqp.config`) - Configuration classes and settings
- **Adapter** (`amqp.adapter`) - Core adapter classes and utilities
- **Service** (`amqp.service`) - Service layer and message handling
- **Router** (`amqp.router`) - Routing and service discovery
- **Model** (`amqp.model`) - Data models and message structures
- **RabbitMQ** (`amqp.rabbitmq`) - RabbitMQ client implementations
## Features
- **Search functionality**: Use the search box to find specific classes, methods, or functions
- **Module index**: Browse all modules alphabetically
- **Class inheritance**: View class hierarchies and inheritance relationships
- **Source code**: Click "View page source" to see the actual Python code
- **Cross-references**: Navigate between related classes and methods
## Rebuilding the Documentation
To regenerate the documentation after code changes:
```bash
cd api-doc
make html
```
## Documentation Structure
The documentation is organized by module:
- **amqp.config**: Configuration management and settings
- **amqp.adapter**: Core adapter functionality and service integration
- **amqp.service**: Service layer, message handling, and tracing
- **amqp.router**: Routing, service discovery, and queue management
- **amqp.model**: Data models, message structures, and serialization
- **amqp.rabbitmq**: RabbitMQ client implementations and utilities
## Key Classes
- `CleverThisServiceAdapter`: Base class for service adapters
- `AMQService`: Main service class for AMQP operations
- `AMQConfiguration`: Configuration management
- `DataMessage`: Message structure for AMQP communication
- `RouterConsumer`: Service discovery and routing
- `RabbitMQClient`: RabbitMQ connection management
api-doc/_build/doctrees/environment.pickle
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/index.doctree
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/modules/amqp.adapter.doctree
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/modules/amqp.config.doctree
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/modules/amqp.doctree
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/modules/amqp.model.doctree
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/modules/amqp.rabbitmq.doctree
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/modules/amqp.router.doctree
BIN
View File
Binary file not shown.
api-doc/_build/doctrees/modules/amqp.service.doctree
BIN
View File
Binary file not shown.
api-doc/_build/html/.buildinfo
-4
View File
@@ -1,4 +0,0 @@
# Sphinx build info version 1
# This file records the configuration used when building these files. When it is not found, a full rebuild will be done.
config: bf29a028b890480b74024cefba26652a
tags: 645f666f9bcd5a90fca523b33c5a78b7
api-doc/_build/html/_modules/amqp/adapter/amq_route_factory.html
-193
View File
@@ -1,193 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.amq_route_factory &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.amq_route_factory</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.amq_route_factory</h1><div class="highlight"><pre>
<span></span><span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_error</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQRoute</span>
<span class="c1"># record separator</span>
<span class="n">RS</span> <span class="o">=</span> <span class="s2">&quot;:&quot;</span>
<span class="c1"># null route, in lieu of None value</span>
<span class="n">NULL_ROUTE</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;componentName&quot;</span><span class="p">:</span> <span class="s2">&quot;&quot;</span><span class="p">,</span>
<span class="s2">&quot;exchange&quot;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
<span class="s2">&quot;queue&quot;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
<span class="s2">&quot;routingKey&quot;</span><span class="p">:</span> <span class="s2">&quot;#&quot;</span><span class="p">,</span>
<span class="s2">&quot;timeout&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="s2">&quot;validUntil&quot;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="p">}</span>
<div class="viewcode-block" id="AMQRouteFactory">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.amq_route_factory.AMQRouteFactory">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">AMQRouteFactory</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Define the methods to create and validate AMQRoute objects.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="AMQRouteFactory.from_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.amq_route_factory.AMQRouteFactory.from_string">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">from_string</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Builds the AMQRoute from its string form.</span>
<span class="sd"> :param data: route string</span>
<span class="sd"> :return: AMQRoute object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_keys</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">RS</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">_keys</span><span class="p">)</span> <span class="o">&gt;=</span> <span class="mi">5</span><span class="p">:</span>
<span class="n">_i</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">_component_name</span> <span class="o">=</span> <span class="n">_keys</span><span class="p">[</span><span class="n">_i</span><span class="p">]</span>
<span class="n">_i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">_exchange</span> <span class="o">=</span> <span class="n">_keys</span><span class="p">[</span><span class="n">_i</span><span class="p">]</span>
<span class="n">_i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">_queue</span> <span class="o">=</span> <span class="n">_keys</span><span class="p">[</span><span class="n">_i</span><span class="p">]</span> <span class="k">if</span> <span class="n">_i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">_keys</span><span class="p">)</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<span class="n">_i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">_routing_key</span> <span class="o">=</span> <span class="n">_keys</span><span class="p">[</span><span class="n">_i</span><span class="p">]</span> <span class="k">if</span> <span class="n">_i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">_keys</span><span class="p">)</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<span class="n">_i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">_timeout</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">_keys</span><span class="p">[</span><span class="n">_i</span><span class="p">])</span> <span class="k">if</span> <span class="n">_i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">_keys</span><span class="p">)</span> <span class="k">else</span> <span class="mi">0</span>
<span class="n">_i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">_valid_until</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">_keys</span><span class="p">[</span><span class="n">_i</span><span class="p">])</span> <span class="k">if</span> <span class="n">_i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">_keys</span><span class="p">)</span> <span class="k">else</span> <span class="mi">0</span>
<span class="k">return</span> <span class="n">AMQRoute</span><span class="p">(</span>
<span class="n">component_name</span><span class="o">=</span><span class="n">_component_name</span><span class="p">,</span>
<span class="n">exchange</span><span class="o">=</span><span class="n">_exchange</span><span class="p">,</span>
<span class="n">queue</span><span class="o">=</span><span class="n">_queue</span><span class="p">,</span>
<span class="n">key</span><span class="o">=</span><span class="n">_routing_key</span><span class="p">,</span>
<span class="n">timeout</span><span class="o">=</span><span class="n">_timeout</span><span class="p">,</span>
<span class="n">valid_until</span><span class="o">=</span><span class="n">_valid_until</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="s2">&quot;AMQRoute incorrect format. AMQRoute expects input as: &quot;</span>
<span class="s2">&quot;&#39;Component-Name:AMQExchange-Name:Queue-Name:Routing-Key:Timeout:Valid-Until&#39;, &quot;</span>
<span class="s2">&quot;got: [&#39;</span><span class="si">%s</span><span class="s2">&#39;]&quot;</span><span class="p">,</span>
<span class="n">data</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="s2">&quot;</span><span class="si">%s</span><span class="s2">: AMQRoute incorrect format. AMQRoute expects input as: &quot;</span>
<span class="s2">&quot;&#39;Component-Name:AMQExchange-Name:Queue-Name:Routing-Key:Timeout:Valid-Until&#39;, &quot;</span>
<span class="s2">&quot;got: [&#39;</span><span class="si">%s</span><span class="s2">&#39;]&quot;</span><span class="p">,</span>
<span class="n">err</span><span class="p">,</span>
<span class="n">data</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">NULL_ROUTE</span></div>
<div class="viewcode-block" id="AMQRouteFactory.validate">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.amq_route_factory.AMQRouteFactory.validate">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">validate</span><span class="p">(</span><span class="n">route_str</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Validates the route string.</span>
<span class="sd"> :param route_str: route string</span>
<span class="sd"> :return: boolean validity indicator</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="n">route_str</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="n">route_str</span><span class="o">.</span><span class="n">count</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">4</span>
<span class="k">return</span> <span class="kc">False</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/backpressure_handler.html
-466
View File
@@ -1,466 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.backpressure_handler &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.backpressure_handler</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.backpressure_handler</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">time</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">asyncio</span><span class="w"> </span><span class="kn">import</span> <span class="n">AbstractEventLoop</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">threading</span><span class="w"> </span><span class="kn">import</span> <span class="n">Thread</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Optional</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika</span><span class="w"> </span><span class="kn">import</span> <span class="n">Message</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika.abc</span><span class="w"> </span><span class="kn">import</span> <span class="n">AbstractRobustChannel</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_info</span><span class="p">,</span> <span class="n">logging_warning</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.config.amq_configuration</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQConfiguration</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">ScalingRequestAlert</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">await_future</span><span class="p">,</span> <span class="n">await_result</span>
<div class="viewcode-block" id="ScaleRequestV1">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.ScaleRequestV1">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">ScaleRequestV1</span><span class="p">:</span>
<div class="viewcode-block" id="ScaleRequestV1.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.ScaleRequestV1.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">serviceId</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">taskId</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">max_availability</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
<span class="n">current_availability</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
<span class="n">requestType</span><span class="p">:</span> <span class="n">ScalingRequestAlert</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">version</span> <span class="o">=</span> <span class="mi">1</span> <span class="c1"># Version of the request, currently 1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">serviceId</span> <span class="o">=</span> <span class="n">serviceId</span>
<span class="bp">self</span><span class="o">.</span><span class="n">taskId</span> <span class="o">=</span> <span class="n">taskId</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span> <span class="o">=</span> <span class="n">max_availability</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span> <span class="o">=</span> <span class="n">current_availability</span>
<span class="bp">self</span><span class="o">.</span><span class="n">requestType</span> <span class="o">=</span> <span class="n">requestType</span>
<span class="bp">self</span><span class="o">.</span><span class="n">thread</span> <span class="o">=</span> <span class="kc">None</span></div>
<div class="viewcode-block" id="ScaleRequestV1.to_json">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.ScaleRequestV1.to_json">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">to_json</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Converts the object to a JSON string matching the Java structure&quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span>
<span class="p">{</span>
<span class="s2">&quot;version&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">version</span><span class="p">,</span>
<span class="s2">&quot;serviceId&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">serviceId</span><span class="p">,</span>
<span class="s2">&quot;taskId&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">taskId</span><span class="p">,</span>
<span class="s2">&quot;maxAvailability&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">,</span>
<span class="s2">&quot;currentAvailability&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span><span class="p">,</span>
<span class="s2">&quot;requestType&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">requestType</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="c1"># Using .value for Enum serialization</span>
<span class="p">},</span>
<span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
<span class="p">)</span></div>
</div>
<div class="viewcode-block" id="BackpressureHandler">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">BackpressureHandler</span><span class="p">:</span>
<span class="c1"># Track the number of messages currently being processed</span>
<span class="n">current_availability</span> <span class="o">=</span> <span class="mi">0</span>
<span class="c1"># helps detect IDLE condition</span>
<span class="c1"># helps prevent flooding the system with backpressure events</span>
<span class="n">last_backpressure_event_time</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">last_backpressure_event</span> <span class="o">=</span> <span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">UPDATE</span>
<span class="c1"># _callback_list is used in unit tests to record the invoked callbacks</span>
<span class="n">_callback_list</span> <span class="o">=</span> <span class="kc">None</span>
<div class="viewcode-block" id="BackpressureHandler.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">channel</span><span class="p">:</span> <span class="n">AbstractRobustChannel</span><span class="p">,</span>
<span class="n">loop</span><span class="p">:</span> <span class="n">AbstractEventLoop</span><span class="p">,</span>
<span class="n">config</span><span class="p">:</span> <span class="n">AMQConfiguration</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">lock</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">channel</span> <span class="o">=</span> <span class="n">channel</span>
<span class="bp">self</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="n">loop</span>
<span class="bp">self</span><span class="o">.</span><span class="n">config</span> <span class="o">=</span> <span class="n">config</span>
<span class="bp">self</span><span class="o">.</span><span class="n">exchange</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_service_id</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">amq_adapter</span><span class="o">.</span><span class="n">swarm_service_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_task_id</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">amq_adapter</span><span class="o">.</span><span class="n">swarm_task_id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">do_loop</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_resource_usage_changed</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_resource_average_value</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_last_resource_max_value</span> <span class="o">=</span> <span class="mi">100</span> <span class="c1"># Default max value for CPU usage</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span> <span class="o">=</span> <span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_load</span> <span class="o">=</span> <span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_load</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idle_state_detected_time</span> <span class="o">=</span> <span class="mi">0</span></div>
<div class="viewcode-block" id="BackpressureHandler.increase_current_load">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.increase_current_load">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">increase_current_load</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Increase the number of parallel executions, or current_availability load&quot;&quot;&quot;</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;Backpressure: Increase current load (</span><span class="si">%s</span><span class="s2">) by 1&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_load</span><span class="p">,</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_load</span> <span class="o">+=</span> <span class="mi">1</span></div>
<div class="viewcode-block" id="BackpressureHandler.decrease_current_load">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.decrease_current_load">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">decrease_current_load</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Decrease the number of parallel executions, or current load&quot;&quot;&quot;</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;Backpressure: Decrease current load(</span><span class="si">%s</span><span class="s2">) by 1&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_load</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_load</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_load</span> <span class="o">-=</span> <span class="mi">1</span></div>
<div class="viewcode-block" id="BackpressureHandler.update_current_availability">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.update_current_availability">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">update_current_availability</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">count</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Update the number of parallel executions, or current load&quot;&quot;&quot;</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span> <span class="o">=</span> <span class="n">count</span></div>
<div class="viewcode-block" id="BackpressureHandler.update_last_backpressure_event_time">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.update_last_backpressure_event_time">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">update_last_backpressure_event_time</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="c1"># logging_info(&quot;Backpressure: Update last data message time&quot;)</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Update the last data message time&quot;&quot;&quot;</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span></div>
<div class="viewcode-block" id="BackpressureHandler.update_backpressure_value">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.update_backpressure_value">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">update_backpressure_value</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">current_availability</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span> <span class="n">maximum</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Update the current backpressure value and check for overload conditions.</span>
<span class="sd"> This method is called by the AMQService.backpressure() method to update</span>
<span class="sd"> the current parallel executions count and check for overload conditions.</span>
<span class="sd"> Args:</span>
<span class="sd"> current_availability: Current value of the backpressure metric</span>
<span class="sd"> maximum: Maximum value of the backpressure metric</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;Backpressure: Updating backpressure value, current_availability=</span><span class="si">%s</span><span class="s2">, maximum=</span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">current_availability</span><span class="p">,</span>
<span class="n">maximum</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">maximum</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="ow">and</span> <span class="n">maximum</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span> <span class="o">=</span> <span class="n">maximum</span>
<span class="c1"># Update the current_availability / parallel executions count</span>
<span class="bp">self</span><span class="o">.</span><span class="n">update_current_availability</span><span class="p">(</span><span class="n">current_availability</span><span class="p">)</span>
<span class="c1"># Check for overload conditions</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_overload_condition</span><span class="p">()</span></div>
<div class="viewcode-block" id="BackpressureHandler.start_backpressure_monitor">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.start_backpressure_monitor">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">start_backpressure_monitor</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Optional</span><span class="p">[</span><span class="n">Thread</span><span class="p">]:</span>
<span class="c1"># Start the Backpressure monitor loop</span>
<span class="bp">self</span><span class="o">.</span><span class="n">thread</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">backpressure_monitor_loop</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">thread</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="kc">True</span> <span class="c1"># This makes it a daemon thread</span>
<span class="bp">self</span><span class="o">.</span><span class="n">thread</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">thread</span></div>
<div class="viewcode-block" id="BackpressureHandler.check_overload_condition">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.check_overload_condition">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">check_overload_condition</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Check if the current_availability availability is too low (OVERLOAD)</span>
<span class="sd"> or if the service has been idle for too long with high availability (IDLE).</span>
<span class="sd"> Note: current_availability represents available capacity, not used capacity.</span>
<span class="sd"> - Low availability (close to 0) means OVERLOAD</span>
<span class="sd"> - High availability (close to maximum) with no activity means IDLE</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">current_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="n">last_event_delta</span><span class="p">:</span> <span class="nb">float</span> <span class="o">=</span> <span class="n">current_time</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event_time</span>
<span class="c1"># logging_info(</span>
<span class="c1"># &quot;Backpressure: Check conditions, current_availability=%s, max=%s, last_activity=%s&quot;,</span>
<span class="c1"># self.current_availability,</span>
<span class="c1"># self.max_availability,</span>
<span class="c1"># last_event_delta,</span>
<span class="c1"># )</span>
<span class="c1"># Check for OVERLOAD condition - low availability (less than 20% of maximum)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span> <span class="o">&lt;=</span> <span class="nb">round</span><span class="p">(</span><span class="mf">0.1</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">):</span>
<span class="c1"># Check if the last backpressure event was not an overload</span>
<span class="k">if</span> <span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event</span> <span class="o">!=</span> <span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">OVERLOAD</span>
<span class="ow">or</span> <span class="n">last_event_delta</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">backpressure</span><span class="o">.</span><span class="n">idle_duration</span>
<span class="p">):</span>
<span class="c1"># Trigger the overload event</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_backpressure_overload_event</span><span class="p">()</span>
<span class="c1"># update / reset time-window so that the OVERLOAD is not sent too often</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event_time</span> <span class="o">=</span> <span class="n">current_time</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event</span> <span class="o">=</span> <span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">OVERLOAD</span>
<span class="c1"># Check for IDLE condition - high availability (more than 80% of maximum) with no activity</span>
<span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span> <span class="o">&gt;=</span> <span class="nb">round</span><span class="p">(</span><span class="mf">0.8</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">):</span>
<span class="n">idle_duration</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">backpressure</span><span class="o">.</span><span class="n">idle_duration</span>
<span class="c1"># Check if service has been idle for longer than the configured duration</span>
<span class="k">if</span> <span class="p">(</span>
<span class="n">current_time</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">idle_state_detected_time</span> <span class="o">&gt;</span> <span class="n">idle_duration</span>
<span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event</span> <span class="o">==</span> <span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">IDLE</span>
<span class="p">):</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;Backpressure: Service has been idle for </span><span class="si">%s</span><span class="s2"> seconds (threshold: </span><span class="si">%s</span><span class="s2"> seconds)&quot;</span><span class="p">,</span>
<span class="n">last_event_delta</span><span class="p">,</span>
<span class="n">idle_duration</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Trigger the idle event</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_handle_backpressure_idle_event</span><span class="p">()</span>
<span class="c1"># update / reset time-window so that the IDLE is not sent too often</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event_time</span> <span class="o">=</span> <span class="n">current_time</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idle_state_detected_time</span> <span class="o">=</span> <span class="n">current_time</span>
<span class="c1"># send IDLE even only once</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event</span> <span class="o">=</span> <span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">UPDATE</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># Don&#39;t send IDLE right away when the availability increases, but wait for a while, send UPDATE instead</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event</span> <span class="o">=</span> <span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">IDLE</span>
<span class="k">if</span> <span class="n">current_time</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">idle_state_detected_time</span> <span class="o">&gt;</span> <span class="n">idle_duration</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idle_state_detected_time</span> <span class="o">=</span> <span class="n">current_time</span>
<span class="k">if</span> <span class="n">last_event_delta</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">backpressure</span><span class="o">.</span><span class="n">time_window</span><span class="p">:</span>
<span class="n">_scaling_request</span><span class="p">:</span> <span class="n">ScaleRequestV1</span> <span class="o">=</span> <span class="n">ScaleRequestV1</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_service_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_task_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span><span class="p">,</span> <span class="c1"># Current availability is passed directly</span>
<span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">UPDATE</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Address the message to any adapter capable of supporting BACKPRESSURE request</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">publish_backpressure_request</span><span class="p">(</span><span class="n">_scaling_request</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event_time</span> <span class="o">=</span> <span class="n">current_time</span>
<span class="c1"># If neither OVERLOAD nor IDLE, and it&#39;s time for an update, send UPDATE event</span>
<span class="k">elif</span> <span class="n">last_event_delta</span> <span class="o">&gt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">backpressure</span><span class="o">.</span><span class="n">time_window</span><span class="p">:</span>
<span class="n">_scaling_request</span><span class="p">:</span> <span class="n">ScaleRequestV1</span> <span class="o">=</span> <span class="n">ScaleRequestV1</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_service_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_task_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span><span class="p">,</span> <span class="c1"># Current availability is passed directly</span>
<span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">UPDATE</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Address the message to any adapter capable of supporting BACKPRESSURE request</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">publish_backpressure_request</span><span class="p">(</span><span class="n">_scaling_request</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event_time</span> <span class="o">=</span> <span class="n">current_time</span>
<span class="bp">self</span><span class="o">.</span><span class="n">last_backpressure_event</span> <span class="o">=</span> <span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">UPDATE</span></div>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_backpressure_monitor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Periodically monitor the backpressure conditions and trigger events accordingly&quot;&quot;&quot;</span>
<span class="n">_monitor_interval</span> <span class="o">=</span> <span class="mf">0.5</span> <span class="c1"># Monitor every 500ms</span>
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">_do_loop</span><span class="p">():</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_overload_condition</span><span class="p">()</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">_monitor_interval</span><span class="p">)</span>
<div class="viewcode-block" id="BackpressureHandler.backpressure_monitor_loop">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.backpressure_monitor_loop">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">backpressure_monitor_loop</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">_loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">new_event_loop</span><span class="p">()</span>
<span class="n">_loop</span><span class="o">.</span><span class="n">run_until_complete</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_backpressure_monitor</span><span class="p">())</span></div>
<div class="viewcode-block" id="BackpressureHandler.handle_backpressure_overload_event">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.handle_backpressure_overload_event">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">handle_backpressure_overload_event</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">logging_warning</span><span class="p">(</span><span class="s2">&quot;Backpressure: Capacity close to depleted!&quot;</span><span class="p">)</span>
<span class="c1"># Send an Overload event to the Management service</span>
<span class="n">scaling_request</span><span class="p">:</span> <span class="n">ScaleRequestV1</span> <span class="o">=</span> <span class="n">ScaleRequestV1</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_service_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_task_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span><span class="p">,</span> <span class="c1"># Current availability is passed directly</span>
<span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">OVERLOAD</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Address the message to any management service instance capable of supporting BACKPRESSURE request</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">publish_backpressure_request</span><span class="p">(</span><span class="n">scaling_request</span><span class="p">)</span></div>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_handle_backpressure_idle_event</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">logging_warning</span><span class="p">(</span><span class="s2">&quot;Backpressure: Service is idle.&quot;</span><span class="p">)</span>
<span class="c1"># Send an Idle event to the Management service</span>
<span class="n">scaling_request</span><span class="p">:</span> <span class="n">ScaleRequestV1</span> <span class="o">=</span> <span class="n">ScaleRequestV1</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_service_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_task_id</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span><span class="p">,</span> <span class="c1"># Current availability is passed directly</span>
<span class="n">ScalingRequestAlert</span><span class="o">.</span><span class="n">IDLE</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Address the message to any adapter capable of supporting BACKPRESSURE request</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">publish_backpressure_request</span><span class="p">(</span><span class="n">scaling_request</span><span class="p">)</span>
<div class="viewcode-block" id="BackpressureHandler.publish_backpressure_request">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.backpressure_handler.BackpressureHandler.publish_backpressure_request">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">publish_backpressure_request</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">scaling_request</span><span class="p">:</span> <span class="n">ScaleRequestV1</span><span class="p">):</span>
<span class="c1"># Publish the backpressure request to the management service</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Publishing backpressure for </span><span class="si">{</span><span class="n">scaling_request</span><span class="o">.</span><span class="n">serviceId</span><span class="si">}</span><span class="s2"> with request type </span><span class="si">{</span><span class="n">scaling_request</span><span class="o">.</span><span class="n">requestType</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">exchange</span><span class="p">:</span>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_wrap_rabbit_mq_api_init</span><span class="p">(</span><span class="n">channel</span><span class="p">):</span>
<span class="n">_exchange</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">get_exchange</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;cleverthis.clevermicro.management&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_exchange</span>
<span class="k">if</span> <span class="n">BackpressureHandler</span><span class="o">.</span><span class="n">_callback_list</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">BackpressureHandler</span><span class="o">.</span><span class="n">_callback_list</span><span class="p">[</span><span class="s2">&quot;_wrap_rabbit_mq_api_init&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="bp">self</span><span class="o">.</span><span class="n">exchange</span> <span class="o">=</span> <span class="k">await</span> <span class="n">await_result</span><span class="p">(</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span><span class="n">_wrap_rabbit_mq_api_init</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">exchange</span><span class="p">:</span>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_wrap_rabbit_mq_api</span><span class="p">():</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">is_closed</span><span class="p">:</span>
<span class="n">binary_content</span><span class="p">:</span> <span class="nb">bytes</span> <span class="o">=</span> <span class="n">scaling_request</span><span class="o">.</span><span class="n">to_json</span><span class="p">()</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span>
<span class="n">pika_message</span><span class="p">:</span> <span class="n">Message</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span>
<span class="n">body</span><span class="o">=</span><span class="n">binary_content</span><span class="p">,</span>
<span class="n">content_encoding</span><span class="o">=</span><span class="s2">&quot;utf-8&quot;</span><span class="p">,</span>
<span class="n">delivery_mode</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
<span class="n">content_type</span><span class="o">=</span><span class="s2">&quot;application/octet-stream&quot;</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="n">priority</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
<span class="n">correlation_id</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">exchange</span><span class="o">.</span><span class="n">publish</span><span class="p">(</span>
<span class="n">message</span><span class="o">=</span><span class="n">pika_message</span><span class="p">,</span> <span class="n">routing_key</span><span class="o">=</span><span class="s2">&quot;backpressure-scaling-v1&quot;</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;Service Message Published to </span><span class="si">%s</span><span class="s2">, msg: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">exchange</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="nb">str</span><span class="p">(</span><span class="n">binary_content</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">if</span> <span class="n">BackpressureHandler</span><span class="o">.</span><span class="n">_callback_list</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">BackpressureHandler</span><span class="o">.</span><span class="n">_callback_list</span><span class="p">[</span><span class="s2">&quot;_wrap_rabbit_mq_api&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">await</span> <span class="n">await_future</span><span class="p">(</span><span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span><span class="n">_wrap_rabbit_mq_api</span><span class="p">(),</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">))</span></div>
<span class="k">def</span><span class="w"> </span><span class="nf">_do_loop</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Helper function for unit tests to perform several loops only. Check if the loop should continue running.</span>
<span class="sd"> Positive value of do_loop indicates the number of iterations left.</span>
<span class="sd"> Negative value indicates the loop should run indefinitely.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_val</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">do_loop</span> <span class="o">!=</span> <span class="mi">0</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">do_loop</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">do_loop</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="n">_val</span></div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/cleverthis_service_adapter.html
-948
View File
@@ -1,948 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.cleverthis_service_adapter &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.cleverthis_service_adapter</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.cleverthis_service_adapter</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">The layer that interfaces with the CleverThis service that this Adapter is integrating with.</span>
<span class="sd">Methods here will invoke the actual CleverThis Service code, which are passed as Callable object.</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">enum</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">inspect</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">pathlib</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">re</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">asyncio</span><span class="w"> </span><span class="kn">import</span> <span class="n">AbstractEventLoop</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">collections</span><span class="w"> </span><span class="kn">import</span> <span class="n">defaultdict</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Callable</span><span class="p">,</span> <span class="n">Coroutine</span><span class="p">,</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">List</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika.abc</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">AbstractExchange</span><span class="p">,</span>
<span class="n">AbstractRobustChannel</span><span class="p">,</span>
<span class="n">AbstractRobustConnection</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aiohttp</span><span class="w"> </span><span class="kn">import</span> <span class="n">ClientResponse</span><span class="p">,</span> <span class="n">ClientSession</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">fastapi</span><span class="w"> </span><span class="kn">import</span> <span class="n">HTTPException</span><span class="p">,</span> <span class="n">UploadFile</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">fastapi.params</span><span class="w"> </span><span class="kn">import</span> <span class="n">File</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">fastapi.routing</span><span class="w"> </span><span class="kn">import</span> <span class="n">APIRoute</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">fastapi.security</span><span class="w"> </span><span class="kn">import</span> <span class="n">OAuth2PasswordRequestForm</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">pydantic</span><span class="w"> </span><span class="kn">import</span> <span class="n">BaseModel</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">starlette.datastructures</span><span class="w"> </span><span class="kn">import</span> <span class="n">Headers</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">starlette.responses</span><span class="w"> </span><span class="kn">import</span> <span class="n">FileResponse</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter</span><span class="w"> </span><span class="kn">import</span> <span class="n">serializer</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.data_parser</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQDataParser</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.data_response_factory</span><span class="w"> </span><span class="kn">import</span> <span class="n">DataResponseFactory</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.file_handler</span><span class="w"> </span><span class="kn">import</span> <span class="n">FileHandler</span><span class="p">,</span> <span class="n">StreamingFileHandler</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">logging_debug</span><span class="p">,</span>
<span class="n">logging_error</span><span class="p">,</span>
<span class="n">logging_info</span><span class="p">,</span>
<span class="n">logging_warning</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.pydantic_serializer</span><span class="w"> </span><span class="kn">import</span> <span class="n">python_type_to_json</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.type_descriptor</span><span class="w"> </span><span class="kn">import</span> <span class="n">TypeDescriptor</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.config.amq_configuration</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQConfiguration</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQErrorMessage</span><span class="p">,</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">DataMessage</span><span class="p">,</span> <span class="n">DataResponse</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.router_base</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQ_REPLY_TO_EXCHANGE</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">await_result</span>
<span class="n">g_extra_init_data</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_create_reply_exchange_and_queue</span><span class="p">(</span>
<span class="n">connection</span><span class="p">:</span> <span class="n">AbstractRobustConnection</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="n">AbstractRobustChannel</span><span class="p">,</span> <span class="n">AbstractExchange</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Create objects for reply.</span>
<span class="sd"> Return (channel for reply queue, exchange for reply, reply queue name)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Wrap calls to RabbitMQ API to make it Awaitable, so it can be run thread-safe with</span>
<span class="c1"># correct event loop. This is needed because the RabbitMQ API is not thread-safe and needs</span>
<span class="c1"># to be called in the context of event loop the first RabbitMQ connection was created with.</span>
<span class="n">_channel</span> <span class="o">=</span> <span class="k">await</span> <span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span>
<span class="n">_exchange_name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">AMQ_REPLY_TO_EXCHANGE</span>
<span class="n">_exchange</span> <span class="o">=</span> <span class="k">await</span> <span class="n">_channel</span><span class="o">.</span><span class="n">get_exchange</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">_exchange_name</span><span class="p">,</span> <span class="n">ensure</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">_queue</span> <span class="o">=</span> <span class="k">await</span> <span class="n">_channel</span><span class="o">.</span><span class="n">declare_queue</span><span class="p">(</span><span class="n">exclusive</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="c1"># Declare a unique, non-exclusive queue</span>
<span class="n">_queue_name</span> <span class="o">=</span> <span class="n">_queue</span><span class="o">.</span><span class="n">name</span>
<span class="k">await</span> <span class="n">_queue</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span>
<span class="n">exchange</span><span class="o">=</span><span class="n">_exchange_name</span><span class="p">,</span>
<span class="n">routing_key</span><span class="o">=</span><span class="n">_queue_name</span><span class="p">,</span> <span class="c1"># Use queue name as routing key</span>
<span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Publishing to exchange: </span><span class="si">{</span><span class="n">_exchange_name</span><span class="si">}</span><span class="s2">, q: </span><span class="si">{</span><span class="n">_queue_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_channel</span><span class="p">,</span> <span class="n">_exchange</span><span class="p">,</span> <span class="n">_queue_name</span>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_convert_file_response</span><span class="p">(</span>
<span class="n">obj</span><span class="p">:</span> <span class="n">FileResponse</span><span class="p">,</span>
<span class="n">connection</span><span class="p">:</span> <span class="n">AbstractRobustConnection</span><span class="p">,</span>
<span class="n">file_handler</span><span class="p">:</span> <span class="n">FileHandler</span><span class="p">,</span>
<span class="n">loop</span><span class="p">:</span> <span class="n">AbstractEventLoop</span> <span class="o">|</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Converts a FileResponse object to a JSON string containing the filename and stream name.</span>
<span class="sd"> The file content itself is transmitted through the named stream (which is implemented as</span>
<span class="sd"> temporary dedicated queue.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># All calls to RabbitMQ API have to be called threadsafe, because those appear to be attached</span>
<span class="c1"># to parent event loop, and need to execute at that event loop, not the current_availability one</span>
<span class="c1"># allow coroutine to execute. calling _future.result() will block the entire thread</span>
<span class="c1"># and the coroutine will never execute</span>
<span class="n">_channel</span><span class="p">,</span> <span class="n">_exchange</span><span class="p">,</span> <span class="n">_routing_key</span> <span class="o">=</span> <span class="k">await</span> <span class="n">await_result</span><span class="p">(</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span><span class="n">_create_reply_exchange_and_queue</span><span class="p">(</span><span class="n">connection</span><span class="p">),</span> <span class="n">loop</span><span class="p">)</span>
<span class="p">)</span>
<span class="c1"># publish_file also needs RabbitMQ API, ensure the loop is passed as argument</span>
<span class="p">(</span><span class="n">_stream_name</span><span class="p">,</span> <span class="n">_file_size</span><span class="p">)</span> <span class="o">=</span> <span class="k">await</span> <span class="n">file_handler</span><span class="o">.</span><span class="n">publish_file</span><span class="p">(</span>
<span class="n">_exchange</span><span class="p">,</span> <span class="n">obj</span><span class="o">.</span><span class="n">path</span><span class="p">,</span> <span class="n">_routing_key</span><span class="p">,</span> <span class="n">loop</span>
<span class="p">)</span>
<span class="c1"># as above, but luckily this time we don&#39;t need to wait for the result</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span><span class="n">_channel</span><span class="o">.</span><span class="n">close</span><span class="p">(),</span> <span class="n">loop</span><span class="p">)</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span>
<span class="p">{</span>
<span class="s2">&quot;files&quot;</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="s2">&quot;filename&quot;</span><span class="p">:</span> <span class="n">obj</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span>
<span class="s2">&quot;mediaType&quot;</span><span class="p">:</span> <span class="s2">&quot;application/octet-stream&quot;</span><span class="p">,</span>
<span class="s2">&quot;streamName&quot;</span><span class="p">:</span> <span class="n">_stream_name</span><span class="p">,</span>
<span class="s2">&quot;fileSize&quot;</span><span class="p">:</span> <span class="n">_file_size</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">)</span>
<div class="viewcode-block" id="is_likely_json">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.is_likely_json">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">is_likely_json</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Quick test to see if should convert the response string to JSON format.&quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="n">_text</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">return</span> <span class="p">(</span><span class="n">_text</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;{&quot;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">_text</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">&quot;}&quot;</span><span class="p">))</span> <span class="ow">or</span> <span class="p">(</span>
<span class="n">_text</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;[&quot;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">_text</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">&quot;]&quot;</span><span class="p">)</span>
<span class="p">)</span></div>
<span class="c1"># ============= CleverThisServiceAdapter =============</span>
<div class="viewcode-block" id="CleverThisServiceAdapter">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">CleverThisServiceAdapter</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> A base class for CleverThis service adapter that handles the incoming AMQP messages and routes</span>
<span class="sd"> them to the appropriate CleverThisService API endpoint.</span>
<span class="sd"> IMPORTANT: This class is not intended to be used directly. It should be subclassed for</span>
<span class="sd"> each CleverThis service and override the methods to handle the specific API calls.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="CleverThisServiceAdapter.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">amq_configuration</span><span class="p">:</span> <span class="n">AMQConfiguration</span><span class="p">,</span>
<span class="n">routes</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">APIRoute</span><span class="p">],</span>
<span class="n">session</span><span class="p">:</span> <span class="n">ClientSession</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">authenticated_user_arg_name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;authenticated_user&quot;</span><span class="p">,</span>
<span class="n">extra_init_data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Initializes the CleverThisServiceAdapter with the given AMQ configuration and optional HTTP</span>
<span class="sd"> session. The HTTP session is for the case when the adapter uses loose coupling with</span>
<span class="sd"> the CleverThis service via HTTP REST API.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="n">extra_init_data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">extra_init_data</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">authenticated_user_arg_name</span> <span class="o">=</span> <span class="n">authenticated_user_arg_name</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span> <span class="o">=</span> <span class="n">amq_configuration</span>
<span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="o">=</span> <span class="n">session</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service_host</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">service_host</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service_port</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">service_port</span>
<span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">:</span> <span class="n">AbstractEventLoop</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">require_authenticated_user</span> <span class="o">=</span> <span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">amq_adapter</span><span class="o">.</span><span class="n">require_authenticated_user</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">routes</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">group_routes_by_method</span><span class="p">(</span><span class="n">api_routes</span><span class="o">=</span><span class="n">routes</span><span class="p">)</span>
<span class="k">global</span> <span class="n">g_extra_init_data</span>
<span class="n">g_extra_init_data</span> <span class="o">=</span> <span class="n">extra_init_data</span> <span class="ow">or</span> <span class="p">{}</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.get_endpoint">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.get_endpoint">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_endpoint</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">endpoint</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Any</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Optionally, Service can override this method to alter the endpoint mapping.</span>
<span class="sd"> This is useful for example when the CleverThis service has a JSON wrapper</span>
<span class="sd"> around the endpoint, like CleverSwarm, which might be the preferred endpoint to call.</span>
<span class="sd"> :param endpoint: The original endpoint to be called.</span>
<span class="sd"> :return: The actual endpoint to be called.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">endpoint</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.clone_and_adapt_route">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.clone_and_adapt_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">clone_and_adapt_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">original_route</span><span class="p">:</span> <span class="n">APIRoute</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">APIRoute</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Creates a clone of APIRoute. Checks if there&#39;s preferred endpoint with json serialization.</span>
<span class="sd"> It also adjusts the path regex to ensure it matches the path in different API versions.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Need to adapt also the path regex to ensure it matches the path in different API versions</span>
<span class="c1"># Ensures the leading &#39;^&#39; is followed by any match and there&#39;s no &#39;/&#39; before final &#39;$&#39;</span>
<span class="n">_new_regex_string</span> <span class="o">=</span> <span class="s2">&quot;^.*&quot;</span> <span class="o">+</span> <span class="n">original_route</span><span class="o">.</span><span class="n">path_regex</span><span class="o">.</span><span class="n">pattern</span><span class="o">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s2">&quot;^&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s2">&quot;/$&quot;</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;$&quot;</span>
<span class="n">_new_route</span><span class="p">:</span> <span class="n">APIRoute</span> <span class="o">=</span> <span class="n">APIRoute</span><span class="p">(</span>
<span class="n">path</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">path</span><span class="p">,</span>
<span class="n">endpoint</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">get_endpoint</span><span class="p">(</span>
<span class="n">original_route</span><span class="o">.</span><span class="n">endpoint</span>
<span class="p">),</span> <span class="c1"># Service can alter the endpoint mapping here</span>
<span class="n">response_model</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_model</span><span class="p">,</span>
<span class="n">status_code</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span>
<span class="n">tags</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">tags</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="k">if</span> <span class="n">original_route</span><span class="o">.</span><span class="n">tags</span> <span class="k">else</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">dependencies</span><span class="o">=</span><span class="p">(</span>
<span class="n">original_route</span><span class="o">.</span><span class="n">dependencies</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="k">if</span> <span class="n">original_route</span><span class="o">.</span><span class="n">dependencies</span> <span class="k">else</span> <span class="kc">None</span>
<span class="p">),</span>
<span class="n">summary</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">summary</span><span class="p">,</span>
<span class="n">description</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">description</span><span class="p">,</span>
<span class="n">response_description</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_description</span><span class="p">,</span>
<span class="n">responses</span><span class="o">=</span><span class="p">(</span><span class="n">original_route</span><span class="o">.</span><span class="n">responses</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="k">if</span> <span class="n">original_route</span><span class="o">.</span><span class="n">responses</span> <span class="k">else</span> <span class="kc">None</span><span class="p">),</span>
<span class="n">deprecated</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">deprecated</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="n">methods</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">methods</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="k">if</span> <span class="n">original_route</span><span class="o">.</span><span class="n">methods</span> <span class="k">else</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">operation_id</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">operation_id</span><span class="p">,</span>
<span class="n">response_model_include</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_model_include</span><span class="p">,</span>
<span class="n">response_model_exclude</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_model_exclude</span><span class="p">,</span>
<span class="n">response_model_by_alias</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_model_by_alias</span><span class="p">,</span>
<span class="n">response_model_exclude_unset</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_model_exclude_unset</span><span class="p">,</span>
<span class="n">response_model_exclude_defaults</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_model_exclude_defaults</span><span class="p">,</span>
<span class="n">response_model_exclude_none</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_model_exclude_none</span><span class="p">,</span>
<span class="n">include_in_schema</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">include_in_schema</span><span class="p">,</span>
<span class="n">response_class</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">response_class</span><span class="p">,</span>
<span class="n">dependency_overrides_provider</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">dependency_overrides_provider</span><span class="p">,</span>
<span class="n">callbacks</span><span class="o">=</span><span class="p">(</span><span class="n">original_route</span><span class="o">.</span><span class="n">callbacks</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="k">if</span> <span class="n">original_route</span><span class="o">.</span><span class="n">callbacks</span> <span class="k">else</span> <span class="kc">None</span><span class="p">),</span>
<span class="n">openapi_extra</span><span class="o">=</span><span class="p">(</span>
<span class="n">original_route</span><span class="o">.</span><span class="n">openapi_extra</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="k">if</span> <span class="n">original_route</span><span class="o">.</span><span class="n">openapi_extra</span> <span class="k">else</span> <span class="kc">None</span>
<span class="p">),</span>
<span class="n">generate_unique_id_function</span><span class="o">=</span><span class="n">original_route</span><span class="o">.</span><span class="n">generate_unique_id_function</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">_new_route</span><span class="o">.</span><span class="n">path_regex</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">_new_regex_string</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_new_route</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.group_routes_by_method">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.group_routes_by_method">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">group_routes_by_method</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">api_routes</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="n">APIRoute</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="n">APIRoute</span><span class="p">]]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Groups API routes by their HTTP method. This is important to ensure that URLPath is matched</span>
<span class="sd"> correctly, as different methods may share the same URLPath (e.g. GET, POST PUT, DELETE).</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_method_to_routes</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span> <span class="c1"># Automatically initializes new lists for new keys</span>
<span class="k">for</span> <span class="n">_route</span> <span class="ow">in</span> <span class="n">api_routes</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span>
<span class="nb">isinstance</span><span class="p">(</span><span class="n">_route</span><span class="p">,</span> <span class="n">APIRoute</span><span class="p">)</span> <span class="ow">and</span> <span class="n">_route</span><span class="o">.</span><span class="n">methods</span>
<span class="p">):</span> <span class="c1"># Ensure methods exist (should always be true for valid routes)</span>
<span class="k">for</span> <span class="n">method</span> <span class="ow">in</span> <span class="n">_route</span><span class="o">.</span><span class="n">methods</span><span class="p">:</span>
<span class="n">_method_to_routes</span><span class="p">[</span><span class="n">method</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">clone_and_adapt_route</span><span class="p">(</span><span class="n">_route</span><span class="p">))</span>
<span class="k">return</span> <span class="nb">dict</span><span class="p">(</span><span class="n">_method_to_routes</span><span class="p">)</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.get_content_type">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.get_content_type">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_content_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message_headers</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Pull the value of the Content-Type header from the message headers.</span>
<span class="sd"> :param message_headers: The headers of the message.</span>
<span class="sd"> return: The content type of the message.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">for</span> <span class="n">header</span><span class="p">,</span> <span class="n">values</span> <span class="ow">in</span> <span class="n">message_headers</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="n">header</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s2">&quot;content-type&quot;</span><span class="p">:</span>
<span class="k">return</span> <span class="n">values</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;;&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">return</span> <span class="s2">&quot;application/json&quot;</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.get_current_user">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.get_current_user">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">get_current_user</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">object</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> To Be Overriden in subclass for given CleverThis Service, to return the current_availability user based</span>
<span class="sd"> on the token, in format appropriate for the service.</span>
<span class="sd"> :param token: The token to be used for authentication.</span>
<span class="sd"> :return: A user object or None if the user is not authenticated.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="kc">None</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.on_message">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.on_message">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">on_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Called when a message is received from the AMQP.</span>
<span class="sd"> :param message:</span>
<span class="sd"> :return: AMQP response class with operation status code and optional error details.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_auth_user</span><span class="p">:</span> <span class="n">Any</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">_amq_response</span><span class="p">:</span> <span class="n">Any</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;on_message: require_authenticated_user=</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">require_authenticated_user</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">require_authenticated_user</span><span class="p">:</span>
<span class="n">_auth</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;Authorization&quot;</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Auth: </span><span class="si">{</span><span class="n">_auth</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">_auth</span><span class="p">,</span> <span class="n">List</span><span class="p">):</span>
<span class="n">_auth</span> <span class="o">=</span> <span class="n">_auth</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;Bearer&quot;</span> <span class="ow">in</span> <span class="n">_auth</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">token</span> <span class="o">=</span> <span class="n">_auth</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot; &quot;</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Token: </span><span class="si">{</span><span class="n">token</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">_auth_user</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_current_user</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">error</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;on_message: Error getting user: </span><span class="si">{</span><span class="n">error</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_auth_user</span> <span class="ow">or</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">require_authenticated_user</span> <span class="ow">or</span> <span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">&quot;/login&quot;</span><span class="p">):</span>
<span class="n">method</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">method</span><span class="p">()</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
<span class="k">if</span> <span class="n">method</span> <span class="o">==</span> <span class="s2">&quot;get&quot;</span><span class="p">:</span>
<span class="n">_amq_response</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">_auth_user</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">method</span> <span class="o">==</span> <span class="s2">&quot;put&quot;</span><span class="p">:</span>
<span class="n">_amq_response</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">_auth_user</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">method</span> <span class="o">==</span> <span class="s2">&quot;post&quot;</span><span class="p">:</span>
<span class="n">_amq_response</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">post</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">_auth_user</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">method</span> <span class="o">==</span> <span class="s2">&quot;head&quot;</span><span class="p">:</span>
<span class="n">_amq_response</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">head</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">_auth_user</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">method</span> <span class="o">==</span> <span class="s2">&quot;delete&quot;</span><span class="p">:</span>
<span class="n">_amq_response</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">_auth_user</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unexpected HTTP method: </span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">method</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">_amq_response</span> <span class="o">=</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of_error_message</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">AMQErrorMessage</span><span class="p">(</span><span class="mi">401</span><span class="p">,</span> <span class="s2">&quot;Unauthorized&quot;</span><span class="p">,</span> <span class="s2">&quot;Unauthorized&quot;</span><span class="p">),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;ALL DONE in on_message id:</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">()</span><span class="si">}</span><span class="s2">, resp code:</span><span class="si">{</span><span class="n">_amq_response</span><span class="o">.</span><span class="n">response_code</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">_amq_response</span></div>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_authorize_forward_request</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">,</span> <span class="n">request_coroutine</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> This method is used to authorize the request and forward it to the CleverThis Service API.</span>
<span class="sd"> Used only for loose / 3rd party mode of coupling with the Service</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># AMQDataParser.parse_request_body(message)</span>
<span class="c1"># TODO: implement authorization logic here - ensure the security related headers are set</span>
<span class="c1"># This depends on what kind of security is required by the CleverThis service.</span>
<span class="c1"># However at this time there is no loosely coupled Service.</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">request_coroutine</span>
<span class="c1"># ==============================================================================================</span>
<span class="c1"># ===================== C l e v e r T h i s A P I m e t h o d s ========================</span>
<span class="c1"># ==============================================================================================</span>
<div class="viewcode-block" id="CleverThisServiceAdapter.get">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.get">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the GET requests for the CleverThis API. In tight coupling (self.session is None),</span>
<span class="sd"> use path, path params and query string to determine the endpoint method to invoke.</span>
<span class="sd"> In loose coupling (self.session is not None), forward the request as REST request using</span>
<span class="sd"> the session, i.e. attempt to call the REST endpoint directly using HTTP protocol.</span>
<span class="sd"> :param message: message from the AMQP that contains the GET request</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchem</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_no_body_payload</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">,</span> <span class="s2">&quot;GET&quot;</span><span class="p">)</span>
<span class="c1"># Forwards as REST - try to invoke the service directly on this path</span>
<span class="n">url</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;http://</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_port</span><span class="si">}{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">(),</span> <span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">()}</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;SEND HEADERS out: </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;</span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s1">=</span><span class="si">{</span><span class="n">v</span><span class="si">}</span><span class="s1">&#39;</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">k</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="w"> </span><span class="ow">in</span><span class="w"> </span><span class="n">headers</span><span class="o">.</span><span class="n">items</span><span class="p">())</span><span class="si">}</span><span class="s2">, REQ=</span><span class="si">{</span><span class="n">url</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">_http_resp</span><span class="p">:</span> <span class="n">ClientResponse</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">_http_resp</span><span class="o">.</span><span class="n">status</span><span class="p">,</span>
<span class="n">_http_resp</span><span class="o">.</span><span class="n">content_type</span><span class="p">,</span>
<span class="k">await</span> <span class="n">_http_resp</span><span class="o">.</span><span class="n">read</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.put">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.put">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">put</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the PUT requests for the CleverThis API. Uses path, path params, query string and</span>
<span class="sd"> body payload to determine the endpoint method to invoke and its input arguments</span>
<span class="sd"> In loose coupling (self.session is not None), attempt to call the REST endpoint directly</span>
<span class="sd"> using HTTP protocol.</span>
<span class="sd"> :param message: message from the AMQP that contains the PUT request</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchem</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_body_payload</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">,</span> <span class="s2">&quot;PUT&quot;</span><span class="p">)</span>
<span class="c1"># Forwards as REST - try to invoke the service directly on this path</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize_forward_request</span><span class="p">(</span>
<span class="n">message</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">put</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;http://</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_port</span><span class="si">}{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">(),</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">(),</span> <span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">()},</span>
<span class="p">),</span>
<span class="n">auth_user</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.post">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.post">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">post</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the POST requests for the CleverThis API. Uses path, path params, query string and</span>
<span class="sd"> body payload to determine the endpoint method to invoke and its input arguments</span>
<span class="sd"> In loose coupling (self.session is not None), attempt to call the REST endpoint directly</span>
<span class="sd"> using HTTP protocol.</span>
<span class="sd"> :param message: message from AMQP that contains the POST request, including body and headers</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchem</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_body_payload</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">,</span> <span class="s2">&quot;POST&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize_forward_request</span><span class="p">(</span>
<span class="n">message</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;http://</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_port</span><span class="si">}{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">(),</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">(),</span> <span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">()},</span>
<span class="p">),</span>
<span class="n">auth_user</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.head">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.head">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">head</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the HEAD requests for the CleverThis API. In tight coupling (self.session is None),</span>
<span class="sd"> use path, path params and query string to determine the endpoint method to invoke.</span>
<span class="sd"> In loose coupling (self.session is not None), HEAD call is not supported.</span>
<span class="sd"> :param message: message from AMQP containing the HEAD request, including body and headers</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchema</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_no_body_payload</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">,</span> <span class="s2">&quot;HEAD&quot;</span><span class="p">)</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.delete">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.delete">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">delete</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the DELETE requests for the CleverThis API.</span>
<span class="sd"> :param message: message from AMQP that contains the DELETE request, including body &amp; headers</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchema</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_no_body_payload</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">,</span> <span class="s2">&quot;DELETE&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize_forward_request</span><span class="p">(</span>
<span class="n">message</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;http://</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_port</span><span class="si">}{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">(),</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">(),</span> <span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">()},</span>
<span class="p">),</span>
<span class="n">auth_user</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.options">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.options">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">options</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the OPTIONS requests. In tight coupling (self.session is None),</span>
<span class="sd"> use path, path params and query string to determine the endpoint method to invoke.</span>
<span class="sd"> In loose coupling (self.session is not None), forward the request as REST request using</span>
<span class="sd"> the session, i.e. attempt to call the REST endpoint directly using HTTP protocol.</span>
<span class="sd"> :param message: message from AMQP that contains the OPTIONS request, including headers</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchema</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_no_body_payload</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">,</span> <span class="s2">&quot;OPTIONS&quot;</span><span class="p">)</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.patch">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.patch">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">patch</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the PATCH requests for the CleverBRAG API.</span>
<span class="sd"> :param message: message from AMQP that contains the PATCH request, including body &amp; headers</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchema</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">session</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">handle_no_body_payload</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">,</span> <span class="s2">&quot;PATCH&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize_forward_request</span><span class="p">(</span>
<span class="n">message</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">session</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;http://</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_port</span><span class="si">}{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">data</span><span class="o">=</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">(),</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span><span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">(),</span> <span class="o">**</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">()},</span>
<span class="p">),</span>
<span class="n">auth_user</span><span class="p">,</span>
<span class="p">)</span></div>
<span class="c1"># ==============================================================================================</span>
<span class="c1"># ================= C l e v e r S w a r m p a y l o a d h a n d l e r s ==================</span>
<span class="c1"># ==============================================================================================</span>
<div class="viewcode-block" id="CleverThisServiceAdapter.handle_no_body_payload">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.handle_no_body_payload">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">handle_no_body_payload</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="n">method</span><span class="p">:</span> <span class="nb">str</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the requests that do not have a body payload.</span>
<span class="sd"> :param message: message from the AMQP that contains the request</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchem</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_routes</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">method</span><span class="p">,</span> <span class="p">[])</span>
<span class="n">_path</span><span class="p">,</span> <span class="n">_args</span> <span class="o">=</span> <span class="n">AMQDataParser</span><span class="o">.</span><span class="n">parse_get_url</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">())</span>
<span class="k">for</span> <span class="n">_route</span> <span class="ow">in</span> <span class="n">_routes</span><span class="p">:</span>
<span class="c1"># TODO: here the regex will match with path prefix,</span>
<span class="c1"># but it might not be a stable feature that we could rely on.</span>
<span class="c1"># Need to figure out something stable for the prefix path.</span>
<span class="n">_match</span> <span class="o">=</span> <span class="n">_route</span><span class="o">.</span><span class="n">path_regex</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">_path</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_match</span><span class="p">:</span>
<span class="n">_args</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">_match</span><span class="o">.</span><span class="n">groupdict</span><span class="p">())</span>
<span class="n">_signature_args</span> <span class="o">=</span> <span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">_route</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span><span class="o">.</span><span class="n">parameters</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">authenticated_user_arg_name</span> <span class="ow">in</span> <span class="n">_signature_args</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="n">_args</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">authenticated_user_arg_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">auth_user</span>
<span class="c1"># If the path matches, call the corresponding function</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">call_cleverthis_api</span><span class="p">(</span>
<span class="n">message</span><span class="p">,</span>
<span class="n">_args</span><span class="p">,</span>
<span class="n">api_method</span><span class="o">=</span><span class="n">_route</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
<span class="n">success_code</span><span class="o">=</span><span class="n">_route</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="mi">404</span><span class="p">,</span>
<span class="s2">&quot;text/plain&quot;</span><span class="p">,</span>
<span class="nb">str</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Destination location [</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">] does not exists&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="CleverThisServiceAdapter.handle_body_payload">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.handle_body_payload">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">handle_body_payload</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span> <span class="n">auth_user</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="n">method</span><span class="p">:</span> <span class="nb">str</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles the requests that have a body payload, including Multipart data.</span>
<span class="sd"> :param message: message from the AMQP that contains the request</span>
<span class="sd"> :param auth_user: user authenticated by the API Gateway, as UserSchem</span>
<span class="sd"> :param method: HTTP method (POST, PUT, etc.)</span>
<span class="sd"> :return: DataResponse to be sent back to the API Gateway via AMQP</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_form_data</span> <span class="o">=</span> <span class="n">AMQDataParser</span><span class="o">.</span><span class="n">parse_request_body</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="n">_routes</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">method</span><span class="p">,</span> <span class="p">[])</span>
<span class="n">_path</span><span class="p">,</span> <span class="n">_query_args</span> <span class="o">=</span> <span class="n">AMQDataParser</span><span class="o">.</span><span class="n">parse_get_url</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">())</span>
<span class="k">for</span> <span class="n">_route</span> <span class="ow">in</span> <span class="n">_routes</span><span class="p">:</span>
<span class="n">_match</span> <span class="o">=</span> <span class="n">_route</span><span class="o">.</span><span class="n">path_regex</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">_path</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_match</span><span class="p">:</span>
<span class="c1"># If the path matches, call the corresponding function</span>
<span class="n">_form_data</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">_match</span><span class="o">.</span><span class="n">groupdict</span><span class="p">())</span>
<span class="n">_form_data</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">_query_args</span><span class="p">)</span>
<span class="k">for</span> <span class="n">_body_param</span> <span class="ow">in</span> <span class="n">_route</span><span class="o">.</span><span class="n">dependant</span><span class="o">.</span><span class="n">body_params</span><span class="p">:</span>
<span class="k">if</span> <span class="n">_body_param</span><span class="o">.</span><span class="n">name</span> <span class="ow">in</span> <span class="n">_form_data</span><span class="p">:</span>
<span class="n">_file_data</span> <span class="o">=</span> <span class="n">_form_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">_body_param</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="n">actual_filepath</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">extra_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">_body_param</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="n">_form_data</span><span class="p">[</span><span class="n">_body_param</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span>
<span class="p">(</span>
<span class="n">_body_param</span><span class="o">.</span><span class="n">type_</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;UploadFile&quot;</span>
<span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">_body_param</span><span class="o">.</span><span class="n">field_info</span><span class="p">,</span> <span class="n">File</span><span class="p">)</span>
<span class="p">)</span>
<span class="ow">and</span> <span class="n">_file_data</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="ow">and</span> <span class="n">actual_filepath</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="ow">and</span> <span class="n">UploadFile</span><span class="p">(</span>
<span class="n">file</span><span class="o">=</span><span class="n">pathlib</span><span class="o">.</span><span class="n">Path</span><span class="p">(</span><span class="n">actual_filepath</span><span class="p">)</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s2">&quot;rb&quot;</span><span class="p">),</span>
<span class="n">size</span><span class="o">=</span><span class="n">_file_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;size&quot;</span><span class="p">),</span>
<span class="n">filename</span><span class="o">=</span><span class="n">_file_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;filename&quot;</span><span class="p">),</span>
<span class="n">headers</span><span class="o">=</span><span class="n">Headers</span><span class="p">({</span><span class="s2">&quot;Content-Type&quot;</span><span class="p">:</span> <span class="n">_file_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;mediaType&quot;</span><span class="p">)}),</span>
<span class="p">)</span>
<span class="ow">or</span> <span class="n">_form_data</span><span class="p">[</span><span class="n">_body_param</span><span class="o">.</span><span class="n">name</span><span class="p">]</span>
<span class="p">)</span>
<span class="n">_verified_args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">_required_args</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">_route</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span> <span class="s2">&quot;__annotations__&quot;</span><span class="p">,</span> <span class="p">{})</span>
<span class="n">_signature_args</span> <span class="o">=</span> <span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">_route</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span><span class="o">.</span><span class="n">parameters</span>
<span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">_required_args</span><span class="p">:</span>
<span class="k">if</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">_form_data</span><span class="p">:</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="p">]</span> <span class="o">=</span> <span class="n">_form_data</span><span class="p">[</span><span class="n">arg</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># needed for CleverSwarm /login input</span>
<span class="k">if</span> <span class="n">_required_args</span><span class="p">[</span><span class="n">arg</span><span class="p">]</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&quot;OAuth2PasswordRequestForm&quot;</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="p">]</span> <span class="o">=</span> <span class="n">OAuth2PasswordRequestForm</span><span class="p">(</span><span class="o">**</span><span class="n">_form_data</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Error: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">authenticated_user_arg_name</span> <span class="ow">in</span> <span class="n">_signature_args</span><span class="o">.</span><span class="n">keys</span><span class="p">():</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">authenticated_user_arg_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">auth_user</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">call_cleverthis_api</span><span class="p">(</span>
<span class="n">message</span><span class="p">,</span>
<span class="n">_verified_args</span><span class="p">,</span>
<span class="n">api_method</span><span class="o">=</span><span class="n">_route</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
<span class="n">success_code</span><span class="o">=</span><span class="n">_route</span><span class="o">.</span><span class="n">status_code</span> <span class="k">if</span> <span class="n">_route</span><span class="o">.</span><span class="n">status_code</span> <span class="k">else</span> <span class="mi">200</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="mi">404</span><span class="p">,</span>
<span class="s2">&quot;text/plain&quot;</span><span class="p">,</span>
<span class="nb">str</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Destination location [</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">] does not exists&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span></div>
<span class="k">def</span><span class="w"> </span><span class="nf">_verify_and_instantiate_arguments</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span>
<span class="n">api_method</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[</span><span class="n">Any</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">Coroutine</span><span class="p">],</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Check the provided arguments against the API method signature and convert the incoming string value</span>
<span class="sd"> into correct expected input object.</span>
<span class="sd"> The logic used FastAPI annotation and would use the value from &#39;default_factory&#39; invocation (where defined)</span>
<span class="sd"> or &#39;default&#39; value, as defined in the FastAPI annotation, to create default value in case the object is not in</span>
<span class="sd"> the input values provided by the caller.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_verified_args</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_sig</span> <span class="o">=</span> <span class="n">inspect</span><span class="o">.</span><span class="n">signature</span><span class="p">(</span><span class="n">api_method</span><span class="p">)</span>
<span class="n">_required_args</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">_sig</span><span class="o">.</span><span class="n">parameters</span><span class="o">.</span><span class="n">values</span><span class="p">())</span>
<span class="c1"># _required_args = getattr(api_method, &#39;__annotations__&#39;, {})</span>
<span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">_required_args</span><span class="p">:</span>
<span class="k">if</span> <span class="n">arg</span><span class="o">.</span><span class="n">name</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
<span class="n">_annotation</span> <span class="o">=</span> <span class="n">arg</span><span class="o">.</span><span class="n">annotation</span>
<span class="n">_str_annotation</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">_annotation</span><span class="p">)</span>
<span class="c1"># convert to another type if arg is not a str</span>
<span class="k">if</span> <span class="n">arg</span><span class="o">.</span><span class="n">name</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">authenticated_user_arg_name</span><span class="p">:</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">_annotation</span> <span class="o">==</span> <span class="nb">int</span><span class="p">:</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">_annotation</span> <span class="o">==</span> <span class="nb">float</span><span class="p">:</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">_annotation</span> <span class="o">==</span> <span class="nb">bool</span><span class="p">:</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">serializer</span><span class="o">.</span><span class="n">str_to_bool</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
<span class="k">elif</span> <span class="p">(</span>
<span class="s2">&quot;Optional&quot;</span> <span class="ow">in</span> <span class="n">_str_annotation</span>
<span class="ow">or</span> <span class="n">_str_annotation</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;list&quot;</span><span class="p">)</span>
<span class="ow">or</span> <span class="n">_str_annotation</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.List&quot;</span><span class="p">)</span>
<span class="ow">or</span> <span class="s2">&quot;class&quot;</span> <span class="ow">in</span> <span class="n">_str_annotation</span>
<span class="p">):</span>
<span class="n">_type_descr</span> <span class="o">=</span> <span class="n">TypeDescriptor</span><span class="p">(</span>
<span class="n">_str_annotation</span><span class="p">,</span> <span class="n">extra_init_data</span><span class="o">=</span><span class="n">g_extra_init_data</span>
<span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_payload</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">)</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">_type_descr</span><span class="o">.</span><span class="n">instantiate</span><span class="p">(</span>
<span class="n">_payload</span><span class="p">,</span> <span class="n">is_json</span><span class="o">=</span><span class="n">is_likely_json</span><span class="p">(</span><span class="n">_payload</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Could not instantiate argument </span><span class="si">{</span><span class="n">arg</span><span class="si">}</span><span class="s2">, defaulting to its string value&quot;</span><span class="p">,</span>
<span class="n">err</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">_temp</span> <span class="o">=</span> <span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">arg</span><span class="o">.</span><span class="n">annotation</span><span class="p">,</span> <span class="n">enum</span><span class="o">.</span><span class="n">EnumType</span><span class="p">)</span> <span class="ow">or</span> <span class="nb">isinstance</span><span class="p">(</span>
<span class="n">arg</span><span class="o">.</span><span class="n">annotation</span><span class="p">,</span> <span class="n">enum</span><span class="o">.</span><span class="n">Enum</span>
<span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_temp</span> <span class="o">=</span> <span class="n">arg</span><span class="o">.</span><span class="n">annotation</span><span class="p">[</span><span class="n">_temp</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">KeyError</span> <span class="k">as</span> <span class="n">ke</span><span class="p">:</span>
<span class="c1"># some enums have string value different to key spelling, try different thing</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_temp</span> <span class="o">=</span> <span class="n">arg</span><span class="o">.</span><span class="n">annotation</span><span class="p">(</span><span class="n">_temp</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">ve</span><span class="p">:</span>
<span class="n">logging_warning</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;cannot convert &#39;</span><span class="si">{</span><span class="n">_temp</span><span class="si">}</span><span class="s2">&#39; to Enum type </span><span class="si">{</span><span class="n">arg</span><span class="o">.</span><span class="n">annotation</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">ke</span><span class="si">}</span><span class="s2">, </span><span class="si">{</span><span class="n">ve</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;ENUM -&gt; </span><span class="si">{</span><span class="n">_temp</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">_temp</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">arg</span><span class="p">,</span> <span class="s2">&quot;default&quot;</span><span class="p">):</span>
<span class="n">_temp</span> <span class="o">=</span> <span class="n">arg</span><span class="o">.</span><span class="n">default</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_temp</span><span class="p">,</span> <span class="s2">&quot;default_factory&quot;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">_temp</span><span class="o">.</span><span class="n">default_factory</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">_temp</span> <span class="o">=</span> <span class="n">_temp</span><span class="o">.</span><span class="n">default_factory</span><span class="p">()</span>
<span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_temp</span><span class="p">,</span> <span class="s2">&quot;default&quot;</span><span class="p">):</span>
<span class="n">_temp</span> <span class="o">=</span> <span class="n">_temp</span><span class="o">.</span><span class="n">default</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">str</span><span class="p">(</span><span class="n">_temp</span><span class="p">)</span> <span class="o">==</span> <span class="s2">&quot;&lt;class &#39;inspect._empty&#39;&gt;&quot;</span><span class="p">:</span>
<span class="n">_verified_args</span><span class="p">[</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="p">]</span> <span class="o">=</span> <span class="n">_temp</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_warning</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">api_method</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2">: Missing required argument: </span><span class="si">{</span><span class="n">arg</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;Failed to validate and instantiate input args&quot;</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_verified_args</span>
<div class="viewcode-block" id="CleverThisServiceAdapter.call_cleverthis_api">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.cleverthis_service_adapter.CleverThisServiceAdapter.call_cleverthis_api">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">call_cleverthis_api</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">,</span>
<span class="n">args</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span>
<span class="n">api_method</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[</span><span class="n">Any</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">Coroutine</span><span class="p">],</span>
<span class="n">success_code</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles POST and PUT requests, which typically contain possibly complex payload data,</span>
<span class="sd"> passed as a dictionary. Invokes the provided API method with the given arguments and</span>
<span class="sd"> authenticated user, and returns the response as DataResponse.</span>
<span class="sd"> :param message: DataMessage containing the details of the call and the payload</span>
<span class="sd"> :param args: map of the arguments to pass to the API method</span>
<span class="sd"> :param api_method: Callable - the method to invoke. Last parameter is the authenticated user</span>
<span class="sd"> :param success_code: HTTP code to be returned when operation suceeds</span>
<span class="sd"> :return: DataResponse class</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_verified_args</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_verify_and_instantiate_arguments</span><span class="p">(</span>
<span class="n">args</span><span class="o">=</span><span class="n">args</span><span class="p">,</span> <span class="n">api_method</span><span class="o">=</span><span class="n">api_method</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;INVOKE ENDPOINT: [</span><span class="si">%s</span><span class="s2">], ARGS: {</span><span class="si">%s</span><span class="s2">}&quot;</span><span class="p">,</span>
<span class="n">api_method</span><span class="o">.</span><span class="vm">__name__</span><span class="p">,</span>
<span class="s2">&quot;, &quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2">=</span><span class="si">{</span><span class="n">v</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">_verified_args</span><span class="o">.</span><span class="n">items</span><span class="p">()),</span>
<span class="p">)</span>
<span class="n">_result</span><span class="p">:</span> <span class="n">Any</span> <span class="o">=</span> <span class="k">await</span> <span class="n">api_method</span><span class="p">(</span><span class="o">**</span><span class="n">_verified_args</span><span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;ENDPOINT RESULT: [</span><span class="si">{</span><span class="n">api_method</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2">], RESULT: </span><span class="si">{</span><span class="nb">str</span><span class="p">(</span><span class="n">_result</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">_result</span><span class="p">,</span> <span class="n">FileResponse</span><span class="p">):</span>
<span class="n">_json_result</span><span class="p">,</span> <span class="n">_content_type</span> <span class="o">=</span> <span class="p">(</span>
<span class="k">await</span> <span class="n">_convert_file_response</span><span class="p">(</span>
<span class="n">_result</span><span class="p">,</span>
<span class="n">message</span><span class="o">.</span><span class="n">connection</span><span class="p">,</span>
<span class="n">file_handler</span><span class="o">=</span><span class="n">StreamingFileHandler</span><span class="p">(),</span>
<span class="n">loop</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">,</span>
<span class="p">),</span>
<span class="s2">&quot;application/octet-stream&quot;</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">_result</span><span class="p">,</span> <span class="n">BaseModel</span><span class="p">):</span>
<span class="n">_json_result</span> <span class="o">=</span> <span class="n">_result</span><span class="o">.</span><span class="n">json</span><span class="p">()</span>
<span class="n">_content_type</span> <span class="o">=</span> <span class="s2">&quot;application/json&quot;</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">_result</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">_json_result</span> <span class="o">=</span> <span class="n">_result</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">_json_result</span> <span class="o">=</span> <span class="n">python_type_to_json</span><span class="p">(</span><span class="n">_result</span><span class="p">)</span>
<span class="n">_content_type</span> <span class="o">=</span> <span class="s2">&quot;application/json&quot;</span>
<span class="n">_binary_result</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">_json_result</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_json_result</span><span class="p">,</span> <span class="s2">&quot;encode&quot;</span><span class="p">)</span>
<span class="k">else</span> <span class="p">(</span>
<span class="n">_json_result</span><span class="o">.</span><span class="n">body</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_json_result</span><span class="p">,</span> <span class="s2">&quot;body&quot;</span><span class="p">)</span>
<span class="k">else</span> <span class="p">(</span><span class="n">_json_result</span><span class="o">.</span><span class="n">read</span><span class="p">()</span> <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">_json_result</span><span class="p">,</span> <span class="s2">&quot;read&quot;</span><span class="p">)</span> <span class="k">else</span> <span class="n">_json_result</span><span class="p">)</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;ALL DONE in call_cleverthis_api id:</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">()</span><span class="si">}</span><span class="s2">, resp code:</span><span class="si">{</span><span class="n">success_code</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">success_code</span><span class="p">,</span>
<span class="n">_content_type</span><span class="p">,</span>
<span class="n">_binary_result</span><span class="p">,</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span>
<span class="k">except</span> <span class="n">HTTPException</span> <span class="k">as</span> <span class="n">http_error</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Service REST endpoint invocation failed: </span><span class="si">{</span><span class="n">http_error</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">_content</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">http_error</span><span class="o">.</span><span class="n">detail</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">is_likely_json</span><span class="p">(</span><span class="n">_content</span><span class="p">):</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of_error_message</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">AMQErrorMessage</span><span class="p">(</span>
<span class="n">http_error</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span>
<span class="s2">&quot;Service Invocation Failed&quot;</span><span class="p">,</span>
<span class="n">_content</span><span class="p">,</span>
<span class="p">),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">http_error</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span>
<span class="s2">&quot;application/json&quot;</span><span class="p">,</span>
<span class="n">_content</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">error</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Service endpoint invocation failed: </span><span class="si">{</span><span class="n">error</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">_content</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">error</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">is_likely_json</span><span class="p">(</span><span class="n">_content</span><span class="p">):</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of_error_message</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">AMQErrorMessage</span><span class="p">(</span>
<span class="mi">500</span><span class="p">,</span>
<span class="s2">&quot;Service Invocation Failed&quot;</span><span class="p">,</span>
<span class="n">_content</span><span class="p">,</span>
<span class="p">),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="mi">500</span><span class="p">,</span>
<span class="s2">&quot;application/json&quot;</span><span class="p">,</span>
<span class="n">_content</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/consul_global_id_generator.html
-214
View File
@@ -1,214 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.consul_global_id_generator &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.consul_global_id_generator</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.consul_global_id_generator</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">logging</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">os</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">random</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">socket</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">time</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">consul_kv</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.config.amq_configuration</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQAdapter</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
<div class="viewcode-block" id="get_config_values">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.consul_global_id_generator.get_config_values">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_config_values</span><span class="p">(</span><span class="n">config</span><span class="p">:</span> <span class="n">AMQAdapter</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Get configuration values from AMQConfiguration or environment variables.</span>
<span class="sd"> Returns:</span>
<span class="sd"> tuple: (consul_host, consul_port, consul_counter_key, max_retries, retry_delay_seconds)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">consul_host</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">consul_host</span>
<span class="n">consul_port</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">consul_port</span>
<span class="n">consul_counter_key</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">consul_counter_key</span>
<span class="n">max_retries</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">consul_max_retries</span>
<span class="n">retry_delay_seconds</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">consul_initial_retry_delay</span>
<span class="k">return</span> <span class="n">consul_host</span><span class="p">,</span> <span class="n">consul_port</span><span class="p">,</span> <span class="n">consul_counter_key</span><span class="p">,</span> <span class="n">max_retries</span><span class="p">,</span> <span class="n">retry_delay_seconds</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to load configuration: </span><span class="si">{</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="si">}</span><span class="s2">. Using default values.&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CONSUL_HOST&quot;</span><span class="p">,</span> <span class="n">consul_kv</span><span class="o">.</span><span class="n">settings</span><span class="o">.</span><span class="n">DEFAULT_ENDPOINT</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]),</span>
<span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CONSUL_PORT&quot;</span><span class="p">,</span> <span class="n">consul_kv</span><span class="o">.</span><span class="n">settings</span><span class="o">.</span><span class="n">DEFAULT_ENDPOINT</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)[</span><span class="mi">1</span><span class="p">])),</span>
<span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CONSUL_COUNTER_KEY&quot;</span><span class="p">,</span> <span class="s2">&quot;service/ids/counter&quot;</span><span class="p">),</span>
<span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CONSUL_MAX_RETRIES&quot;</span><span class="p">,</span> <span class="mi">5</span><span class="p">)),</span>
<span class="nb">float</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CONSUL_RETRY_DELAY_SECONDS&quot;</span><span class="p">,</span> <span class="mf">0.2</span><span class="p">)),</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="generate_fallback_id">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.consul_global_id_generator.generate_fallback_id">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">generate_fallback_id</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Generate a fallback ID when Consul is not available.</span>
<span class="sd"> Uses a combination of timestamp, hostname hash, and random number.</span>
<span class="sd"> Returns:</span>
<span class="sd"> int: A reasonably unique integer ID</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">timestamp</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">)</span>
<span class="n">hostname</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">gethostname</span><span class="p">()</span>
<span class="n">hostname_hash</span> <span class="o">=</span> <span class="nb">hash</span><span class="p">(</span><span class="n">hostname</span><span class="p">)</span> <span class="o">%</span> <span class="mi">10000</span>
<span class="n">random_part</span> <span class="o">=</span> <span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">9999</span><span class="p">)</span>
<span class="n">unique_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">timestamp</span> <span class="o">*</span> <span class="mi">100000000</span><span class="p">)</span> <span class="o">+</span> <span class="p">(</span><span class="n">hostname_hash</span> <span class="o">*</span> <span class="mi">10000</span><span class="p">)</span> <span class="o">+</span> <span class="n">random_part</span>
<span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Using fallback ID generation method: </span><span class="si">{</span><span class="n">unique_id</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">unique_id</span></div>
<div class="viewcode-block" id="get_unique_instance_id">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.consul_global_id_generator.get_unique_instance_id">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_unique_instance_id</span><span class="p">(</span><span class="n">config</span><span class="p">:</span> <span class="n">AMQAdapter</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Get a globally unique ID from Consul.</span>
<span class="sd"> Uses Consul&#39;s atomic Compare-And-Set operations to safely increment a counter.</span>
<span class="sd"> Falls back to a local generation method if Consul is unavailable.</span>
<span class="sd"> Returns:</span>
<span class="sd"> int: A globally unique integer ID</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">consul_host</span><span class="p">,</span> <span class="n">consul_port</span><span class="p">,</span> <span class="n">consul_counter_key</span><span class="p">,</span> <span class="n">max_retries</span><span class="p">,</span> <span class="n">retry_delay_seconds</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">get_config_values</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">consul_kv</span><span class="o">.</span><span class="n">Connection</span><span class="p">(</span><span class="n">endpoint</span><span class="o">=</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">consul_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">consul_port</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="n">index</span><span class="p">,</span> <span class="n">data</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">consul_counter_key</span><span class="p">)</span>
<span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Initializing Consul counter at </span><span class="si">{</span><span class="n">consul_counter_key</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">consul_counter_key</span><span class="p">,</span> <span class="s2">&quot;1&quot;</span><span class="p">,</span> <span class="n">cas</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">&quot;Failed to initialize Consul counter&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">generate_fallback_id</span><span class="p">()</span>
<span class="n">current_value</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;Value&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">))</span>
<span class="n">new_value</span> <span class="o">=</span> <span class="n">current_value</span> <span class="o">+</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">attempt</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">max_retries</span><span class="p">):</span>
<span class="n">success</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">consul_counter_key</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">new_value</span><span class="p">),</span> <span class="n">cas</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;ModifyIndex&quot;</span><span class="p">])</span>
<span class="k">if</span> <span class="n">success</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Successfully obtained unique ID: </span><span class="si">{</span><span class="n">new_value</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">new_value</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;CAS update failed on attempt </span><span class="si">{</span><span class="n">attempt</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="mi">1</span><span class="si">}</span><span class="s2">, retrying...&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">attempt</span> <span class="o">&lt;</span> <span class="n">max_retries</span> <span class="o">-</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">retry_delay_seconds</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="n">attempt</span><span class="p">))</span> <span class="c1"># Exponential backoff</span>
<span class="n">index</span><span class="p">,</span> <span class="n">data</span> <span class="o">=</span> <span class="n">c</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">consul_counter_key</span><span class="p">)</span>
<span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">&quot;Counter disappeared during update&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">generate_fallback_id</span><span class="p">()</span>
<span class="n">current_value</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;Value&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">))</span>
<span class="n">new_value</span> <span class="o">=</span> <span class="n">current_value</span> <span class="o">+</span> <span class="mi">1</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to update counter after </span><span class="si">{</span><span class="n">max_retries</span><span class="si">}</span><span class="s2"> attempts&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">generate_fallback_id</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Error accessing Consul: </span><span class="si">{</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">generate_fallback_id</span><span class="p">()</span></div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/data_message_factory.html
-449
View File
@@ -1,449 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.data_message_factory &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.data_message_factory</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.data_message_factory</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">methods to create, serialize and deserialize DataMessage objects.</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">base64</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">re</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">datetime</span><span class="w"> </span><span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timezone</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">io</span><span class="w"> </span><span class="kn">import</span> <span class="n">BytesIO</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">List</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">amqp.adapter.file_handler</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.file_handler</span><span class="w"> </span><span class="kn">import</span> <span class="n">FileHandler</span><span class="p">,</span> <span class="n">StreamingFileHandler</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.pydantic_serializer</span><span class="w"> </span><span class="kn">import</span> <span class="n">python_type_to_json</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.serializer</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">map_as_string</span><span class="p">,</span>
<span class="n">map_with_list_as_string</span><span class="p">,</span>
<span class="n">parse_map_list_string</span><span class="p">,</span>
<span class="n">parse_map_string</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">DataMessage</span><span class="p">,</span> <span class="n">DataMessageMagicByte</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.snowflake_id</span><span class="w"> </span><span class="kn">import</span> <span class="n">SnowflakeId</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">sanitize_as_rabbitmq_name</span>
<div class="viewcode-block" id="DataMessageFactory">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">DataMessageFactory</span><span class="p">:</span>
<span class="n">SNOWFLAKE_ID_BITS</span> <span class="o">=</span> <span class="mi">80</span>
<span class="n">SEQUENCE_BITS</span> <span class="o">=</span> <span class="mi">12</span>
<span class="n">MACHINE_ID_BITS</span> <span class="o">=</span> <span class="mi">24</span>
<span class="n">TIMESTAMP_BITS</span> <span class="o">=</span> <span class="n">SNOWFLAKE_ID_BITS</span> <span class="o">-</span> <span class="n">SEQUENCE_BITS</span> <span class="o">-</span> <span class="n">MACHINE_ID_BITS</span>
<span class="n">MACHINE_ID_MASK</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="n">MACHINE_ID_BITS</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span>
<span class="n">SEQUENCE_MASK</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span> <span class="o">&lt;&lt;</span> <span class="n">SEQUENCE_BITS</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span>
<span class="n">_instance</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">snowflake_epoch</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2024</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">tzinfo</span><span class="o">=</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span><span class="o">.</span><span class="n">timestamp</span><span class="p">()</span>
<div class="viewcode-block" id="DataMessageFactory.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">machine_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">machine_id</span> <span class="o">=</span> <span class="p">(</span><span class="n">machine_id</span> <span class="o">&amp;</span> <span class="bp">self</span><span class="o">.</span><span class="n">MACHINE_ID_MASK</span><span class="p">)</span> <span class="o">&lt;&lt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">SEQUENCE_BITS</span>
<span class="bp">self</span><span class="o">.</span><span class="n">snowflake_sequence</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">snowflake_last_timestamp</span> <span class="o">=</span> <span class="mi">0</span></div>
<div class="viewcode-block" id="DataMessageFactory.get_instance">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.get_instance">[docs]</a>
<span class="nd">@classmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">get_instance</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">machine_id</span><span class="p">:</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> return reusable factory instance</span>
<span class="sd"> :param machine_id: a generator ID, should be unique for each component to prevent duplicate IDs</span>
<span class="sd"> :return: factory instance</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">cls</span><span class="o">.</span><span class="n">_instance</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="bp">cls</span><span class="o">.</span><span class="n">_instance</span> <span class="o">=</span> <span class="n">DataMessageFactory</span><span class="p">(</span><span class="n">machine_id</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">_instance</span></div>
<div class="viewcode-block" id="DataMessageFactory.generate_snowflake_id">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.generate_snowflake_id">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">generate_snowflake_id</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">SnowflakeId</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Generate an unique ID following the SnowflakeID structure, see https://en.wikipedia.org/wiki/Snowflake_ID</span>
<span class="sd"> :return: 80-bit wide ID</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">timestamp</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">(</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span><span class="o">.</span><span class="n">timestamp</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">)</span>
<span class="k">if</span> <span class="n">timestamp</span> <span class="o">&lt;</span> <span class="bp">self</span><span class="o">.</span><span class="n">snowflake_last_timestamp</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="s2">&quot;Clock moved backwards. Refusing to generate id&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">timestamp</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">snowflake_last_timestamp</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">snowflake_sequence</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">snowflake_sequence</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span> <span class="o">&amp;</span> <span class="bp">self</span><span class="o">.</span><span class="n">SEQUENCE_MASK</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">snowflake_sequence</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">while</span> <span class="n">timestamp</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">snowflake_last_timestamp</span><span class="p">:</span>
<span class="n">timestamp</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">(</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span><span class="o">.</span><span class="n">timestamp</span><span class="p">()</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">snowflake_last_timestamp</span> <span class="o">=</span> <span class="n">timestamp</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">snowflake_sequence</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">snowflake_last_timestamp</span> <span class="o">=</span> <span class="n">timestamp</span>
<span class="n">time</span> <span class="o">=</span> <span class="n">timestamp</span> <span class="o">-</span> <span class="mi">1000</span> <span class="o">*</span> <span class="nb">int</span><span class="p">(</span><span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">snowflake_epoch</span><span class="p">)</span>
<span class="k">return</span> <span class="n">SnowflakeId</span><span class="p">(</span>
<span class="n">time</span> <span class="o">&gt;&gt;</span> <span class="p">(</span><span class="mi">64</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">SEQUENCE_BITS</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">MACHINE_ID_BITS</span><span class="p">),</span>
<span class="p">(</span>
<span class="p">(</span><span class="n">time</span> <span class="o">&lt;&lt;</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">SEQUENCE_BITS</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">MACHINE_ID_BITS</span><span class="p">))</span>
<span class="o">|</span> <span class="bp">self</span><span class="o">.</span><span class="n">machine_id</span>
<span class="o">|</span> <span class="bp">self</span><span class="o">.</span><span class="n">snowflake_sequence</span>
<span class="p">)</span>
<span class="o">&amp;</span> <span class="mh">0xFFFFFFFFFFFFFFFF</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataMessageFactory.create_request_message">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.create_request_message">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">create_request_message</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">method</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">headers</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]],</span>
<span class="n">message</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Encapsulate / hide the system level inputs, create the AMQ message from business relevant input</span>
<span class="sd"> :param method: a method requested (e.g. HTTP method like GET, POST in case of REST request)</span>
<span class="sd"> :param domain: service host name</span>
<span class="sd"> :param path: path at the service</span>
<span class="sd"> :param headers: HTTP headers</span>
<span class="sd"> :param message: payload</span>
<span class="sd"> :return: DataMessage object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">serializable_headers</span> <span class="o">=</span> <span class="n">headers</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> <span class="k">if</span> <span class="n">headers</span> <span class="k">else</span> <span class="p">{}</span>
<span class="n">payload</span> <span class="o">=</span> <span class="n">base64</span><span class="o">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">else</span> <span class="n">message</span><span class="p">)</span>
<span class="n">trace_info</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">return</span> <span class="n">DataMessage</span><span class="p">(</span>
<span class="n">DataMessageMagicByte</span><span class="o">.</span><span class="n">HTTP_REQUEST</span><span class="o">.</span><span class="n">value</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">generate_snowflake_id</span><span class="p">(),</span>
<span class="n">method</span><span class="p">,</span>
<span class="n">domain</span><span class="p">,</span>
<span class="n">path</span><span class="p">,</span>
<span class="n">serializable_headers</span><span class="p">,</span>
<span class="n">trace_info</span><span class="p">,</span>
<span class="n">payload</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataMessageFactory.safe_rsplit">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.safe_rsplit">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">safe_rsplit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">fq_method_name</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">&quot;_&quot;</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Safely split a fully qualified method name into package, class, and method</span>
<span class="sd"> Args:</span>
<span class="sd"> fq_method_name (str): Fully qualified method name</span>
<span class="sd"> default (str, optional): Default value for missing tokens. Defaults to &#39;&#39;.</span>
<span class="sd"> Returns:</span>
<span class="sd"> tuple: (package, class_name, method)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">fq_method_name</span><span class="p">:</span>
<span class="k">return</span> <span class="n">default</span><span class="p">,</span> <span class="n">default</span><span class="p">,</span> <span class="n">default</span>
<span class="n">parts</span> <span class="o">=</span> <span class="n">fq_method_name</span><span class="o">.</span><span class="n">rsplit</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="c1"># Pad with default values if insufficient tokens</span>
<span class="k">while</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">parts</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span></div>
<div class="viewcode-block" id="DataMessageFactory.create_rpc_message">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.create_rpc_message">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">create_rpc_message</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">fq_method_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">swarm_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">args</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Create a RPC message with the given parameters.</span>
<span class="sd"> :param fq_method_name: fully qualified method name (e.g. &#39;com.example.ServiceClass.method_name&#39;)</span>
<span class="sd"> :param args: arguments whose serialized form will form the request body</span>
<span class="sd"> :return: DataMessage object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">package</span><span class="p">,</span> <span class="n">class_name</span><span class="p">,</span> <span class="n">method</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">safe_rsplit</span><span class="p">(</span><span class="n">fq_method_name</span><span class="p">)</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;Content-Type&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;application/octet-stream&quot;</span><span class="p">],</span> <span class="s2">&quot;Return-Address&quot;</span><span class="p">:</span> <span class="p">[</span><span class="n">swarm_id</span><span class="p">]}</span>
<span class="n">trace_info</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">body</span> <span class="o">=</span> <span class="n">python_type_to_json</span><span class="p">(</span><span class="n">args</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span> <span class="k">if</span> <span class="n">args</span> <span class="k">else</span> <span class="sa">b</span><span class="s2">&quot;&quot;</span>
<span class="nb">print</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Creating RPC message for method: </span><span class="si">{</span><span class="n">fq_method_name</span><span class="si">}</span><span class="s2">, args: </span><span class="si">{</span><span class="n">args</span><span class="si">}</span><span class="s2"> body: </span><span class="si">{</span><span class="n">body</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">body</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="s1">&#39;None&#39;</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">DataMessage</span><span class="p">(</span>
<span class="n">DataMessageMagicByte</span><span class="o">.</span><span class="n">RPC_REQUEST</span><span class="o">.</span><span class="n">value</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">generate_snowflake_id</span><span class="p">(),</span>
<span class="n">package</span><span class="p">,</span>
<span class="n">class_name</span><span class="p">,</span>
<span class="n">method</span><span class="p">,</span>
<span class="n">headers</span><span class="p">,</span>
<span class="n">trace_info</span><span class="p">,</span>
<span class="n">body</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataMessageFactory.amq_message_routing_key">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.amq_message_routing_key">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">amq_message_routing_key</span><span class="p">(</span><span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Constructs the message specific routing key.</span>
<span class="sd"> :param message: DataMessage input</span>
<span class="sd"> :return: routing key (as string) compliant to RabbitMQ allowed character set.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">sanitize_as_rabbitmq_name</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">domain</span><span class="p">()</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="DataMessageFactory.get_timestamp_from_id">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.get_timestamp_from_id">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">get_timestamp_from_id</span><span class="p">(</span><span class="n">_id</span><span class="p">:</span> <span class="n">SnowflakeId</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Helper method - retrieve the timestamp from the ID</span>
<span class="sd"> :param _id: SnowflakeID (includes encoded timestamp)</span>
<span class="sd"> :return: the message timestamp</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">bits</span> <span class="o">=</span> <span class="n">_id</span><span class="o">.</span><span class="n">get_bits</span><span class="p">()</span>
<span class="n">low</span> <span class="o">=</span> <span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&gt;&gt;</span> <span class="p">(</span><span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">SEQUENCE_BITS</span> <span class="o">+</span> <span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">MACHINE_ID_BITS</span><span class="p">)</span>
<span class="n">high</span> <span class="o">=</span> <span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&lt;&lt;</span> <span class="p">(</span>
<span class="mi">64</span> <span class="o">-</span> <span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">SEQUENCE_BITS</span> <span class="o">-</span> <span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">MACHINE_ID_BITS</span>
<span class="p">)</span>
<span class="k">return</span> <span class="p">(</span><span class="n">high</span> <span class="o">|</span> <span class="n">low</span><span class="p">)</span> <span class="o">+</span> <span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">snowflake_epoch</span> <span class="o">*</span> <span class="mi">1000</span></div>
<div class="viewcode-block" id="DataMessageFactory.to_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.to_string">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">to_string</span><span class="p">(</span><span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Human-readable representation.</span>
<span class="sd"> :param message: DataMessage input</span>
<span class="sd"> :return: as string</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;DataMessage</span><span class="se">{{</span><span class="s2">&quot;</span>
<span class="sa">f</span><span class="s2">&quot;id=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">()</span><span class="si">}</span><span class="s2">, &quot;</span>
<span class="sa">f</span><span class="s2">&quot;method=&#39;</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">method</span><span class="p">()</span><span class="si">}</span><span class="s2">&#39;, &quot;</span>
<span class="sa">f</span><span class="s2">&quot;domain=&#39;</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">domain</span><span class="p">()</span><span class="si">}</span><span class="s2">&#39;, &quot;</span>
<span class="sa">f</span><span class="s2">&quot;path=&#39;</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&#39;, &quot;</span>
<span class="sa">f</span><span class="s2">&quot;headers=</span><span class="se">{{</span><span class="si">{</span><span class="n">map_with_list_as_string</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">())</span><span class="si">}</span><span class="se">}}</span><span class="s2">, &quot;</span>
<span class="sa">f</span><span class="s2">&quot;traceInfo=</span><span class="se">{{</span><span class="si">{</span><span class="n">map_as_string</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">())</span><span class="si">}</span><span class="se">}}</span><span class="s2">, &quot;</span>
<span class="sa">f</span><span class="s2">&quot;base64body=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">base64body</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="se">}}</span><span class="s2">&quot;</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataMessageFactory.serialize">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.serialize">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">serialize</span><span class="p">(</span><span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Converts DataMessage to byte array</span>
<span class="sd"> :param message: message to serialize</span>
<span class="sd"> :return: messages as bytes</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">id_bytes</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">())</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>
<span class="n">prolog</span> <span class="o">=</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;|m=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">method</span><span class="p">()</span><span class="si">}</span><span class="s2">|d=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">domain</span><span class="p">()</span><span class="si">}</span><span class="s2">|p=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">|h=</span><span class="se">{{</span><span class="si">{</span><span class="n">map_with_list_as_string</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">())</span><span class="si">}</span><span class="se">}}</span><span class="s2">|t=</span><span class="se">{{</span><span class="si">{</span><span class="n">map_as_string</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">())</span><span class="si">}</span><span class="se">}}</span><span class="s2">|b=&quot;</span>
<span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span>
<span class="n">header_length</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">id_bytes</span><span class="p">)</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">prolog</span><span class="p">)</span>
<span class="n">prolog_len</span> <span class="o">=</span> <span class="n">header_length</span><span class="o">.</span><span class="n">to_bytes</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">byteorder</span><span class="o">=</span><span class="s2">&quot;big&quot;</span><span class="p">)</span>
<span class="n">msg_length</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">+</span> <span class="n">header_length</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">base64body</span><span class="p">())</span>
<span class="n">baos</span> <span class="o">=</span> <span class="n">BytesIO</span><span class="p">(</span><span class="nb">bytearray</span><span class="p">(</span><span class="n">msg_length</span><span class="p">))</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">prolog_len</span><span class="p">)</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">magic</span><span class="p">()</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">))</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">id_bytes</span><span class="p">)</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">prolog</span><span class="p">)</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">base64body</span><span class="p">())</span>
<span class="k">return</span> <span class="n">baos</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span></div>
<div class="viewcode-block" id="DataMessageFactory.from_bytes">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.from_bytes">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">from_bytes</span><span class="p">(</span><span class="n">input_bytes</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Builds the DataMessage from its serialized (byte array) form.</span>
<span class="sd"> :param input_bytes: serialized message</span>
<span class="sd"> :return: DataMessage instance</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">prolog_len</span> <span class="o">=</span> <span class="p">(</span><span class="n">input_bytes</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&lt;&lt;</span> <span class="mi">8</span><span class="p">)</span> <span class="o">+</span> <span class="n">input_bytes</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">metadata</span> <span class="o">=</span> <span class="n">input_bytes</span><span class="p">[</span><span class="mi">2</span> <span class="p">:</span> <span class="n">prolog_len</span> <span class="o">+</span> <span class="mi">2</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="n">pattern</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span>
<span class="sa">r</span><span class="s2">&quot;([AR])&quot;</span>
<span class="sa">r</span><span class="s2">&quot;([0-9A-Fa-f]+)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;m=([^|]*)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;d=([^|]*)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;p=([^|]*)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;h=(\{[^}]*\})\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;t=(\{[^}]*\})\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;b=&quot;</span>
<span class="p">)</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">pattern</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">metadata</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">match</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;DataMessage format does not match expected format. &quot;</span>
<span class="sa">f</span><span class="s2">&quot;Input string [</span><span class="si">{</span><span class="n">metadata</span><span class="si">}</span><span class="s2">] does not match pattern: </span><span class="si">{</span><span class="n">pattern</span><span class="o">.</span><span class="n">pattern</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">id_str</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="n">method</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="n">domain</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="n">headers_str</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">6</span><span class="p">)</span>
<span class="n">trace_info_str</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">7</span><span class="p">)</span>
<span class="n">body_b64</span> <span class="o">=</span> <span class="n">input_bytes</span><span class="p">[</span><span class="n">prolog_len</span> <span class="o">+</span> <span class="mi">2</span> <span class="p">:]</span>
<span class="n">headers</span> <span class="o">=</span> <span class="n">parse_map_list_string</span><span class="p">(</span><span class="n">headers_str</span><span class="p">)</span>
<span class="n">trace_info</span> <span class="o">=</span> <span class="n">parse_map_string</span><span class="p">(</span><span class="n">trace_info_str</span><span class="p">)</span>
<span class="k">return</span> <span class="n">DataMessage</span><span class="p">(</span>
<span class="k">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span> <span class="c1"># Magic byte (R for RPC, A for HTTP)</span>
<span class="n">SnowflakeId</span><span class="o">.</span><span class="n">from_hex</span><span class="p">(</span><span class="n">id_str</span><span class="p">),</span>
<span class="n">method</span><span class="p">,</span>
<span class="n">domain</span><span class="p">,</span>
<span class="n">path</span><span class="p">,</span>
<span class="n">headers</span><span class="p">,</span>
<span class="n">trace_info</span><span class="p">,</span>
<span class="n">body_b64</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataMessageFactory.from_stream">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_message_factory.DataMessageFactory.from_stream">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">from_stream</span><span class="p">(</span>
<span class="n">stream</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span>
<span class="n">rabbitmq_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">loop</span><span class="p">,</span>
<span class="n">file_handler</span><span class="p">:</span> <span class="n">FileHandler</span> <span class="o">=</span> <span class="n">StreamingFileHandler</span><span class="p">(),</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataMessage</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Builds the DataMessage from its serialized (byte array) form.</span>
<span class="sd"> :param stream: serialized message</span>
<span class="sd"> :param rabbitmq_url: RabbitMQ connection URL</span>
<span class="sd"> :param loop: event loop</span>
<span class="sd"> :param file_handler: file handler for downloading the message</span>
<span class="sd"> :return: DataMessage instance</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># async implementation</span>
<span class="n">_message_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">stream</span><span class="o">.</span><span class="n">decode</span><span class="p">())</span>
<span class="n">_req_info</span> <span class="o">=</span> <span class="n">_message_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;req-info&quot;</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_req_info</span><span class="p">:</span>
<span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="n">eof</span><span class="p">)</span> <span class="o">=</span> <span class="k">await</span> <span class="n">file_handler</span><span class="o">.</span><span class="n">download_buffer</span><span class="p">(</span>
<span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="o">=</span><span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">queue_name</span><span class="o">=</span><span class="n">_req_info</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">eof</span> <span class="o">==</span> <span class="n">amqp</span><span class="o">.</span><span class="n">adapter</span><span class="o">.</span><span class="n">file_handler</span><span class="o">.</span><span class="n">END_OF_FILE</span><span class="p">:</span>
<span class="k">return</span> <span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">body</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">None</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/data_parser.html
-325
View File
@@ -1,325 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.data_parser &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.data_parser</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.data_parser</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">io</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">Tuple</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">urllib.parse</span><span class="w"> </span><span class="kn">import</span> <span class="n">parse_qs</span><span class="p">,</span> <span class="n">urlparse</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">xml.etree</span><span class="w"> </span><span class="kn">import</span> <span class="n">ElementTree</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">python_multipart</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">fastapi</span><span class="w"> </span><span class="kn">import</span> <span class="n">UploadFile</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">python_multipart.multipart</span><span class="w"> </span><span class="kn">import</span> <span class="n">Field</span><span class="p">,</span> <span class="n">File</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_debug</span><span class="p">,</span> <span class="n">logging_error</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQMessage</span>
<div class="viewcode-block" id="MultipartFormDataParser">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.MultipartFormDataParser">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">MultipartFormDataParser</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Parses multipart/form-data content and extracts form fields (key/value pairs) and files.</span>
<span class="sd"> Note: multipart message that contains file(s) to upload are converted to CleverMicro JSON-based streaming format,</span>
<span class="sd"> where files are transmitted separately via stream using dedicated queue, so the message content is no longer</span>
<span class="sd"> the HTTP multipart format, but the Content-Type is being preserved for compatibility with the original message.</span>
<span class="sd"> This class supports both formats (multipart and JSON) depending on the message content, because if no file is</span>
<span class="sd"> included, the message is transmitted in the unchanged multipart/form-data format.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="MultipartFormDataParser.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.MultipartFormDataParser.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_multipart</span> <span class="o">=</span> <span class="kc">False</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_json</span> <span class="o">=</span> <span class="kc">False</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_valid</span> <span class="o">=</span> <span class="kc">True</span>
<span class="c1"># Convert each list of strings to a single string joined by newline, then to bytes</span>
<span class="n">_headers</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">key</span><span class="p">:</span> <span class="s2">&quot;</span><span class="se">\n</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">value</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">()</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">if</span> <span class="p">(</span>
<span class="nb">hasattr</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="s2">&quot;extra_data&quot;</span><span class="p">)</span>
<span class="ow">and</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">extra_data</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span>
<span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">extra_data</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span>
<span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_json</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">[</span><span class="s2">&quot;files&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">[</span><span class="s2">&quot;form&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_data</span> <span class="o">=</span> <span class="n">AMQDataParser</span><span class="o">.</span><span class="n">process_form_data</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">jsonerror</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Parsing JSON: </span><span class="si">{</span><span class="n">jsonerror</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_valid</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">python_multipart</span><span class="o">.</span><span class="n">parse_form</span><span class="p">(</span>
<span class="n">_headers</span><span class="p">,</span> <span class="n">io</span><span class="o">.</span><span class="n">BytesIO</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">()),</span> <span class="bp">self</span><span class="o">.</span><span class="n">on_field</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">on_file</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_multipart</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">__error</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;Error parsing multipart/form-data: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">. Trying parsing as JSON.&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_json</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">[</span><span class="s2">&quot;files&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">[</span><span class="s2">&quot;form&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_data</span> <span class="o">=</span> <span class="n">AMQDataParser</span><span class="o">.</span><span class="n">process_form_data</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">jsonerror</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="n">__error</span><span class="p">)</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Parsing JSON: </span><span class="si">{</span><span class="n">jsonerror</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_valid</span> <span class="o">=</span> <span class="kc">False</span></div>
<div class="viewcode-block" id="MultipartFormDataParser.on_field">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.MultipartFormDataParser.on_field">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">on_field</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">field</span><span class="p">:</span> <span class="n">Field</span><span class="p">):</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">field</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">[</span><span class="n">field</span><span class="o">.</span><span class="n">field_name</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)]</span> <span class="o">=</span> <span class="n">field</span><span class="o">.</span><span class="n">value</span></div>
<div class="viewcode-block" id="MultipartFormDataParser.on_file">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.MultipartFormDataParser.on_file">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">on_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">file</span><span class="p">:</span> <span class="n">File</span><span class="p">):</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">file</span><span class="p">))</span>
<span class="n">file</span><span class="o">.</span><span class="n">file_object</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">form_data</span><span class="p">[</span><span class="n">file</span><span class="o">.</span><span class="n">field_name</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)]</span> <span class="o">=</span> <span class="n">UploadFile</span><span class="p">(</span>
<span class="n">file</span><span class="o">.</span><span class="n">file_object</span><span class="p">,</span> <span class="n">size</span><span class="o">=</span><span class="n">file</span><span class="o">.</span><span class="n">size</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="n">file</span><span class="o">.</span><span class="n">file_name</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span>
<span class="p">)</span></div>
</div>
<div class="viewcode-block" id="AMQDataParser">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.AMQDataParser">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">AMQDataParser</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> A class for parsing AMQ data messages. It provides methods to parse form data,</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="AMQDataParser.process_form_data">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.AMQDataParser.process_form_data">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">process_form_data</span><span class="p">(</span><span class="n">json_input</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Processes the input JSON to extract form data. The JSON has two top entries: &#39;files&#39; and &#39;form&#39;.</span>
<span class="sd"> This is part of the file stream implementation in AMQ adapter to stream large files in chunks.</span>
<span class="sd"> &#39;files&#39; contains information about the AMQ queue from which read the file, while &#39;form&#39; contains</span>
<span class="sd"> additional non-file form values.</span>
<span class="sd"> This function combines the file and values at the top level in the returned dictionary.</span>
<span class="sd"> Args:</span>
<span class="sd"> json_input: JSON string or dict containing the form and files data</span>
<span class="sd"> Returns:</span>
<span class="sd"> dict: Processed form data with files data overriding non-empty values</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Load JSON if input is string</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">json_input</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">json_input</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">json_input</span>
<span class="n">result</span> <span class="o">=</span> <span class="p">{}</span>
<span class="c1"># Process form data - take first element of each list</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">data</span><span class="p">[</span><span class="s2">&quot;form&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="n">value</span><span class="p">:</span> <span class="c1"># Only process if list is not empty</span>
<span class="n">result</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># Override with files data where available</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">data</span><span class="p">[</span><span class="s2">&quot;files&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="n">value</span><span class="p">:</span> <span class="c1"># Only override if files data is non-empty</span>
<span class="n">result</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">return</span> <span class="n">result</span></div>
<div class="viewcode-block" id="AMQDataParser.parse_get_url">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.AMQDataParser.parse_get_url">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">parse_get_url</span><span class="p">(</span><span class="n">url</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Parses an HTTP GET URL and returns (context_path, query_params_dict)</span>
<span class="sd"> Args:</span>
<span class="sd"> url: The URL string to parse (e.g., &quot;http://example.com/path?foo=1&amp;bar=2&amp;foo=3&quot;)</span>
<span class="sd"> Returns:</span>
<span class="sd"> tuple: (context_path, query_params_dict)</span>
<span class="sd"> - context_path: The path part of the URL (e.g., &quot;/path&quot;)</span>
<span class="sd"> - query_params_dict: Query parameters where values are str for single value or list of str for multivalue</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_parsed</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="n">_query_params</span> <span class="o">=</span> <span class="n">parse_qs</span><span class="p">(</span><span class="n">_parsed</span><span class="o">.</span><span class="n">query</span><span class="p">)</span>
<span class="c1"># Convert values from lists to single values when there&#39;s only one value</span>
<span class="n">_simplified_params</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">_query_params</span><span class="o">.</span><span class="n">items</span><span class="p">()}</span>
<span class="k">return</span> <span class="n">_parsed</span><span class="o">.</span><span class="n">path</span><span class="p">,</span> <span class="n">_simplified_params</span></div>
<div class="viewcode-block" id="AMQDataParser.parse_request_body">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_parser.AMQDataParser.parse_request_body">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">parse_request_body</span><span class="p">(</span><span class="n">message</span><span class="p">:</span> <span class="n">AMQMessage</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Parses the HTTP POST request body based on the MIME type.</span>
<span class="sd"> This is typically called fom the concrete CleverThis Service to extract parameters from the message body.</span>
<span class="sd"> Args:</span>
<span class="sd"> message (DataMessage): Contains raw request and MIME headers.</span>
<span class="sd"> Returns:</span>
<span class="sd"> dict: Parsed data as a dictionary.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_content_type</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;Content-Type&quot;</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">_content_type</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">&quot;Content-Type header is required&quot;</span><span class="p">)</span>
<span class="n">content_type</span> <span class="o">=</span> <span class="n">_content_type</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># Parse JSON</span>
<span class="k">if</span> <span class="n">content_type</span> <span class="o">==</span> <span class="s2">&quot;application/json&quot;</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">()</span> <span class="ow">or</span> <span class="s2">&quot;</span><span class="si">{}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="n">json</span><span class="o">.</span><span class="n">JSONDecodeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid JSON: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid JSON: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># Parse URL-encoded form data</span>
<span class="k">elif</span> <span class="n">content_type</span> <span class="o">==</span> <span class="s2">&quot;application/x-www-form-urlencoded&quot;</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_form_data</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">k</span><span class="p">:</span> <span class="n">v</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">parse_qs</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">())</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">_form_data</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">())</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">_form_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">())</span>
<span class="k">if</span> <span class="n">_form_data</span><span class="p">[</span><span class="s2">&quot;files&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">_form_data</span><span class="p">[</span><span class="s2">&quot;form&quot;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">_form_data</span> <span class="o">=</span> <span class="n">AMQDataParser</span><span class="o">.</span><span class="n">process_form_data</span><span class="p">(</span><span class="n">_form_data</span><span class="p">)</span>
<span class="k">return</span> <span class="n">_form_data</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid URL-encoded form data: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid URL-encoded form data: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># Parse multipart/form-data</span>
<span class="k">elif</span> <span class="n">content_type</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;multipart/form-data&quot;</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">parser</span><span class="p">:</span> <span class="n">MultipartFormDataParser</span> <span class="o">=</span> <span class="n">MultipartFormDataParser</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="k">return</span> <span class="n">parser</span><span class="o">.</span><span class="n">form_data</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid multipart/form-data: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid multipart/form-data: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># Parse plain text</span>
<span class="k">elif</span> <span class="n">content_type</span> <span class="o">==</span> <span class="s2">&quot;text/plain&quot;</span><span class="p">:</span>
<span class="k">return</span> <span class="p">{</span><span class="s2">&quot;text&quot;</span><span class="p">:</span> <span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">()}</span>
<span class="c1"># Parse XML</span>
<span class="k">elif</span> <span class="n">content_type</span> <span class="o">==</span> <span class="s2">&quot;application/xml&quot;</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_root</span> <span class="o">=</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">fromstring</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">())</span>
<span class="k">return</span> <span class="p">{</span><span class="n">elem</span><span class="o">.</span><span class="n">tag</span><span class="p">:</span> <span class="n">elem</span><span class="o">.</span><span class="n">text</span> <span class="k">for</span> <span class="n">elem</span> <span class="ow">in</span> <span class="n">_root</span><span class="p">}</span>
<span class="k">except</span> <span class="n">ElementTree</span><span class="o">.</span><span class="n">ParseError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Invalid XML: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unsupported Content-Type: </span><span class="si">{</span><span class="n">content_type</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/data_response_factory.html
-276
View File
@@ -1,276 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.data_response_factory &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.data_response_factory</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.data_response_factory</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">method to create, serialize and deserialize DataResponse message</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">base64</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">re</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">io</span><span class="w"> </span><span class="kn">import</span> <span class="n">BytesIO</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.serializer</span><span class="w"> </span><span class="kn">import</span> <span class="n">map_as_string</span><span class="p">,</span> <span class="n">parse_map_string</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQErrorMessage</span><span class="p">,</span> <span class="n">DataMessageMagicByte</span><span class="p">,</span> <span class="n">DataResponse</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.snowflake_id</span><span class="w"> </span><span class="kn">import</span> <span class="n">SnowflakeId</span>
<div class="viewcode-block" id="DataResponseFactory">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_response_factory.DataResponseFactory">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">DataResponseFactory</span><span class="p">:</span>
<div class="viewcode-block" id="DataResponseFactory.create_async_response_message">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_response_factory.DataResponseFactory.create_async_response_message">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">create_async_response_message</span><span class="p">(</span><span class="n">_id</span><span class="p">:</span> <span class="n">SnowflakeId</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Helper method to create standard OK response.</span>
<span class="sd"> :param _id: ID</span>
<span class="sd"> :return: Response message</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="n">_id</span><span class="p">,</span>
<span class="n">response_code</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span>
<span class="n">content_type</span><span class="o">=</span><span class="s2">&quot;application/text&quot;</span><span class="p">,</span>
<span class="n">body</span><span class="o">=</span><span class="s2">&quot;OK&quot;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">),</span>
<span class="n">trace_info</span><span class="o">=</span><span class="p">{},</span>
<span class="n">error</span><span class="o">=</span><span class="s2">&quot;&quot;</span><span class="p">,</span>
<span class="n">error_cause</span><span class="o">=</span><span class="s2">&quot;&quot;</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataResponseFactory.of">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_response_factory.DataResponseFactory.of">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">of</span><span class="p">(</span>
<span class="nb">id</span><span class="p">:</span> <span class="n">SnowflakeId</span><span class="p">,</span>
<span class="n">response_code</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
<span class="n">content_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">body</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span>
<span class="n">trace_info</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span>
<span class="n">error</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="n">error_cause</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Create the DataResponse message supplying all key values</span>
<span class="sd"> :param id: SnowflakeID</span>
<span class="sd"> :param response_code: HTTP response code</span>
<span class="sd"> :param content_type: MIME content type</span>
<span class="sd"> :param body: payload</span>
<span class="sd"> :param trace_info: Jaeger trace info (id)</span>
<span class="sd"> :return: DataResponseMessage object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">DataResponse</span><span class="p">(</span>
<span class="n">magic</span><span class="o">=</span><span class="n">DataMessageMagicByte</span><span class="o">.</span><span class="n">HTTP_REQUEST</span><span class="o">.</span><span class="n">value</span><span class="p">,</span>
<span class="nb">id</span><span class="o">=</span><span class="nb">id</span><span class="p">,</span>
<span class="n">response_code</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">response_code</span><span class="p">),</span>
<span class="n">content_type</span><span class="o">=</span><span class="n">content_type</span><span class="p">,</span>
<span class="n">error</span><span class="o">=</span><span class="n">error</span><span class="p">,</span>
<span class="n">error_cause</span><span class="o">=</span><span class="n">error_cause</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{},</span>
<span class="n">trace_info</span><span class="o">=</span><span class="n">trace_info</span><span class="p">,</span>
<span class="n">base64body</span><span class="o">=</span><span class="n">base64</span><span class="o">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">body</span><span class="p">),</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataResponseFactory.of_error_message">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_response_factory.DataResponseFactory.of_error_message">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">of_error_message</span><span class="p">(</span>
<span class="nb">id</span><span class="p">:</span> <span class="n">SnowflakeId</span><span class="p">,</span>
<span class="n">error_message</span><span class="p">:</span> <span class="n">AMQErrorMessage</span><span class="p">,</span>
<span class="n">trace_info</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="nb">id</span><span class="p">,</span>
<span class="n">response_code</span><span class="o">=</span><span class="n">error_message</span><span class="o">.</span><span class="n">response_code</span><span class="p">,</span>
<span class="n">content_type</span><span class="o">=</span><span class="n">error_message</span><span class="o">.</span><span class="n">content_type</span><span class="p">,</span>
<span class="n">body</span><span class="o">=</span><span class="n">error_message</span><span class="o">.</span><span class="n">serialize</span><span class="p">(),</span>
<span class="n">trace_info</span><span class="o">=</span><span class="n">trace_info</span><span class="p">,</span>
<span class="n">error</span><span class="o">=</span><span class="n">error_message</span><span class="o">.</span><span class="n">error</span><span class="p">,</span>
<span class="n">error_cause</span><span class="o">=</span><span class="n">error_message</span><span class="o">.</span><span class="n">detail</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataResponseFactory.serialize">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_response_factory.DataResponseFactory.serialize">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">serialize</span><span class="p">(</span><span class="n">response</span><span class="p">:</span> <span class="n">DataResponse</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Serialize the message to byte array (to be sent over RabbitMQ)</span>
<span class="sd"> :param response: Object to serialize</span>
<span class="sd"> :return: byte array as serialized message</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">id_bytes</span> <span class="o">=</span> <span class="n">response</span><span class="o">.</span><span class="n">id</span><span class="p">()</span><span class="o">.</span><span class="n">as_string</span><span class="p">()</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span>
<span class="n">prolog</span> <span class="o">=</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;|r=</span><span class="si">{</span><span class="n">response</span><span class="o">.</span><span class="n">response_code</span><span class="p">()</span><span class="si">}</span><span class="s2">|c=</span><span class="si">{</span><span class="n">response</span><span class="o">.</span><span class="n">content_type</span><span class="p">()</span><span class="si">}</span><span class="s2">|e=</span><span class="si">{</span><span class="n">response</span><span class="o">.</span><span class="n">error</span><span class="p">()</span><span class="si">}</span><span class="s2">|f=</span><span class="si">{</span><span class="n">response</span><span class="o">.</span><span class="n">error_cause</span><span class="p">()</span><span class="si">}</span><span class="s2">|t=</span><span class="se">{{</span><span class="si">{</span><span class="n">map_as_string</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">trace_info</span><span class="p">())</span><span class="si">}</span><span class="se">}}</span><span class="s2">|b=&quot;</span>
<span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span>
<span class="n">header_length</span> <span class="o">=</span> <span class="mi">1</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">id_bytes</span><span class="p">)</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">prolog</span><span class="p">)</span>
<span class="n">prolog_len</span> <span class="o">=</span> <span class="n">header_length</span><span class="o">.</span><span class="n">to_bytes</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="n">byteorder</span><span class="o">=</span><span class="s2">&quot;big&quot;</span><span class="p">)</span>
<span class="n">msg_length</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">+</span> <span class="n">header_length</span> <span class="o">+</span> <span class="nb">len</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">base64body</span><span class="p">())</span>
<span class="k">with</span> <span class="n">BytesIO</span><span class="p">(</span><span class="nb">bytearray</span><span class="p">(</span><span class="n">msg_length</span><span class="p">))</span> <span class="k">as</span> <span class="n">baos</span><span class="p">:</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">prolog_len</span><span class="p">)</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">DataMessageMagicByte</span><span class="o">.</span><span class="n">HTTP_REQUEST</span><span class="o">.</span><span class="n">value</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">))</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">id_bytes</span><span class="p">)</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">prolog</span><span class="p">)</span>
<span class="n">baos</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">base64body</span><span class="p">())</span>
<span class="k">return</span> <span class="n">baos</span><span class="o">.</span><span class="n">getvalue</span><span class="p">()</span></div>
<div class="viewcode-block" id="DataResponseFactory.from_bytes">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.data_response_factory.DataResponseFactory.from_bytes">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">from_bytes</span><span class="p">(</span><span class="n">input_bytes</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">DataResponse</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Buil;ds the DataResponse object from its serialized form</span>
<span class="sd"> :param input_bytes: byte array - serialized response</span>
<span class="sd"> :return: DataResponse object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">prolog_len</span> <span class="o">=</span> <span class="p">(</span><span class="n">input_bytes</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&lt;&lt;</span> <span class="mi">8</span><span class="p">)</span> <span class="o">+</span> <span class="n">input_bytes</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="n">metadata</span> <span class="o">=</span> <span class="n">input_bytes</span><span class="p">[</span><span class="mi">2</span> <span class="p">:</span> <span class="n">prolog_len</span> <span class="o">+</span> <span class="mi">2</span><span class="p">]</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="n">pattern</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">DataMessageMagicByte</span><span class="o">.</span><span class="n">HTTP_REQUEST</span><span class="o">.</span><span class="n">value</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="sa">r</span><span class="s2">&quot;([0-9A-Fa-f]+)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;r=([^|]*)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;c=([^|]*)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;e=([^|]*)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;f=([^|]*)\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;t=(\{[^}]*\})\|&quot;</span>
<span class="sa">r</span><span class="s2">&quot;b=&quot;</span>
<span class="p">)</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">pattern</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">metadata</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">match</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;DataResponse format does not match expected format. &quot;</span>
<span class="sa">f</span><span class="s2">&quot;Input string [</span><span class="si">{</span><span class="n">metadata</span><span class="si">}</span><span class="s2">] does not match pattern: </span><span class="si">{</span><span class="n">pattern</span><span class="o">.</span><span class="n">pattern</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">id_str</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">response_code</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">2</span><span class="p">))</span>
<span class="n">content_type</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">3</span><span class="p">)</span>
<span class="n">error</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">4</span><span class="p">)</span>
<span class="n">error_cause</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
<span class="n">trace_info_str</span> <span class="o">=</span> <span class="n">match</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">6</span><span class="p">)</span>
<span class="n">body_b64</span> <span class="o">=</span> <span class="n">input_bytes</span><span class="p">[</span><span class="n">prolog_len</span> <span class="o">+</span> <span class="mi">2</span> <span class="p">:]</span>
<span class="n">trace_info</span> <span class="o">=</span> <span class="n">parse_map_string</span><span class="p">(</span><span class="n">trace_info_str</span><span class="p">)</span>
<span class="k">return</span> <span class="n">DataResponseFactory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">id_str</span><span class="p">,</span>
<span class="n">response_code</span><span class="p">,</span>
<span class="n">content_type</span><span class="p">,</span>
<span class="n">base64</span><span class="o">.</span><span class="n">b64decode</span><span class="p">(</span><span class="n">body_b64</span><span class="p">),</span>
<span class="n">trace_info</span><span class="p">,</span>
<span class="n">error</span><span class="p">,</span>
<span class="n">error_cause</span><span class="p">,</span>
<span class="p">)</span></div>
</div>
<span class="c1"># Ensure backward compatibility</span>
<span class="n">AMQResponseFactory</span> <span class="o">=</span> <span class="n">DataResponseFactory</span>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/file_handler.html
-561
View File
@@ -1,561 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.file_handler &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.file_handler</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.file_handler</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">implements the file downloader for the amqp adapter, where file is sent in chunks over RabbitMQ</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">logging</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">os</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">asyncio</span><span class="w"> </span><span class="kn">import</span> <span class="n">AbstractEventLoop</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">collections</span><span class="w"> </span><span class="kn">import</span> <span class="n">defaultdict</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">pathlib</span><span class="w"> </span><span class="kn">import</span> <span class="n">Path</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika</span><span class="w"> </span><span class="kn">import</span> <span class="n">Message</span><span class="p">,</span> <span class="n">connect_robust</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika.abc</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">AbstractExchange</span><span class="p">,</span>
<span class="n">AbstractIncomingMessage</span><span class="p">,</span>
<span class="n">AbstractQueue</span><span class="p">,</span>
<span class="n">AbstractRobustChannel</span><span class="p">,</span>
<span class="n">DeliveryMode</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_debug</span><span class="p">,</span> <span class="n">logging_error</span><span class="p">,</span> <span class="n">logging_info</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">await_result</span>
<span class="c1"># Global dictionary to track completed downloads</span>
<span class="n">download_locations</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">str</span><span class="p">)</span>
<span class="n">IN_PROGRESS</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">END_OF_FILE</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">CANCELLED</span> <span class="o">=</span> <span class="mi">2</span>
<div class="viewcode-block" id="FileHandler">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.FileHandler">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">FileHandler</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> The abstract base class for file handling. This is mainly to make the code testable, as there are only 2</span>
<span class="sd"> options for pushing (uploading) a file(s) to a service: as HTTP multipart message delivered via RabbitMQ</span>
<span class="sd"> or via stream in RabbitMQ. Downloading a file is always done via RabbitMQ stream. Thus the method signiture</span>
<span class="sd"> is tighly coupled to the RabbitMQ API in v1. Only alternative implementation is a mock object in unittests.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="FileHandler.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.FileHandler.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">pass</span></div>
<div class="viewcode-block" id="FileHandler.ensure_directory_exists">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.FileHandler.ensure_directory_exists">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">ensure_directory_exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Ensures that the directory for the given filepath exists, and handles file versioning.</span>
<span class="sd"> Args:</span>
<span class="sd"> filepath: The path to the file.</span>
<span class="sd"> Returns:</span>
<span class="sd"> The original filepath if the file does not exist, or a modified filepath</span>
<span class="sd"> pointing to the next available version of the file if it does.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># 1. Ensure directory exists</span>
<span class="n">directory</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">filepath</span><span class="p">)</span>
<span class="k">if</span> <span class="n">directory</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">directory</span><span class="p">):</span>
<span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">directory</span><span class="p">)</span>
<span class="c1"># 2. Handle file versioning</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">filepath</span><span class="p">):</span>
<span class="k">return</span> <span class="n">filepath</span> <span class="c1"># File doesn&#39;t exist, return original path</span>
<span class="c1"># File exists, find next available version</span>
<span class="n">base</span><span class="p">,</span> <span class="n">ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">filepath</span><span class="p">)</span>
<span class="n">version</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">new_filepath</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">base</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">version</span><span class="si">}{</span><span class="n">ext</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">new_filepath</span><span class="p">):</span>
<span class="k">return</span> <span class="n">new_filepath</span> <span class="c1"># Found a free version, return this path</span>
<span class="n">version</span> <span class="o">+=</span> <span class="mi">1</span></div>
<div class="viewcode-block" id="FileHandler.download_buffer">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.FileHandler.download_buffer">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">download_buffer</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">queue_name</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="p">(</span><span class="nb">bytearray</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Asynchronously downloads a bytearray from RabbitMQ based on the provided file information.</span>
<span class="sd"> It assumes entire array is in the first (and only) chunk</span>
<span class="sd"> Args:</span>
<span class="sd"> loop: The asyncio event loop.</span>
<span class="sd"> rabbitmq_url: The RabbitMQ connection URL.</span>
<span class="sd"> queue_name (str): The directory where the file should be saved.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">&quot;Subclasses must implement this method.&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="FileHandler.download_file">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.FileHandler.download_file">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">download_file</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">file_info</span><span class="p">,</span> <span class="n">message_data</span><span class="p">,</span> <span class="n">output_dir</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Asynchronously downloads a file from RabbitMQ based on the provided file information.</span>
<span class="sd"> takes the file_info and message_data and output_dir, creates queue consumer and waits for the file</span>
<span class="sd"> to be downloaded. File may consist of multiple chunks, so the function will wait until all chunks are received.</span>
<span class="sd"> Args:</span>
<span class="sd"> loop: The asyncio event loop.</span>
<span class="sd"> rabbitmq_url: The RabbitMQ connection URL.</span>
<span class="sd"> file_info (dict): A dictionary containing file metadata</span>
<span class="sd"> message_data (dict): The entire message</span>
<span class="sd"> output_dir (str): The directory where the file should be saved.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">&quot;Subclasses must implement this method.&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="FileHandler.on_message">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.FileHandler.on_message">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">on_message</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">message</span><span class="p">:</span> <span class="n">AbstractIncomingMessage</span><span class="p">,</span>
<span class="n">json_data</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span>
<span class="n">rabbitmq_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">loop</span><span class="p">,</span>
<span class="n">output_dir</span><span class="o">=</span><span class="s2">&quot;downloaded_files&quot;</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">defaultdict</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Asynchronous callback function for handling messages from the &#39;amq-cleverswarm&#39; queue.</span>
<span class="sd"> Args:</span>
<span class="sd"> message: The incoming RabbitMQ message.</span>
<span class="sd"> rabbitmq_url: The RabbitMQ connection URL.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">&quot;Subclasses must implement this method.&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="FileHandler.publish_file">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.FileHandler.publish_file">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">publish_file</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">exchange</span><span class="p">:</span> <span class="n">AbstractExchange</span><span class="p">,</span>
<span class="n">file_path</span><span class="p">:</span> <span class="n">Path</span><span class="p">,</span>
<span class="n">routing_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">loop</span><span class="p">:</span> <span class="n">AbstractEventLoop</span><span class="p">,</span>
<span class="n">max_chunk_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">2</span><span class="o">**</span><span class="mi">20</span><span class="p">,</span> <span class="c1"># 1MB default chunk size</span>
<span class="n">content_type</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;application/octet-stream&quot;</span><span class="p">,</span> <span class="c1"># Default content type</span>
<span class="n">delivery_mode</span><span class="p">:</span> <span class="n">DeliveryMode</span> <span class="o">=</span> <span class="n">DeliveryMode</span><span class="o">.</span><span class="n">PERSISTENT</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">int</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Reads a file from the file system and publishes its content to a RabbitMQ exchange</span>
<span class="sd"> in chunks. It declares a queue with a random name, binds it to the specified</span>
<span class="sd"> exchange, and uses the queue name as the routing key.</span>
<span class="sd"> Args:</span>
<span class="sd"> exchange: The aio_pika RobustChannel to use for communication with RabbitMQ.</span>
<span class="sd"> file_path: The path to the file to publish.</span>
<span class="sd"> max_chunk_size: The maximum size of each chunk (in bytes).</span>
<span class="sd"> routing_key: The name of the RabbitMQ exchange routing key to publish to correct queue.</span>
<span class="sd"> loop: correct event loop</span>
<span class="sd"> max_chunk_size: chunk size to send</span>
<span class="sd"> content_type: The content type of the file data.</span>
<span class="sd"> delivery_mode: The delivery mode (e.g., PERSISTENT or TRANSIENT).</span>
<span class="sd"> Returns:</span>
<span class="sd"> The name of the randomly generated queue where the file content was published.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">(</span><span class="s2">&quot;Subclasses must implement this method.&quot;</span><span class="p">)</span></div>
</div>
<span class="c1"># The StreamingFileHandler class is a subclass of FileHandler that implements the download_file method</span>
<div class="viewcode-block" id="StreamingFileHandler">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.StreamingFileHandler">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">StreamingFileHandler</span><span class="p">(</span><span class="n">FileHandler</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handles file downloads from RabbitMQ streams.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="StreamingFileHandler.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.StreamingFileHandler.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">()</span></div>
<span class="c1"># This method MUST NOT be static.</span>
<span class="c1"># For testing purpose, we will mock this method and inject a mock connection.</span>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_get_connection</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">loop</span><span class="p">):</span>
<span class="k">return</span> <span class="k">await</span> <span class="n">connect_robust</span><span class="p">(</span><span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">)</span>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_download_from_queue_to_file</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">queue_name</span><span class="p">,</span> <span class="n">filepath</span><span class="p">,</span> <span class="n">loop</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Internal method for downloading a file from a queue to disk.</span>
<span class="sd"> Return (file size, eof status).</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Here the normal mock doesn&#39;t work since this method</span>
<span class="c1"># is async and will be run in another loop,</span>
<span class="c1"># out of the scope of method mock.</span>
<span class="n">connection</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_connection</span><span class="p">(</span><span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">)</span>
<span class="n">_file_size</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">_eof</span> <span class="o">=</span> <span class="n">IN_PROGRESS</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">channel</span><span class="p">:</span> <span class="n">AbstractRobustChannel</span> <span class="o">=</span> <span class="k">await</span> <span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span>
<span class="n">queue</span><span class="p">:</span> <span class="n">AbstractQueue</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">get_queue</span><span class="p">(</span><span class="n">queue_name</span><span class="p">,</span> <span class="n">ensure</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Awaiting all file chunks being downloaded for </span><span class="si">{</span><span class="n">filepath</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">queue</span><span class="o">.</span><span class="n">iterator</span><span class="p">()</span> <span class="k">as</span> <span class="n">queue_iter</span><span class="p">:</span>
<span class="k">async</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">queue_iter</span><span class="p">:</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">message</span><span class="o">.</span><span class="n">process</span><span class="p">():</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filepath</span><span class="p">,</span> <span class="s2">&quot;ab&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">)</span>
<span class="n">_file_size</span> <span class="o">=</span> <span class="n">_file_size</span> <span class="o">+</span> <span class="n">message</span><span class="o">.</span><span class="n">body_size</span>
<span class="n">_eof</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;eof&quot;</span><span class="p">,</span> <span class="n">END_OF_FILE</span><span class="p">))</span>
<span class="k">if</span> <span class="n">_eof</span> <span class="o">!=</span> <span class="n">IN_PROGRESS</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;File download failed: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">finally</span><span class="p">:</span>
<span class="k">await</span> <span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">return</span> <span class="n">_file_size</span><span class="p">,</span> <span class="n">_eof</span>
<div class="viewcode-block" id="StreamingFileHandler.download_file">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.StreamingFileHandler.download_file">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">download_file</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">file_info</span><span class="p">,</span> <span class="n">message_data</span><span class="p">,</span> <span class="n">output_dir</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Asynchronously downloads a file from RabbitMQ based on the provided file information.</span>
<span class="sd"> takes the file_info and message_data and output_dir, creates queue consumer and waits for the file to be downloaded</span>
<span class="sd"> Args:</span>
<span class="sd"> loop: The asyncio event loop.</span>
<span class="sd"> rabbitmq_url: The RabbitMQ connection URL.</span>
<span class="sd"> file_info (dict): A dictionary containing file metadata</span>
<span class="sd"> message_data (dict): The entire message</span>
<span class="sd"> output_dir (str): The directory where the file should be saved.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">file_info</span><span class="p">[</span><span class="s2">&quot;filename&quot;</span><span class="p">]</span>
<span class="n">size</span> <span class="o">=</span> <span class="n">file_info</span><span class="p">[</span><span class="s2">&quot;size&quot;</span><span class="p">]</span>
<span class="n">stream_name</span> <span class="o">=</span> <span class="n">file_info</span><span class="p">[</span><span class="s2">&quot;streamName&quot;</span><span class="p">]</span>
<span class="n">queue_name</span> <span class="o">=</span> <span class="n">file_info</span><span class="p">[</span><span class="s2">&quot;queue_name&quot;</span><span class="p">]</span>
<span class="n">filepath</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">output_dir</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span>
<span class="n">filepath</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ensure_directory_exists</span><span class="p">(</span>
<span class="n">filepath</span>
<span class="p">)</span> <span class="c1"># if filename exists, create a new (next) version</span>
<span class="n">download_locations</span><span class="p">[</span><span class="n">file_info</span><span class="p">[</span><span class="s2">&quot;key&quot;</span><span class="p">]]</span> <span class="o">=</span> <span class="n">filepath</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span>
<span class="n">filepath</span>
<span class="p">)</span> <span class="c1"># if a new version was created, ensure to use that new version</span>
<span class="n">file_info</span><span class="p">[</span><span class="s2">&quot;filename&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">filename</span>
<span class="n">logging_debug</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Downloading file: </span><span class="si">{</span><span class="n">filename</span><span class="si">}</span><span class="s2"> (size: </span><span class="si">{</span><span class="n">size</span><span class="si">}</span><span class="s2"> bytes) from stream: </span><span class="si">{</span><span class="n">stream_name</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">queue_name</span><span class="si">}</span><span class="s2"> to </span><span class="si">{</span><span class="n">filepath</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">queue_name</span><span class="p">:</span>
<span class="n">_local_file_size</span><span class="p">,</span> <span class="n">_local_eof</span> <span class="o">=</span> <span class="k">await</span> <span class="n">await_result</span><span class="p">(</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_download_from_queue_to_file</span><span class="p">(</span><span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">queue_name</span><span class="p">,</span> <span class="n">filepath</span><span class="p">,</span> <span class="n">loop</span><span class="p">),</span>
<span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">_local_file_size</span><span class="p">,</span> <span class="n">_local_eof</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">,</span> <span class="n">CANCELLED</span></div>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_download_from_queue_to_buffer</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">queue_name</span><span class="p">,</span> <span class="n">loop</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">bytearray</span><span class="p">,</span> <span class="nb">int</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Internal method for downloading a file from a queue to RAM.</span>
<span class="sd"> Return (bytearray, eof status).</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_connection</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">_get_connection</span><span class="p">(</span><span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">)</span>
<span class="n">_eof</span> <span class="o">=</span> <span class="n">IN_PROGRESS</span>
<span class="n">_res</span> <span class="o">=</span> <span class="nb">bytearray</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">channel</span><span class="p">:</span> <span class="n">AbstractRobustChannel</span> <span class="o">=</span> <span class="k">await</span> <span class="n">_connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span>
<span class="n">queue</span><span class="p">:</span> <span class="n">AbstractQueue</span> <span class="o">=</span> <span class="k">await</span> <span class="n">channel</span><span class="o">.</span><span class="n">get_queue</span><span class="p">(</span><span class="n">queue_name</span><span class="p">,</span> <span class="n">ensure</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Awaiting all buffer chunks being downloaded for </span><span class="si">{</span><span class="n">queue_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">queue</span><span class="o">.</span><span class="n">iterator</span><span class="p">()</span> <span class="k">as</span> <span class="n">queue_iter</span><span class="p">:</span>
<span class="k">async</span> <span class="k">for</span> <span class="n">message</span> <span class="ow">in</span> <span class="n">queue_iter</span><span class="p">:</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">message</span><span class="o">.</span><span class="n">process</span><span class="p">():</span>
<span class="n">_res</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">)</span>
<span class="n">_eof</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;eof&quot;</span><span class="p">,</span> <span class="n">END_OF_FILE</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_eof</span> <span class="o">!=</span> <span class="n">IN_PROGRESS</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Finished downloading buffer </span><span class="si">{</span><span class="n">queue_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Downloading buffer: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">await</span> <span class="n">_connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">return</span> <span class="n">_res</span><span class="p">,</span> <span class="n">_eof</span>
<div class="viewcode-block" id="StreamingFileHandler.download_buffer">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.StreamingFileHandler.download_buffer">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">download_buffer</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">queue_name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="p">(</span><span class="nb">bytearray</span><span class="p">,</span> <span class="nb">int</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Asynchronously downloads a bytearay from RabbitMQ based on the provided file information.</span>
<span class="sd"> It assumes entire array is in the first (and only) chunk</span>
<span class="sd"> Args:</span>
<span class="sd"> loop: The asyncio event loop.</span>
<span class="sd"> rabbitmq_url: The RabbitMQ connection URL.</span>
<span class="sd"> queue_name (str): The directory where the file should be saved.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Downloading buffer from stream: </span><span class="si">{</span><span class="n">queue_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">queue_name</span><span class="p">:</span>
<span class="n">_local_res</span><span class="p">,</span> <span class="n">_local_eof</span> <span class="o">=</span> <span class="k">await</span> <span class="n">await_result</span><span class="p">(</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_download_from_queue_to_buffer</span><span class="p">(</span><span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">queue_name</span><span class="p">,</span> <span class="n">loop</span><span class="p">),</span> <span class="n">loop</span><span class="o">=</span><span class="n">loop</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">_local_res</span><span class="p">,</span> <span class="n">_local_eof</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">,</span> <span class="n">CANCELLED</span></div>
<div class="viewcode-block" id="StreamingFileHandler.on_message">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.StreamingFileHandler.on_message">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">on_message</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">message</span><span class="p">:</span> <span class="n">AbstractIncomingMessage</span><span class="p">,</span>
<span class="n">json_data</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span>
<span class="n">rabbitmq_url</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">loop</span><span class="p">,</span>
<span class="n">output_dir</span><span class="o">=</span><span class="s2">&quot;downloaded_files&quot;</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">defaultdict</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Asynchronous callback function for handling messages from the &#39;amq-cleverswarm&#39; queue.</span>
<span class="sd"> Args:</span>
<span class="sd"> message: The incoming RabbitMQ message.</span>
<span class="sd"> json_data: The JSON data from the message.</span>
<span class="sd"> rabbitmq_url: The RabbitMQ connection URL.</span>
<span class="sd"> loop: The asyncio event loop of the RabbitMQ API</span>
<span class="sd"> output_dir</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">_message_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="o">.</span><span class="n">decode</span><span class="p">())</span>
<span class="n">_amq_message_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">json_data</span><span class="o">.</span><span class="n">decode</span><span class="p">())</span>
<span class="n">files_to_download</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">_key</span><span class="p">,</span> <span class="n">_file_type</span> <span class="ow">in</span> <span class="n">_amq_message_data</span><span class="p">[</span><span class="s2">&quot;files&quot;</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">_file_type</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">files_info</span> <span class="o">=</span> <span class="n">_file_type</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">stream_name</span> <span class="o">=</span> <span class="n">files_info</span><span class="p">[</span><span class="s2">&quot;streamName&quot;</span><span class="p">]</span>
<span class="n">files_info</span><span class="p">[</span><span class="s2">&quot;queue_name&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">_message_data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">stream_name</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">)</span> <span class="k">if</span> <span class="n">stream_name</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<span class="p">)</span>
<span class="n">files_info</span><span class="p">[</span><span class="s2">&quot;key&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="n">_key</span>
<span class="n">files_to_download</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">files_info</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">files_to_download</span><span class="p">:</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot; [</span><span class="si">%s</span><span class="s2">] No files to download in this message.&quot;</span><span class="p">,</span> <span class="n">message</span><span class="o">.</span><span class="n">delivery_tag</span><span class="p">)</span>
<span class="c1"># await message.ack()</span>
<span class="k">return</span> <span class="n">defaultdict</span><span class="p">()</span>
<span class="c1"># Create a task for each file download</span>
<span class="n">_tasks</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">file_info</span> <span class="ow">in</span> <span class="n">files_to_download</span><span class="p">:</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot; [</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">delivery_tag</span><span class="si">}</span><span class="s2">] about to create task for </span><span class="si">{</span><span class="n">file_info</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">_task</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">create_task</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">download_file</span><span class="p">(</span><span class="n">loop</span><span class="p">,</span> <span class="n">rabbitmq_url</span><span class="p">,</span> <span class="n">file_info</span><span class="p">,</span> <span class="n">_message_data</span><span class="p">,</span> <span class="n">output_dir</span><span class="p">)</span>
<span class="p">)</span>
<span class="n">_tasks</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">_task</span><span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot; [</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">delivery_tag</span><span class="si">}</span><span class="s2">] Waiting for all files to download...gather(*tasks)&quot;</span>
<span class="p">)</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">gather</span><span class="p">(</span><span class="o">*</span><span class="n">_tasks</span><span class="p">)</span> <span class="c1"># Wait for all downloads to complete</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;All files downloaded. d-tag: </span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">delivery_tag</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Building DataMessage: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">(),</span> <span class="n">loop</span><span class="o">=</span><span class="n">loop</span>
<span class="p">)</span> <span class="c1"># ACK -&gt; ignore the message (do not redeliver)</span>
<span class="k">return</span> <span class="n">download_locations</span></div>
<div class="viewcode-block" id="StreamingFileHandler.publish_file">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.file_handler.StreamingFileHandler.publish_file">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">publish_file</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">exchange</span><span class="p">:</span> <span class="n">AbstractExchange</span><span class="p">,</span>
<span class="n">file_path</span><span class="p">:</span> <span class="n">Path</span><span class="p">,</span>
<span class="n">routing_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">loop</span><span class="p">:</span> <span class="n">AbstractEventLoop</span><span class="p">,</span>
<span class="n">max_chunk_size</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">2</span><span class="o">**</span><span class="mi">20</span><span class="p">,</span> <span class="c1"># 1MB default chunk size</span>
<span class="n">content_type</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;application/octet-stream&quot;</span><span class="p">,</span> <span class="c1"># Default content type</span>
<span class="n">delivery_mode</span><span class="p">:</span> <span class="n">DeliveryMode</span> <span class="o">=</span> <span class="n">DeliveryMode</span><span class="o">.</span><span class="n">PERSISTENT</span><span class="p">,</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">int</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Reads a file from the file system and publishes its content to a RabbitMQ exchange</span>
<span class="sd"> in chunks. It declares a queue with a random name, binds it to the specified</span>
<span class="sd"> exchange, and uses the queue name as the routing key.</span>
<span class="sd"> Args:</span>
<span class="sd"> exchange: The aio_pika RobustChannel to use for communication with RabbitMQ.</span>
<span class="sd"> file_path: The path to the file to publish.</span>
<span class="sd"> max_chunk_size: The maximum size of each chunk (in bytes).</span>
<span class="sd"> routing_key: The name of the RabbitMQ exchange routing key to publish to correct queue.</span>
<span class="sd"> loop: correct event loop</span>
<span class="sd"> max_chunk_size: chunk size to send</span>
<span class="sd"> content_type: The content type of the file data.</span>
<span class="sd"> delivery_mode: The delivery mode (e.g., PERSISTENT or TRANSIENT).</span>
<span class="sd"> Returns:</span>
<span class="sd"> The name of the randomly generated queue where the file content was published.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">file_path</span><span class="o">.</span><span class="n">is_file</span><span class="p">():</span>
<span class="k">raise</span> <span class="ne">FileNotFoundError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;File not found: </span><span class="si">{</span><span class="n">file_path</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">_file_size</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">getsize</span><span class="p">(</span><span class="n">file_path</span><span class="p">)</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">file_path</span><span class="p">,</span> <span class="s2">&quot;rb&quot;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">_offset</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">_chunk</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">max_chunk_size</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">_chunk</span><span class="p">:</span>
<span class="k">break</span> <span class="c1"># End of file</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span>
<span class="n">body</span><span class="o">=</span><span class="n">_chunk</span><span class="p">,</span>
<span class="n">content_type</span><span class="o">=</span><span class="n">content_type</span><span class="p">,</span>
<span class="n">delivery_mode</span><span class="o">=</span><span class="n">delivery_mode</span><span class="p">,</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span>
<span class="s2">&quot;file_name&quot;</span><span class="p">:</span> <span class="n">file_path</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="s2">&quot;pos&quot;</span><span class="p">:</span> <span class="n">_offset</span><span class="p">,</span>
<span class="s2">&quot;total_size&quot;</span><span class="p">:</span> <span class="n">_file_size</span><span class="p">,</span>
<span class="s2">&quot;chunk_size&quot;</span><span class="p">:</span> <span class="nb">len</span><span class="p">(</span><span class="n">_chunk</span><span class="p">),</span>
<span class="s2">&quot;eof&quot;</span><span class="p">:</span> <span class="p">(</span>
<span class="n">END_OF_FILE</span> <span class="k">if</span> <span class="n">max_chunk_size</span> <span class="o">+</span> <span class="n">_offset</span> <span class="o">&gt;=</span> <span class="n">_file_size</span> <span class="k">else</span> <span class="n">IN_PROGRESS</span>
<span class="p">),</span>
<span class="p">},</span>
<span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Going to Publish chunk </span><span class="si">{</span><span class="n">_offset</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">_file_size</span><span class="si">}</span><span class="s2"> to </span><span class="si">{</span><span class="n">exchange</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">routing_key</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">_future</span> <span class="o">=</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">run_coroutine_threadsafe</span><span class="p">(</span>
<span class="n">exchange</span><span class="o">.</span><span class="n">publish</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">routing_key</span><span class="o">=</span><span class="n">routing_key</span><span class="p">),</span> <span class="n">loop</span>
<span class="p">)</span>
<span class="k">while</span> <span class="ow">not</span> <span class="n">_future</span><span class="o">.</span><span class="n">done</span><span class="p">():</span>
<span class="c1"># sleep allows coroutine to execute. must wait for completion to ensure all chunks written in right order</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.02</span><span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Waiting for chunk </span><span class="si">{</span><span class="n">_offset</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">_file_size</span><span class="si">}</span><span class="s2"> future= </span><span class="si">{</span><span class="n">_future</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">_offset</span> <span class="o">+=</span> <span class="nb">len</span><span class="p">(</span><span class="n">_chunk</span><span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;Published chunk </span><span class="si">{</span><span class="n">_offset</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">_file_size</span><span class="si">}</span><span class="s2"> to </span><span class="si">{</span><span class="n">exchange</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">routing_key</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;File </span><span class="si">{</span><span class="n">file_path</span><span class="si">}</span><span class="s2"> published to RabbitMQ exchange </span><span class="si">{</span><span class="n">exchange</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">, queue </span><span class="si">{</span><span class="n">routing_key</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">routing_key</span><span class="p">,</span> <span class="n">_file_size</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/logging_utils.html
-250
View File
@@ -1,250 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.logging_utils &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.logging_utils</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.logging_utils</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">inspect</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">logging</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">os</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">sys</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">threading</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">traceback</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">importlib.metadata</span><span class="w"> </span><span class="kn">import</span> <span class="n">version</span>
<span class="c1"># Get version from installed package metadata</span>
<span class="n">__version__</span> <span class="o">=</span> <span class="n">version</span><span class="p">(</span><span class="s2">&quot;amq_adapter&quot;</span><span class="p">)</span> <span class="c1"># Name must match pyproject.toml [project] name</span>
<span class="n">__verbose__</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;AMQ_VERBOSE_LOGGING&quot;</span><span class="p">,</span> <span class="s2">&quot;0&quot;</span><span class="p">)</span> <span class="o">==</span> <span class="s2">&quot;1&quot;</span>
<div class="viewcode-block" id="get_context_info">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.logging_utils.get_context_info">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_context_info</span><span class="p">():</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Retrieves the current_availability module, line number, and function name.</span>
<span class="sd"> Returns:</span>
<span class="sd"> A tuple containing (module name, line number, function name).</span>
<span class="sd"> Returns (None, None, None) if it fails to retrieve the information.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Get the frame object for the current_availability function call</span>
<span class="n">frame</span> <span class="o">=</span> <span class="n">inspect</span><span class="o">.</span><span class="n">currentframe</span><span class="p">()</span>
<span class="k">if</span> <span class="n">frame</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="c1"># Get the frame object for the caller of the current_availability function</span>
<span class="n">frame</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">f_back</span><span class="o">.</span><span class="n">f_back</span>
<span class="k">if</span> <span class="n">frame</span><span class="p">:</span>
<span class="n">code_context</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">f_code</span>
<span class="n">module_name</span> <span class="o">=</span> <span class="n">code_context</span><span class="o">.</span><span class="n">co_filename</span>
<span class="n">module_name</span> <span class="o">=</span> <span class="p">(</span>
<span class="s2">&quot;amqp&quot;</span> <span class="o">+</span> <span class="n">module_name</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;/amqp&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">&quot;/amqp&quot;</span> <span class="ow">in</span> <span class="n">module_name</span>
<span class="k">else</span> <span class="n">module_name</span>
<span class="p">)</span>
<span class="n">line_number</span> <span class="o">=</span> <span class="n">frame</span><span class="o">.</span><span class="n">f_lineno</span>
<span class="n">function_name</span> <span class="o">=</span> <span class="n">code_context</span><span class="o">.</span><span class="n">co_name</span>
<span class="k">return</span> <span class="n">module_name</span><span class="p">,</span> <span class="n">line_number</span><span class="p">,</span> <span class="n">function_name</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
<span class="k">return</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span></div>
<div class="viewcode-block" id="logging_error">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.logging_utils.logging_error">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">logging_error</span><span class="p">(</span><span class="n">msg</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Log an error message according to current_availability logging for &#39;ERROR&#39; level.</span>
<span class="sd"> Add module name, line and function name where the error occurred, on single line.</span>
<span class="sd"> Args:</span>
<span class="sd"> msg (str): The error message to log.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_exec_info</span> <span class="o">=</span> <span class="n">sys</span><span class="o">.</span><span class="n">exc_info</span><span class="p">()[</span><span class="mi">2</span><span class="p">]</span>
<span class="n">_thread_id</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">get_ident</span><span class="p">()</span>
<span class="k">if</span> <span class="n">_exec_info</span><span class="p">:</span>
<span class="n">_tb</span> <span class="o">=</span> <span class="n">traceback</span><span class="o">.</span><span class="n">extract_tb</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">exc_info</span><span class="p">()[</span><span class="mi">2</span><span class="p">])[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [E] &#39;</span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&#39; ---&gt; &#39;</span><span class="si">{</span><span class="n">_tb</span><span class="o">.</span><span class="n">filename</span><span class="si">}</span><span class="s2">&#39;:</span><span class="si">{</span><span class="n">_tb</span><span class="o">.</span><span class="n">lineno</span><span class="si">}</span><span class="s2">, &#39;</span><span class="si">{</span><span class="n">_tb</span><span class="o">.</span><span class="n">name</span><span class="si">}</span><span class="s2">()&#39;&quot;</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">module</span><span class="p">,</span> <span class="n">line_number</span><span class="p">,</span> <span class="n">function_name</span> <span class="o">=</span> <span class="n">get_context_info</span><span class="p">()</span>
<span class="k">if</span> <span class="n">module</span> <span class="ow">and</span> <span class="n">line_number</span> <span class="ow">and</span> <span class="n">function_name</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [E] &#39;</span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&#39; ---&gt; &#39;</span><span class="si">{</span><span class="n">module</span><span class="si">}</span><span class="s2">&#39;:</span><span class="si">{</span><span class="n">line_number</span><span class="si">}</span><span class="s2">, &#39;</span><span class="si">{</span><span class="n">function_name</span><span class="si">}</span><span class="s2">()&#39;&quot;</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [E] </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span></div>
<div class="viewcode-block" id="logging_warning">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.logging_utils.logging_warning">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">logging_warning</span><span class="p">(</span><span class="n">msg</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Log a warning message according to current_availability logging for &#39;WARN&#39; level.</span>
<span class="sd"> Add module name, line and function name where the print occurred, on single line.</span>
<span class="sd"> Args:</span>
<span class="sd"> msg (str): The warning message to log.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_thread_id</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">get_ident</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">__verbose__</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [W] </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">return</span>
<span class="n">module</span><span class="p">,</span> <span class="n">line_number</span><span class="p">,</span> <span class="n">function_name</span> <span class="o">=</span> <span class="n">get_context_info</span><span class="p">()</span>
<span class="k">if</span> <span class="n">module</span> <span class="ow">and</span> <span class="n">line_number</span> <span class="ow">and</span> <span class="n">function_name</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [W] &#39;</span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&#39; ---&gt; &#39;</span><span class="si">{</span><span class="n">module</span><span class="si">}</span><span class="s2">&#39;:</span><span class="si">{</span><span class="n">line_number</span><span class="si">}</span><span class="s2">, &#39;</span><span class="si">{</span><span class="n">function_name</span><span class="si">}</span><span class="s2">()&#39;&quot;</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [W] </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span></div>
<div class="viewcode-block" id="logging_info">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.logging_utils.logging_info">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">logging_info</span><span class="p">(</span><span class="n">msg</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Log an info message according to current_availability logging for &#39;INFO&#39; level.</span>
<span class="sd"> Add module name, line and function name where the print occurred, on single line.</span>
<span class="sd"> Args:</span>
<span class="sd"> msg (str): The info message to log.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_thread_id</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">get_ident</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">__verbose__</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [*] </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">return</span>
<span class="n">module</span><span class="p">,</span> <span class="n">line_number</span><span class="p">,</span> <span class="n">function_name</span> <span class="o">=</span> <span class="n">get_context_info</span><span class="p">()</span>
<span class="k">if</span> <span class="n">module</span> <span class="ow">and</span> <span class="n">line_number</span> <span class="ow">and</span> <span class="n">function_name</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [*] &#39;</span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&#39; ---&gt; &#39;</span><span class="si">{</span><span class="n">module</span><span class="si">}</span><span class="s2">&#39;:</span><span class="si">{</span><span class="n">line_number</span><span class="si">}</span><span class="s2">, &#39;</span><span class="si">{</span><span class="n">function_name</span><span class="si">}</span><span class="s2">()&#39;&quot;</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [*] </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span></div>
<div class="viewcode-block" id="logging_debug">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.logging_utils.logging_debug">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">logging_debug</span><span class="p">(</span><span class="n">msg</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Log a debug message according to current_availability logging for &#39;DEBUG&#39; level.</span>
<span class="sd"> Add module name, line and function name where the print occurred, on single line.</span>
<span class="sd"> Args:</span>
<span class="sd"> msg (str): The debug message to log.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_thread_id</span> <span class="o">=</span> <span class="n">threading</span><span class="o">.</span><span class="n">get_ident</span><span class="p">()</span>
<span class="n">module</span><span class="p">,</span> <span class="n">line_number</span><span class="p">,</span> <span class="n">function_name</span> <span class="o">=</span> <span class="n">get_context_info</span><span class="p">()</span>
<span class="k">if</span> <span class="n">module</span> <span class="ow">and</span> <span class="n">line_number</span> <span class="ow">and</span> <span class="n">function_name</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [DBG] &#39;</span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&#39; ---&gt; &#39;</span><span class="si">{</span><span class="n">module</span><span class="si">}</span><span class="s2">&#39;:</span><span class="si">{</span><span class="n">line_number</span><span class="si">}</span><span class="s2">, &#39;</span><span class="si">{</span><span class="n">function_name</span><span class="si">}</span><span class="s2">()&#39;&quot;</span><span class="p">,</span>
<span class="o">*</span><span class="n">args</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">_thread_id</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">__version__</span><span class="si">}</span><span class="s2"> [DBG] </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span></div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/pydantic_serializer.html
-304
View File
@@ -1,304 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.pydantic_serializer &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.pydantic_serializer</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.pydantic_serializer</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">This module provides serialization and deserialization functions for Pydantic models that extend BaseModel</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">importlib</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">uuid</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">datetime</span><span class="w"> </span><span class="kn">import</span> <span class="n">date</span><span class="p">,</span> <span class="n">datetime</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">types</span><span class="w"> </span><span class="kn">import</span> <span class="n">UnionType</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">Union</span><span class="p">,</span> <span class="n">get_args</span><span class="p">,</span> <span class="n">get_origin</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">urllib.parse</span><span class="w"> </span><span class="kn">import</span> <span class="n">parse_qs</span><span class="p">,</span> <span class="n">urlparse</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">pydantic</span><span class="w"> </span><span class="kn">import</span> <span class="n">BaseModel</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_debug</span>
<div class="viewcode-block" id="serialize_object">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.pydantic_serializer.serialize_object">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">serialize_object</span><span class="p">(</span><span class="n">obj</span><span class="p">:</span> <span class="n">BaseModel</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Serializes an R2RSerializable object to a string in the format</span>
<span class="sd"> &quot;&lt;ClassName&gt;:{key1:value1, key2:value2,...}&quot;. Handles UUIDs and datetimes,</span>
<span class="sd"> and nested Pydantic models.</span>
<span class="sd"> Args:</span>
<span class="sd"> obj: The BaseModel object to serialize.</span>
<span class="sd"> Returns:</span>
<span class="sd"> A string representation of the object.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">class_name</span> <span class="o">=</span> <span class="n">obj</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span>
<span class="n">module_name</span> <span class="o">=</span> <span class="n">obj</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__module__</span>
<span class="n">obj_dict</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">obj</span><span class="o">.</span><span class="n">model_dump</span><span class="p">()</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="c1"># here we use the original value</span>
<span class="c1"># the one returned by model_dump will convert everything into dict recursively</span>
<span class="n">value</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">uuid</span><span class="o">.</span><span class="n">UUID</span><span class="p">):</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">datetime</span><span class="p">):</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span> <span class="c1"># Ensure dictionaries are handled</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="n">BaseModel</span><span class="p">):</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">serialize_object</span><span class="p">(</span><span class="n">value</span><span class="p">)</span> <span class="c1"># Recursively serialize</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="n">serialized_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">value</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="n">BaseModel</span><span class="p">):</span>
<span class="n">serialized_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">serialize_object</span><span class="p">(</span><span class="n">item</span><span class="p">))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">serialized_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">serialized_list</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="n">dict_string</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">obj_dict</span><span class="p">)</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">module_name</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">class_name</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">dict_string</span><span class="si">}</span><span class="s2">&quot;</span></div>
<div class="viewcode-block" id="python_type_to_json">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.pydantic_serializer.python_type_to_json">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">python_type_to_json</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Convert Python objects (dict, list, tuple, set, datetime, etc.) to a JSON string.</span>
<span class="sd"> Handles common non-JSON-serializable types like datetime, date, and sets.</span>
<span class="sd"> Args:</span>
<span class="sd"> data: Python object to be converted (dict, list, tuple, set, etc.)</span>
<span class="sd"> Returns:</span>
<span class="sd"> str: JSON string representation of the input data</span>
<span class="sd"> Raises:</span>
<span class="sd"> TypeError: If the object contains non-serializable types (e.g., custom classes)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">def</span><span class="w"> </span><span class="nf">default_serializer</span><span class="p">(</span><span class="n">obj</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Custom serializer for non-JSON-serializable objects.&quot;&quot;&quot;</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="p">(</span><span class="n">datetime</span><span class="p">,</span> <span class="n">date</span><span class="p">)):</span>
<span class="k">return</span> <span class="n">obj</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span> <span class="c1"># Convert datetime/date to ISO string</span>
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="nb">set</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">list</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span> <span class="c1"># Convert sets to lists</span>
<span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">obj</span><span class="p">,</span> <span class="s2">&quot;__dict__&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="n">obj</span><span class="o">.</span><span class="vm">__dict__</span> <span class="c1"># Attempt to serialize custom objects</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Object of type </span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="n">obj</span><span class="p">)</span><span class="si">}</span><span class="s2"> is not JSON serializable&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="n">default_serializer</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">ensure_ascii</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span></div>
<div class="viewcode-block" id="try_deserialize_object">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.pydantic_serializer.try_deserialize_object">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">try_deserialize_object</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">BaseModel</span> <span class="o">|</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Try deserialize object, if failed, return the input string as-is.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
<span class="k">return</span> <span class="n">data</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">return</span> <span class="n">deserialize_object</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to deserialize object: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">data</span></div>
<div class="viewcode-block" id="deserialize_object">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.pydantic_serializer.deserialize_object">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">deserialize_object</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">BaseModel</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Deserializes a string in the format &quot;&lt;ClassName&gt;:{key1:value1, key2:value2,...}&quot;</span>
<span class="sd"> back into an BaseModel object. Handles UUIDs, datetimes, and</span>
<span class="sd"> nested Pydantic models.</span>
<span class="sd"> Args:</span>
<span class="sd"> data: The string to deserialize.</span>
<span class="sd"> Returns:</span>
<span class="sd"> An R2RSerializable object.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">class_path</span><span class="p">,</span> <span class="n">dict_str</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">module_path</span><span class="p">,</span> <span class="n">class_name</span> <span class="o">=</span> <span class="n">class_path</span><span class="o">.</span><span class="n">rsplit</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">obj_dict</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">dict_str</span><span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Deserialized object: </span><span class="si">{</span><span class="n">class_path</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="s2">&quot;[&quot;</span> <span class="ow">in</span> <span class="n">class_name</span><span class="p">:</span>
<span class="n">class_name</span> <span class="o">=</span> <span class="n">class_name</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;[&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;trying object: </span><span class="si">{</span><span class="n">class_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">module</span> <span class="o">=</span> <span class="n">importlib</span><span class="o">.</span><span class="n">import_module</span><span class="p">(</span><span class="n">module_path</span><span class="p">)</span>
<span class="bp">cls</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">module</span><span class="p">,</span> <span class="n">class_name</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">cls</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unknown class: </span><span class="si">{</span><span class="n">class_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># process nested model, dict and list</span>
<span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">obj_dict</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">key_type</span><span class="p">:</span> <span class="nb">type</span><span class="p">[</span><span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="bp">cls</span><span class="o">.</span><span class="n">model_fields</span><span class="p">[</span><span class="n">key</span><span class="p">]</span><span class="o">.</span><span class="n">annotation</span>
<span class="c1"># handle union with none, this will give the true type</span>
<span class="c1"># for example, return `A` if the type is `A | None`</span>
<span class="k">if</span> <span class="n">get_origin</span><span class="p">(</span><span class="n">key_type</span><span class="p">)</span> <span class="ow">is</span> <span class="n">UnionType</span> <span class="ow">or</span> <span class="n">get_origin</span><span class="p">(</span><span class="n">key_type</span><span class="p">)</span> <span class="ow">is</span> <span class="n">Union</span><span class="p">:</span>
<span class="n">union_types</span> <span class="o">=</span> <span class="n">get_args</span><span class="p">(</span><span class="n">key_type</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">union_types</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">2</span><span class="p">:</span>
<span class="c1"># if there are multiple unions, like `A | B | None`</span>
<span class="c1"># then we have no way to figure out which one to use</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Unsupported Union type: </span><span class="si">{</span><span class="n">key_type</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># return the none-None one as the type for processing</span>
<span class="k">if</span> <span class="n">union_types</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">is</span> <span class="nb">type</span><span class="p">(</span><span class="kc">None</span><span class="p">):</span>
<span class="n">key_type</span> <span class="o">=</span> <span class="n">union_types</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">key_type</span> <span class="o">=</span> <span class="n">union_types</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># skip if the value is None</span>
<span class="k">if</span> <span class="n">value</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">key_type</span> <span class="o">==</span> <span class="n">uuid</span><span class="o">.</span><span class="n">UUID</span><span class="p">:</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">uuid</span><span class="o">.</span><span class="n">UUID</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">key_type</span> <span class="o">==</span> <span class="n">datetime</span><span class="p">:</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">key_type</span> <span class="o">==</span> <span class="nb">dict</span><span class="p">:</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">elif</span> <span class="nb">issubclass</span><span class="p">(</span><span class="n">key_type</span><span class="p">,</span> <span class="n">BaseModel</span><span class="p">):</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">deserialize_object</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">key_type</span> <span class="o">==</span> <span class="nb">list</span><span class="p">:</span>
<span class="n">obj_dict</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">map</span><span class="p">(</span><span class="n">try_deserialize_object</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span>
<span class="k">return</span> <span class="bp">cls</span><span class="p">(</span><span class="o">**</span><span class="n">obj_dict</span><span class="p">)</span></div>
<div class="viewcode-block" id="parse_url_query">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.pydantic_serializer.parse_url_query">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">parse_url_query</span><span class="p">(</span><span class="n">url</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Parses a HTTP GET URL and returns (context_path, query_params_dict)</span>
<span class="sd"> Args:</span>
<span class="sd"> url: The URL string to parse (e.g., &quot;http://example.com/path?foo=1&amp;bar=2&amp;foo=3&quot;)</span>
<span class="sd"> Returns:</span>
<span class="sd"> tuple: (context_path, query_params_dict)</span>
<span class="sd"> - context_path: The path part of the URL (e.g., &quot;/path&quot;)</span>
<span class="sd"> - query_params_dict: Dictionary of query parameters where values are always lists</span>
<span class="sd"> (e.g., {&quot;foo&quot;: [&quot;1&quot;, &quot;3&quot;], &quot;bar&quot;: [&quot;2&quot;]})</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">parsed</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="n">query_params</span> <span class="o">=</span> <span class="n">parse_qs</span><span class="p">(</span><span class="n">parsed</span><span class="o">.</span><span class="n">query</span><span class="p">)</span>
<span class="c1"># Convert values from lists to single values when there&#39;s only one value</span>
<span class="c1"># If you prefer to always keep values as lists, remove this comprehension</span>
<span class="n">simplified_params</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span> <span class="k">else</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">query_params</span><span class="o">.</span><span class="n">items</span><span class="p">()}</span>
<span class="k">return</span> <span class="p">(</span><span class="n">parsed</span><span class="o">.</span><span class="n">path</span><span class="p">,</span> <span class="n">simplified_params</span><span class="p">)</span></div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/serializer.html
-216
View File
@@ -1,216 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.serializer &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.serializer</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.serializer</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd">Helper functions used to aid with (de)serialization of messages sent over AMQP.</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">struct</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">io</span><span class="w"> </span><span class="kn">import</span> <span class="n">BytesIO</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Union</span>
<div class="viewcode-block" id="long_to_bytes">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.long_to_bytes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">long_to_bytes</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="k">return</span> <span class="n">struct</span><span class="o">.</span><span class="n">pack</span><span class="p">(</span><span class="s2">&quot;q&quot;</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span></div>
<div class="viewcode-block" id="bytes_to_long">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.bytes_to_long">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">bytes_to_long</span><span class="p">(</span><span class="n">b</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
<span class="k">return</span> <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s2">&quot;q&quot;</span><span class="p">,</span> <span class="n">b</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span></div>
<div class="viewcode-block" id="read_long">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.read_long">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">read_long</span><span class="p">(</span><span class="n">bis</span><span class="p">:</span> <span class="n">BytesIO</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">return</span> <span class="n">struct</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="s2">&quot;q&quot;</span><span class="p">,</span> <span class="n">bis</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">8</span><span class="p">))[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">except</span> <span class="n">struct</span><span class="o">.</span><span class="n">error</span><span class="p">:</span>
<span class="k">return</span> <span class="mi">0</span></div>
<div class="viewcode-block" id="object_or_list_as_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.object_or_list_as_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">object_or_list_as_string</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;[</span><span class="si">{</span><span class="s1">&#39;,&#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">value</span><span class="p">)</span><span class="si">}</span><span class="s2">]&quot;</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">value</span><span class="p">)</span></div>
<div class="viewcode-block" id="map_as_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.map_as_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">map_as_string</span><span class="p">(</span><span class="n">map_</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="s2">&quot;,&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2">=</span><span class="si">{</span><span class="n">object_or_list_as_string</span><span class="p">(</span><span class="n">v</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">map_</span><span class="o">.</span><span class="n">items</span><span class="p">())</span></div>
<div class="viewcode-block" id="map_with_list_as_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.map_with_list_as_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">map_with_list_as_string</span><span class="p">(</span><span class="n">map_</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="s2">&quot;,&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2">=</span><span class="si">{</span><span class="n">object_or_list_as_string</span><span class="p">(</span><span class="n">v</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">map_</span><span class="o">.</span><span class="n">items</span><span class="p">())</span></div>
<div class="viewcode-block" id="parse_map_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.parse_map_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">parse_map_string</span><span class="p">(</span><span class="n">input_str</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
<span class="n">map_</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">if</span> <span class="n">input_str</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;{&quot;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">input_str</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">&quot;}&quot;</span><span class="p">):</span>
<span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="nb">str</span><span class="p">(</span><span class="n">input_str</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">&quot;</span><span class="se">\n</span><span class="s2">&quot;</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">entry</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span>
<span class="k">if</span> <span class="n">entry</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">&#39;&quot;&#39;</span><span class="p">):</span>
<span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;:&quot;</span><span class="p">)</span>
<span class="n">map_</span><span class="p">[</span><span class="n">key</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">&#39;&quot;&#39;</span><span class="p">)]</span> <span class="o">=</span> <span class="n">value</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">&#39;&quot;&#39;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;=&quot;</span><span class="p">)</span>
<span class="n">map_</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
<span class="k">return</span> <span class="n">map_</span></div>
<div class="viewcode-block" id="csv_as_list">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.csv_as_list">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">csv_as_list</span><span class="p">(</span><span class="n">input_str</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
<span class="k">return</span> <span class="p">[</span><span class="n">item</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">input_str</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s2">&quot;[]&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span></div>
<div class="viewcode-block" id="add_to_map">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.add_to_map">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">add_to_map</span><span class="p">(</span><span class="n">map_</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]],</span> <span class="n">token</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">eq_idx</span> <span class="o">=</span> <span class="n">token</span><span class="o">.</span><span class="n">index</span><span class="p">(</span><span class="s2">&quot;=&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="n">eq_idx</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span> <span class="c1"># set to -1 when not found</span>
<span class="k">if</span> <span class="n">eq_idx</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">token</span><span class="p">[:</span><span class="n">eq_idx</span><span class="p">]</span>
<span class="n">map_</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">csv_as_list</span><span class="p">(</span><span class="n">token</span><span class="p">[</span><span class="n">eq_idx</span> <span class="o">+</span> <span class="mi">1</span> <span class="p">:])</span></div>
<div class="viewcode-block" id="parse_map_list_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.parse_map_list_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">parse_map_list_string</span><span class="p">(</span><span class="n">input_str</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]:</span>
<span class="n">map_</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">if</span> <span class="n">input_str</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;{&quot;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">input_str</span><span class="o">.</span><span class="n">endswith</span><span class="p">(</span><span class="s2">&quot;}&quot;</span><span class="p">):</span>
<span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">input_str</span><span class="p">[</span><span class="mi">1</span><span class="p">:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;],&quot;</span><span class="p">):</span>
<span class="n">add_to_map</span><span class="p">(</span><span class="n">map_</span><span class="p">,</span> <span class="n">token</span><span class="p">)</span>
<span class="k">return</span> <span class="n">map_</span></div>
<div class="viewcode-block" id="str_to_bool">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.serializer.str_to_bool">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">str_to_bool</span><span class="p">(</span><span class="n">value</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">return</span> <span class="n">value</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&quot;true&quot;</span><span class="p">,</span> <span class="s2">&quot;yes&quot;</span><span class="p">,</span> <span class="s2">&quot;1&quot;</span><span class="p">,</span> <span class="s2">&quot;t&quot;</span><span class="p">,</span> <span class="s2">&quot;y&quot;</span><span class="p">,</span> <span class="s2">&quot;on&quot;</span><span class="p">)</span></div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/service_message_factory.html
-278
View File
@@ -1,278 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.service_message_factory &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.service_message_factory</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.service_message_factory</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">base64</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.data_message_factory</span><span class="w"> </span><span class="kn">import</span> <span class="n">DataMessageFactory</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_error</span><span class="p">,</span> <span class="n">logging_info</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.serializer</span><span class="w"> </span><span class="kn">import</span> <span class="n">map_as_string</span><span class="p">,</span> <span class="n">parse_map_string</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.trace_info_adapter</span><span class="w"> </span><span class="kn">import</span> <span class="n">TraceInfoAdapter</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">ServiceMessage</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.service_message_type</span><span class="w"> </span><span class="kn">import</span> <span class="n">ServiceMessageType</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.snowflake_id</span><span class="w"> </span><span class="kn">import</span> <span class="n">SnowflakeId</span>
<span class="n">RS</span> <span class="o">=</span> <span class="s2">&quot;|&quot;</span>
<span class="n">SPLIT_RS</span> <span class="o">=</span> <span class="sa">r</span><span class="s2">&quot;\|&quot;</span>
<span class="n">INVALID_MESSAGE</span> <span class="o">=</span> <span class="n">ServiceMessage</span><span class="p">()</span> <span class="c1"># default values mean invalid message</span>
<div class="viewcode-block" id="ServiceMessageFactory">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">ServiceMessageFactory</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> build/serialize/deserialize CleverMicro Service Message class</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="ServiceMessageFactory.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">process_name</span> <span class="o">=</span> <span class="n">name</span></div>
<div class="viewcode-block" id="ServiceMessageFactory.to_string">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory.to_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">to_string</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">ServiceMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Convert ServiceMessage to human-readable format</span>
<span class="sd"> :param message: ServiceMessage object</span>
<span class="sd"> :return: as string</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="n">message</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;ServiceMessage</span><span class="se">{{</span><span class="s2">id=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">as_string</span><span class="p">()</span><span class="si">}</span><span class="s2">|type=</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">format_message_type</span><span class="p">(</span><span class="n">message</span><span class="p">)</span><span class="si">}</span><span class="s2">|&quot;</span>
<span class="sa">f</span><span class="s2">&quot;recipient=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">recipient_name</span><span class="si">}</span><span class="s2">|replyTo=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">reply_to</span><span class="si">}</span><span class="s2">|&quot;</span>
<span class="sa">f</span><span class="s2">&quot;traceInfo=</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="si">}</span><span class="s2">|body=</span><span class="si">{</span><span class="n">base64</span><span class="o">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">encode</span><span class="p">())</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">message</span><span class="o">.</span><span class="n">payload_type</span><span class="w"> </span><span class="o">&amp;</span><span class="w"> </span><span class="mh">0x80</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="n">message</span><span class="o">.</span><span class="n">message</span><span class="si">}</span><span class="se">}}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">return</span> <span class="s2">&quot;null&quot;</span></div>
<div class="viewcode-block" id="ServiceMessageFactory.of">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory.of">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">of</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">message_type</span><span class="p">:</span> <span class="n">ServiceMessageType</span><span class="p">,</span> <span class="n">payload</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">recipient_name</span><span class="p">:</span> <span class="nb">str</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">ServiceMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Builds ServiceMessage object from relevant values.</span>
<span class="sd"> :param message_type: enum Message type</span>
<span class="sd"> :param payload: Message content, stored as-is</span>
<span class="sd"> :param recipient_name: Recipient name (from the routing expression)</span>
<span class="sd"> :return: ServiceMessage object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_id</span> <span class="o">=</span> <span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">get_instance</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">generate_snowflake_id</span><span class="p">()</span>
<span class="k">return</span> <span class="n">ServiceMessage</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="n">_id</span><span class="p">,</span>
<span class="n">payload_type</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span>
<span class="n">message_type</span><span class="o">=</span><span class="n">message_type</span><span class="p">,</span>
<span class="n">recipient_name</span><span class="o">=</span><span class="n">recipient_name</span><span class="p">,</span>
<span class="n">reply_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">process_name</span><span class="p">,</span>
<span class="n">trace_info</span><span class="o">=</span><span class="n">TraceInfoAdapter</span><span class="o">.</span><span class="n">create_produce_span</span><span class="p">(</span><span class="n">_id</span><span class="p">),</span>
<span class="n">message</span><span class="o">=</span><span class="n">payload</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="ServiceMessageFactory.as_base64">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory.as_base64">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">as_base64</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">message_type</span><span class="p">:</span> <span class="n">ServiceMessageType</span><span class="p">,</span> <span class="n">payload</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">recipient_name</span><span class="p">:</span> <span class="nb">str</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">ServiceMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Builds ServiceMessage object from relevant values, stores content Base64 encoded.</span>
<span class="sd"> :param message_type: enum Message type</span>
<span class="sd"> :param payload: Message content, stored as Base64 string</span>
<span class="sd"> :param recipient_name: Recipient name (from the routing expression)</span>
<span class="sd"> :return: ServiceMessage object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_id</span> <span class="o">=</span> <span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">get_instance</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">generate_snowflake_id</span><span class="p">()</span>
<span class="k">return</span> <span class="n">ServiceMessage</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="n">_id</span><span class="p">,</span>
<span class="n">payload_type</span><span class="o">=</span><span class="mh">0x80</span><span class="p">,</span>
<span class="n">message_type</span><span class="o">=</span><span class="n">message_type</span><span class="p">,</span>
<span class="n">recipient_name</span><span class="o">=</span><span class="n">recipient_name</span><span class="p">,</span>
<span class="n">reply_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">process_name</span><span class="p">,</span>
<span class="n">trace_info</span><span class="o">=</span><span class="n">TraceInfoAdapter</span><span class="o">.</span><span class="n">create_produce_span</span><span class="p">(</span><span class="n">_id</span><span class="o">.</span><span class="n">as_string</span><span class="p">()),</span>
<span class="n">message</span><span class="o">=</span><span class="n">payload</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="ServiceMessageFactory.from_bytes">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory.from_bytes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">from_bytes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">input_bytes</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">ServiceMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Deserialize ServiceMessage from byte array serialized form.</span>
<span class="sd"> :param input_bytes: serialized ServiceMessage</span>
<span class="sd"> :return: ServiceMessage object represented by the input.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;ServiceMessage &gt;&gt;&gt; [</span><span class="si">{</span><span class="n">input_bytes</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="si">}</span><span class="s2">]&quot;</span><span class="p">)</span>
<span class="n">input_str_array</span> <span class="o">=</span> <span class="n">input_bytes</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">RS</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">_id</span> <span class="o">=</span> <span class="n">input_str_array</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">payload_type</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">input_str_array</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="mi">16</span><span class="p">)</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">message_type</span> <span class="o">=</span> <span class="n">ServiceMessageType</span><span class="p">(</span><span class="n">payload_type</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">&amp;</span> <span class="mh">0x7F</span><span class="p">)</span>
<span class="n">recipient_name</span> <span class="o">=</span> <span class="n">input_str_array</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">if</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">input_str_array</span><span class="p">)</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">reply_to</span> <span class="o">=</span> <span class="n">input_str_array</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">if</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">input_str_array</span><span class="p">)</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">trace_info</span> <span class="o">=</span> <span class="n">parse_map_string</span><span class="p">(</span><span class="n">input_str_array</span><span class="p">[</span><span class="n">i</span><span class="p">])</span> <span class="k">if</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">input_str_array</span><span class="p">)</span> <span class="k">else</span> <span class="p">{}</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">input_str_array</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">if</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">input_str_array</span><span class="p">)</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<span class="k">return</span> <span class="n">ServiceMessage</span><span class="p">(</span>
<span class="nb">id</span><span class="o">=</span><span class="n">SnowflakeId</span><span class="o">.</span><span class="n">from_hex</span><span class="p">(</span><span class="n">_id</span><span class="p">),</span>
<span class="n">payload_type</span><span class="o">=</span><span class="n">payload_type</span><span class="p">,</span>
<span class="n">message_type</span><span class="o">=</span><span class="n">message_type</span><span class="p">,</span>
<span class="n">recipient_name</span><span class="o">=</span><span class="n">recipient_name</span><span class="p">,</span>
<span class="n">reply_to</span><span class="o">=</span><span class="n">reply_to</span><span class="p">,</span>
<span class="n">trace_info</span><span class="o">=</span><span class="n">trace_info</span><span class="p">,</span>
<span class="n">message</span><span class="o">=</span><span class="p">(</span><span class="n">base64</span><span class="o">.</span><span class="n">b64decode</span><span class="p">(</span><span class="n">message</span><span class="p">)</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span> <span class="k">if</span> <span class="n">payload_type</span> <span class="o">&amp;</span> <span class="mh">0x80</span> <span class="k">else</span> <span class="n">message</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="s2">&quot;Unexpected ServiceMessage format. Expected 6 tokens separated by &#39;|&#39; in format: &quot;</span>
<span class="s2">&quot;[ID(64bit number)|MessageType(8bit number)|RecipientName(String)|&quot;</span>
<span class="s2">&quot;ReplyTo(String)|TraceInfo|Payload(String or Base64)], got: &quot;</span>
<span class="s2">&quot;[</span><span class="si">%s</span><span class="s2">], reported problem: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">input_bytes</span><span class="o">.</span><span class="n">decode</span><span class="p">(),</span>
<span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">INVALID_MESSAGE</span></div>
<div class="viewcode-block" id="ServiceMessageFactory.serialize">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory.serialize">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">serialize</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">ServiceMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Serialize ServiceMessage to byte array.</span>
<span class="sd"> :param message: ServiceMessage to serialize.</span>
<span class="sd"> :return: serialized object.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">as_string</span><span class="p">()</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">format_message_type</span><span class="p">(</span><span class="n">message</span><span class="p">)</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="n">message</span><span class="o">.</span><span class="n">recipient_name</span><span class="si">}{</span><span class="n">RS</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">reply_to</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="n">map_as_string</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">)</span><span class="si">}{</span><span class="n">RS</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">base64</span><span class="o">.</span><span class="n">b64encode</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">encode</span><span class="p">())</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">message</span><span class="o">.</span><span class="n">payload_type</span><span class="w"> </span><span class="o">!=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="n">message</span><span class="o">.</span><span class="n">message</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="ServiceMessageFactory.format_message_type">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.service_message_factory.ServiceMessageFactory.format_message_type">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">format_message_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">ServiceMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Formats the Enum message type and decorates it with base64 flag at highest bit.</span>
<span class="sd"> For compatibility with 0-based enum ordinals in Java subtract 1 from ordinal value.</span>
<span class="sd"> :param message: Service message to send</span>
<span class="sd"> :return: formatted message type</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">num_type</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">message_type</span><span class="o">.</span><span class="n">value</span> <span class="k">if</span> <span class="n">message</span><span class="o">.</span><span class="n">message_type</span> <span class="k">else</span> <span class="n">ServiceMessageType</span><span class="o">.</span><span class="n">UNKNOWN</span><span class="o">.</span><span class="n">value</span>
<span class="p">)</span> <span class="o">-</span> <span class="mi">1</span>
<span class="n">fmt</span> <span class="o">=</span> <span class="p">(</span><span class="n">num_type</span> <span class="o">&amp;</span> <span class="mh">0x7F</span><span class="p">)</span> <span class="o">|</span> <span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">payload_type</span> <span class="o">&amp;</span> <span class="mh">0x80</span><span class="p">)</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">fmt</span><span class="si">:</span><span class="s2">02X</span><span class="si">}</span><span class="s2">&quot;</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/trace_info_adapter.html
-126
View File
@@ -1,126 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.trace_info_adapter &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.trace_info_adapter</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.trace_info_adapter</h1><div class="highlight"><pre>
<span></span><span class="kn">from</span><span class="w"> </span><span class="nn">dataclasses</span><span class="w"> </span><span class="kn">import</span> <span class="n">dataclass</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Dict</span>
<div class="viewcode-block" id="TraceInfoAdapter">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.trace_info_adapter.TraceInfoAdapter">[docs]</a>
<span class="nd">@dataclass</span>
<span class="k">class</span><span class="w"> </span><span class="nc">TraceInfoAdapter</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> TODO This class should set up the Open Telemetry tracing stack. To be done later.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="TraceInfoAdapter.create_produce_span">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.trace_info_adapter.TraceInfoAdapter.create_produce_span">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">create_produce_span</span><span class="p">(</span><span class="n">message_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
<span class="k">return</span> <span class="p">{</span><span class="s2">&quot;trace-id&quot;</span><span class="p">:</span> <span class="n">message_id</span><span class="p">}</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/adapter/type_descriptor.html
-324
View File
@@ -1,324 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.adapter.type_descriptor &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.adapter.type_descriptor</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.adapter.type_descriptor</h1><div class="highlight"><pre>
<span></span><span class="sd">&quot;&quot;&quot; &quot;&quot;&quot;</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">importlib</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Any</span>
<div class="viewcode-block" id="TypeDescriptor">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.type_descriptor.TypeDescriptor">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">TypeDescriptor</span><span class="p">:</span>
<div class="viewcode-block" id="TypeDescriptor.__init__">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.type_descriptor.TypeDescriptor.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">type_string</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">extra_init_data</span><span class="p">:</span> <span class="nb">dict</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">type_string</span> <span class="o">=</span> <span class="n">type_string</span>
<span class="bp">self</span><span class="o">.</span><span class="n">extra_init_data</span> <span class="o">=</span> <span class="n">extra_init_data</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_json</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span>
<span class="bp">self</span><span class="o">.</span><span class="n">type</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_extract_type</span><span class="p">()</span></div>
<span class="k">def</span><span class="w"> </span><span class="nf">_extract_type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;&lt;class &#39;&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_string</span><span class="p">[</span><span class="mi">8</span><span class="p">:</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
<span class="c1"># Handle Annotated case of FastAPI annotation</span>
<span class="k">if</span> <span class="s2">&quot;typing.Optional&quot;</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_string</span><span class="p">:</span>
<span class="n">annotated</span> <span class="o">=</span> <span class="n">_extract_type_of</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">type_string</span><span class="p">)</span>
<span class="k">if</span> <span class="s2">&quot;typing.Annotated&quot;</span> <span class="ow">in</span> <span class="n">annotated</span><span class="p">:</span>
<span class="c1"># Extract content between the first square brackets after Annotated</span>
<span class="n">match</span> <span class="o">=</span> <span class="n">_extract_type_of</span><span class="p">(</span><span class="n">annotated</span><span class="p">)</span>
<span class="k">if</span> <span class="n">match</span><span class="p">:</span>
<span class="c1"># Split by comma and get first part, then strip whitespace</span>
<span class="n">parts</span> <span class="o">=</span> <span class="p">[</span><span class="n">p</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">match</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_json</span> <span class="o">=</span> <span class="n">parts</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s2">&quot;json&quot;</span>
<span class="k">return</span> <span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">if</span> <span class="n">parts</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># Handle simple Optional case</span>
<span class="k">return</span> <span class="n">annotated</span>
<span class="k">return</span> <span class="n">_extract_type_of</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">type_string</span><span class="p">)</span>
<div class="viewcode-block" id="TypeDescriptor.instantiate">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.type_descriptor.TypeDescriptor.instantiate">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">instantiate</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">payload</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="n">is_json</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Any</span><span class="p">:</span>
<span class="k">return</span> <span class="n">_instantiate</span><span class="p">(</span>
<span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">type</span>
<span class="k">if</span> <span class="s2">&quot;typing.Optional&quot;</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_string</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;&lt;class &quot;</span><span class="p">)</span>
<span class="k">else</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_string</span>
<span class="p">),</span>
<span class="n">payload</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_json</span> <span class="ow">or</span> <span class="n">is_json</span><span class="p">,</span>
<span class="n">extra_init_data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">extra_init_data</span><span class="p">,</span>
<span class="p">)</span></div>
<span class="k">def</span><span class="w"> </span><span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;TypeDescriptor(type=&#39;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">type</span><span class="si">}</span><span class="s2">&#39;, is_json=</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">is_json</span><span class="si">}</span><span class="s2">)&quot;</span></div>
<span class="k">def</span><span class="w"> </span><span class="nf">_extract_type_of</span><span class="p">(</span><span class="n">input_string</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="n">stack</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">start_index</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="n">end_index</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
<span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">char</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">input_string</span><span class="p">):</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span> <span class="c1"># this assumes paramerized type, that must be at lest 1 char long</span>
<span class="k">if</span> <span class="n">char</span> <span class="o">==</span> <span class="s2">&quot;[&quot;</span><span class="p">:</span>
<span class="k">if</span> <span class="n">start_index</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
<span class="n">start_index</span> <span class="o">=</span> <span class="n">i</span>
<span class="n">stack</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">char</span><span class="p">)</span>
<span class="k">elif</span> <span class="n">char</span> <span class="o">==</span> <span class="s2">&quot;]&quot;</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">stack</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="n">end_index</span> <span class="o">=</span> <span class="n">i</span>
<span class="k">break</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">stack</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">stack</span><span class="o">.</span><span class="n">pop</span><span class="p">()</span>
<span class="k">if</span> <span class="n">start_index</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span> <span class="ow">and</span> <span class="n">end_index</span> <span class="o">!=</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
<span class="k">return</span> <span class="n">input_string</span><span class="p">[</span><span class="n">start_index</span> <span class="o">+</span> <span class="mi">1</span> <span class="p">:</span> <span class="n">end_index</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">return</span> <span class="n">input_string</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">if</span> <span class="n">start_index</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">end_index</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span> <span class="k">else</span> <span class="s2">&quot;&quot;</span>
<div class="viewcode-block" id="merge_extra_init_data">
<a class="viewcode-back" href="../../../modules/amqp.adapter.html#amqp.adapter.type_descriptor.merge_extra_init_data">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">merge_extra_init_data</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">json_payload</span><span class="p">,</span> <span class="n">extra_init_data</span><span class="p">:</span> <span class="nb">dict</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Merges extra_init_data into json_payload only when:</span>
<span class="sd"> 1. cls has an attribute named like the key</span>
<span class="sd"> 2. json_payload doesn&#39;t already have the key</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="p">{</span>
<span class="o">**</span><span class="n">json_payload</span><span class="p">,</span>
<span class="o">**</span><span class="p">{</span>
<span class="n">k</span><span class="p">:</span> <span class="n">v</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">extra_init_data</span><span class="o">.</span><span class="n">items</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span><span class="nb">hasattr</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">k</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="nb">hasattr</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="s2">&quot;model_fields&quot;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">k</span> <span class="ow">in</span> <span class="bp">cls</span><span class="o">.</span><span class="n">model_fields</span><span class="p">))</span>
<span class="ow">and</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">json_payload</span>
<span class="p">},</span>
<span class="p">}</span></div>
<span class="k">def</span><span class="w"> </span><span class="nf">_instantiate</span><span class="p">(</span>
<span class="n">type_string</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">payload</span><span class="p">:</span> <span class="n">Any</span><span class="p">,</span> <span class="n">is_json</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">,</span> <span class="n">extra_init_data</span><span class="p">:</span> <span class="nb">dict</span> <span class="o">=</span> <span class="kc">None</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Any</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Dynamically imports and instantiates a class from a TypeDescriptor&#39;s type_string attribute.</span>
<span class="sd"> Args:</span>
<span class="sd"> type_string: TypeDescriptor instance with the class path</span>
<span class="sd"> payload: Argument to pass to the constructor</span>
<span class="sd"> Returns:</span>
<span class="sd"> Instance of the target class initialized with payload</span>
<span class="sd"> Raises:</span>
<span class="sd"> ImportError: If the module/class cannot be imported</span>
<span class="sd"> TypeError: If instantiation fails</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Handle JSON payload first</span>
<span class="n">json_payload</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span> <span class="k">if</span> <span class="n">is_json</span> <span class="ow">and</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">else</span> <span class="kc">None</span>
<span class="c1"># Handle basic types</span>
<span class="k">if</span> <span class="n">type_string</span> <span class="o">==</span> <span class="s2">&quot;str&quot;</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="k">if</span> <span class="n">type_string</span> <span class="o">==</span> <span class="s2">&quot;int&quot;</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="k">if</span> <span class="n">type_string</span> <span class="o">==</span> <span class="s2">&quot;float&quot;</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="k">if</span> <span class="n">type_string</span> <span class="o">==</span> <span class="s2">&quot;bool&quot;</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="c1"># Handle typing module types</span>
<span class="k">if</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;list&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="p">(</span>
<span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.List&quot;</span><span class="p">)</span>
<span class="ow">or</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.Sequence&quot;</span><span class="p">)</span>
<span class="ow">or</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;list&quot;</span><span class="p">)</span>
<span class="p">):</span>
<span class="k">if</span> <span class="n">json_payload</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="n">json_payload</span>
<span class="n">inner_type</span> <span class="o">=</span> <span class="n">_extract_type_of</span><span class="p">(</span><span class="n">type_string</span><span class="p">)</span>
<span class="k">if</span> <span class="n">inner_type</span><span class="p">:</span>
<span class="k">return</span> <span class="p">[</span>
<span class="n">_instantiate</span><span class="p">(</span><span class="n">inner_type</span><span class="p">,</span> <span class="n">p</span><span class="p">,</span> <span class="n">extra_init_data</span><span class="o">=</span><span class="n">extra_init_data</span><span class="p">)</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="p">(</span><span class="n">payload</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="n">payload</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">))</span>
<span class="p">]</span>
<span class="k">return</span> <span class="n">payload</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="k">else</span> <span class="n">payload</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.Dict&quot;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.Mapping&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="n">json_payload</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="n">json_payload</span>
<span class="k">return</span> <span class="nb">dict</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="k">if</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.Set&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="n">json_payload</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">set</span><span class="p">(</span><span class="n">json_payload</span><span class="p">)</span>
<span class="n">_set_type</span> <span class="o">=</span> <span class="n">_extract_type_of</span><span class="p">(</span><span class="n">type_string</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">set</span><span class="p">(</span>
<span class="n">payload</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span>
<span class="k">else</span> <span class="p">[</span><span class="n">_instantiate</span><span class="p">(</span><span class="n">_set_type</span><span class="p">,</span> <span class="n">p</span><span class="p">)</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">payload</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.Tuple&quot;</span><span class="p">):</span>
<span class="k">if</span> <span class="n">json_payload</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">json_payload</span><span class="p">)</span>
<span class="n">_tuple_type</span> <span class="o">=</span> <span class="n">_extract_type_of</span><span class="p">(</span><span class="n">type_string</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">tuple</span><span class="p">(</span>
<span class="n">payload</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span>
<span class="k">else</span> <span class="p">[</span><span class="n">_instantiate</span><span class="p">(</span><span class="n">_tuple_type</span><span class="p">,</span> <span class="n">p</span><span class="p">)</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">payload</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">type_string</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&quot;typing.Optional&quot;</span><span class="p">):</span>
<span class="n">inner_type</span> <span class="o">=</span> <span class="n">_extract_type_of</span><span class="p">(</span><span class="n">type_string</span><span class="p">)</span>
<span class="k">if</span> <span class="n">inner_type</span><span class="p">:</span>
<span class="k">return</span> <span class="n">_instantiate</span><span class="p">(</span><span class="n">inner_type</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">is_json</span><span class="p">,</span> <span class="n">extra_init_data</span><span class="p">)</span>
<span class="k">return</span> <span class="n">payload</span>
<span class="c1"># Handle regular class instantiation</span>
<span class="n">parts</span> <span class="o">=</span> <span class="n">type_string</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="p">)</span>
<span class="n">module_path</span> <span class="o">=</span> <span class="s2">&quot;.&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">parts</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="n">class_name</span> <span class="o">=</span> <span class="n">parts</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Import the module</span>
<span class="n">module</span> <span class="o">=</span> <span class="n">importlib</span><span class="o">.</span><span class="n">import_module</span><span class="p">(</span><span class="n">module_path</span><span class="p">)</span>
<span class="c1"># Get the class</span>
<span class="bp">cls</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">module</span><span class="p">,</span> <span class="n">class_name</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="bp">cls</span><span class="p">):</span>
<span class="k">return</span> <span class="n">payload</span>
<span class="c1"># Instantiate with payload</span>
<span class="k">if</span> <span class="n">json_payload</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">_args</span> <span class="o">=</span> <span class="n">merge_extra_init_data</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">json_payload</span><span class="p">,</span> <span class="n">extra_init_data</span> <span class="ow">or</span> <span class="p">{})</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="s2">&quot;from_dict&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">from_dict</span><span class="p">(</span><span class="n">_args</span><span class="p">)</span>
<span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="s2">&quot;create&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="o">**</span><span class="n">_args</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">cls</span><span class="p">(</span><span class="o">**</span><span class="n">_args</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">payload</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
<span class="n">_args</span> <span class="o">=</span> <span class="n">merge_extra_init_data</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">payload</span><span class="p">,</span> <span class="n">extra_init_data</span> <span class="ow">or</span> <span class="p">{})</span>
<span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="s2">&quot;from_dict&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">from_dict</span><span class="p">(</span><span class="n">_args</span><span class="p">)</span>
<span class="k">elif</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="s2">&quot;create&quot;</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="o">**</span><span class="n">_args</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">cls</span><span class="p">(</span><span class="o">**</span><span class="n">_args</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">cls</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">ImportError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ImportError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Could not import module &#39;</span><span class="si">{</span><span class="n">module_path</span><span class="si">}</span><span class="s2">&#39;&quot;</span><span class="p">)</span> <span class="kn">from</span><span class="w"> </span><span class="nn">e</span>
<span class="k">except</span> <span class="ne">AttributeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ImportError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Module &#39;</span><span class="si">{</span><span class="n">module_path</span><span class="si">}</span><span class="s2">&#39; has no class &#39;</span><span class="si">{</span><span class="n">class_name</span><span class="si">}</span><span class="s2">&#39;&quot;</span><span class="p">)</span> <span class="kn">from</span><span class="w"> </span><span class="nn">e</span>
<span class="k">except</span> <span class="ne">TypeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to instantiate </span><span class="si">{</span><span class="n">type_string</span><span class="si">}</span><span class="s2"> with payload </span><span class="si">{</span><span class="n">payload</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span> <span class="kn">from</span><span class="w"> </span><span class="nn">e</span>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/config/amq_configuration.html
-383
View File
@@ -1,383 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.config.amq_configuration &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.config.amq_configuration</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.config.amq_configuration</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">configparser</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">inspect</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">os</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_error</span><span class="p">,</span> <span class="n">logging_warning</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd">Convert configuration from properties file to an object.</span>
<span class="sd">&quot;&quot;&quot;</span>
<div class="viewcode-block" id="AMQAdapter">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.AMQAdapter">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">AMQAdapter</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> configuration with prefix &#39;cm.amq-adapter&#39;</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">PREFIX</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;cm.amq-adapter.&quot;</span>
<div class="viewcode-block" id="AMQAdapter.__init__">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.AMQAdapter.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">config</span><span class="p">:</span> <span class="n">configparser</span><span class="o">.</span><span class="n">ConfigParser</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> cm.amq-adapter.service-name=amq-adapter</span>
<span class="sd"> cm.amq-adapter.generator-id=164</span>
<span class="sd"> cm.amq-adapter.route-mapping=</span>
<span class="sd"> :param config: config parser to read configuration values</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="bp">self</span><span class="o">.</span><span class="n">generator_id</span> <span class="o">=</span> <span class="mi">0</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service_name</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;service-name&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;amq-python-adapter-&quot;</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">generator_id</span><span class="p">),</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_mapping</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;route-mapping&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="kc">None</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">require_authenticated_user</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;require-authenticated-user&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">backpressure_enabled</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;default-backpressure-enabled&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">callbacks_enabled</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;callbacks-enabled&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># these 2 shall be provided by Docker swarm, format is a random alphanumeric string</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_service_id</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;SWARM_SERVICE_ID&quot;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">&quot;clever-amqp-service&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">swarm_task_id</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;SWARM_TASK_ID&quot;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">&quot;python-adapter&quot;</span><span class="p">)</span>
<span class="c1">#</span>
<span class="c1"># Consul related configuration</span>
<span class="bp">self</span><span class="o">.</span><span class="n">consul_port</span> <span class="o">=</span> <span class="p">(</span>
<span class="nb">int</span><span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span>
<span class="s2">&quot;CONSUL_PORT&quot;</span><span class="p">,</span>
<span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-port&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;8500&quot;</span><span class="p">),</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;CONSUL_PORT&quot;</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="k">else</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-port&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">8500</span><span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">consul_host</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span>
<span class="s2">&quot;CONSUL_HOST&quot;</span><span class="p">,</span>
<span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-host&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;consul&quot;</span><span class="p">),</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">consul_max_retries</span> <span class="o">=</span> <span class="p">(</span>
<span class="nb">int</span><span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span>
<span class="s2">&quot;CONSUL_MAX_RETRIES&quot;</span><span class="p">,</span>
<span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-max-retries&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;5&quot;</span><span class="p">),</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;CONSUL_MAX_RETRIES&quot;</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="k">else</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-max-retries&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">5</span><span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">consul_initial_retry_delay</span> <span class="o">=</span> <span class="p">(</span>
<span class="nb">float</span><span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span>
<span class="s2">&quot;CONSUL_INITIAL_RETRY_DELAY&quot;</span><span class="p">,</span>
<span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-initial-retry-delay&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;0.2&quot;</span><span class="p">,</span>
<span class="p">),</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;CONSUL_INITIAL_RETRY_DELAY&quot;</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="k">else</span> <span class="n">config</span><span class="o">.</span><span class="n">getfloat</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-initial-retry-delay&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mf">0.2</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">consul_counter_key</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span>
<span class="s2">&quot;CONSUL_COUNTER_KEY&quot;</span><span class="p">,</span>
<span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-counter-key&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;service/ids/counter&quot;</span><span class="p">,</span>
<span class="p">),</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;CONSUL_COUNTER_KEY&quot;</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span>
<span class="k">else</span> <span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;consul-counter-key&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;service/ids/counter&quot;</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="AMQAdapter.adapter_prefix">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.AMQAdapter.adapter_prefix">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">adapter_prefix</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">service_name</span><span class="si">}</span><span class="s2">-</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">generator_id</span><span class="si">}</span><span class="s2">-&quot;</span></div>
</div>
<div class="viewcode-block" id="Dispatch">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.Dispatch">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">Dispatch</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> configuration with prefix &#39;cm.dispatch&#39;</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">PREFIX</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;cm.dispatch.&quot;</span>
<div class="viewcode-block" id="Dispatch.__init__">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.Dispatch.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">config</span><span class="p">:</span> <span class="n">configparser</span><span class="o">.</span><span class="n">ConfigParser</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> cm.dispatch.use-dlq=true</span>
<span class="sd"> cm.dispatch.amq-host=rabbitmq</span>
<span class="sd"> cm.dispatch.amq-port=5672</span>
<span class="sd"> cm.dispatch.amq-port-tls=5671</span>
<span class="sd"> cm.dispatch.service-host=</span>
<span class="sd"> cm.dispatch.service-port=8080</span>
<span class="sd"> :param config: config parser to read configuration values</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="bp">self</span><span class="o">.</span><span class="n">use_dlq</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;use-dlq&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_host</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;amq-host&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;rabbitmq&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_port</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;amq-port&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">5672</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_port_tls</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;amq-port-tls&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">5671</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service_host</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;service-host&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;localhost&quot;</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service_port</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;service-port&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">8080</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rabbit_mq_user</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;RABBIT_MQ_USER&quot;</span><span class="p">,</span> <span class="s2">&quot;springuser&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rabbit_mq_password</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;RABBIT_MQ_PASSWORD&quot;</span><span class="p">,</span> <span class="s2">&quot;TheCleverWho&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rabbit_mq_url</span> <span class="o">=</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;amqp://</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">rabbit_mq_user</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">rabbit_mq_password</span><span class="si">}</span><span class="s2">@</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_host</span><span class="si">}</span><span class="s2">/&quot;</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">download_dir</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;download-dir&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="s2">&quot;downloaded_files&quot;</span>
<span class="p">)</span></div>
</div>
<div class="viewcode-block" id="Backpressure">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.Backpressure">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">Backpressure</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> configuration with prefix &#39;cm.backpressure&#39;</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">PREFIX</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;cm.backpressure.&quot;</span>
<div class="viewcode-block" id="Backpressure.__init__">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.Backpressure.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">config</span><span class="p">:</span> <span class="n">configparser</span><span class="o">.</span><span class="n">ConfigParser</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> cm.backpressure.threshold=20</span>
<span class="sd"> cm.backpressure.time-window=3000</span>
<span class="sd"> :param config: config parser to read configuration values</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="bp">self</span><span class="o">.</span><span class="n">threshold_load</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;threshold&quot;</span><span class="p">,</span>
<span class="n">fallback</span><span class="o">=</span><span class="nb">int</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&quot;PARALLEL_WORKERS&quot;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)),</span>
<span class="p">)</span>
<span class="c1"># time-window in seconds, duration between backpressure reports</span>
<span class="bp">self</span><span class="o">.</span><span class="n">time_window</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;time-window&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">10</span>
<span class="p">)</span>
<span class="c1"># Define maximum CPU usage before triggering backpressure OVERLOAD alert</span>
<span class="bp">self</span><span class="o">.</span><span class="n">threshold_cpu</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;threshold-cpu&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">90</span>
<span class="p">)</span>
<span class="c1"># Define the time window length, in seconds, during which, if CPU usage is consistently above the threshold, the system will trigger OVERLOAD alert</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cpu_overload_duration</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;cpu-overload-duration&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">30</span>
<span class="p">)</span>
<span class="c1"># Define the time window length, in seconds, during which, if no request is made to the Service, the system will trigger IDLE alert</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idle_duration</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;idle-duration&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="mi">30</span>
<span class="p">)</span>
<span class="c1"># Flag to enable/disable CPU monitoring</span>
<span class="bp">self</span><span class="o">.</span><span class="n">cpu_monitoring_enabled</span> <span class="o">=</span> <span class="n">config</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span>
<span class="s2">&quot;CleverMicro-AMQ&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">PREFIX</span> <span class="o">+</span> <span class="s2">&quot;cpu-monitoring-enabled&quot;</span><span class="p">,</span> <span class="n">fallback</span><span class="o">=</span><span class="kc">False</span>
<span class="p">)</span></div>
</div>
<div class="viewcode-block" id="AMQConfiguration">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.AMQConfiguration">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">AMQConfiguration</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Top level configuration context holder / object</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="AMQConfiguration.__init__">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.AMQConfiguration.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">config_file_name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="kc">None</span><span class="p">):</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">configparser</span><span class="o">.</span><span class="n">ConfigParser</span><span class="p">()</span>
<span class="c1"># Read the application.properties file</span>
<span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">):</span>
<span class="n">config</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># Get the directory of the caller (the module where this function is called)</span>
<span class="n">caller_frame</span> <span class="o">=</span> <span class="n">inspect</span><span class="o">.</span><span class="n">stack</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span> <span class="c1"># 0 is current_availability frame, 1 is caller</span>
<span class="n">caller_module</span> <span class="o">=</span> <span class="n">inspect</span><span class="o">.</span><span class="n">getmodule</span><span class="p">(</span><span class="n">caller_frame</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">if</span> <span class="n">caller_module</span><span class="p">:</span>
<span class="n">caller_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">caller_module</span><span class="o">.</span><span class="vm">__file__</span><span class="p">))</span>
<span class="n">config_file_path</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">caller_dir</span><span class="p">,</span> <span class="n">config_file_name</span><span class="p">)</span>
<span class="n">config</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">config_file_path</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Configuration file not found: </span><span class="si">{</span><span class="n">config_file_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">logging_warning</span><span class="p">(</span>
<span class="s2">&quot;Defaulting to preset values, which will likely be wrong, causing errors.&quot;</span>
<span class="p">)</span>
<span class="c1"># Override values with environment variables</span>
<span class="k">for</span> <span class="n">section</span> <span class="ow">in</span> <span class="n">config</span><span class="o">.</span><span class="n">sections</span><span class="p">():</span>
<span class="k">for</span> <span class="n">option</span> <span class="ow">in</span> <span class="n">config</span><span class="o">.</span><span class="n">options</span><span class="p">(</span><span class="n">section</span><span class="p">):</span>
<span class="c1"># env_var = f&quot;{section.upper()}_{option.upper()}&quot;</span>
<span class="k">if</span> <span class="n">option</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">:</span>
<span class="n">config</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">section</span><span class="p">,</span> <span class="n">option</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">[</span><span class="n">option</span><span class="p">])</span>
<span class="c1">#</span>
<span class="c1"># Add individual config areas</span>
<span class="c1">#</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_adapter</span><span class="p">:</span> <span class="n">AMQAdapter</span> <span class="o">=</span> <span class="n">AMQAdapter</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">dispatch</span><span class="p">:</span> <span class="n">Dispatch</span> <span class="o">=</span> <span class="n">Dispatch</span><span class="p">(</span><span class="n">config</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">backpressure</span><span class="p">:</span> <span class="n">Backpressure</span> <span class="o">=</span> <span class="n">Backpressure</span><span class="p">(</span><span class="n">config</span><span class="p">)</span></div>
<div class="viewcode-block" id="AMQConfiguration.get_unique_instance_id">
<a class="viewcode-back" href="../../../modules/amqp.config.html#amqp.config.amq_configuration.AMQConfiguration.get_unique_instance_id">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_unique_instance_id</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Get a unique instance ID for this AMQ service.</span>
<span class="sd"> Returns:</span>
<span class="sd"> int: A unique instance ID</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">instance_id</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
<span class="n">logging_warning</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Generated unique instance ID: </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">instance_id</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">instance_id</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/model/model.html
-589
View File
@@ -1,589 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.model.model &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.model.model</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.model.model</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">base64</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">dataclasses</span><span class="w"> </span><span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">enum</span><span class="w"> </span><span class="kn">import</span> <span class="n">Enum</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">List</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika.abc</span><span class="w"> </span><span class="kn">import</span> <span class="n">AbstractRobustConnection</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.service_message_type</span><span class="w"> </span><span class="kn">import</span> <span class="n">ServiceMessageType</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.snowflake_id</span><span class="w"> </span><span class="kn">import</span> <span class="n">SnowflakeId</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">sanitize_as_rabbitmq_name</span>
<span class="n">RS</span> <span class="o">=</span> <span class="s2">&quot;:&quot;</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd">Define the structure of messages used in Clever Micro</span>
<span class="sd">&quot;&quot;&quot;</span>
<div class="viewcode-block" id="DataMessageMagicByte">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataMessageMagicByte">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">DataMessageMagicByte</span><span class="p">(</span><span class="n">Enum</span><span class="p">):</span>
<span class="n">HTTP_REQUEST</span> <span class="o">=</span> <span class="s2">&quot;A&quot;</span>
<span class="n">RPC_REQUEST</span> <span class="o">=</span> <span class="s2">&quot;R&quot;</span></div>
<div class="viewcode-block" id="CleverMicroMessage">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage">[docs]</a>
<span class="nd">@dataclass</span>
<span class="k">class</span><span class="w"> </span><span class="nc">CleverMicroMessage</span><span class="p">:</span>
<span class="n">DEFAULT_CONTENT_TYPE</span> <span class="o">=</span> <span class="p">[</span><span class="s2">&quot;application/octet-stream&quot;</span><span class="p">]</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> The common structure of a message that transfers the user request to the Service and back. All data related messages</span>
<span class="sd"> conform to this base structure:</span>
<span class="sd"> magic - identifies the message type (REST, RPC, DataResponse)</span>
<span class="sd"> id - unique message ID</span>
<span class="sd"> m_field - method name (for REST style calls) or service name (for RPC style calls) or response code for DataResponse</span>
<span class="sd"> d_field - domain name (for REST style calls) or module name (for RPC style calls) or error code for DataResponse</span>
<span class="sd"> p_field - path (for REST style calls) or method name (for RPC style calls) or error cause for DataResponse</span>
<span class="sd"> headers - headers of the message (e.g. Content-Type, Accept, etc.)</span>
<span class="sd"> trace_info - trace information (e.g. trace ID, span ID, etc. for Jaeger trace monitor)</span>
<span class="sd"> base64body - the body of the message encoded in base64</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="CleverMicroMessage.__init__">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">magic</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="nb">id</span><span class="p">:</span> <span class="n">SnowflakeId</span><span class="p">,</span>
<span class="n">m_field</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">d_field</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">p_field</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">headers</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]],</span>
<span class="n">trace_info</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span>
<span class="n">base64body</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_magic</span> <span class="o">=</span> <span class="n">magic</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_id</span><span class="p">:</span> <span class="n">SnowflakeId</span> <span class="o">=</span> <span class="nb">id</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_m_field</span> <span class="o">=</span> <span class="n">m_field</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_d_field</span> <span class="o">=</span> <span class="n">d_field</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_p_field</span> <span class="o">=</span> <span class="n">p_field</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_headers</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]</span> <span class="o">=</span> <span class="n">headers</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_trace_info</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="n">trace_info</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_base64body</span><span class="p">:</span> <span class="nb">bytes</span> <span class="o">=</span> <span class="n">base64body</span></div>
<div class="viewcode-block" id="CleverMicroMessage.magic">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.magic">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">magic</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_magic</span></div>
<div class="viewcode-block" id="CleverMicroMessage.id">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.id">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">id</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_id</span></div>
<div class="viewcode-block" id="CleverMicroMessage.m_field">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.m_field">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">m_field</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_m_field</span></div>
<div class="viewcode-block" id="CleverMicroMessage.d_field">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.d_field">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">d_field</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_d_field</span></div>
<div class="viewcode-block" id="CleverMicroMessage.p_field">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.p_field">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">p_field</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_p_field</span></div>
<div class="viewcode-block" id="CleverMicroMessage.headers">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.headers">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">headers</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_headers</span></div>
<div class="viewcode-block" id="CleverMicroMessage.trace_info">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.trace_info">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">trace_info</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_trace_info</span></div>
<div class="viewcode-block" id="CleverMicroMessage.base64body">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.base64body">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">base64body</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_base64body</span></div>
<div class="viewcode-block" id="CleverMicroMessage.body">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.body">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">body</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="k">return</span> <span class="n">base64</span><span class="o">.</span><span class="n">b64decode</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">base64body</span><span class="p">())</span></div>
<div class="viewcode-block" id="CleverMicroMessage.content_type">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.content_type">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">content_type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;Content-Type&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">DEFAULT_CONTENT_TYPE</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span></div>
<div class="viewcode-block" id="CleverMicroMessage.return_address">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.CleverMicroMessage.return_address">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">return_address</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">()</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;Return-Address&quot;</span><span class="p">,</span> <span class="s2">&quot;&quot;</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span></div>
</div>
<div class="viewcode-block" id="DataMessage">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataMessage">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">DataMessage</span><span class="p">(</span><span class="n">CleverMicroMessage</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> The class represents a structure of a message that transfers the user request to the Service.</span>
<span class="sd"> It may also be used to make a call/request between services.</span>
<span class="sd"> The response to this call is in format of DataResponse class.</span>
<span class="sd"> See DataMessageFactory for code to instantiate, serialize and deserialize the class</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="DataMessage.__init__">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataMessage.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">magic</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="nb">id</span><span class="p">:</span> <span class="n">SnowflakeId</span><span class="p">,</span>
<span class="n">method</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">headers</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]],</span>
<span class="n">trace_info</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span>
<span class="n">base64body</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span>
<span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">magic</span><span class="p">,</span> <span class="nb">id</span><span class="p">,</span> <span class="n">method</span><span class="p">,</span> <span class="n">domain</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">trace_info</span><span class="p">,</span> <span class="n">base64body</span><span class="p">)</span></div>
<div class="viewcode-block" id="DataMessage.method">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataMessage.method">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">method</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">m_field</span><span class="p">()</span></div>
<div class="viewcode-block" id="DataMessage.domain">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataMessage.domain">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">domain</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_field</span><span class="p">()</span></div>
<div class="viewcode-block" id="DataMessage.path">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataMessage.path">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">path</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">p_field</span><span class="p">()</span></div>
<div class="viewcode-block" id="DataMessage.routing_key">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataMessage.routing_key">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">routing_key</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="n">sanitize_as_rabbitmq_name</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">d_field</span><span class="p">()</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">p_field</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
</div>
<div class="viewcode-block" id="MultipartDataMessage">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.MultipartDataMessage">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">MultipartDataMessage</span><span class="p">(</span><span class="n">DataMessage</span><span class="p">):</span>
<div class="viewcode-block" id="MultipartDataMessage.__init__">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.MultipartDataMessage.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">,</span> <span class="n">extra_data</span><span class="p">:</span> <span class="nb">dict</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">message</span><span class="o">.</span><span class="n">magic</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">method</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">domain</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">headers</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">base64body</span><span class="p">(),</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">extra_data</span> <span class="o">=</span> <span class="n">extra_data</span></div>
</div>
<div class="viewcode-block" id="AMQMessage">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.AMQMessage">[docs]</a>
<span class="nd">@dataclass</span>
<span class="k">class</span><span class="w"> </span><span class="nc">AMQMessage</span><span class="p">(</span><span class="n">DataMessage</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Expand the core DataMessage type to include the channel, which is required to create a dedicated queue</span>
<span class="sd"> in case the response should initiate file download.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">_connection</span><span class="p">:</span> <span class="n">AbstractRobustConnection</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="nb">repr</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="c1"># Don&#39;t show in repr</span>
<div class="viewcode-block" id="AMQMessage.__init__">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.AMQMessage.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data_message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">,</span> <span class="n">connection</span><span class="p">:</span> <span class="n">AbstractRobustConnection</span><span class="p">):</span>
<span class="c1"># Initialize the parent DataMessage with all properties from the input data_message</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">magic</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">magic</span><span class="p">(),</span>
<span class="nb">id</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">id</span><span class="p">(),</span>
<span class="n">method</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">method</span><span class="p">(),</span>
<span class="n">domain</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">domain</span><span class="p">(),</span>
<span class="n">path</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">path</span><span class="p">(),</span>
<span class="n">headers</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">headers</span><span class="p">(),</span>
<span class="n">trace_info</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">trace_info</span><span class="p">(),</span>
<span class="n">base64body</span><span class="o">=</span><span class="n">data_message</span><span class="o">.</span><span class="n">base64body</span><span class="p">(),</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_connection</span> <span class="o">=</span> <span class="n">connection</span>
<span class="bp">self</span><span class="o">.</span><span class="n">extra_data</span> <span class="o">=</span> <span class="n">data_message</span><span class="o">.</span><span class="n">extra_data</span> <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">data_message</span><span class="p">,</span> <span class="s2">&quot;extra_data&quot;</span><span class="p">)</span> <span class="k">else</span> <span class="p">{}</span></div>
<span class="nd">@property</span>
<span class="k">def</span><span class="w"> </span><span class="nf">connection</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">AbstractRobustConnection</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Get the AMQP connection associated with this message&quot;&quot;&quot;</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_connection</span></div>
<div class="viewcode-block" id="DataResponse">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataResponse">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">DataResponse</span><span class="p">(</span><span class="n">CleverMicroMessage</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> The response message to REST &amp; RPC style calls (note: the request is made with DataMessage)</span>
<span class="sd"> See DataResponseFactory for code to instantiate, serialize and deserialize the class</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="DataResponse.__init__">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataResponse.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">magic</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="nb">id</span><span class="p">:</span> <span class="n">SnowflakeId</span><span class="p">,</span>
<span class="n">response_code</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">content_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">error</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">error_cause</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">headers</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">List</span><span class="p">[</span><span class="nb">str</span><span class="p">]],</span>
<span class="n">trace_info</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">],</span>
<span class="n">base64body</span><span class="p">:</span> <span class="nb">bytes</span><span class="p">,</span>
<span class="p">):</span>
<span class="k">if</span> <span class="n">headers</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">headers</span><span class="p">[</span><span class="s2">&quot;Content-Type&quot;</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span><span class="n">content_type</span><span class="p">]</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span>
<span class="n">magic</span><span class="p">,</span>
<span class="nb">id</span><span class="p">,</span>
<span class="n">response_code</span><span class="p">,</span>
<span class="n">error</span><span class="p">,</span>
<span class="n">error_cause</span><span class="p">,</span>
<span class="n">headers</span><span class="p">,</span>
<span class="n">trace_info</span><span class="p">,</span>
<span class="n">base64body</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="DataResponse.response_code">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataResponse.response_code">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">response_code</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">m_field</span><span class="p">()</span></div>
<div class="viewcode-block" id="DataResponse.error">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataResponse.error">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">error</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">d_field</span><span class="p">()</span></div>
<div class="viewcode-block" id="DataResponse.error_cause">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.DataResponse.error_cause">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">error_cause</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">p_field</span><span class="p">()</span></div>
</div>
<div class="viewcode-block" id="AMQRoute">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.AMQRoute">[docs]</a>
<span class="nd">@dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">AMQRoute</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Configuration item representing one AMQRoute. Adapter uses this to declare the Exchange</span>
<span class="sd"> and/or queue and bind the key as routing expression.</span>
<span class="sd"> :param component_name: a service/component name that this route points to (owner of the route)</span>
<span class="sd"> :param exchange: RabbitMQ exchange name</span>
<span class="sd"> :param queue: RabbitMQ queue name</span>
<span class="sd"> :param key: RabbitMQ TOPIC Exchange routing key</span>
<span class="sd"> :param timeout: Timeout in milliseconds, how long to wait for reply, or not waiting if less then 1</span>
<span class="sd"> :param valid_until: optional timestamp (millis from epoch) after which the route is ignored</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">component_name</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">exchange</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">queue</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">key</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">timeout</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">valid_until</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">FORMAT</span> <span class="o">=</span> <span class="s2">&quot;Component-Name:Exchange-Name:Queue-Name:Routing-Key:Timeout:Valid-Until&quot;</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">component_name</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">exchange</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">key</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">valid_until</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">other</span><span class="p">,</span> <span class="n">AMQRoute</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">component_name</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">component_name</span>
<span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">queue</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">queue</span>
<span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">key</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">key</span>
<span class="p">)</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="nd">@property</span>
<span class="k">def</span><span class="w"> </span><span class="nf">primary_key</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Build the unique identifier of the route</span>
<span class="sd"> :return primary key</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">component_name</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="si">}{</span><span class="n">RS</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">key</span><span class="si">}</span><span class="s2">&quot;</span>
<div class="viewcode-block" id="AMQRoute.as_string">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.AMQRoute.as_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">as_string</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span></div>
</div>
<span class="c1"># Ensure backward compatibility</span>
<span class="n">AMQResponse</span> <span class="o">=</span> <span class="n">DataResponse</span>
<div class="viewcode-block" id="ServiceMessage">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.ServiceMessage">[docs]</a>
<span class="nd">@dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">ServiceMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> The CleverMicro service message that is used to communicate the internal state of the AMQ channel.</span>
<span class="sd"> In v0.2 it supports service discovery / route registration and Backpressure status notification.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="nb">id</span><span class="p">:</span> <span class="n">SnowflakeId</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">payload_type</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">message_type</span><span class="p">:</span> <span class="n">ServiceMessageType</span> <span class="o">=</span> <span class="n">ServiceMessageType</span><span class="o">.</span><span class="n">UNKNOWN</span>
<span class="n">recipient_name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">reply_to</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">trace_info</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default_factory</span><span class="o">=</span><span class="nb">dict</span><span class="p">)</span>
<span class="n">message</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="kc">None</span>
<div class="viewcode-block" id="ServiceMessage.is_valid">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.ServiceMessage.is_valid">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">is_valid</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">message_type</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span></div>
</div>
<div class="viewcode-block" id="ScalingRequestAlert">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.ScalingRequestAlert">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">ScalingRequestAlert</span><span class="p">(</span><span class="n">Enum</span><span class="p">):</span>
<span class="n">OVERLOAD</span> <span class="o">=</span> <span class="s2">&quot;OVERLOAD&quot;</span>
<span class="n">IDLE</span> <span class="o">=</span> <span class="s2">&quot;IDLE&quot;</span>
<span class="n">UPDATE</span> <span class="o">=</span> <span class="s2">&quot;UPDATE&quot;</span>
<span class="n">SCALE_UP</span> <span class="o">=</span> <span class="s2">&quot;SCALE_UP&quot;</span>
<span class="n">SCALE_DOWN</span> <span class="o">=</span> <span class="s2">&quot;SCALE_DOWN&quot;</span></div>
<div class="viewcode-block" id="ScalingRequest">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.ScalingRequest">[docs]</a>
<span class="nd">@dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="k">class</span><span class="w"> </span><span class="nc">ScalingRequest</span><span class="p">:</span>
<span class="n">service_id</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">task_id</span><span class="p">:</span> <span class="nb">str</span>
<span class="n">max_availability</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">current_availability</span><span class="p">:</span> <span class="nb">int</span>
<span class="n">request_type</span><span class="p">:</span> <span class="n">ScalingRequestAlert</span>
<span class="n">version</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">1</span>
<div class="viewcode-block" id="ScalingRequest.to_json">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.ScalingRequest.to_json">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">to_json</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Converts the object to a JSON string matching the Java structure&quot;&quot;&quot;</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span>
<span class="p">{</span>
<span class="s2">&quot;version&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">version</span><span class="p">,</span>
<span class="s2">&quot;serviceId&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">service_id</span><span class="p">,</span>
<span class="s2">&quot;taskId&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">,</span>
<span class="s2">&quot;maxAvailability&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">max_availability</span><span class="p">,</span>
<span class="s2">&quot;currentAvailability&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_availability</span><span class="p">,</span>
<span class="s2">&quot;requestType&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">request_type</span><span class="o">.</span><span class="n">value</span><span class="p">,</span> <span class="c1"># Using .value for Enum serialization</span>
<span class="p">},</span>
<span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="ScalingRequest.from_json">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.ScalingRequest.from_json">[docs]</a>
<span class="nd">@classmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">from_json</span><span class="p">(</span><span class="bp">cls</span><span class="p">,</span> <span class="n">json_str</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="s2">&quot;ScalingRequest&quot;</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Creates a ScalingRequest from a JSON string&quot;&quot;&quot;</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">json_str</span><span class="p">)</span>
<span class="k">return</span> <span class="bp">cls</span><span class="p">(</span>
<span class="n">service_id</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;serviceId&quot;</span><span class="p">],</span>
<span class="n">task_id</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;taskId&quot;</span><span class="p">],</span>
<span class="n">max_availability</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;maxAvailability&quot;</span><span class="p">],</span>
<span class="n">current_availability</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;currentAvailability&quot;</span><span class="p">],</span>
<span class="n">request_type</span><span class="o">=</span><span class="n">ScalingRequestAlert</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s2">&quot;requestType&quot;</span><span class="p">]),</span> <span class="c1"># Convert string to Enum</span>
<span class="p">)</span></div>
</div>
<div class="viewcode-block" id="AMQErrorMessage">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.AMQErrorMessage">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">AMQErrorMessage</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> The error response message to RPC style call (note: the request is made with DataMessage)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="AMQErrorMessage.__init__">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.AMQErrorMessage.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span>
<span class="n">response_code</span><span class="p">:</span> <span class="nb">int</span><span class="p">,</span>
<span class="n">error</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">detail</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span>
<span class="n">content_type</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&quot;application/json&quot;</span><span class="p">,</span>
<span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">response_code</span> <span class="o">=</span> <span class="n">response_code</span>
<span class="bp">self</span><span class="o">.</span><span class="n">error</span> <span class="o">=</span> <span class="n">error</span>
<span class="bp">self</span><span class="o">.</span><span class="n">detail</span> <span class="o">=</span> <span class="n">detail</span>
<span class="bp">self</span><span class="o">.</span><span class="n">content_type</span> <span class="o">=</span> <span class="n">content_type</span></div>
<div class="viewcode-block" id="AMQErrorMessage.serialize">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.AMQErrorMessage.serialize">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">serialize</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bytes</span><span class="p">:</span>
<span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span>
<span class="p">{</span>
<span class="s2">&quot;response_code&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_code</span><span class="p">,</span>
<span class="s2">&quot;error&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">error</span><span class="p">,</span>
<span class="s2">&quot;detail&quot;</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">detail</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s2">&quot;utf-8&quot;</span><span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/model/service_message_type.html
-120
View File
@@ -1,120 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.model.service_message_type &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.model.service_message_type</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.model.service_message_type</h1><div class="highlight"><pre>
<span></span><span class="kn">from</span><span class="w"> </span><span class="nn">enum</span><span class="w"> </span><span class="kn">import</span> <span class="n">Enum</span><span class="p">,</span> <span class="n">auto</span>
<div class="viewcode-block" id="ServiceMessageType">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.model.ServiceMessageType">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">ServiceMessageType</span><span class="p">(</span><span class="n">Enum</span><span class="p">):</span>
<span class="n">ROUTING_DATA_REQUEST</span> <span class="o">=</span> <span class="n">auto</span><span class="p">()</span>
<span class="n">ROUTING_DATA</span> <span class="o">=</span> <span class="n">auto</span><span class="p">()</span>
<span class="n">ROUTING_DATA_REMOVE</span> <span class="o">=</span> <span class="n">auto</span><span class="p">()</span>
<span class="n">ROUTING_DATA_REPLACE</span> <span class="o">=</span> <span class="n">auto</span><span class="p">()</span>
<span class="n">BACKPRESSURE</span> <span class="o">=</span> <span class="n">auto</span><span class="p">()</span>
<span class="n">UNKNOWN</span> <span class="o">=</span> <span class="n">auto</span><span class="p">()</span></div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/model/snowflake_id.html
-204
View File
@@ -1,204 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.model.snowflake_id &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.model.snowflake_id</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.model.snowflake_id</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">struct</span>
<div class="viewcode-block" id="SnowflakeId">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.snowflake_id.SnowflakeId">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">SnowflakeId</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> see https://en.wikipedia.org/wiki/Snowflake_ID</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">SNOWFLAKE_ID_WIDTH</span> <span class="o">=</span> <span class="p">(</span>
<span class="mi">10</span> <span class="c1"># with in bytes (8 bits each), can be more than long int width (8 bytes)</span>
<span class="p">)</span>
<div class="viewcode-block" id="SnowflakeId.__init__">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.snowflake_id.SnowflakeId.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hi_bits_or_bytes</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">lo_bits</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bits</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span>
<span class="k">if</span> <span class="n">lo_bits</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">if</span> <span class="n">hi_bits_or_bytes</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">shift</span> <span class="o">=</span> <span class="mi">8</span> <span class="c1"># starting from Hi value, which is only 2 bytes wide</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">hi_bits_or_bytes</span><span class="p">)</span> <span class="ow">and</span> <span class="n">i</span> <span class="o">&lt;=</span> <span class="bp">self</span><span class="o">.</span><span class="n">SNOWFLAKE_ID_WIDTH</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">|=</span> <span class="n">hi_bits_or_bytes</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">&lt;&lt;</span> <span class="n">shift</span>
<span class="k">if</span> <span class="n">shift</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">n</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="n">shift</span> <span class="o">=</span> <span class="mi">56</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">shift</span> <span class="o">-=</span> <span class="mi">8</span>
<span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">else</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">bits</span> <span class="o">=</span> <span class="p">[</span><span class="n">lo_bits</span><span class="p">,</span> <span class="n">hi_bits_or_bytes</span><span class="p">]</span></div>
<div class="viewcode-block" id="SnowflakeId.from_hex">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.snowflake_id.SnowflakeId.from_hex">[docs]</a>
<span class="nd">@staticmethod</span>
<span class="k">def</span><span class="w"> </span><span class="nf">from_hex</span><span class="p">(</span><span class="n">_input</span><span class="p">):</span>
<span class="n">expected_length</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">SnowflakeId</span><span class="o">.</span><span class="n">SNOWFLAKE_ID_WIDTH</span>
<span class="k">if</span> <span class="n">_input</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">or</span> <span class="nb">len</span><span class="p">(</span><span class="n">_input</span><span class="p">)</span> <span class="o">&lt;</span> <span class="n">expected_length</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;MessageId is </span><span class="si">{</span><span class="n">SnowflakeId</span><span class="o">.</span><span class="n">SNOWFLAKE_ID_WIDTH</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">8</span><span class="si">}</span><span class="s2"> bits wide, it is represented by HEX string &quot;</span>
<span class="sa">f</span><span class="s2">&quot;of length: </span><span class="si">{</span><span class="n">expected_length</span><span class="si">}</span><span class="s2">, real input: [</span><span class="si">{</span><span class="n">_input</span><span class="si">}</span><span class="s2">] of length: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">_input</span><span class="p">)</span><span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="n">_input</span><span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="mi">0</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="n">bits</span> <span class="o">=</span> <span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">]</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">expected_length</span><span class="p">,</span> <span class="mi">2</span><span class="p">):</span>
<span class="k">if</span> <span class="n">i</span> <span class="o">==</span> <span class="n">expected_length</span> <span class="o">-</span> <span class="p">(</span><span class="mi">2</span> <span class="o">*</span> <span class="mi">8</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">bits</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">=</span> <span class="p">(</span><span class="n">bits</span><span class="p">[</span><span class="n">n</span><span class="p">]</span> <span class="o">&lt;&lt;</span> <span class="mi">8</span><span class="p">)</span> <span class="o">|</span> <span class="nb">int</span><span class="p">(</span><span class="n">_input</span><span class="p">[</span><span class="n">i</span> <span class="p">:</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">2</span><span class="p">],</span> <span class="mi">16</span><span class="p">)</span>
<span class="k">return</span> <span class="n">SnowflakeId</span><span class="p">(</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span></div>
<span class="k">def</span><span class="w"> </span><span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">_hex</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="si">:</span><span class="s2">016X</span><span class="si">}{</span><span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="si">:</span><span class="s2">016X</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="n">_val</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">_hex</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">_hex</span><span class="p">)</span> <span class="o">&lt;=</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">SnowflakeId</span><span class="o">.</span><span class="n">SNOWFLAKE_ID_WIDTH</span>
<span class="k">else</span> <span class="n">_hex</span><span class="p">[(</span><span class="o">-</span><span class="mi">2</span> <span class="o">*</span> <span class="n">SnowflakeId</span><span class="o">.</span><span class="n">SNOWFLAKE_ID_WIDTH</span><span class="p">)</span> <span class="p">:]</span>
<span class="p">)</span>
<span class="k">return</span> <span class="n">_val</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__eq__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">other</span><span class="p">,</span> <span class="n">SnowflakeId</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="n">other</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">def</span><span class="w"> </span><span class="fm">__lt__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">other</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&lt;</span> <span class="n">other</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">other</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">]:</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">&lt;</span> <span class="n">other</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<div class="viewcode-block" id="SnowflakeId.get_bytes">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.snowflake_id.SnowflakeId.get_bytes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_bytes</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">high</span> <span class="o">=</span> <span class="n">struct</span><span class="o">.</span><span class="n">pack</span><span class="p">(</span><span class="s2">&quot;!Q&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">low</span> <span class="o">=</span> <span class="n">struct</span><span class="o">.</span><span class="n">pack</span><span class="p">(</span><span class="s2">&quot;!Q&quot;</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="k">return</span> <span class="n">high</span><span class="p">[</span><span class="o">-</span><span class="mi">2</span><span class="p">:]</span> <span class="o">+</span> <span class="n">low</span></div>
<div class="viewcode-block" id="SnowflakeId.get_bits">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.snowflake_id.SnowflakeId.get_bits">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_bits</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">bits</span></div>
<div class="viewcode-block" id="SnowflakeId.as_string">
<a class="viewcode-back" href="../../../modules/amqp.model.html#amqp.model.snowflake_id.SnowflakeId.as_string">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">as_string</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/rabbitmq/rabbit_mq_client.html
-374
View File
@@ -1,374 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.rabbitmq.rabbit_mq_client &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.rabbitmq.rabbit_mq_client</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.rabbitmq.rabbit_mq_client</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">logging</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Optional</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">aio_pika</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika.abc</span><span class="w"> </span><span class="kn">import</span> <span class="n">AbstractRobustExchange</span><span class="p">,</span> <span class="n">AbstractRobustQueue</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">pika.exchange_type</span><span class="w"> </span><span class="kn">import</span> <span class="n">ExchangeType</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.data_message_factory</span><span class="w"> </span><span class="kn">import</span> <span class="n">DataMessageFactory</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">logging_debug</span><span class="p">,</span>
<span class="n">logging_error</span><span class="p">,</span>
<span class="n">logging_info</span><span class="p">,</span>
<span class="n">logging_warning</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQRoute</span><span class="p">,</span> <span class="n">DataMessage</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.router_base</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">AMQ_REPLY_TO_EXCHANGE</span><span class="p">,</span>
<span class="n">AMQ_RPC_EXCHANGE</span><span class="p">,</span>
<span class="n">DLQ_EXCHANGE</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">sanitize_as_rabbitmq_name</span>
<div class="viewcode-block" id="RabbitMQClient">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">RabbitMQClient</span><span class="p">:</span>
<span class="n">PREFETCH_COUNT</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">DLQ_ARGS</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;x-dead-letter-exchange&quot;</span><span class="p">:</span> <span class="n">DLQ_EXCHANGE</span><span class="p">}</span>
<div class="viewcode-block" id="RabbitMQClient.__init__">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">configuration</span><span class="p">,</span> <span class="n">router</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">:</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">RobustChannel</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="p">:</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">RobustConnection</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">rpc_exchange</span><span class="p">:</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">RobustExchange</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reply_exchange</span><span class="p">:</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">RobustExchange</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span> <span class="o">=</span> <span class="n">configuration</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&quot;*******************************************&quot;</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="s2">&quot;******** RabbitMQClient.init(): </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">amq_host</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&quot;*******************************************&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">router</span> <span class="o">=</span> <span class="n">router</span>
<span class="c1"># this.meterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reply_to_exchange</span><span class="p">:</span> <span class="n">AbstractRobustExchange</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="p">(</span>
<span class="kc">None</span> <span class="c1"># when sending response to RPC request</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reply_to_queue</span><span class="p">:</span> <span class="n">AbstractRobustQueue</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="p">(</span>
<span class="kc">None</span> <span class="c1"># when receiving response to RPC request</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="RabbitMQClient.init">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.init">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">init</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">AbstractRobustExchange</span><span class="p">:</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">setup_amq_channel</span><span class="p">(</span><span class="n">loop</span><span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">init_routing_paths_consumer</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">channel</span><span class="p">)</span>
<span class="n">cm_reply_to_queue</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">get_reply_to_queue_name</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reply_to_queue</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">declare_queue</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="n">cm_reply_to_queue</span><span class="p">,</span> <span class="n">durable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">exclusive</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">auto_delete</span><span class="o">=</span><span class="kc">False</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reply_to_exchange</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">declare_exchange</span><span class="p">(</span>
<span class="n">AMQ_REPLY_TO_EXCHANGE</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">ExchangeType</span><span class="o">.</span><span class="n">direct</span>
<span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">reply_to_queue</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span>
<span class="n">exchange</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">reply_to_exchange</span><span class="p">,</span> <span class="n">routing_key</span><span class="o">=</span><span class="n">cm_reply_to_queue</span>
<span class="p">)</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">reply_to_exchange</span></div>
<div class="viewcode-block" id="RabbitMQClient.cleanup">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.cleanup">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">cleanup</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RabbitMQClient.cleanup()&quot;</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">cleanup</span><span class="p">()</span> <span class="c1"># de-register route(s)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">is_open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">is_closed</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="s2">&quot;RabbitMQClient.PreDestroy cleanup - connection close error: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">e</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="RabbitMQClient.register_inbound_routes">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.register_inbound_routes">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">register_inbound_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route_mapping</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">callback</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Register inbound routes for the AMQ adapter. The route mapping is a comma-separated string that contains</span>
<span class="sd"> list of routes. A route is the mapping of domain and path to the exchange and queue name.</span>
<span class="sd"> This method declare (and thus create) the queue and exchange, bind them together using the</span>
<span class="sd"> regex-like routing expression given in the route.</span>
<span class="sd"> After this the Service is listening on (consuming from) the queue for incoming messages.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">is_open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">):</span>
<span class="n">queue_args</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">DLQ_ARGS</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">use_dlq</span> <span class="k">else</span> <span class="kc">None</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RabbitMQClient register routes, cfg mapping: [</span><span class="si">%s</span><span class="s2">]&quot;</span><span class="p">,</span> <span class="n">route_mapping</span><span class="p">)</span>
<span class="c1"># convert the comma-separated string into list of AMQRoute objects</span>
<span class="n">requested_routes</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">unwrap_route_list</span><span class="p">(</span><span class="n">route_mapping</span><span class="p">)</span>
<span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="n">requested_routes</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Note: AMQRoute allows one of the exchange or queue to be None, in which case the default is used</span>
<span class="n">queue_name</span> <span class="o">=</span> <span class="p">(</span>
<span class="n">route</span><span class="o">.</span><span class="n">queue</span> <span class="k">if</span> <span class="n">route</span><span class="o">.</span><span class="n">queue</span> <span class="k">else</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">get_consume_queue_name</span><span class="p">()</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RabbitMQClient register route: [</span><span class="si">%s</span><span class="s2">]&quot;</span><span class="p">,</span> <span class="n">route</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">amq_adapter</span><span class="o">.</span><span class="n">callbacks_enabled</span><span class="p">:</span>
<span class="n">queue</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">AbstractRobustQueue</span><span class="p">]</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">declare_queue</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="n">queue_name</span><span class="p">,</span>
<span class="n">durable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">exclusive</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">auto_delete</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">arguments</span><span class="o">=</span><span class="n">queue_args</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">queue</span> <span class="o">=</span> <span class="kc">None</span>
<span class="n">_exchange_name</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="n">route</span><span class="o">.</span><span class="n">exchange</span> <span class="k">if</span> <span class="n">route</span><span class="o">.</span><span class="n">exchange</span> <span class="k">else</span> <span class="n">AMQ_RPC_EXCHANGE</span>
<span class="n">exchange</span><span class="p">:</span> <span class="n">AbstractRobustExchange</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">declare_exchange</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="n">_exchange_name</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="n">ExchangeType</span><span class="o">.</span><span class="n">topic</span>
<span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">amq_adapter</span><span class="o">.</span><span class="n">callbacks_enabled</span><span class="p">:</span>
<span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">exchange</span><span class="p">,</span> <span class="n">route</span><span class="o">.</span><span class="n">key</span><span class="p">)</span>
<span class="n">_c_tag</span> <span class="o">=</span> <span class="k">await</span> <span class="n">queue</span><span class="o">.</span><span class="n">consume</span><span class="p">(</span><span class="n">callback</span><span class="p">,</span> <span class="n">no_ack</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">_c_tag</span> <span class="o">=</span> <span class="kc">None</span>
<span class="c1"># Register the route with the router and publish its detail to other adapters</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">register_route</span><span class="p">(</span>
<span class="n">AMQRoute</span><span class="p">(</span>
<span class="n">route</span><span class="o">.</span><span class="n">component_name</span><span class="p">,</span>
<span class="n">_exchange_name</span><span class="p">,</span>
<span class="n">queue_name</span><span class="p">,</span>
<span class="n">sanitize_as_rabbitmq_name</span><span class="p">(</span><span class="n">route</span><span class="o">.</span><span class="n">key</span><span class="p">),</span>
<span class="n">route</span><span class="o">.</span><span class="n">timeout</span><span class="p">,</span>
<span class="n">route</span><span class="o">.</span><span class="n">valid_until</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot; [</span><span class="si">%s</span><span class="s2">] AMQ connection initialized for route: </span><span class="si">%s</span><span class="s2">, listens on: </span><span class="si">%s</span><span class="s2">/</span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">_c_tag</span><span class="p">,</span>
<span class="n">route</span><span class="p">,</span>
<span class="n">exchange</span><span class="p">,</span>
<span class="n">queue_name</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;Callback registration INCOMPLETE, </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_warning</span><span class="p">(</span><span class="s2">&quot;Channel is null, cannot register routes.&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="RabbitMQClient.register_data_reply_callback">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.register_data_reply_callback">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">register_data_reply_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">rpc_callback</span><span class="p">):</span>
<span class="n">_c_tag</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">reply_to_queue</span><span class="o">.</span><span class="n">consume</span><span class="p">(</span><span class="n">callback</span><span class="o">=</span><span class="n">rpc_callback</span><span class="p">,</span> <span class="n">no_ack</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot; [</span><span class="si">%s</span><span class="s2">] RabbitMQClient register data reply callback on queue: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">_c_tag</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">reply_to_queue</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="RabbitMQClient.setup_amq_channel">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.setup_amq_channel">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">setup_amq_channel</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">loop</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Set up the RabbitMQ connection</span>
<span class="sd"> :return: nothing, it applies changes to instance variables</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
<span class="sa">f</span><span class="s2">&quot;RabbitMQClient.setupAMQ, connecting to RabbitMQ at </span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">amq_host</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">amq_port</span><span class="si">}</span><span class="s2">, loop=</span><span class="si">{</span><span class="n">loop</span><span class="si">}</span><span class="s2">&quot;</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="o">=</span> <span class="k">await</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">connect_robust</span><span class="p">(</span>
<span class="n">host</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">amq_host</span><span class="p">,</span>
<span class="n">port</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">amq_port</span><span class="p">,</span>
<span class="n">login</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">rabbit_mq_user</span><span class="p">,</span>
<span class="n">password</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">rabbit_mq_password</span><span class="p">,</span>
<span class="n">loop</span><span class="o">=</span><span class="n">loop</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">logging_debug</span><span class="p">(</span>
<span class="s2">&quot;RabbitMQClient.setupAMQ, got connection, open=</span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">is_closed</span><span class="p">,</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">channel</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">set_qos</span><span class="p">(</span><span class="n">prefetch_count</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">PREFETCH_COUNT</span><span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">init_internal_routing_paths</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">set_route_added_notifier</span><span class="p">(</span>
<span class="k">lambda</span> <span class="n">route</span><span class="p">:</span> <span class="n">logging_debug</span><span class="p">(</span>
<span class="s2">&quot; RouteAdded, setting up metrics: amqp.adapter.exchange.</span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">route</span><span class="o">.</span><span class="n">exchange</span><span class="p">,</span>
<span class="p">)</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="RabbitMQClient.send_message">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.send_message">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">send_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> The API routine that sends a datagram message (DataMessage) to the destination. The Destination</span>
<span class="sd"> is determined from the DataMessage metadata section, using `domain` and `path` as key.</span>
<span class="sd"> :param message: message to send</span>
<span class="sd"> :return: nothing (void) as submitting message to RabbitMQ exchange does not return anything</span>
<span class="sd"> meaningful. There will be asynch ACK later that will confirm the message was sent to RMQ.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">route</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">find_route_by_message</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="k">if</span> <span class="n">route</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">rpc_exchange</span><span class="p">:</span>
<span class="k">if</span> <span class="n">route</span><span class="o">.</span><span class="n">timeout</span> <span class="o">&lt;=</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1"># No response expected</span>
<span class="n">pika_message</span><span class="p">:</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">Message</span> <span class="o">=</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">Message</span><span class="p">(</span>
<span class="n">body</span><span class="o">=</span><span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">serialize</span><span class="p">(</span><span class="n">message</span><span class="p">),</span>
<span class="n">content_encoding</span><span class="o">=</span><span class="s2">&quot;utf-8&quot;</span><span class="p">,</span>
<span class="n">delivery_mode</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">rpc_exchange</span><span class="o">.</span><span class="n">publish</span><span class="p">(</span>
<span class="n">message</span><span class="o">=</span><span class="n">pika_message</span><span class="p">,</span>
<span class="n">routing_key</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">get_routing_key</span><span class="p">(</span>
<span class="n">message</span>
<span class="p">),</span> <span class="c1"># context path is a routing key</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Msg Publish (no-reply), messageId: </span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">()</span><span class="si">}</span><span class="s2">, to: </span><span class="si">{</span><span class="n">route</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="c1"># This is RPC call, there should be response</span>
<span class="n">pika_message</span><span class="p">:</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">Message</span> <span class="o">=</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">Message</span><span class="p">(</span>
<span class="n">body</span><span class="o">=</span><span class="n">DataMessageFactory</span><span class="o">.</span><span class="n">serialize</span><span class="p">(</span><span class="n">message</span><span class="p">),</span>
<span class="n">content_encoding</span><span class="o">=</span><span class="s2">&quot;utf-8&quot;</span><span class="p">,</span>
<span class="n">delivery_mode</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span>
<span class="n">correlation_id</span><span class="o">=</span><span class="nb">str</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">()),</span>
<span class="n">reply_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">get_reply_to_queue_name</span><span class="p">(),</span>
<span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">rpc_exchange</span><span class="o">.</span><span class="n">publish</span><span class="p">(</span>
<span class="n">message</span><span class="o">=</span><span class="n">pika_message</span><span class="p">,</span>
<span class="n">routing_key</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">get_routing_key</span><span class="p">(</span>
<span class="n">message</span>
<span class="p">),</span> <span class="c1"># context path is a routing key</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;Msg Publish (RPC call), messageId: </span><span class="si">%s</span><span class="s2">, to: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">message</span><span class="o">.</span><span class="n">id</span><span class="p">()</span><span class="o">.</span><span class="n">as_string</span><span class="p">(),</span>
<span class="n">route</span><span class="o">.</span><span class="n">as_string</span><span class="p">(),</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_warning</span><span class="p">(</span>
<span class="s2">&quot;Message not sent, no route for </span><span class="si">%s</span><span class="s2">/</span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">message</span><span class="o">.</span><span class="n">domain</span><span class="p">(),</span>
<span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">(),</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;Send message failed: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span></div>
<div class="viewcode-block" id="RabbitMQClient.request_routes">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.request_routes">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">request_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">request_routes_from_adapters</span><span class="p">()</span></div>
<div class="viewcode-block" id="RabbitMQClient.find_route">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.rabbit_mq_client.RabbitMQClient.find_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">find_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">AMQRoute</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">router</span><span class="o">.</span><span class="n">find_route</span><span class="p">(</span><span class="n">domain</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/rabbitmq/user_management_service_client.html
-397
View File
@@ -1,397 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.rabbitmq.user_management_service_client &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.rabbitmq.user_management_service_client</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.rabbitmq.user_management_service_client</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">asyncio</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">json</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">logging</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">uuid</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Dict</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Optional</span>
<span class="kn">import</span><span class="w"> </span><span class="nn">aio_pika</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika</span><span class="w"> </span><span class="kn">import</span> <span class="n">Message</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika.abc</span><span class="w"> </span><span class="kn">import</span> <span class="n">AbstractIncomingMessage</span><span class="p">,</span> <span class="n">AbstractRobustChannel</span><span class="p">,</span> <span class="n">AbstractRobustConnection</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_debug</span><span class="p">,</span> <span class="n">logging_error</span><span class="p">,</span> <span class="n">logging_info</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.config.amq_configuration</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQConfiguration</span>
<div class="viewcode-block" id="UserManagementServiceException">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceException">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">UserManagementServiceException</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Exception raised for errors in the User Management Service.&quot;&quot;&quot;</span>
<div class="viewcode-block" id="UserManagementServiceException.__init__">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceException.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">exception_type</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">additional_info</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">exception_type</span> <span class="o">=</span> <span class="n">exception_type</span>
<span class="bp">self</span><span class="o">.</span><span class="n">message</span> <span class="o">=</span> <span class="n">message</span>
<span class="bp">self</span><span class="o">.</span><span class="n">additional_info</span> <span class="o">=</span> <span class="n">additional_info</span> <span class="ow">or</span> <span class="p">{}</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">exception_type</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">message</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
</div>
<div class="viewcode-block" id="UserManagementServiceClient">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">UserManagementServiceClient</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Client for the User Management Service that communicates over RabbitMQ.</span>
<span class="sd"> Supports operations for token, user, group, tenant, and permission services.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Exchange name for User Management Service</span>
<span class="n">EXCHANGE_NAME</span> <span class="o">=</span> <span class="s2">&quot;cleverthis.clevermicro.auth.endpoints&quot;</span>
<span class="c1"># Service routing keys</span>
<span class="n">TOKEN_SERVICE</span> <span class="o">=</span> <span class="s2">&quot;token-service-v1&quot;</span>
<span class="n">USER_SERVICE</span> <span class="o">=</span> <span class="s2">&quot;user-service-v1&quot;</span>
<span class="n">GROUP_SERVICE</span> <span class="o">=</span> <span class="s2">&quot;group-service-v1&quot;</span>
<span class="n">TENANT_SERVICE</span> <span class="o">=</span> <span class="s2">&quot;tenant-service-v1&quot;</span>
<span class="n">PERMISSION_SERVICE</span> <span class="o">=</span> <span class="s2">&quot;permission-service-v1&quot;</span>
<span class="c1"># Response queue name</span>
<span class="n">RESPONSE_QUEUE_NAME</span> <span class="o">=</span> <span class="s2">&quot;user-management-service-cleverswarm&quot;</span>
<div class="viewcode-block" id="UserManagementServiceClient.__init__">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">configuration</span><span class="p">:</span> <span class="n">AMQConfiguration</span><span class="p">,</span> <span class="n">loop</span><span class="p">:</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">AbstractEventLoop</span> <span class="o">=</span> <span class="kc">None</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Initialize the User Management Service client.</span>
<span class="sd"> </span>
<span class="sd"> Args:</span>
<span class="sd"> configuration: AMQ configuration</span>
<span class="sd"> loop: Event loop to use (optional)</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span> <span class="o">=</span> <span class="n">configuration</span>
<span class="bp">self</span><span class="o">.</span><span class="n">loop</span> <span class="o">=</span> <span class="n">loop</span> <span class="ow">or</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">AbstractRobustConnection</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">AbstractRobustChannel</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">exchange</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">aio_pika</span><span class="o">.</span><span class="n">RobustExchange</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">response_queue</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">aio_pika</span><span class="o">.</span><span class="n">RobustQueue</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">response_consumer_tag</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">futures</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">Future</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
<span class="bp">self</span><span class="o">.</span><span class="n">initialized</span> <span class="o">=</span> <span class="kc">False</span></div>
<div class="viewcode-block" id="UserManagementServiceClient.initialize">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient.initialize">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">initialize</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Initialize the RabbitMQ connection and set up the exchange and response queue.&quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">initialized</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># Connect to RabbitMQ</span>
<span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="o">=</span> <span class="k">await</span> <span class="n">aio_pika</span><span class="o">.</span><span class="n">connect_robust</span><span class="p">(</span>
<span class="n">host</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">amq_host</span><span class="p">,</span>
<span class="n">port</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">amq_port</span><span class="p">,</span>
<span class="n">login</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">rabbit_mq_user</span><span class="p">,</span>
<span class="n">password</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">dispatch</span><span class="o">.</span><span class="n">rabbit_mq_password</span><span class="p">,</span>
<span class="n">loop</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Create channel</span>
<span class="bp">self</span><span class="o">.</span><span class="n">channel</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">channel</span><span class="p">()</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">set_qos</span><span class="p">(</span><span class="n">prefetch_count</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Declare exchange</span>
<span class="bp">self</span><span class="o">.</span><span class="n">exchange</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">declare_exchange</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">EXCHANGE_NAME</span><span class="p">,</span>
<span class="nb">type</span><span class="o">=</span><span class="n">aio_pika</span><span class="o">.</span><span class="n">ExchangeType</span><span class="o">.</span><span class="n">TOPIC</span><span class="p">,</span>
<span class="n">durable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Declare response queue</span>
<span class="bp">self</span><span class="o">.</span><span class="n">response_queue</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">declare_queue</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">RESPONSE_QUEUE_NAME</span><span class="p">,</span>
<span class="n">durable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">exclusive</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">auto_delete</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Start consuming responses</span>
<span class="bp">self</span><span class="o">.</span><span class="n">response_consumer_tag</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_queue</span><span class="o">.</span><span class="n">consume</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_on_response_callback</span><span class="p">,</span>
<span class="n">no_ack</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">initialized</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;User Management Service client initialized&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Failed to initialize User Management Service client: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">is_closed</span><span class="p">:</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">raise</span></div>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">_on_response_callback</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AbstractIncomingMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Handle responses from the User Management Service.</span>
<span class="sd"> </span>
<span class="sd"> Args:</span>
<span class="sd"> message: The incoming message</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">correlation_id</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">correlation_id</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">correlation_id</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;Received response without correlation ID&quot;</span><span class="p">)</span>
<span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">()</span>
<span class="k">return</span>
<span class="n">future</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">futures</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">correlation_id</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">future</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;No pending request found for correlation ID: </span><span class="si">{</span><span class="n">correlation_id</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">()</span>
<span class="k">return</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">response_data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">))</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Received response: </span><span class="si">{</span><span class="n">response_data</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">future</span><span class="o">.</span><span class="n">set_result</span><span class="p">(</span><span class="n">response_data</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">future</span><span class="o">.</span><span class="n">set_exception</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
<span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Error processing response: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">()</span>
<div class="viewcode-block" id="UserManagementServiceClient.close">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient.close">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">close</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;Close the RabbitMQ connection.&quot;&quot;&quot;</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="ow">and</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">is_closed</span><span class="p">:</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">initialized</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;User Management Service client closed&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="UserManagementServiceClient.call_service">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient.call_service">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">call_service</span><span class="p">(</span>
<span class="bp">self</span><span class="p">,</span> <span class="n">service</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">method</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">args</span><span class="p">:</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="kc">None</span>
<span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Call a method on a User Management Service.</span>
<span class="sd"> </span>
<span class="sd"> Args:</span>
<span class="sd"> service: Service routing key (e.g., &#39;token-service-v1&#39;)</span>
<span class="sd"> method: Method name to call</span>
<span class="sd"> args: Method arguments</span>
<span class="sd"> </span>
<span class="sd"> Returns:</span>
<span class="sd"> Response data</span>
<span class="sd"> </span>
<span class="sd"> Raises:</span>
<span class="sd"> UserManagementServiceException: If the service returns an error</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">initialized</span><span class="p">:</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">initialize</span><span class="p">()</span>
<span class="n">correlation_id</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">uuid</span><span class="o">.</span><span class="n">uuid4</span><span class="p">())</span>
<span class="n">future</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">loop</span><span class="o">.</span><span class="n">create_future</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">futures</span><span class="p">[</span><span class="n">correlation_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">future</span>
<span class="n">request_data</span> <span class="o">=</span> <span class="p">{</span>
<span class="s2">&quot;method&quot;</span><span class="p">:</span> <span class="n">method</span><span class="p">,</span>
<span class="s2">&quot;args&quot;</span><span class="p">:</span> <span class="n">args</span> <span class="ow">or</span> <span class="p">{},</span>
<span class="p">}</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">Message</span><span class="p">(</span>
<span class="n">body</span><span class="o">=</span><span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">request_data</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">&#39;utf-8&#39;</span><span class="p">),</span>
<span class="n">content_type</span><span class="o">=</span><span class="s1">&#39;application/json&#39;</span><span class="p">,</span>
<span class="n">correlation_id</span><span class="o">=</span><span class="n">correlation_id</span><span class="p">,</span>
<span class="n">reply_to</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">RESPONSE_QUEUE_NAME</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">exchange</span><span class="o">.</span><span class="n">publish</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">routing_key</span><span class="o">=</span><span class="n">service</span><span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Sent request to </span><span class="si">{</span><span class="n">service</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">method</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">args</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># Wait for response</span>
<span class="n">response</span> <span class="o">=</span> <span class="k">await</span> <span class="n">future</span>
<span class="c1"># Process response</span>
<span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;type&quot;</span><span class="p">)</span> <span class="o">==</span> <span class="s2">&quot;OK&quot;</span><span class="p">:</span>
<span class="k">return</span> <span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;result&quot;</span><span class="p">,</span> <span class="p">[])</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">UserManagementServiceException</span><span class="p">(</span>
<span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;exception&quot;</span><span class="p">,</span> <span class="s2">&quot;unknown_error&quot;</span><span class="p">),</span>
<span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;message&quot;</span><span class="p">,</span> <span class="s2">&quot;Unknown error&quot;</span><span class="p">),</span>
<span class="n">response</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&quot;additional_info&quot;</span><span class="p">,</span> <span class="p">{})</span>
<span class="p">)</span>
<span class="k">except</span> <span class="n">asyncio</span><span class="o">.</span><span class="n">CancelledError</span><span class="p">:</span>
<span class="k">raise</span>
<span class="k">except</span> <span class="n">UserManagementServiceException</span><span class="p">:</span>
<span class="k">raise</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Error calling service </span><span class="si">{</span><span class="n">service</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">method</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">raise</span>
<span class="k">finally</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">futures</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="n">correlation_id</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span></div>
<div class="viewcode-block" id="UserManagementServiceClient.validate_token">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient.validate_token">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">validate_token</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Validate a JWT token.</span>
<span class="sd"> </span>
<span class="sd"> Args:</span>
<span class="sd"> token: JWT token to validate</span>
<span class="sd"> </span>
<span class="sd"> Returns:</span>
<span class="sd"> Token information</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">call_service</span><span class="p">(</span>
<span class="n">service</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">TOKEN_SERVICE</span><span class="p">,</span>
<span class="n">method</span><span class="o">=</span><span class="s2">&quot;validateToken&quot;</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;token&quot;</span><span class="p">:</span> <span class="n">token</span><span class="p">}</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="UserManagementServiceClient.query_user_by_id">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient.query_user_by_id">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">query_user_by_id</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">user_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Query user information by user ID.</span>
<span class="sd"> </span>
<span class="sd"> Args:</span>
<span class="sd"> user_id: User ID</span>
<span class="sd"> </span>
<span class="sd"> Returns:</span>
<span class="sd"> User information</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">call_service</span><span class="p">(</span>
<span class="n">service</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">USER_SERVICE</span><span class="p">,</span>
<span class="n">method</span><span class="o">=</span><span class="s2">&quot;queryUserById&quot;</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;userId&quot;</span><span class="p">:</span> <span class="n">user_id</span><span class="p">}</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="UserManagementServiceClient.list_users_in_group">
<a class="viewcode-back" href="../../../modules/amqp.rabbitmq.html#amqp.rabbitmq.user_management_service_client.UserManagementServiceClient.list_users_in_group">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">list_users_in_group</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">group_id</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="n">Dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]]:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> List users in a group.</span>
<span class="sd"> </span>
<span class="sd"> Args:</span>
<span class="sd"> group_id: Group ID</span>
<span class="sd"> </span>
<span class="sd"> Returns:</span>
<span class="sd"> List of users</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">return</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">call_service</span><span class="p">(</span>
<span class="n">service</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">GROUP_SERVICE</span><span class="p">,</span>
<span class="n">method</span><span class="o">=</span><span class="s2">&quot;listUsersInGroup&quot;</span><span class="p">,</span>
<span class="n">args</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;groupId&quot;</span><span class="p">:</span> <span class="n">group_id</span><span class="p">}</span>
<span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/router/route_database.html
-209
View File
@@ -1,209 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.router.route_database &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.router.route_database</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.router.route_database</h1><div class="highlight"><pre>
<span></span><span class="kn">import</span><span class="w"> </span><span class="nn">re</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Set</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_info</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQRoute</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">sanitize_as_rabbitmq_name</span>
<div class="viewcode-block" id="RouteDatabase">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">RouteDatabase</span><span class="p">:</span>
<div class="viewcode-block" id="RouteDatabase.__init__">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="p">:</span> <span class="n">Set</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span></div>
<div class="viewcode-block" id="RouteDatabase.get_routes">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.get_routes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Set</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span></div>
<div class="viewcode-block" id="RouteDatabase.pattern">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.pattern">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">pattern</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">routing_key</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">re</span><span class="o">.</span><span class="n">Pattern</span><span class="p">:</span>
<span class="n">breaker</span> <span class="o">=</span> <span class="kc">False</span>
<span class="n">tokens</span> <span class="o">=</span> <span class="n">routing_key</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;.&quot;</span><span class="p">)</span>
<span class="n">counter</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">tokens</span><span class="p">)</span>
<span class="n">regex_parts</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">token</span> <span class="ow">in</span> <span class="n">tokens</span><span class="p">:</span>
<span class="n">counter</span> <span class="o">-=</span> <span class="mi">1</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">breaker</span><span class="p">:</span>
<span class="k">if</span> <span class="n">token</span> <span class="o">==</span> <span class="s2">&quot;&quot;</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="s2">&quot;[^</span><span class="se">\\</span><span class="s2">.]*&quot;</span>
<span class="k">elif</span> <span class="n">token</span> <span class="o">==</span> <span class="s2">&quot;#&quot;</span><span class="p">:</span>
<span class="n">breaker</span> <span class="o">=</span> <span class="kc">True</span>
<span class="n">res</span> <span class="o">=</span> <span class="s2">&quot;.*$&quot;</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">res</span> <span class="o">=</span> <span class="n">token</span>
<span class="k">if</span> <span class="n">counter</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">breaker</span><span class="p">:</span>
<span class="n">res</span> <span class="o">+=</span> <span class="s2">&quot;</span><span class="se">\\</span><span class="s2">.&quot;</span>
<span class="n">regex_parts</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">res</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">regex_parts</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s2">&quot;&quot;</span><span class="p">)</span>
<span class="n">regex</span> <span class="o">=</span> <span class="s2">&quot;&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">filter</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span> <span class="o">!=</span> <span class="s2">&quot;&quot;</span><span class="p">,</span> <span class="n">regex_parts</span><span class="p">))</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">routing_key</span><span class="p">:</span>
<span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="s2">&quot;.*&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">regex</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouteDatabase.matches">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.matches">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">matches</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">routing_key_from_route</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">routing_key_from_message</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="n">route_pattern</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pattern</span><span class="p">(</span><span class="n">routing_key_from_route</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">route_pattern</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="n">sanitize_as_rabbitmq_name</span><span class="p">(</span><span class="n">routing_key_from_message</span><span class="p">)))</span></div>
<div class="viewcode-block" id="RouteDatabase.wrap_routes">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.wrap_routes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">wrap_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="s2">&quot;,&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">route</span><span class="p">)</span> <span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouteDatabase.find_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.find_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">find_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">AMQRoute</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">routing_key</span> <span class="o">=</span> <span class="n">sanitize_as_rabbitmq_name</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">domain</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;findRoute key:</span><span class="si">{</span><span class="n">routing_key</span><span class="si">}</span><span class="s2"> in:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">wrap_routes</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="p">:</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">matches</span><span class="p">(</span><span class="n">route</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">routing_key</span><span class="p">):</span>
<span class="k">return</span> <span class="n">route</span>
<span class="k">return</span> <span class="kc">None</span></div>
<div class="viewcode-block" id="RouteDatabase.find_route_by_service">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.find_route_by_service">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">find_route_by_service</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Optional</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]:</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;findRoute service_name:</span><span class="si">{</span><span class="n">service_name</span><span class="si">}</span><span class="s2"> in:</span><span class="si">{</span><span class="bp">self</span><span class="o">.</span><span class="n">wrap_routes</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="p">:</span>
<span class="k">if</span> <span class="n">route</span><span class="o">.</span><span class="n">component_name</span> <span class="o">==</span> <span class="n">service_name</span><span class="p">:</span>
<span class="k">return</span> <span class="n">route</span>
<span class="k">return</span> <span class="kc">None</span></div>
<div class="viewcode-block" id="RouteDatabase.add_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.add_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">add_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route</span><span class="p">:</span> <span class="n">AMQRoute</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">if</span> <span class="n">route</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">route</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">True</span>
<span class="k">return</span> <span class="kc">False</span></div>
<div class="viewcode-block" id="RouteDatabase.remove_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.route_database.RouteDatabase.remove_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">remove_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route</span><span class="p">:</span> <span class="n">AMQRoute</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="k">return</span> <span class="n">route</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">routes</span><span class="o">.</span><span class="n">remove</span><span class="p">(</span><span class="n">route</span><span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/router/router_base.html
-258
View File
@@ -1,258 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.router.router_base &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.router.router_base</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.router.router_base</h1><div class="highlight"><pre>
<span></span><span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Callable</span><span class="p">,</span> <span class="n">List</span><span class="p">,</span> <span class="n">Optional</span><span class="p">,</span> <span class="n">Set</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.amq_route_factory</span><span class="w"> </span><span class="kn">import</span> <span class="n">NULL_ROUTE</span><span class="p">,</span> <span class="n">AMQRouteFactory</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">logging_debug</span><span class="p">,</span> <span class="n">logging_error</span><span class="p">,</span> <span class="n">logging_info</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQRoute</span><span class="p">,</span> <span class="n">DataMessage</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.route_database</span><span class="w"> </span><span class="kn">import</span> <span class="n">RouteDatabase</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.utils</span><span class="w"> </span><span class="kn">import</span> <span class="n">sanitize_as_rabbitmq_name</span>
<span class="n">CM_REQUEST_EXCHANGE</span> <span class="o">=</span> <span class="s2">&quot;CM-Request&quot;</span>
<span class="n">CM_RESPONSE_EXCHANGE</span> <span class="o">=</span> <span class="s2">&quot;CM-Response&quot;</span>
<span class="n">CM_C_REQ_QUEUE</span> <span class="o">=</span> <span class="s2">&quot;C-REQ&quot;</span>
<span class="n">CM_C_RESP_QUEUE</span> <span class="o">=</span> <span class="s2">&quot;C-RESP&quot;</span>
<span class="n">CM_C_AMQ_QUEUE</span> <span class="o">=</span> <span class="s2">&quot;C-AMQ&quot;</span>
<span class="n">CM_P_RESP_QUEUE</span> <span class="o">=</span> <span class="s2">&quot;P-RESP&quot;</span>
<span class="n">CM_P_REPLY_TO</span> <span class="o">=</span> <span class="s2">&quot;P-REPLY-TO&quot;</span>
<span class="n">ANY_RECIPIENT</span> <span class="o">=</span> <span class="s2">&quot;*&quot;</span>
<span class="n">AMQ_RPC_EXCHANGE</span> <span class="o">=</span> <span class="s2">&quot;AMQ-RPC&quot;</span>
<span class="n">AMQ_REPLY_TO_EXCHANGE</span> <span class="o">=</span> <span class="s2">&quot;AMQ-REPLY-TO&quot;</span>
<span class="n">DLQ_EXCHANGE</span> <span class="o">=</span> <span class="s2">&quot;DLX&quot;</span>
<div class="viewcode-block" id="RouterBase">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">RouterBase</span><span class="p">:</span>
<div class="viewcode-block" id="RouterBase.__init__">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_database</span> <span class="o">=</span> <span class="n">RouteDatabase</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">own_routes</span><span class="p">:</span> <span class="n">Set</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_added_notifier</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[</span><span class="n">AMQRoute</span><span class="p">],</span> <span class="kc">None</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_removed_notifier</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[</span><span class="n">AMQRoute</span><span class="p">],</span> <span class="kc">None</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span></div>
<div class="viewcode-block" id="RouterBase.get_routing_key">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.get_routing_key">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_routing_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="n">sanitize_as_rabbitmq_name</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">domain</span><span class="p">()</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">()</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterBase.wrap_routes">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.wrap_routes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">wrap_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">wrap_routes</span><span class="p">()</span></div>
<div class="viewcode-block" id="RouterBase.wrap_own_routes">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.wrap_own_routes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">wrap_own_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
<span class="k">return</span> <span class="s2">&quot;,&quot;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">route</span><span class="p">)</span> <span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">own_routes</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterBase.unwrap_route_list">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.unwrap_route_list">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">unwrap_route_list</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route_string</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">List</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]:</span>
<span class="n">logging_debug</span><span class="p">(</span><span class="s2">&quot;RouterMaster.unwrapRoutes, route(s): </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">route_string</span><span class="p">)</span>
<span class="n">routes</span> <span class="o">=</span> <span class="p">[</span><span class="n">AMQRouteFactory</span><span class="o">.</span><span class="n">from_string</span><span class="p">(</span><span class="n">route_str</span><span class="p">)</span> <span class="k">for</span> <span class="n">route_str</span> <span class="ow">in</span> <span class="n">route_string</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
<span class="k">return</span> <span class="p">[</span><span class="n">route</span> <span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="n">routes</span> <span class="k">if</span> <span class="n">route</span> <span class="o">!=</span> <span class="n">NULL_ROUTE</span><span class="p">]</span></div>
<div class="viewcode-block" id="RouterBase.get_routes">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.get_routes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">get_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Set</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">get_routes</span><span class="p">()</span></div>
<div class="viewcode-block" id="RouterBase.find_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.find_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">find_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">domain</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">AMQRoute</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">find_route</span><span class="p">(</span><span class="n">domain</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterBase.find_route_by_message">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.find_route_by_message">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">find_route_by_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">DataMessage</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">AMQRoute</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">find_route</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">domain</span><span class="p">(),</span> <span class="n">message</span><span class="o">.</span><span class="n">path</span><span class="p">())</span></div>
<div class="viewcode-block" id="RouterBase.find_route_by_service">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.find_route_by_service">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">find_route_by_service</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">service_name</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Optional</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]:</span>
<span class="n">_route</span><span class="p">:</span> <span class="n">Optional</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">find_route_by_service</span><span class="p">(</span><span class="n">service_name</span><span class="p">)</span>
<span class="k">if</span> <span class="n">_route</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">own_routes</span><span class="p">:</span>
<span class="k">if</span> <span class="n">route</span><span class="o">.</span><span class="n">component_name</span> <span class="o">==</span> <span class="n">service_name</span><span class="p">:</span>
<span class="n">_route</span> <span class="o">=</span> <span class="n">route</span>
<span class="k">return</span> <span class="n">_route</span></div>
<div class="viewcode-block" id="RouterBase.add_routes">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.add_routes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">add_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">amq_route_list</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">unwrap_route_list</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot; [DBG] RouterMaster.addRoute [</span><span class="si">%d</span><span class="s2">]: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">amq_route_list</span><span class="p">),</span> <span class="n">message</span><span class="p">)</span>
<span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="n">amq_route_list</span><span class="p">:</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RouterMaster.addRoute, adding route: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">route</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">route</span><span class="o">.</span><span class="n">exchange</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">route</span><span class="o">.</span><span class="n">queue</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">route</span><span class="o">.</span><span class="n">key</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="s2">&quot;Cannot setup AMQ Exchange for &#39;</span><span class="si">%s</span><span class="s2">&#39;, at least one of &quot;</span>
<span class="s2">&quot;Exchange, Queue and/or Key must be present in the routing string. &quot;</span>
<span class="s2">&quot;Expected format is &#39;</span><span class="si">%s</span><span class="s2">&#39;&quot;</span><span class="p">,</span>
<span class="n">route</span><span class="p">,</span>
<span class="n">AMQRoute</span><span class="o">.</span><span class="n">FORMAT</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">if</span> <span class="n">route</span><span class="o">.</span><span class="n">exchange</span> <span class="o">!=</span> <span class="n">DLQ_EXCHANGE</span><span class="p">:</span>
<span class="n">is_new_route</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">add_route</span><span class="p">(</span><span class="n">route</span><span class="p">)</span>
<span class="k">if</span> <span class="n">is_new_route</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_added_notifier</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_added_notifier</span><span class="p">(</span><span class="n">route</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;add_routes error: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterBase.set_route_added_notifier">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.set_route_added_notifier">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">set_route_added_notifier</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">callback</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[</span><span class="n">AMQRoute</span><span class="p">],</span> <span class="kc">None</span><span class="p">]):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_added_notifier</span> <span class="o">=</span> <span class="n">callback</span></div>
<div class="viewcode-block" id="RouterBase.remove_routes">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.remove_routes">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">remove_routes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">to_remove</span> <span class="o">=</span> <span class="p">[</span><span class="n">AMQRouteFactory</span><span class="o">.</span><span class="n">from_string</span><span class="p">(</span><span class="n">route_str</span><span class="p">)</span> <span class="k">for</span> <span class="n">route_str</span> <span class="ow">in</span> <span class="n">message</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;,&quot;</span><span class="p">)]</span>
<span class="n">removed_count</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="mi">1</span> <span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="n">to_remove</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">remove_route</span><span class="p">(</span><span class="n">route</span><span class="p">))</span>
<span class="n">logging_info</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;RouterMaster.removeRoutes(): </span><span class="si">{</span><span class="n">message</span><span class="si">}</span><span class="s2">, removed=</span><span class="si">{</span><span class="n">removed_count</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">IOError</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Can&#39;t remove route(s): </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterBase.remove_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.remove_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">remove_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route</span><span class="p">:</span> <span class="n">AMQRoute</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="n">is_route_removed</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">remove_route</span><span class="p">(</span><span class="n">route</span><span class="p">)</span>
<span class="k">if</span> <span class="n">is_route_removed</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">route_removed_notifier</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_removed_notifier</span><span class="p">(</span><span class="n">route</span><span class="p">)</span>
<span class="k">return</span> <span class="n">is_route_removed</span></div>
<div class="viewcode-block" id="RouterBase.add_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.add_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">add_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route</span><span class="p">:</span> <span class="n">AMQRoute</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">route_database</span><span class="o">.</span><span class="n">add_route</span><span class="p">(</span><span class="n">route</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterBase.add_own_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_base.RouterBase.add_own_route">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">add_own_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route</span><span class="p">:</span> <span class="n">AMQRoute</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">own_routes</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">route</span><span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>
api-doc/_build/html/_modules/amqp/router/router_consumer.html
-328
View File
@@ -1,328 +0,0 @@
<!DOCTYPE html>
<html class="writer-html5" lang="en" data-content_root="../../../">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>amqp.router.router_consumer &mdash; AMQ Adapter Python 1.0 documentation</title>
<link rel="stylesheet" type="text/css" href="../../../_static/pygments.css?v=03e43079" />
<link rel="stylesheet" type="text/css" href="../../../_static/css/theme.css?v=e59714d7" />
<script src="../../../_static/jquery.js?v=5d32c60e"></script>
<script src="../../../_static/_sphinx_javascript_frameworks_compat.js?v=2cd50e6c"></script>
<script src="../../../_static/documentation_options.js?v=f2a433a1"></script>
<script src="../../../_static/doctools.js?v=9bcbadda"></script>
<script src="../../../_static/sphinx_highlight.js?v=dc90522c"></script>
<script src="../../../_static/js/theme.js"></script>
<link rel="index" title="Index" href="../../../genindex.html" />
<link rel="search" title="Search" href="../../../search.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="../../../index.html" class="icon icon-home">
AMQ Adapter Python
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
<input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<p class="caption" role="heading"><span class="caption-text">Contents:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.html">AMQP Package</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.config.html">Configuration Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.adapter.html">Adapter Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.service.html">Service Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.router.html">Router Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.model.html">Model Module</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../modules/amqp.rabbitmq.html">RabbitMQ Module</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="../../../index.html">AMQ Adapter Python</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="../../../index.html" class="icon icon-home" aria-label="Home"></a></li>
<li class="breadcrumb-item"><a href="../../index.html">Module code</a></li>
<li class="breadcrumb-item active">amqp.router.router_consumer</li>
<li class="wy-breadcrumbs-aside">
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<h1>Source code for amqp.router.router_consumer</h1><div class="highlight"><pre>
<span></span><span class="kn">from</span><span class="w"> </span><span class="nn">typing</span><span class="w"> </span><span class="kn">import</span> <span class="n">Set</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">aio_pika.abc</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">AbstractIncomingMessage</span><span class="p">,</span>
<span class="n">AbstractRobustChannel</span><span class="p">,</span>
<span class="n">AbstractRobustQueue</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.amq_route_factory</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQRouteFactory</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.adapter.logging_utils</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">logging_debug</span><span class="p">,</span>
<span class="n">logging_error</span><span class="p">,</span>
<span class="n">logging_info</span><span class="p">,</span>
<span class="n">logging_warning</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.config.amq_configuration</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQConfiguration</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.model</span><span class="w"> </span><span class="kn">import</span> <span class="n">AMQRoute</span><span class="p">,</span> <span class="n">ServiceMessage</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.model.service_message_type</span><span class="w"> </span><span class="kn">import</span> <span class="n">ServiceMessageType</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.router_base</span><span class="w"> </span><span class="kn">import</span> <span class="p">(</span>
<span class="n">ANY_RECIPIENT</span><span class="p">,</span>
<span class="n">CM_C_REQ_QUEUE</span><span class="p">,</span>
<span class="n">CM_REQUEST_EXCHANGE</span><span class="p">,</span>
<span class="n">CM_RESPONSE_EXCHANGE</span><span class="p">,</span>
<span class="p">)</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">amqp.router.router_producer</span><span class="w"> </span><span class="kn">import</span> <span class="n">RouterProducer</span>
<div class="viewcode-block" id="RouterConsumer">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_consumer.RouterConsumer">[docs]</a>
<span class="k">class</span><span class="w"> </span><span class="nc">RouterConsumer</span><span class="p">(</span><span class="n">RouterProducer</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> * Contains Consumer logic. The logic here:</span>
<span class="sd"> * Sets a listener on CM_C_REQ_QUEUE queue to receive routing data requests</span>
<span class="sd"> * Upon receipt of routing data request, responds on CM_RESPONSE_EXCHANGE with local routing data.</span>
<span class="sd"> * It also proactively pushes the local routing data to CM_RESPONSE_EXCHANGE on startup to</span>
<span class="sd"> * notify the other (excluding self) Producers about its presence.</span>
<span class="sd"> *</span>
<span class="sd"> * Consumer is also Producer (Consumer sits alongside the CM Service, and anytime the Service needs</span>
<span class="sd"> * to communicate with other Service, it can do so via the Adapter). Therefore, the Producer</span>
<span class="sd"> * capability is also required.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="RouterConsumer.__init__">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_consumer.RouterConsumer.__init__">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">configuration</span><span class="p">:</span> <span class="n">AMQConfiguration</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amqRouteFactory</span> <span class="o">=</span> <span class="n">AMQRouteFactory</span><span class="p">()</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RouterConsumer.&lt;init&gt;&quot;</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterConsumer.init_routing_paths_consumer">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_consumer.RouterConsumer.init_routing_paths_consumer">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">init_routing_paths_consumer</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">_channel</span><span class="p">:</span> <span class="n">AbstractRobustChannel</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> * Initialize router according to Consumer mode.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;RouterConsumer.initConsumerRouter, channel open: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">is_open</span><span class="p">(</span><span class="n">_channel</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">is_open</span><span class="p">(</span><span class="n">_channel</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="c1"># 1. declare a Queue to receive the RoutingData requests</span>
<span class="n">_cm_request_queue</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">amq_adapter</span><span class="o">.</span><span class="n">adapter_prefix</span><span class="p">()</span> <span class="o">+</span> <span class="n">CM_C_REQ_QUEUE</span>
<span class="p">)</span>
<span class="n">_queue</span><span class="p">:</span> <span class="n">AbstractRobustQueue</span> <span class="o">=</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="o">.</span><span class="n">declare_queue</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="n">_cm_request_queue</span><span class="p">,</span>
<span class="n">durable</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">exclusive</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">auto_delete</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># 1a. bind queue to CM_REQUEST_EXCHANGE w/o any conditions (empty routing key)</span>
<span class="k">await</span> <span class="n">_queue</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">exchange</span><span class="o">=</span><span class="n">CM_REQUEST_EXCHANGE</span><span class="p">,</span> <span class="n">routing_key</span><span class="o">=</span><span class="s2">&quot;&quot;</span><span class="p">)</span>
<span class="c1"># 1b. Service adapter mode - listens on Request queue and replies to Response queue</span>
<span class="n">_c_tag</span> <span class="o">=</span> <span class="k">await</span> <span class="n">_queue</span><span class="o">.</span><span class="n">consume</span><span class="p">(</span>
<span class="n">callback</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">handle_routing_data_request_message</span><span class="p">,</span> <span class="n">no_ack</span><span class="o">=</span><span class="kc">False</span>
<span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot; [</span><span class="si">%s</span><span class="s2">] RouterConsumer listens for RouteDataRequests on: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">_c_tag</span><span class="p">,</span>
<span class="n">_cm_request_queue</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">ioe</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;RouterConsumer FATAL: cannot initialize, </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">ioe</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;RouterConsumer FATAL: cannot initialize, RabbitMQ channel is not open.&quot;</span><span class="p">)</span></div>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> * ServiceMessageType.ROUTING_DATA_REQUEST handler listening on queue &#39;CleverMicroRequest&#39;</span>
<span class="sd"> &quot;&quot;&quot;</span>
<div class="viewcode-block" id="RouterConsumer.handle_routing_data_request_message">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_consumer.RouterConsumer.handle_routing_data_request_message">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">handle_routing_data_request_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">message</span><span class="p">:</span> <span class="n">AbstractIncomingMessage</span><span class="p">):</span>
<span class="c1"># some debug statements, TODO - remove for prod release</span>
<span class="n">delivery_tag</span> <span class="o">=</span> <span class="n">message</span><span class="o">.</span><span class="n">delivery_tag</span>
<span class="n">debug</span> <span class="o">=</span> <span class="s2">&quot;Delivery.Properties = &quot;</span> <span class="o">+</span> <span class="n">message</span><span class="o">.</span><span class="n">properties</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span>
<span class="n">logging_debug</span><span class="p">(</span>
<span class="s2">&quot; ******[</span><span class="si">%s</span><span class="s2">] (ROUTING_DATA_REQ) tag: </span><span class="si">%s</span><span class="s2">, env: </span><span class="si">%s</span><span class="s2">, props: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">delivery_tag</span><span class="p">,</span>
<span class="n">message</span><span class="o">.</span><span class="n">consumer_tag</span><span class="p">,</span>
<span class="n">message</span><span class="o">.</span><span class="n">properties</span><span class="p">,</span>
<span class="n">debug</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">await</span> <span class="n">message</span><span class="o">.</span><span class="n">ack</span><span class="p">(</span><span class="kc">False</span><span class="p">)</span>
<span class="n">cm_message</span><span class="p">:</span> <span class="n">ServiceMessage</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service_message_factory</span><span class="o">.</span><span class="n">from_bytes</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">)</span>
<span class="n">route_mapping</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">wrap_own_routes</span><span class="p">()</span>
<span class="n">logging_debug</span><span class="p">(</span>
<span class="s2">&quot;******[</span><span class="si">%s</span><span class="s2">] (ROUTING_DATA_REQ) msg=</span><span class="si">%s</span><span class="s2">, routeMap=[</span><span class="si">%s</span><span class="s2">]&quot;</span><span class="p">,</span>
<span class="n">delivery_tag</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">service_message_factory</span><span class="o">.</span><span class="n">to_string</span><span class="p">(</span><span class="n">cm_message</span><span class="p">),</span>
<span class="n">route_mapping</span><span class="p">,</span>
<span class="p">)</span>
<span class="c1"># Reply only if self adapter has own routeMapping (e.g. API-gateway-side adapter has none)</span>
<span class="k">if</span> <span class="p">(</span>
<span class="n">cm_message</span><span class="o">.</span><span class="n">is_valid</span><span class="p">()</span>
<span class="ow">and</span> <span class="n">cm_message</span><span class="o">.</span><span class="n">message_type</span> <span class="o">==</span> <span class="n">ServiceMessageType</span><span class="o">.</span><span class="n">ROUTING_DATA_REQUEST</span>
<span class="ow">and</span> <span class="n">route_mapping</span>
<span class="p">):</span>
<span class="k">if</span> <span class="n">cm_message</span><span class="o">.</span><span class="n">reply_to</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">amq_configuration</span><span class="o">.</span><span class="n">amq_adapter</span><span class="o">.</span><span class="n">service_name</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">service_message</span><span class="p">:</span> <span class="n">ServiceMessage</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service_message_factory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">ServiceMessageType</span><span class="o">.</span><span class="n">ROUTING_DATA</span><span class="p">,</span>
<span class="n">route_mapping</span><span class="p">,</span>
<span class="n">cm_message</span><span class="o">.</span><span class="n">reply_to</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">publish_service_message</span><span class="p">(</span><span class="n">service_message</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cm_response_exchange</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">err</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="s2">&quot;******[</span><span class="si">%s</span><span class="s2">] (ROUTING_DATA_REQ) Response to queue: </span><span class="si">%s</span><span class="s2"> failed: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span>
<span class="n">delivery_tag</span><span class="p">,</span>
<span class="n">CM_RESPONSE_EXCHANGE</span><span class="p">,</span>
<span class="n">err</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_info</span><span class="p">(</span>
<span class="s2">&quot;******[</span><span class="si">%s</span><span class="s2">] (ROUTING_DATA_REQ) [</span><span class="si">%s</span><span class="s2">] Ignore request originated by ourselves.&quot;</span><span class="p">,</span>
<span class="n">delivery_tag</span><span class="p">,</span>
<span class="nb">str</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">),</span>
<span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span>
<span class="s2">&quot;******[</span><span class="si">%s</span><span class="s2">] (ROUTING_DATA_REQ) [</span><span class="si">%s</span><span class="s2">] produces invalid or no CM msg type.&quot;</span><span class="p">,</span>
<span class="n">delivery_tag</span><span class="p">,</span>
<span class="nb">str</span><span class="p">(</span><span class="n">message</span><span class="o">.</span><span class="n">body</span><span class="p">),</span>
<span class="p">)</span></div>
<div class="viewcode-block" id="RouterConsumer.register_route">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_consumer.RouterConsumer.register_route">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">register_route</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route</span><span class="p">:</span> <span class="n">AMQRoute</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> * Called in Consumer (Service Adapter) context. Registers the routing data for self Service</span>
<span class="sd"> * with the Master router by sending own routing data to Master via CleverMicroResponse queue.</span>
<span class="sd"> * :param route: route to register with Master</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">is_open</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">)</span> <span class="ow">and</span> <span class="n">route</span><span class="o">.</span><span class="n">queue</span><span class="p">:</span>
<span class="n">serviceMessage</span><span class="p">:</span> <span class="n">ServiceMessage</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service_message_factory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">ServiceMessageType</span><span class="o">.</span><span class="n">ROUTING_DATA</span><span class="p">,</span> <span class="n">route</span><span class="o">.</span><span class="n">as_string</span><span class="p">(),</span> <span class="n">ANY_RECIPIENT</span>
<span class="p">)</span>
<span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">publish_service_message</span><span class="p">(</span><span class="n">serviceMessage</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cm_response_exchange</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">add_own_route</span><span class="p">(</span><span class="n">route</span><span class="p">)</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RouterConsumer registered Route: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">route</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logging_warning</span><span class="p">(</span>
<span class="s2">&quot;RouterConsumer couldn&#39;t register route data: </span><span class="si">%s</span><span class="s2"> channel isn&#39;t open.&quot;</span><span class="p">,</span>
<span class="n">CM_RESPONSE_EXCHANGE</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">ioe</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;RouterConsumer failed register route data: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">ioe</span><span class="p">)</span></div>
<div class="viewcode-block" id="RouterConsumer.remove_route_on_cleanup">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_consumer.RouterConsumer.remove_route_on_cleanup">[docs]</a>
<span class="k">async</span> <span class="k">def</span><span class="w"> </span><span class="nf">remove_route_on_cleanup</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">route</span><span class="p">:</span> <span class="n">AMQRoute</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> * Unregister / remove route from the list</span>
<span class="sd"> * :param route: route to remove</span>
<span class="sd"> * :return result of remove operation</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RouterConsumer.removeRoute, route: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">route</span><span class="o">.</span><span class="n">as_string</span><span class="p">())</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">service_message</span><span class="p">:</span> <span class="n">ServiceMessage</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">service_message_factory</span><span class="o">.</span><span class="n">of</span><span class="p">(</span>
<span class="n">ServiceMessageType</span><span class="o">.</span><span class="n">ROUTING_DATA_REMOVE</span><span class="p">,</span> <span class="n">route</span><span class="o">.</span><span class="n">as_string</span><span class="p">(),</span> <span class="n">ANY_RECIPIENT</span>
<span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="k">await</span> <span class="bp">self</span><span class="o">.</span><span class="n">publish_service_message</span><span class="p">(</span><span class="n">service_message</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">cm_response_exchange</span><span class="p">):</span>
<span class="n">logging_warning</span><span class="p">(</span>
<span class="s2">&quot;RouterConsumer couldn&#39;t remove current_availability route: </span><span class="si">%s</span><span class="s2">, channel not open.&quot;</span><span class="p">,</span>
<span class="n">route</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">ioe</span><span class="p">:</span>
<span class="n">logging_error</span><span class="p">(</span><span class="s2">&quot;RouterConsumer failed remove current_availability route: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="n">ioe</span><span class="p">)</span>
<span class="n">result</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">remove_route</span><span class="p">(</span><span class="n">route</span><span class="p">)</span>
<span class="k">return</span> <span class="n">result</span></div>
<div class="viewcode-block" id="RouterConsumer.cleanup">
<a class="viewcode-back" href="../../../modules/amqp.router.html#amqp.router.router_consumer.RouterConsumer.cleanup">[docs]</a>
<span class="k">def</span><span class="w"> </span><span class="nf">cleanup</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> * Cleanup - de-register routes with master.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">routes</span><span class="p">:</span> <span class="n">Set</span><span class="p">[</span><span class="n">AMQRoute</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_routes</span><span class="p">()</span>
<span class="n">logging_info</span><span class="p">(</span><span class="s2">&quot;RouterConsumer.cleanup() [</span><span class="si">%d</span><span class="s2">]: </span><span class="si">%s</span><span class="s2">&quot;</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">routes</span><span class="p">),</span> <span class="bp">self</span><span class="o">.</span><span class="n">wrap_routes</span><span class="p">())</span>
<span class="n">not_removed</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span>
<span class="mi">1</span> <span class="k">for</span> <span class="n">r</span> <span class="ow">in</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">remove_route_on_cleanup</span><span class="p">(</span><span class="n">route</span><span class="p">)</span> <span class="k">for</span> <span class="n">route</span> <span class="ow">in</span> <span class="n">routes</span><span class="p">)</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">r</span>
<span class="p">)</span>
<span class="k">if</span> <span class="n">not_removed</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">logging_warning</span><span class="p">(</span>
<span class="s2">&quot;</span><span class="si">%d</span><span class="s2"> route(s) not removed. self may lead to failed delivery on self route&quot;</span><span class="p">,</span>
<span class="n">not_removed</span><span class="p">,</span>
<span class="p">)</span></div>
</div>
</pre></div>
</div>
</div>
<footer>
<hr/>
<div role="contentinfo">
<p>&#169; Copyright 2024, CleverThis.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>

Some files were not shown because too many files have changed in this diff Show More