cleveragents-core/features/async_execution.feature

Feature: Async Job Execution and Workers
  As a developer
  I want plan phases to execute as background jobs
  So that long-running operations don't block the CLI

  # ---- AsyncJob Domain Model ----

  Scenario: Create an async job with default values
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    Then the async job status should be "queued"
    And the async job phase should be "execute"
    And the async job schema_version should be 1

  Scenario: AsyncJob status transitions follow the state machine
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    When I mark the async job as running with worker "worker-001"
    Then the async job status should be "running"
    And the async job worker_id should be "worker-001"
    And the async job started_at should be set

  Scenario: AsyncJob transition from running to succeeded
    Given an async job store is initialised
    And I create a running async job
    When I mark the async job as succeeded
    Then the async job status should be "succeeded"
    And the async job finished_at should be set

  Scenario: AsyncJob transition from running to failed
    Given an async job store is initialised
    And I create a running async job
    When I mark the async job as failed
    Then the async job status should be "failed"
    And the async job finished_at should be set

  Scenario: AsyncJob transition from queued to cancelled
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    When I mark the async job as cancelled
    Then the async job status should be "cancelled"
    And the async job finished_at should be set

  Scenario: AsyncJob transition from running to cancelled
    Given an async job store is initialised
    And I create a running async job
    When I mark the async job as cancelled
    Then the async job status should be "cancelled"

  Scenario: Invalid transition from queued to succeeded raises error
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    When I try to mark the async job as succeeded
    Then an InvalidJobTransitionError should be raised

  Scenario: Invalid transition from succeeded to running raises error
    Given an async job store is initialised
    And I create a running async job
    And I mark the async job as succeeded
    When I try to mark the async job as running with worker "worker-002"
    Then an InvalidJobTransitionError should be raised

  Scenario: Invalid transition from failed raises error
    Given an async job store is initialised
    And I create a running async job
    And I mark the async job as failed
    When I try to mark the async job as succeeded
    Then an InvalidJobTransitionError should be raised

  # ---- Worker Pickup and Execution ----

  Scenario: Worker picks up and executes a queued job
    Given an async worker is initialised with max_workers 2
    And a queued async job exists
    When the worker picks up and executes the job
    Then the async job status should be "succeeded"
    And the worker health should show 1 job processed

  Scenario: Worker marks job as failed on execution error
    Given an async worker is initialised with a failing executor
    And a queued async job exists
    When the worker picks up and executes the job
    Then the async job status should be "failed"
    And the worker health should show 1 job failed

  # ---- Job Cancellation ----

  Scenario: Cancel a running job via cancellation token
    Given an async worker is initialised with max_workers 2
    And a queued async job exists
    When I cancel the async job
    Then the async job status should be "cancelled"

  # ---- Stuck Job Detection ----

  Scenario: Detect and fail stuck jobs with expired heartbeat
    Given an async worker is initialised with job_timeout 1
    And a running async job with expired heartbeat exists
    When the worker detects stuck jobs
    Then the stuck job should be marked as "failed"

  # ---- Job TTL and Cleanup ----

  Scenario: Cleanup completed jobs older than retention TTL
    Given an async worker is initialised with job_ttl 1
    And a completed async job older than TTL exists
    When the worker runs job cleanup
    Then the completed job should be removed from the store

  # ---- Graceful Shutdown ----

  Scenario: Worker stops gracefully on shutdown request
    Given an async worker is initialised with max_workers 2
    When the worker is started
    And a shutdown is requested
    Then the worker should stop running

  # ---- Payload Serialization ----

  Scenario: Serialize and deserialize job payload with schema version
    When I serialize a job payload for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    Then the serialized payload should contain schema_version 1
    And the deserialized payload should match the original

  Scenario: Serialize payload with extra data
    When I serialize a job payload with extra data
    Then the serialized payload should contain the extra data

  Scenario: Deserialize invalid JSON raises error
    When I try to deserialize invalid JSON
    Then a ValueError should be raised for async deserialize

  # ---- Worker Health Report ----

  Scenario: Worker health report contains expected fields
    Given an async worker is initialised with max_workers 4
    When I request the worker health report
    Then the health report should contain worker_id
    And the health report should contain config details

  # ---- AsyncWorkerConfig Validation ----

  Scenario: AsyncWorkerConfig rejects invalid max_workers
    When I try to create AsyncWorkerConfig with max_workers 0
    Then a ValueError should be raised for config

  Scenario: AsyncWorkerConfig rejects invalid poll_interval
    When I try to create AsyncWorkerConfig with poll_interval 0
    Then a ValueError should be raised for config

  # ---- InMemoryJobStore ----

  Scenario: Job store rejects duplicate job IDs
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    When I try to add the same job again
    Then a ValueError should be raised for duplicate job

  Scenario: Job store lists jobs by status
    Given an async job store is initialised
    And I create 3 queued async jobs
    When I list jobs by status "queued"
    Then the job list should have 3 entries

  Scenario: AsyncJob heartbeat recording
    Given an async job store is initialised
    And I create a running async job
    When I record a heartbeat on the async job
    Then the async job last_heartbeat should be updated

  Scenario: Heartbeat on non-running job raises error
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    When I try to record a heartbeat on the async job
    Then a ValueError should be raised for heartbeat

  Scenario: AsyncJob is_terminal property
    Given an async job store is initialised
    And I create a running async job
    When I mark the async job as succeeded
    Then the async job should be terminal

  Scenario: AsyncJob is_running property
    Given an async job store is initialised
    And I create a running async job
    Then the async job should be running

  Scenario: AsyncJob as_cli_dict returns expected keys
    Given an async job store is initialised
    And I create a running async job
    When I get the async job CLI dict
    Then the async CLI dict should contain key "job_id"
    And the async CLI dict should contain key "status"
    And the async CLI dict should contain key "worker_id"

  Scenario: Invalid phase raises validation error
    When I try to create an async job with phase "invalid"
    Then a validation error should be raised for phase

  Scenario: Invalid payload JSON raises validation error
    When I try to create an async job with invalid payload JSON
    Then a validation error should be raised for payload

  Scenario: Mark running with empty worker_id raises error
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    When I try to mark the async job as running with an empty worker_id
    Then a ValueError should be raised for worker_id

  Scenario: CancellationToken cooperative cancellation
    Given a cancellation token is created
    When I cancel the async token
    Then the token should be cancelled
    And the token wait should return True

  Scenario: CancellationToken reset
    Given a cancellation token is created
    And I cancel the async token
    When I reset the token
    Then the token should not be cancelled

  Scenario: WorkerHealthReport tracks jobs
    Given a worker health report for "test-worker"
    When I record 3 completed jobs and 1 failed job
    Then the health report should show 3 processed and 1 failed

  Scenario: Serialize payload with empty plan_id raises error
    When I try to serialize a payload with empty plan_id
    Then a ValueError should be raised for serialize

  Scenario: Deserialize empty payload raises error
    When I try to deserialize an empty payload
    Then a ValueError should be raised for deserialize

  Scenario: can_transition_job with invalid types raises TypeError
    When I call can_transition_job with invalid types
    Then a TypeError should be raised for async transition

  # ---- Additional Coverage: Validation Guards ----

  Scenario: WorkerHealthReport with empty worker_id raises error
    When I try to create a WorkerHealthReport with empty worker_id
    Then a ValueError should be raised for empty worker_id

  Scenario: AsyncWorkerConfig with invalid enabled type raises error
    When I try to create AsyncWorkerConfig with non-bool enabled
    Then a TypeError should be raised for async config enabled

  Scenario: AsyncWorkerConfig with invalid job_timeout raises error
    When I try to create AsyncWorkerConfig with job_timeout 0
    Then a ValueError should be raised for async config validation

  Scenario: AsyncWorkerConfig with invalid job_ttl raises error
    When I try to create AsyncWorkerConfig with job_ttl 0
    Then a ValueError should be raised for async config validation

  Scenario: InMemoryJobStore add with non-AsyncJob raises error
    Given an async job store is initialised
    When I try to add a non-AsyncJob to the store
    Then a TypeError should be raised for async store type

  Scenario: InMemoryJobStore get with empty job_id raises error
    Given an async job store is initialised
    When I try to get a job with empty job_id
    Then a ValueError should be raised for async store get

  Scenario: InMemoryJobStore list_by_status with invalid type raises error
    Given an async job store is initialised
    When I try to list jobs by invalid status type
    Then a TypeError should be raised for async store type

  Scenario: InMemoryJobStore update with non-AsyncJob raises error
    Given an async job store is initialised
    When I try to update with a non-AsyncJob
    Then a TypeError should be raised for async store type

  Scenario: InMemoryJobStore update with non-existent job raises error
    Given an async job store is initialised
    When I try to update a non-existent job
    Then a ValueError should be raised for async store update

  Scenario: InMemoryJobStore remove with empty job_id raises error
    Given an async job store is initialised
    When I try to remove a job with empty job_id
    Then a ValueError should be raised for async store remove

  Scenario: AsyncWorker with invalid config type raises error
    When I try to create AsyncWorker with invalid config
    Then a TypeError should be raised for async worker config

  Scenario: AsyncWorker with invalid job_store type raises error
    When I try to create AsyncWorker with invalid job_store
    Then a TypeError should be raised for async worker store

  Scenario: AsyncWorker pickup with non-AsyncJob raises error
    Given an async worker is initialised with max_workers 2
    When I try to pickup a non-AsyncJob
    Then a TypeError should be raised for async worker pickup

  Scenario: AsyncWorker cancel_job with empty job_id raises error
    Given an async worker is initialised with max_workers 2
    When I try to cancel a job with empty job_id
    Then a ValueError should be raised for async worker cancel

  Scenario: AsyncWorker worker_id property returns value
    Given an async worker is initialised with max_workers 2
    Then the worker should have a worker_id starting with "worker-"

  Scenario: AsyncWorker config property returns config
    Given an async worker is initialised with max_workers 2
    Then the worker config max_workers should be 2

  Scenario: AsyncWorker start when already running does nothing
    Given an async worker is initialised with max_workers 2
    When I start the worker
    And I start the worker again
    Then the worker should still be running

  Scenario: AsyncWorker stop after start stops cleanly
    Given an async worker is initialised with max_workers 2
    When I start the worker
    And I stop the worker
    Then the worker should not be running

  Scenario: AsyncWorker stop when not running does nothing
    Given an async worker is initialised with max_workers 2
    When I stop the worker
    Then the worker should not be running

  Scenario: AsyncWorker cancel job that does not exist returns false
    Given an async worker is initialised with max_workers 2
    Then cancelling a non-existent job should return false

  Scenario: AsyncJob payload property deserializes JSON
    Given an async job store is initialised
    And I create a new async job for plan "01HXYZ01HXYZ01HXYZ01HXYZ01" phase "execute"
    Then the async job payload property should return a dict

  Scenario: AsyncJob validate_payload_json rejects invalid JSON
    When I try to create an async job with payload_json "not-json{"
    Then a validation error should be raised for payload

  Scenario: AsyncJob as_cli_dict includes optional fields when set
    Given an async job store is initialised
    And I create a running async job
    And I mark the async job as succeeded
    When I get the async job CLI dict
    Then the async CLI dict should contain key "started_at"
    And the async CLI dict should contain key "finished_at"

  # ---- Review Fixes: Concurrent Execution (T2) ----

  Scenario: Worker executes multiple jobs concurrently
    Given an async worker is initialised with max_workers 4
    And 3 queued async jobs exist
    When the worker executes all queued jobs concurrently
    Then all 3 jobs should be in a terminal state
    And the worker health should show 3 jobs processed total

  # ---- Review Fixes: Cancel records cancelled counter (B2) ----

  Scenario: Cancelled job increments cancelled counter not processed counter
    Given an async worker is initialised with max_workers 2
    And a queued async job exists
    When I cancel the async job
    Then the async job status should be "cancelled"
    And the worker health should show 0 jobs processed
    And the worker health should show 0 jobs failed

  # ---- Review Fixes: Error message audit trail (SEC1) ----

  Scenario: Failed job records error message for audit trail
    Given an async worker is initialised with a failing executor
    And a queued async job exists
    When the worker picks up and executes the job
    Then the async job status should be "failed"
    And the async job should have an error_message set

  # ---- Review Fixes: DB model round-trip (T1) ----

  Scenario: AsyncJobModel round-trip through SQLAlchemy
    When I create and persist an AsyncJobModel to the database
    Then the AsyncJobModel should be retrievable with correct fields
    And the database CHECK constraint should reject invalid status

  # ---- Coverage: Safe Initialisation and Cleanup ----

  Scenario: Worker start creates thread pool and stop destroys it cleanly
    Given an async worker is initialised with max_workers 2
    And a queued async job exists
    When I start the worker
    And the worker processes all queued jobs within 3 seconds
    And I stop the worker
    Then the worker should not be running
    And the async job status should be "succeeded"

  Scenario: Worker poll loop dispatches jobs via thread pool
    Given an async worker is initialised with max_workers 4
    And 2 queued async jobs exist
    When I start the worker
    And all created jobs reach a terminal state within 3 seconds
    And I stop the worker
    Then all 2 jobs should be in a terminal state
    And the worker health should show 2 jobs processed total

  Scenario: Worker stop cancels in-flight jobs via tokens
    Given an async worker is initialised with a blocking executor
    And a queued async job exists
    When I start the worker
    And the blocking executor has been entered
    And I stop the worker
    Then the cancellation token for the job should be signalled

  # ---- Coverage: Cancellation Paths ----

  Scenario: Running job cancelled via token records cancelled count
    Given an async worker is initialised with a blocking executor
    And a queued async job exists
    When I dispatch the job in a background thread
    And the blocking executor has been entered
    And I cancel the running async job via the worker
    And the background job thread completes
    Then the async job status should be "cancelled"
    And the worker health should show 1 jobs cancelled
    And the worker health should show 0 jobs processed

  Scenario: Pickup and execute skips already-terminal job safely
    Given an async worker is initialised with max_workers 2
    And a queued async job exists
    And the async job is already cancelled
    When the worker picks up and executes the job
    Then the async job status should be "cancelled"
    And the worker health should show 0 jobs processed

  Scenario: Executor that moves job to terminal during execution is handled
    Given an async worker is initialised with a terminal-forcing executor
    And a queued async job exists
    When the worker picks up and executes the job
    Then the async job should be in a terminal state
    And the worker health should show 0 jobs processed

  # ---- Coverage: Job Store Operations ----

  Scenario: Job store list_all returns all jobs regardless of status
    Given an async job store is initialised
    And I add 3 jobs in mixed statuses to the store
    When I call list_all on the job store
    Then the store should return 3 jobs
    And the store count should be 3

  Scenario: Job store remove deletes a job and count reflects it
    Given an async job store is initialised
    And a queued async job exists in the store
    When I remove the job from the store
    Then the store count should be 0

  # ---- Coverage: Domain Model Validators ----

  Scenario: transition_to rejects non-AsyncJobStatus target
    Given a queued async job exists standalone
    When I try to transition_to with a string instead of enum
    Then an async TypeError should be raised

  Scenario: can_transition_job rejects non-enum to_status argument
    When I call can_transition_job with a string to_status
    Then an async TypeError should be raised

  Scenario: serialize_job_payload rejects empty phase
    When I try to serialize a payload with an empty phase
    Then an async ValueError should be raised matching "phase"

  Scenario: serialize_job_payload rejects zero schema version
    When I try to serialize a payload with schema_version 0
    Then an async ValueError should be raised matching "schema_version"

  Scenario: deserialize_job_payload rejects non-object JSON
    When I try to deserialize the JSON string "[1,2,3]"
    Then an async ValueError should be raised matching "JSON object"

  Scenario: Pickup handles already-running job gracefully
    Given an async worker is initialised with max_workers 2
    And a queued async job exists
    And the async job is already running with worker "other-worker"
    When the worker picks up and executes the job
    Then the async job status should be "running"
    And the worker health should show 0 jobs processed

  Scenario: Executor exception on already-terminal job skips mark_failed
    Given an async worker is initialised with a crash-after-terminal executor
    And a queued async job exists
    When the worker picks up and executes the job
    Then the async job should be in a terminal state

  Scenario: detect_stuck_jobs tolerates already-terminal jobs
    Given an async worker is initialised with job_timeout 1
    And a running async job with expired heartbeat exists
    And the running async job has already been cancelled externally
    When the worker detects stuck jobs
    Then no additional stuck jobs should be reported

  Scenario: CLI dict includes error_message for failed job
    Given a queued async job exists standalone
    When I mark the async job as running with worker "w-err"
    And I mark the async job as failed with error "SomeError: detail"
    And I get the async job CLI dict
    Then the async CLI dict should contain key "error_message"
    And the async CLI dict error_message should be "SomeError: detail"

  # ---- Review Fix: Plan lifecycle async wiring (P1) ----

  Scenario: execute_plan enqueues async job when async is enabled
    Given a plan lifecycle service with async enabled and a job store
    And a plan in Strategize/COMPLETE state exists
    When I execute the plan via the lifecycle service
    Then the plan should be in Execute/QUEUED state
    And the job store should contain 1 async job for phase "execute"
    And the enqueued job plan_id should match the plan

  Scenario: apply_plan enqueues async job when async is enabled
    Given a plan lifecycle service with async enabled and a job store
    And a plan in Execute/COMPLETE state exists
    When I apply the plan via the lifecycle service
    Then the plan should be in Apply/QUEUED state
    And the job store should contain 1 async job for phase "apply"
    And the enqueued job plan_id should match the plan

  Scenario: execute_plan does not enqueue when async is disabled
    Given a plan lifecycle service with async disabled
    And a plan in Strategize/COMPLETE state exists
    When I execute the plan via the lifecycle service
    Then the plan should be in Execute/QUEUED state
    And no async jobs should be in the job store

  Scenario: apply_plan does not enqueue when async is disabled
    Given a plan lifecycle service with async disabled
    And a plan in Execute/COMPLETE state exists
    When I apply the plan via the lifecycle service
    Then the plan should be in Apply/QUEUED state
    And no async jobs should be in the job store

  Scenario: execute_plan does not enqueue when no job store is configured
    Given a plan lifecycle service with async enabled but no job store
    And a plan in Strategize/COMPLETE state exists
    When I execute the plan via the lifecycle service
    Then the plan should be in Execute/QUEUED state

  # ---- Review Fix: Error message redaction (P2) ----

  Scenario: Failed job error message redacts secrets
    Given an async worker is initialised with a secret-leaking executor
    And a queued async job exists
    When the worker picks up and executes the job
    Then the async job status should be "failed"
    And the async job error_message should not contain the raw secret
    And the async job error_message should contain "***REDACTED***"