freemo
2d345ae31c
CI / build (push) Waiting to run
CI / lint (push) Waiting to run
CI / typecheck (push) Waiting to run
CI / security (push) Waiting to run
CI / quality (push) Waiting to run
CI / unit_tests (push) Waiting to run
CI / integration_tests (push) Waiting to run
CI / coverage (push) Blocked by required conditions
CI / docker (push) Blocked by required conditions
CI / lint (pull_request) Failing after 14s
CI / typecheck (pull_request) Successful in 27s
CI / coverage (pull_request) Has been skipped
CI / security (pull_request) Successful in 22s
CI / quality (pull_request) Successful in 15s
CI / integration_tests (pull_request) Failing after 4m37s
CI / build (pull_request) Successful in 16s
CI / unit_tests (pull_request) Successful in 10m3s
CI / docker (pull_request) Has been skipped
Implement the four-stage tool lifecycle (discover/activate/execute/deactivate) with capability enforcement, per-plan activation caching, JSON Schema validation, execution tracing, and cancellation propagation. New modules: - tool/context.py: ToolExecutionContext, BoundResource, Change, CancellationToken - tool/lifecycle.py: ToolRuntime, ToolInstance protocol, ToolLifecycleCache - tool/schema_validator.py: JSON Schema input/output validation Tests: 57 Behave scenarios, 6 Robot smoke tests, 5 ASV benchmarks Coverage: context 100%, schema_validator 100%, lifecycle 94%, overall 97%
380 lines
19 KiB
Gherkin
380 lines
19 KiB
Gherkin
Feature: Tool lifecycle runtime
|
|
As the tool execution engine
|
|
I need to orchestrate discover/activate/execute/deactivate lifecycle
|
|
With capability enforcement, schema validation, caching, and cancellation
|
|
|
|
# ── ToolExecutionContext ──────────────────────────────────────────────
|
|
|
|
Scenario: Create a tool execution context with defaults
|
|
Given I create a tool execution context with plan_id "plan-001"
|
|
Then the context plan_id should be "plan-001"
|
|
And the context should not be read-only
|
|
And the context should not require checkpoints
|
|
And the context should have 0 changes
|
|
And the context should have 0 traces
|
|
And the context cancellation token should not be cancelled
|
|
|
|
Scenario: Create a read-only execution context
|
|
Given I create a tool execution context with plan_id "plan-002" and read_only True
|
|
Then the context should be read-only
|
|
|
|
Scenario: Create a checkpoint-required execution context
|
|
Given I create a tool execution context with plan_id "plan-003" and require_checkpoints True
|
|
Then the context should require checkpoints
|
|
|
|
Scenario: Record changes in execution context
|
|
Given I create a tool execution context with plan_id "plan-004"
|
|
When I record a change with operation "create" and resource_id "res-001"
|
|
Then the context should have 1 changes
|
|
|
|
Scenario: Add traces to execution context
|
|
Given I create a tool execution context with plan_id "plan-005"
|
|
When I add a trace for tool "test/tool" with success True
|
|
Then the context should have 1 traces
|
|
|
|
Scenario: Get resource from context by slot name
|
|
Given I create a tool execution context with plan_id "plan-006"
|
|
And I bind resource "res-001" to slot "repo" with type "git-checkout"
|
|
When I get the resource for slot "repo"
|
|
Then the bound resource_id should be "res-001"
|
|
|
|
Scenario: Get resource from context with missing slot raises KeyError
|
|
Given I create a tool execution context with plan_id "plan-007"
|
|
When I try to get the resource for slot "missing"
|
|
Then a KeyError should be raised with message containing "missing"
|
|
|
|
Scenario: Context summary includes correct counts
|
|
Given I create a tool execution context with plan_id "plan-008"
|
|
And I bind resource "res-001" to slot "repo" with type "git-checkout"
|
|
When I record a change with operation "modify" and resource_id "res-001"
|
|
And I add a trace for tool "test/tool" with success True
|
|
Then the context summary should show resource_count 1
|
|
And the context summary should show change_count 1
|
|
And the context summary should show trace_count 1
|
|
|
|
# ── BoundResource ────────────────────────────────────────────────────
|
|
|
|
Scenario: Create a BoundResource with all fields
|
|
Given I create a bound resource with slot "db" and resource_id "res-db-001" and type "postgresql"
|
|
Then the bound resource slot_name should be "db"
|
|
And the bound resource access should be "read_only"
|
|
|
|
# ── CancellationToken ────────────────────────────────────────────────
|
|
|
|
Scenario: Cancellation token starts not cancelled
|
|
Given I create a cancellation token
|
|
Then the cancellation token should not be cancelled
|
|
|
|
Scenario: Cancel sets the cancellation flag
|
|
Given I create a cancellation token
|
|
When I cancel the token
|
|
Then the cancellation token should be cancelled
|
|
|
|
Scenario: Check raises ToolCancelledError when cancelled
|
|
Given I create a cancellation token
|
|
When I cancel the token
|
|
Then calling check on the token should raise ToolCancelledError
|
|
|
|
# ── Change tracking ──────────────────────────────────────────────────
|
|
|
|
Scenario: Create a change with all operations
|
|
Given I create a change with operation "create" and resource_id "res-001"
|
|
Then the change operation should be "create"
|
|
And the change resource_id should be "res-001"
|
|
|
|
Scenario: Change has automatic timestamp
|
|
Given I create a change with operation "delete" and resource_id "res-002"
|
|
Then the change timestamp should be a valid ISO-8601 string
|
|
|
|
# ── ToolDescriptor ───────────────────────────────────────────────────
|
|
|
|
Scenario: Create a tool descriptor
|
|
Given I create a tool descriptor with name "builtin/read-file" and description "Read a file"
|
|
Then the descriptor name should be "builtin/read-file"
|
|
And the descriptor source should be "builtin"
|
|
|
|
# ── ToolResult ───────────────────────────────────────────────────────
|
|
|
|
Scenario: Create a successful tool result
|
|
Given I create a tool result with success True and data "hello"
|
|
Then the tool result should indicate success
|
|
And the tool result data should be "hello"
|
|
|
|
Scenario: Create a failed tool result
|
|
Given I create a tool result with success False and error "something broke"
|
|
Then the tool result should indicate failure
|
|
And the tool result error should be "something broke"
|
|
|
|
# ── JSON Schema validation ───────────────────────────────────────────
|
|
|
|
Scenario: Validate tool input against valid schema
|
|
Given I have a JSON schema requiring property "path" of type "string"
|
|
When I validate input with path "test.txt" against the schema
|
|
Then the input validation should succeed
|
|
|
|
Scenario: Validate tool input against invalid data
|
|
Given I have a JSON schema requiring property "path" of type "string"
|
|
When I validate input with path 42 against the schema
|
|
Then the input validation should fail with ToolSchemaValidationError
|
|
|
|
Scenario: Validate tool output against valid schema
|
|
Given I have a JSON schema requiring property "content" of type "string"
|
|
When I validate output with content "file data" against the schema
|
|
Then the output validation should succeed
|
|
|
|
Scenario: Validate tool output against invalid data
|
|
Given I have a JSON schema requiring property "content" of type "string"
|
|
When I validate output with content 123 against the schema
|
|
Then the output validation should fail with ToolSchemaValidationError
|
|
|
|
Scenario: Schema validation error contains error details
|
|
Given I have a JSON schema requiring property "path" of type "string"
|
|
When I validate input with path 42 against the schema
|
|
Then the schema validation error should have errors list
|
|
And the schema validation error should reference the schema
|
|
|
|
# ── ToolLifecycleCache ───────────────────────────────────────────────
|
|
|
|
Scenario: Cache starts empty
|
|
Given I create a tool lifecycle cache
|
|
Then the cache should have 0 plans
|
|
|
|
Scenario: Put and get an instance from cache
|
|
Given I create a tool lifecycle cache
|
|
And I have a mock tool instance named "builtin/read"
|
|
When I put the instance in cache for plan "plan-001" and tool "builtin/read"
|
|
Then getting the instance for plan "plan-001" and tool "builtin/read" should return it
|
|
|
|
Scenario: Get returns None for uncached tool
|
|
Given I create a tool lifecycle cache
|
|
Then getting the instance for plan "plan-001" and tool "builtin/missing" should return None
|
|
|
|
Scenario: Increment execution count
|
|
Given I create a tool lifecycle cache
|
|
And I have a mock tool instance named "builtin/read"
|
|
When I put the instance in cache for plan "plan-001" and tool "builtin/read"
|
|
And I increment execution for plan "plan-001" and tool "builtin/read"
|
|
Then the cache stats for plan "plan-001" should show tool "builtin/read" with execution_count 1
|
|
|
|
Scenario: Get plan tools lists activated tools
|
|
Given I create a tool lifecycle cache
|
|
And I have a mock tool instance named "builtin/read"
|
|
And I have a mock tool instance named "builtin/write"
|
|
When I put the instance in cache for plan "plan-001" and tool "builtin/read"
|
|
And I put the second instance in cache for plan "plan-001" and tool "builtin/write"
|
|
Then the plan tools for plan "plan-001" should include "builtin/read"
|
|
And the plan tools for plan "plan-001" should include "builtin/write"
|
|
|
|
Scenario: Remove a single tool from cache
|
|
Given I create a tool lifecycle cache
|
|
And I have a mock tool instance named "builtin/read"
|
|
When I put the instance in cache for plan "plan-001" and tool "builtin/read"
|
|
And I remove tool "builtin/read" from plan "plan-001"
|
|
Then getting the instance for plan "plan-001" and tool "builtin/read" should return None
|
|
|
|
Scenario: Remove plan clears all tools
|
|
Given I create a tool lifecycle cache
|
|
And I have a mock tool instance named "builtin/read"
|
|
And I have a mock tool instance named "builtin/write"
|
|
When I put the instance in cache for plan "plan-001" and tool "builtin/read"
|
|
And I put the second instance in cache for plan "plan-001" and tool "builtin/write"
|
|
And I remove plan "plan-001" from cache
|
|
Then the cache should have 0 plans
|
|
|
|
# ── ToolRuntime ──────────────────────────────────────────────────────
|
|
|
|
Scenario: Register and list tools
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance
|
|
Then the tool list should contain "builtin/read-file"
|
|
|
|
Scenario: Unregister a tool
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance
|
|
When I unregister tool "builtin/read-file"
|
|
Then the tool list should not contain "builtin/read-file"
|
|
|
|
Scenario: Get a registered tool
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance
|
|
Then getting tool "builtin/read-file" should return the tool model
|
|
|
|
Scenario: Get an unregistered tool returns None
|
|
Given I create a tool runtime
|
|
Then getting tool "builtin/missing" should return None
|
|
|
|
Scenario: Discover returns a ToolDescriptor
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance
|
|
When I discover tool "builtin/read-file"
|
|
Then the discovered descriptor name should be "builtin/read-file"
|
|
|
|
Scenario: Activate a tool for a plan
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-010"
|
|
When I activate tool "builtin/read-file" in the runtime
|
|
Then the mock instance should have been activated
|
|
|
|
Scenario: Activate is idempotent for same plan
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-011"
|
|
When I activate tool "builtin/read-file" in the runtime
|
|
And I activate tool "builtin/read-file" in the runtime again
|
|
Then the mock instance activate count should be 1
|
|
|
|
Scenario: Execute a tool successfully
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-012"
|
|
When I execute tool "builtin/read-file" with params path "test.txt"
|
|
Then the execution result should be successful
|
|
And the context should have 1 traces
|
|
|
|
Scenario: Execute a tool with input schema validation failure
|
|
Given I create a tool runtime
|
|
And I register a builtin tool with input schema "builtin/typed-tool"
|
|
And I create a tool execution context with plan_id "plan-013"
|
|
When I execute tool "builtin/typed-tool" with invalid params
|
|
Then the execution result should not be successful
|
|
And the execution result error should contain "Input validation failed"
|
|
|
|
Scenario: Read-only plan rejects tool with writes
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/write-file" that writes
|
|
And I create a tool execution context with plan_id "plan-014" and read_only True
|
|
When I try to execute tool "builtin/write-file"
|
|
Then a ToolAccessDeniedError should be raised
|
|
|
|
Scenario: Checkpoint-required plan rejects non-checkpointable tool
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-015" and require_checkpoints True
|
|
When I try to activate tool "builtin/read-file" for checkpoint plan
|
|
Then a ToolCheckpointRequiredError should be raised
|
|
|
|
Scenario: Execute records changes from tool result
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/edit-file" that produces changes
|
|
And I create a tool execution context with plan_id "plan-016"
|
|
When I execute tool "builtin/edit-file" with params path "test.txt"
|
|
Then the context should have 1 changes
|
|
|
|
Scenario: Execute with cancelled token raises ToolCancelledError
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-017"
|
|
And I cancel the context cancellation token
|
|
When I try to execute tool "builtin/read-file" after cancellation
|
|
Then a ToolCancelledError should be raised from execution
|
|
|
|
Scenario: Deactivate a single tool
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-018"
|
|
When I activate tool "builtin/read-file" in the runtime
|
|
And I deactivate tool "builtin/read-file" in the runtime
|
|
Then the mock instance should have been deactivated
|
|
|
|
Scenario: Deactivate plan clears all activated tools
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I register a second builtin tool "builtin/list-files" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-019"
|
|
When I activate tool "builtin/read-file" in the runtime
|
|
And I activate tool "builtin/list-files" in the runtime
|
|
And I deactivate plan "plan-019" in the runtime
|
|
Then all mock instances should have been deactivated
|
|
|
|
Scenario: Execute unregistered tool raises ToolRuntimeError
|
|
Given I create a tool runtime
|
|
And I create a tool execution context with plan_id "plan-020"
|
|
When I try to execute unregistered tool "builtin/ghost"
|
|
Then a ToolRuntimeError should be raised with message containing "not registered"
|
|
|
|
Scenario: Cache stats after execution
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-021"
|
|
When I execute tool "builtin/read-file" with params path "test.txt"
|
|
Then the cache stats for plan "plan-021" should show 1 active tools
|
|
|
|
Scenario: Output schema validation failure returns error result
|
|
Given I create a tool runtime
|
|
And I register a builtin tool with output schema "builtin/validated-output"
|
|
And I create a tool execution context with plan_id "plan-022"
|
|
When I execute tool "builtin/validated-output" with params path "test.txt"
|
|
Then the execution result should not be successful
|
|
And the execution result error should contain "Output validation failed"
|
|
|
|
Scenario: Execution trace captures duration
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-023"
|
|
When I execute tool "builtin/read-file" with params path "test.txt"
|
|
Then the last trace should have a non-negative duration_ms
|
|
|
|
Scenario: ToolExecutionTrace fields
|
|
Given I create a tool execution trace for "builtin/read"
|
|
Then the trace tool_name should be "builtin/read"
|
|
And the trace should have a started_at timestamp
|
|
|
|
Scenario: Read-only plan allows read-only tools
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/search" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-024" and read_only True
|
|
When I execute tool "builtin/search" with params path "*.py"
|
|
Then the execution result should be successful
|
|
|
|
Scenario: Activation failure raises ToolActivationError
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/broken" with a failing activate mock
|
|
And I create a tool execution context with plan_id "plan-025"
|
|
When I try to activate the failing tool "builtin/broken"
|
|
Then a ToolActivationError should be raised
|
|
|
|
Scenario: Execution failure raises ToolExecutionError
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/crashing" with a failing execute mock
|
|
And I create a tool execution context with plan_id "plan-026"
|
|
When I try to execute the crashing tool "builtin/crashing"
|
|
Then a ToolExecutionError should be raised
|
|
And the context should have 1 traces
|
|
And the last trace should show failure
|
|
|
|
Scenario: Deactivation failure is non-fatal warning
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/sticky" with a failing deactivate mock
|
|
And I create a tool execution context with plan_id "plan-027"
|
|
When I activate tool "builtin/sticky" in the runtime
|
|
And I deactivate tool "builtin/sticky" in the runtime
|
|
Then no exception should have been raised from deactivation
|
|
|
|
Scenario: Plan deactivation handles failing tools gracefully
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/sticky" with a failing deactivate mock
|
|
And I create a tool execution context with plan_id "plan-028"
|
|
When I activate tool "builtin/sticky" in the runtime
|
|
And I deactivate plan "plan-028" in the runtime
|
|
Then no exception should have been raised from deactivation
|
|
|
|
Scenario: Cache plan_count property tracks plans
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/read-file" with a mock instance that is read_only
|
|
And I create a tool execution context with plan_id "plan-029"
|
|
When I activate tool "builtin/read-file" in the runtime
|
|
Then the runtime cache plan count should be 1
|
|
|
|
Scenario: Discover unregistered tool raises ToolRuntimeError
|
|
Given I create a tool runtime
|
|
When I try to discover unregistered tool "builtin/ghost"
|
|
Then a ToolRuntimeError should be raised with message containing "No implementation registered"
|
|
|
|
Scenario: Execution with result error records trace error field
|
|
Given I create a tool runtime
|
|
And I register a builtin tool "builtin/err-tool" with a mock that returns error result
|
|
And I create a tool execution context with plan_id "plan-030"
|
|
When I execute tool "builtin/err-tool" with params path "test.txt"
|
|
Then the last trace should show the error message
|