HAL9000
155e5dd586
CI / benchmark-publish (pull_request) Has been skipped
CI / lint (pull_request) Failing after 19s
CI / push-validation (pull_request) Successful in 20s
CI / helm (pull_request) Successful in 32s
CI / quality (pull_request) Successful in 33s
CI / typecheck (pull_request) Successful in 53s
CI / security (pull_request) Successful in 1m4s
CI / coverage (pull_request) Has been skipped
CI / benchmark-regression (pull_request) Has been skipped
CI / build (pull_request) Successful in 3m18s
CI / e2e_tests (pull_request) Successful in 6m18s
CI / integration_tests (pull_request) Successful in 7m2s
CI / unit_tests (pull_request) Successful in 8m53s
CI / docker (pull_request) Has been skipped
CI / status-check (pull_request) Failing after 1s
Fixed Gherkin syntax error where the 'LLM actor without tools' scenario had incorrect indentation (3 spaces instead of 2), which prevented the feature file from being parsed correctly by Behave. This was causing the concurrency test scenario to fail during test discovery. ISSUES CLOSED: #8588
272 lines
16 KiB
Gherkin
272 lines
16 KiB
Gherkin
Feature: Actor registry and loader
|
|
As a CleverAgents user
|
|
I want to discover, load, and cache actor YAML configs from the filesystem
|
|
So that actors are available for compilation and execution
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Discovery from search roots
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Discover actors from a single directory
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| reviewer.yaml | assistants/reviewer | llm | gpt-4 |
|
|
| formatter.yaml | utilities/formatter | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then the loader should find 2 actors
|
|
And the loader should contain actor "assistants/reviewer"
|
|
And the loader should contain actor "utilities/formatter"
|
|
|
|
Scenario: Discover actors from multiple search roots
|
|
Given a temporary actors directory "root_a" with files
|
|
| filename | name | type | model |
|
|
| writer.yaml | assistants/writer | llm | gpt-4 |
|
|
And a temporary actors directory "root_b" with files
|
|
| filename | name | type | model |
|
|
| analyst.yaml | data/analyst | llm | gpt-4 |
|
|
When I create an actor loader with both directories as search roots
|
|
And I run discovery
|
|
Then the loader should find 2 actors
|
|
And the loader should contain actor "assistants/writer"
|
|
And the loader should contain actor "data/analyst"
|
|
|
|
Scenario: Only YAML files are discovered
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| valid.yaml | assistants/valid | llm | gpt-4 |
|
|
And a non-YAML file "readme.txt" in the directory
|
|
And a non-YAML file "config.json" in the directory
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then the loader should find 1 actors
|
|
|
|
Scenario: Nested YAML files are discovered recursively
|
|
Given a temporary actors directory with nested structure
|
|
| subdir | filename | name | type | model |
|
|
| sub | nested.yaml | assistants/nested | llm | gpt-4 |
|
|
| sub/deep | deep.yaml | assistants/deep | llm | gpt-4 |
|
|
And a top-level actor file
|
|
| filename | name | type | model |
|
|
| top.yaml | assistants/top | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then the loader should find 3 actors
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Duplicate detection
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Duplicate actor names across search roots emit consolidated error
|
|
Given a temporary actors directory "dup_a" with files
|
|
| filename | name | type | model |
|
|
| writer.yaml | assistants/writer | llm | gpt-4 |
|
|
And a temporary actors directory "dup_b" with files
|
|
| filename | name | type | model |
|
|
| writer2.yaml | assistants/writer | llm | gpt-4 |
|
|
When I create an actor loader with both directories as search roots
|
|
And I run discovery expecting errors
|
|
Then the loader should raise a validation error mentioning "assistants/writer"
|
|
|
|
Scenario: Duplicate actor names within a single search root emit error
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| one.yaml | assistants/helper | llm | gpt-4 |
|
|
| two.yaml | assistants/helper | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery expecting errors
|
|
Then the loader should raise a validation error mentioning "assistants/helper"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Namespace handling
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Default local/ namespace applied when namespace omitted
|
|
Given a temporary actors directory with a nameless actor file
|
|
| filename | raw_name | type | model |
|
|
| bare.yaml | my-agent | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then the loader should contain actor "local/my-agent"
|
|
|
|
Scenario: Explicit namespace preserved as-is
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| custom.yaml | myteam/my-actor | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then the loader should contain actor "myteam/my-actor"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Namespaced lookup
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Get actor by namespaced name
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| helper.yaml | assistants/helper | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then I can retrieve actor "assistants/helper" by name
|
|
And the retrieved actor type should be "llm"
|
|
|
|
Scenario: Get nonexistent actor returns None
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| helper.yaml | assistants/helper | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then getting actor "assistants/missing" should return None
|
|
|
|
Scenario: List actors filtered by namespace
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| rev.yaml | assistants/reviewer | llm | gpt-4 |
|
|
| fmt.yaml | utilities/formatter | llm | gpt-4 |
|
|
| ana.yaml | assistants/analyzer | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then listing loaded actors with namespace "assistants" should return 2 actors
|
|
And listing loaded actors with namespace "utilities" should return 1 actors
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Content hash caching
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Unchanged files are not reloaded on re-discovery
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| cached.yaml | assistants/cached | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
And I run discovery again
|
|
Then the actor "assistants/cached" should have been loaded only once
|
|
|
|
Scenario: Modified files are reloaded on re-discovery
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| mutable.yaml | assistants/mutable | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
And I modify the actor file "mutable.yaml" to change the model to "gpt-3.5-turbo"
|
|
And I run discovery again
|
|
Then the actor "assistants/mutable" model should be "gpt-3.5-turbo"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Validation at load time
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Invalid YAML file is rejected with clear error
|
|
Given a temporary actors directory with an invalid YAML file "bad.yaml"
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery expecting errors
|
|
Then the loader should raise a validation error mentioning "bad.yaml"
|
|
|
|
Scenario: YAML file missing required fields is rejected
|
|
Given a temporary actors directory with a schema-invalid file "incomplete.yaml"
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery expecting errors
|
|
Then the loader should raise a validation error
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Integration with example actors
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Discover all example actors from examples/actors/
|
|
Given the project examples/actors directory exists
|
|
When I create an actor loader with the examples/actors directory
|
|
And I run discovery
|
|
Then the loader should find at least 5 actors
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Cache invalidation
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Clear cache removes all loaded actors
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| test.yaml | assistants/test | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
And I clear the loader cache
|
|
Then the loader should find 0 actors
|
|
|
|
Scenario: Deleted actor file is removed on re-discovery
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| keep.yaml | assistants/keep | llm | gpt-4 |
|
|
| remove.yaml | assistants/removable | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
And I delete the actor file "remove.yaml"
|
|
And I run discovery again
|
|
Then the loader should find 1 actors
|
|
And the loader should contain actor "assistants/keep"
|
|
And getting actor "assistants/removable" should return None
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Tool Registry integration
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Actor loader resolves tool references through tool registry
|
|
Given a tool registry with tool "files/read_file"
|
|
And a temporary actors directory with a tool-actor file
|
|
| filename | name | type | tools |
|
|
| tools.yaml | utilities/file_ops | tool | files/read_file |
|
|
When I create an actor loader with that directory and the tool registry
|
|
And I run discovery
|
|
Then the loader should contain actor "utilities/file_ops"
|
|
And the actor "utilities/file_ops" tools should include "files/read_file"
|
|
|
|
Scenario: Actor with unresolvable tool reference logs warning
|
|
Given an empty tool registry
|
|
And a temporary actors directory with a tool-actor file
|
|
| filename | name | type | tools |
|
|
| tools.yaml | utilities/ops | tool | missing/nonexistent |
|
|
When I create an actor loader with that directory and the tool registry
|
|
And I run discovery
|
|
Then the loader should contain actor "utilities/ops"
|
|
And the loader should have a warning about unresolved tool "missing/nonexistent"
|
|
|
|
# ────────────────────────────────────────────────────────────
|
|
# Edge cases for coverage
|
|
# ────────────────────────────────────────────────────────────
|
|
|
|
Scenario: Non-existent search root is skipped gracefully
|
|
Given a non-existent directory path as search root
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
Then the loader should find 0 actors
|
|
|
|
Scenario: YAML file containing a list instead of a mapping is rejected
|
|
Given a temporary actors directory with a list-content YAML file "list.yaml"
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery expecting errors
|
|
Then the loader should raise a validation error mentioning "list.yaml"
|
|
|
|
Scenario: LLM actor without tools does not produce warnings with tool registry
|
|
Given a tool registry with tool "files/read_file"
|
|
And a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| simple.yaml | assistants/simple | llm | gpt-4 |
|
|
When I create an actor loader with that directory and the tool registry
|
|
And I run discovery
|
|
Then the loader should contain actor "assistants/simple"
|
|
And the loader should have 0 warnings
|
|
|
|
# ────────────────────────────────────────────────────────
|
|
# Concurrency and race conditions
|
|
# ────────────────────────────────────────────────────────
|
|
|
|
@unit @actor @concurrency
|
|
Scenario: list_actors with namespace filter is thread-safe under concurrent clear/discover
|
|
Given a temporary actors directory with files
|
|
| filename | name | type | model |
|
|
| rev.yaml | assistants/reviewer | llm | gpt-4 |
|
|
| fmt.yaml | utilities/formatter | llm | gpt-4 |
|
|
| ana.yaml | assistants/analyzer | llm | gpt-4 |
|
|
When I create an actor loader with that directory as search root
|
|
And I run discovery
|
|
And 5 threads concurrently call list_actors with namespace "assistants" while another thread calls clear
|
|
Then all concurrent list_actors calls should return consistent results
|