UAT: `agents automation-profile add` JSON output uses wrong structure — `phase_transitions`/`decision_automation`/`self_repair`/`execution_controls` instead of spec-required `thresholds`/`flags` #2960

New issue

Open

opened 2026-04-05 02:57:33 +00:00 by freemo · 1 comment

freemo commented

2026-04-05 02:57:33 +00:00

Owner

Metadata

Branch: fix/automation-profile-add-json-output-spec-compliance
Commit Message: fix(cli): correct automation-profile add JSON output to use spec-required thresholds/flags structure
Milestone: v3.7.0
Parent Epic: #362

Background and Context

The agents automation-profile add command's JSON output uses a different data structure than specified in docs/specification.md. The spec requires thresholds and flags top-level keys in the data object, but the implementation reuses the show command's internal _profile_spec_dict() helper which returns a categorised structure (phase_transitions, decision_automation, self_repair, execution_controls). This is a separate issue from the rich output deviation tracked in #2923.

Current Behavior

The implementation's _profile_spec_dict() function (lines 72–110 in src/cleveragents/cli/commands/automation_profile.py) is shared between add, show, list, and remove commands. When --format json is used with automation-profile add, the data object in the response uses the categorised structure:

{
  "name": "...",
  "description": "...",
  "source": "custom",
  "schema_version": "1.0",
  "phase_transitions": {
    "decompose_task": 0.0,
    "create_tool": 0.0,
    "select_tool": 1.0
  },
  "decision_automation": {
    "edit_code": 0.0,
    "execute_command": 0.0
  },
  "self_repair": {
    "create_file": 0.0,
    "delete_content": 1.0,
    "access_network": 1.0,
    "modify_config": 0.0,
    "approve_plan": 1.0
  },
  "execution_controls": {
    "install_dependency": 0.0,
    "require_sandbox": true,
    "require_checkpoints": true,
    "allow_unsafe_tools": false
  },
  "guards": null
}

Differences from spec:

Thresholds are split into 4 categorised sub-objects instead of a flat thresholds object
Safety flags are mixed into execution_controls instead of a separate flags object
No created timestamp field
Extra fields not in spec: source, schema_version, guards
Missing messages array with "Profile registered"

Expected Behavior (from spec)

From docs/specification.md section "agents automation-profile add", the JSON output should be:

{
  "command": "automation-profile add",
  "status": "ok",
  "exit_code": 0,
  "data": {
    "name": "local/careful-auto",
    "description": "Autonomous execution with mandatory sandbox and manual apply",
    "created": "2026-02-08T14:30:00Z",
    "thresholds": {
      "decompose_task": 0.0,
      "create_tool": 0.0,
      "select_tool": 1.0,
      "edit_code": 0.0,
      "execute_command": 0.0,
      "create_file": 0.0,
      "delete_content": 1.0,
      "access_network": 1.0,
      "install_dependency": 0.0,
      "modify_config": 0.0,
      "approve_plan": 0.0
    },
    "flags": {
      "require_sandbox": true,
      "require_checkpoints": true,
      "allow_unsafe_tools": false
    }
  },
  "timing": { "started": "2026-02-08T14:30:00Z", "duration_ms": 70 },
  "messages": ["Profile registered"]
}

Key points:

All thresholds grouped under a single flat thresholds key (not categorised)
Safety flags grouped under a separate flags key
A created timestamp field in data
messages array containing "Profile registered" (not "Profile added")

Note: The automation-profile show JSON output in the spec intentionally uses the categorised structure (phase_transitions, etc.), so the add command's JSON format is deliberately different from show. The implementation incorrectly reuses the show format for add.

Steps to Reproduce

agents automation-profile add --config ./my-profile.yaml --format json

Observe that the JSON output uses phase_transitions/decision_automation/self_repair/execution_controls instead of thresholds/flags.

Impact

Clients parsing the JSON output of automation-profile add will receive a different structure than documented, breaking programmatic integrations that follow the spec.

Code Location

src/cleveragents/cli/commands/automation_profile.py:

Lines 72–110: _profile_spec_dict() — shared between add, show, list, and remove commands, but the spec requires different JSON structures for add vs show

Subtasks

Introduce a dedicated _profile_add_data_dict() helper (or equivalent) that returns the spec-required flat thresholds + flags + created structure for the add command response
Update the automation-profile add command handler to use the new helper instead of _profile_spec_dict() when building the JSON data payload
Ensure the messages field in the JSON envelope contains ["Profile registered"]
Ensure the created timestamp is populated from the profile's creation time
Verify _profile_spec_dict() (used by show, list, remove) is unchanged and continues to pass existing tests
Tests (Behave): Add/update scenarios in features/ covering automation-profile add --format json output structure
Tests (Robot): Add/update integration test in robot/ for automation-profile add JSON output
Verify coverage ≥97% via nox -s coverage_report
Run nox (all default sessions), fix any errors

Definition of Done

This issue is complete when:

All subtasks above are completed and checked off.
The agents automation-profile add --format json output matches the spec-required structure exactly: flat thresholds object, separate flags object, created timestamp, and messages: ["Profile registered"].
The automation-profile show --format json output is unaffected (still uses the categorised structure as per spec).
A Git commit is created where the first line of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation.
The commit is pushed to the remote on the branch matching the Branch in Metadata exactly.
The commit is submitted as a pull request to master, reviewed, and merged before this issue is marked done.
All nox stages pass.
Coverage ≥ 97%.

Automated by CleverAgents Bot
Supervisor: UAT Testing | Agent: ca-new-issue-creator

## Metadata - **Branch**: `fix/automation-profile-add-json-output-spec-compliance` - **Commit Message**: `fix(cli): correct automation-profile add JSON output to use spec-required thresholds/flags structure` - **Milestone**: v3.7.0 - **Parent Epic**: #362 ## Background and Context The `agents automation-profile add` command's JSON output uses a different data structure than specified in `docs/specification.md`. The spec requires `thresholds` and `flags` top-level keys in the `data` object, but the implementation reuses the `show` command's internal `_profile_spec_dict()` helper which returns a categorised structure (`phase_transitions`, `decision_automation`, `self_repair`, `execution_controls`). This is a separate issue from the rich output deviation tracked in #2923. ## Current Behavior The implementation's `_profile_spec_dict()` function (lines 72–110 in `src/cleveragents/cli/commands/automation_profile.py`) is shared between `add`, `show`, `list`, and `remove` commands. When `--format json` is used with `automation-profile add`, the `data` object in the response uses the categorised structure: ```json { "name": "...", "description": "...", "source": "custom", "schema_version": "1.0", "phase_transitions": { "decompose_task": 0.0, "create_tool": 0.0, "select_tool": 1.0 }, "decision_automation": { "edit_code": 0.0, "execute_command": 0.0 }, "self_repair": { "create_file": 0.0, "delete_content": 1.0, "access_network": 1.0, "modify_config": 0.0, "approve_plan": 1.0 }, "execution_controls": { "install_dependency": 0.0, "require_sandbox": true, "require_checkpoints": true, "allow_unsafe_tools": false }, "guards": null } ``` Differences from spec: 1. Thresholds are split into 4 categorised sub-objects instead of a flat `thresholds` object 2. Safety flags are mixed into `execution_controls` instead of a separate `flags` object 3. No `created` timestamp field 4. Extra fields not in spec: `source`, `schema_version`, `guards` 5. Missing `messages` array with `"Profile registered"` ## Expected Behavior (from spec) From `docs/specification.md` section "agents automation-profile add", the JSON output should be: ```json { "command": "automation-profile add", "status": "ok", "exit_code": 0, "data": { "name": "local/careful-auto", "description": "Autonomous execution with mandatory sandbox and manual apply", "created": "2026-02-08T14:30:00Z", "thresholds": { "decompose_task": 0.0, "create_tool": 0.0, "select_tool": 1.0, "edit_code": 0.0, "execute_command": 0.0, "create_file": 0.0, "delete_content": 1.0, "access_network": 1.0, "install_dependency": 0.0, "modify_config": 0.0, "approve_plan": 0.0 }, "flags": { "require_sandbox": true, "require_checkpoints": true, "allow_unsafe_tools": false } }, "timing": { "started": "2026-02-08T14:30:00Z", "duration_ms": 70 }, "messages": ["Profile registered"] } ``` Key points: - All thresholds grouped under a single flat `thresholds` key (not categorised) - Safety flags grouped under a separate `flags` key - A `created` timestamp field in `data` - `messages` array containing `"Profile registered"` (not `"Profile added"`) Note: The `automation-profile show` JSON output in the spec intentionally uses the categorised structure (`phase_transitions`, etc.), so the `add` command's JSON format is deliberately different from `show`. The implementation incorrectly reuses the `show` format for `add`. ## Steps to Reproduce ```bash agents automation-profile add --config ./my-profile.yaml --format json ``` Observe that the JSON output uses `phase_transitions`/`decision_automation`/`self_repair`/`execution_controls` instead of `thresholds`/`flags`. ## Impact Clients parsing the JSON output of `automation-profile add` will receive a different structure than documented, breaking programmatic integrations that follow the spec. ## Code Location `src/cleveragents/cli/commands/automation_profile.py`: - Lines 72–110: `_profile_spec_dict()` — shared between `add`, `show`, `list`, and `remove` commands, but the spec requires different JSON structures for `add` vs `show` ## Subtasks - [ ] Introduce a dedicated `_profile_add_data_dict()` helper (or equivalent) that returns the spec-required flat `thresholds` + `flags` + `created` structure for the `add` command response - [ ] Update the `automation-profile add` command handler to use the new helper instead of `_profile_spec_dict()` when building the JSON `data` payload - [ ] Ensure the `messages` field in the JSON envelope contains `["Profile registered"]` - [ ] Ensure the `created` timestamp is populated from the profile's creation time - [ ] Verify `_profile_spec_dict()` (used by `show`, `list`, `remove`) is unchanged and continues to pass existing tests - [ ] Tests (Behave): Add/update scenarios in `features/` covering `automation-profile add --format json` output structure - [ ] Tests (Robot): Add/update integration test in `robot/` for `automation-profile add` JSON output - [ ] Verify coverage ≥97% via `nox -s coverage_report` - [ ] Run `nox` (all default sessions), fix any errors ## Definition of Done This issue is complete when: - All subtasks above are completed and checked off. - The `agents automation-profile add --format json` output matches the spec-required structure exactly: flat `thresholds` object, separate `flags` object, `created` timestamp, and `messages: ["Profile registered"]`. - The `automation-profile show --format json` output is unaffected (still uses the categorised structure as per spec). - A Git commit is created where the **first line** of the commit message matches the Commit Message in Metadata exactly, followed by a blank line, then additional lines providing relevant details about the implementation. - The commit is pushed to the remote on the branch matching the **Branch** in Metadata exactly. - The commit is submitted as a **pull request** to `master`, reviewed, and **merged** before this issue is marked done. - All nox stages pass. - Coverage ≥ 97%. --- **Automated by CleverAgents Bot** Supervisor: UAT Testing | Agent: ca-new-issue-creator

freemo added this to the v3.7.0 milestone

2026-04-05 02:57:38 +00:00

freemo added a new dependency

2026-04-05 02:57:53 +00:00

#362 Epic: Security & Safety Hardening

freemo commented

2026-04-05 03:00:31 +00:00

Author

Owner

Issue triaged by project owner:

State: Verified
Priority: Confirmed
MoSCoW: Should Have

Valid finding verified during batch triage.

Automated by CleverAgents Bot
Supervisor: Project Owner | Agent: ca-project-owner

Issue triaged by project owner: - **State**: Verified - **Priority**: Confirmed - **MoSCoW**: Should Have Valid finding verified during batch triage. --- **Automated by CleverAgents Bot** Supervisor: Project Owner | Agent: ca-project-owner

freemo referenced this issue

2026-04-05 06:15:22 +00:00

[CA-AUTO] UAT Pool Supervisor — v3.2.0–v3.7.0 — Session Tracker #3094