Robot Framework integration test for Specification Workflow Example 8:
Cloud Infrastructure Management with the supervised automation profile.
Test suite exercises:
- Custom resource type registration (local/terraform-state) via YAML
fixture with copy_on_write sandbox strategy and CLI args validation
- Custom skill creation (local/terraform-ops) with 3 anonymous tools
(terraform_plan, terraform_show, cloud_metrics) and skill composition
via includes (local/file-ops)
- Supervised profile gating: verifies both strategize-to-execute
(create_tool=1.0) and execute-to-apply (select_tool=1.0) transitions
require explicit human approval via should_auto_progress()
- Action creation with supervised profile, typed arguments (STRING,
FLOAT), and invariant propagation to plans via InvariantSource.ACTION
- Mocked infrastructure analysis producing right-sizing recommendations
for over-provisioned resources while skipping critical instances
- Invariant enforcement blocking modifications to resources tagged
critical:true while allowing non-critical changes
Fixture files provide deterministic mock Terraform state (3 AWS
resources), CloudWatch metrics (CPU at 12% avg on m5.2xlarge), and
sample HCL configuration.
ISSUES CLOSED: #772
Reviewed-by: freemo (reviewer-pool-1)
Co-authored-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Co-committed-by: Brent E. Edwards <brent.edwards@cleverthis.com>
Rename the TDD tag system from tdd_bug/tdd_bug_<N> to tdd_issue/tdd_issue_<N>
across the entire codebase. The tdd_expected_fail tag is unchanged.
The TDD expected-failure workflow is not limited to bug fixes — it applies
equally to any issue type (features, tasks, refactors). The _bug suffix was
misleading and narrowed the perceived scope. The new _issue suffix accurately
reflects that the TDD tagging system applies to any Forgejo issue.
Changes span 92 files:
- features/environment.py: validate_tdd_tags(), should_invert_result(), and
apply_tdd_inversion() updated — regex, variables, error messages
- robot/tdd_expected_fail_listener.py: _validate_tdd_tags(), _should_invert_result(),
start_test(), end_test() updated consistently
- 33 Behave .feature files: all @tdd_bug/@tdd_bug_<N> tags renamed
- 29 Robot .robot files: all tdd_bug/tdd_bug_<N> tags renamed
- 3 Robot fixture files renamed (tdd_bug_alone, tdd_missing_tdd_bug,
tdd_expected_fail_missing_bug_n) with content and references updated
- Tag validation tests and helpers updated (function names, command dispatch
keys, output strings, fixture references)
- CONTRIBUTING.md: section renamed from 'TDD Bug Test Tags' to
'TDD Issue Test Tags', all tag references and examples updated
- noxfile.py: comment references updated
- Step definition files, mock helpers, and benchmark files: docstring
references updated
ISSUES CLOSED: #965
Implements the three-tag TDD bug-capture system in Robot Framework via a
Listener v3 module, paralleling the Behave implementation. Tests tagged
tdd_expected_fail that fail have their result inverted to PASS (bug still
exists); tests that unexpectedly pass are inverted to FAIL with guidance.
Addresses all 15 findings from code review (PR !673, reviewer hamza.khyari):
P2 fixes:
- Added idempotency guard (_processed_tests set) to prevent double-inversion
when the listener is loaded twice in the same process.
- Rewrote normal-test-unaffected check to run alongside a tdd_expected_fail
fixture in a single Robot invocation, proving the listener is loaded and
selectively applies rather than being a tautological pass.
P3 fixes:
- Added output.xml existence guard with clear diagnostics in _run_fixture.
- Documented intentional use of data.tags (static definition) vs result.tags
(runtime-modifiable) in end_test docstring.
- Added SKIP status test fixture and integration test case.
- Added message content assertion in cmd_expected_fail_inverted.
- Tightened substring assertions to match specific error text.
- Added tdd_expected_fail-alone fixture (both companions missing).
- Added close() hook to clear _validation_errors and _processed_tests.
- Simplified _run_fixture return type to tuple[str, str].
- Changed listener path resolution from CWD-relative to __file__-relative
in noxfile.py (integration_tests, slow_integration_tests, e2e_tests).
P4 fixes:
- Added __all__ declaration to helper module.
- Changed module docstring from "mirroring" to "paralleling".
- Added comment documenting accepted XML parsing risk (self-generated XML).
Additional fixes:
- Increased M4 E2E plan-tree test timeout from 30s to 120s (pre-existing
timeout failure unrelated to this feature).
Quality gates (post-rebase onto latest master):
- nox -s lint: PASS
- nox -s typecheck: PASS (0 errors)
- nox -s unit_tests: PASS (10,700 scenarios)
- nox -s integration_tests: PASS (1,505 tests)
- nox -s coverage_report: PASS (97.9% >= 97% threshold)
- nox -s benchmark: PASS
- nox -s docs: PASS
- nox -s build: PASS
- nox -s security_scan: PASS
- nox -s dead_code: PASS
ISSUES CLOSED: #628
brent.edwards
hurui200320
Jeffrey Phillips Freeman