brent.edwards
eb46f0ff54
CI / push-validation (push) Successful in 40s
CI / helm (push) Successful in 44s
CI / build (push) Successful in 1m14s
CI / lint (push) Successful in 1m18s
CI / quality (push) Successful in 1m36s
CI / e2e_tests (push) Successful in 1m19s
CI / typecheck (push) Successful in 2m22s
CI / security (push) Successful in 2m23s
CI / benchmark-regression (push) Failing after 41s
CI / integration_tests (push) Successful in 4m46s
CI / unit_tests (push) Failing after 20m13s
CI / benchmark-publish (push) Failing after 28m28s
CI / coverage (push) Has been cancelled
CI / docker (push) Has been cancelled
CI / status-check (push) Has been cancelled
## Summary This PR fixes issue #10878 where architecture reviews were truncated because the regex pattern for parsing file output would stop at the first ``` encountered in the Markdown report. ## Changes - Change file delimiters from ``` to >>>>>>>/<<<<<<< to avoid Markdown conflicts - Add tier hydration before strategize phase in plan_executor.py - Increase max_tokens to 16384 in llm_actors.py for longer outputs - Increase context_max_tokens_hot from 16000 to 32000 in settings.py - Fix get_hot_view → get_hot_fragments in strategy_actor.py and plan_executor.py - Add opencode to skip directories in context_tier_hydrator.py - Change sandbox output location to plan-output/ directory in plan.py - Add get_context_summary stub method to acms_service.py ## Testing Run architecture review action and verify the output report is complete with all sections. Reviewed-on: #10938
84 lines
4.3 KiB
Gherkin
84 lines
4.3 KiB
Gherkin
@tdd @tdd_issue @tdd_issue_10878
|
|
Feature: Delimiter parsing does not trip on triple-backticks in LLM output (#10878)
|
|
Bug #10878 caused architecture reviews to be truncated because the old regex
|
|
delimiter (triple-backtick markdown code fences ```) would stop at the first
|
|
triple-backtick appearing *inside* the LLM-generated file content, cutting off
|
|
all subsequent sections.
|
|
|
|
This test suite re-proves that the new delimiter scheme using unique sentinel
|
|
markers is not vulnerable to markdown collision.
|
|
|
|
Forgejo: #10878
|
|
|
|
# ---------------------------------------------------------------
|
|
# The old backtick-based parser was broken (reproduction of bug)
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Old backtick delimiter parser stops at first triple-backtick in content
|
|
Given a string containing ````
|
|
When I run the command "echo 'backtick-test'"
|
|
And I create file with LLM output containing code fences in body
|
|
When I parse the LLM output as if it were old backtick-delimited file blocks
|
|
Then the parser should return only 2 entries truncated at first triple-backtick
|
|
|
|
Scenario: New CLEVERAGENTS_FILE_START delimiter parser handles code fences inside content
|
|
Given a string containing ````
|
|
When I run the command "echo 'backtick-test'"
|
|
And I create file with LLM output using unique delimiters and code fences embedded in body
|
|
When I parse the LLM output with new CLEVERAGENTS_FILE_START/CLEVERAGENTS_FILE_END delimiters
|
|
Then the parser should return 6 entries not truncated at any triple-backtick
|
|
|
|
|
|
Scenario: Old backtick-delimiter parser fails when markdown report contains triple backticks
|
|
Given a string containing ````
|
|
When I run the command "echo 'old-parser-test'"
|
|
And I create LLM output where file content contains embedded markdown code blocks
|
|
When I parse it with the old backtick-style delimiter pattern
|
|
Then the parser should return only 1 entry instead of the expected 4
|
|
|
|
Scenario: New delimited parser handles markdown code blocks inside file content
|
|
Given a string containing ````
|
|
When I run the command "echo 'new-parser-test'"
|
|
And I create LLM output using unique sentinels with embedded ```python code blocks in body
|
|
When I parse it with CLEVERAGENTS_FILE_START / CLEVERAGENTS_FILE_END delimiters
|
|
Then the parser should return a result count of 4 entries
|
|
|
|
|
|
# ---------------------------------------------------------------
|
|
# Verify new delimiter handles the real-world bug pattern: report + 3+ file blocks each
|
|
# containing inline code examples that use triple-backticks
|
|
# ---------------------------------------------------------------
|
|
|
|
Scenario: Delimiter survives LLM output with architecture-report header and nested code
|
|
Given a string containing ````
|
|
When I run the command "echo 'full-regression-test'"
|
|
And I create full regression file block output that includes report section plus multiple files each with inline ```python examples
|
|
When I parse it with new CLEVERAGENTS_FILE_START / CLEVERAGENTS_FILE_END delimiters
|
|
Then the parser should return the full entry count of all embedded blocks
|
|
|
|
|
|
# ---------------------------------------------------------------
|
|
# Unit tests on LLMExecuteActor._parse_file_blocks directly (unit-test level)
|
|
# ---------------------------------------------------------------
|
|
|
|
@mock_only
|
|
Scenario: _parse_file_blocks handles file content with triple-backtick code fences inside body
|
|
Given a mock plan_id "01HQXXXXX"
|
|
And LLM output containing two FILE blocks each with embedded ```python sections in the content
|
|
When I call _parse_file_blocks on that LLM output
|
|
Then it should return 2 changeset entries not 1
|
|
|
|
@mock_only
|
|
Scenario: _parse_file_blocks handles file content with a trailing triple-backtick line
|
|
Given a mock plan_id "01HYYYYYY"
|
|
And an llm_output where the file body ends on a line that is exactly ``` after content
|
|
When I call _parse_file_blocks on that output
|
|
Then it should return 1 entry
|
|
|
|
Scenario: Backwards compatibility — backtick-delimited file blocks still parse correctly (sanitisation)
|
|
Given a string ````
|
|
When I run the command "echo 'backtick-sanitise'"
|
|
And LLM input with backtick-delimited FILE blocks containing code without triple-backticks inside body
|
|
When I call _parse_file_blocks on that output
|
|
Then it should return 1 entry
|