cleveragents/cleveragents-core
4
0
Fork 3
Code Issues 6.0k Pull Requests 115 Actions Packages Projects Releases Wiki Activity

[AUTO-ARCH-6] spec: add ADR-049 specification restructuring proposal #8891

Closed
HAL9000 wants to merge 1 commits from auto-arch-6/adr-049-spec-restructuring into master
pull from: auto-arch-6/adr-049-spec-restructuring
merge into: cleveragents:master
Conversation 7 Commits 1 Files Changed 2
+252
2 changed files with 252 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/ADR-049-specification-restructuring.md
+251
View File
@@ -0,0 +1,251 @@
---
adr_number: 49
title: "Specification Restructuring — Monolithic to Modular Directory"
status_history:
- ["2026-04-14", "Draft", "CleverThis"]
- ["2026-04-14", "Proposed", "CleverThis"]
tier: 1
authors: ["CleverThis"]
superseded_by:
related_adrs:
- number: 1
title: "Layered Architecture"
relationship: "Specification structure must reflect the layered architecture it describes"
acceptance:
votes_for: []
votes_against: []
abstentions: []
---
## Context
`docs/specification.md` is the single source of truth for the CleverAgents system architecture and behavior. ADRs are accepted and then incorporated into the specification, which has grown continuously as the project has matured. As of April 2026, the file is approximately **3.1 MB** (~30,000+ lines) — an order of magnitude beyond the ~3,000-line threshold at which the architectural guidelines recommend transitioning to a directory structure.
At this scale the monolithic file creates concrete, compounding problems:
- **Tooling limits**: Many editors, diff tools, language servers, and CI linters impose hard or soft limits on single-file size. A 3 MB Markdown file exceeds the comfortable operating range of most static-site generators' incremental build caches, causing full rebuilds on every change.
- **Navigation friction**: Readers must scroll or search through tens of thousands of lines to locate a specific section. The MkDocs table-of-contents sidebar becomes unwieldy with hundreds of headings from a single page.
- **Parallel editing conflicts**: Multiple architecture workers writing to the same file in concurrent branches produce merge conflicts on nearly every PR, even when the changes are logically independent.
- **Cognitive load**: A contributor working on, say, the TUI Persona System must open and parse the entire specification to find the relevant section — there is no scoped entry point.
- **Incremental review**: Reviewers cannot approve a targeted change to one module without reading through the entire file to confirm that surrounding content was not inadvertently altered.
- **Version history noise**: `git log -- docs/specification.md` conflates changes to every module in a single history stream, making it difficult to trace the evolution of any individual concern.
The architectural guidelines already anticipate this transition: workers are instructed to split into `docs/specification/` with one file per module when the spec exceeds ~3,000 lines. This ADR formalizes that decision, defines the target directory structure, and specifies the migration strategy.
## Decision Drivers
- The specification file is ~3.1 MB — more than 1,000× the recommended single-file threshold, and growing with every accepted ADR.
- MkDocs Material's navigation and search work significantly better with multiple focused pages than with a single enormous page.
- Architecture workers operate concurrently; a single shared file is a structural merge-conflict bottleneck.
- The layered architecture (ADR-001) already partitions the system into well-defined modules; the specification structure should mirror that partitioning.
- Cross-references within the specification must remain navigable after the split — anchor-based links must be preserved or redirected.
- The `mkdocs.yml` navigation must be updated to expose the new structure; the current single `Specification: specification.md` entry must be replaced with a structured sub-navigation.
- No content changes may accompany the restructuring — the split is a pure reorganization to avoid conflating structural and substantive changes.
- All internal links (from ADRs, from other docs pages, from the specification itself) that reference `specification.md#<anchor>` must be updated to point to the correct new file and anchor.
## Decision
The CleverAgents specification is restructured from a single `docs/specification.md` file into a **`docs/specification/` directory** containing one Markdown file per major module or concern. The entry point becomes `docs/specification/index.md`. The `mkdocs.yml` navigation is updated to reflect the new structure. No content is changed during the restructuring — only file boundaries and cross-reference paths.
## Design
### Target Directory Structure
```
docs/specification/
├── index.md # Overview, Glossary, Standards Alignment
├── cli-commands.md # CLI Commands
├── plan-lifecycle.md # Plan Lifecycle, Decision Tree, Correction Engine
├── project-resource-model.md # Projects, Resources, Resource Types, DAG
├── actor-session-model.md # Actors, Sessions, LLM Providers
├── tool-skill-system.md # Tools, Skills, Validations, MCP, Agent Skills
├── acms.md # Advanced Context Management System, UKO, CRP
├── invariant-system.md # Invariants, Automation Profiles, Safety Profiles
├── lsp-integration.md # LSP Integration
├── server-client-architecture.md # A2A Protocol, Server Architecture, Client Architecture
├── tui.md # TUI Architecture, Persona System, Reference/Command System
├── security.md # Security, Sandbox, Checkpoints
├── testing-strategy.md # Testing Strategy, Quality Gates
└── milestone-plan.md # Milestone Plan (v3.2.0 through v3.9.0)
```
### File Responsibilities
| File | Sections Migrated from `specification.md` |
|------|-------------------------------------------|
| `index.md` | Overview, Glossary, Standards Alignment, document-level conventions |
| `cli-commands.md` | CLI Commands section in full |
| `plan-lifecycle.md` | Plan Lifecycle, Decision Tree, Correction Engine |
| `project-resource-model.md` | Projects, Resources, Resource Types, Resource DAG |
| `actor-session-model.md` | Actors, Sessions, LLM Providers |
| `tool-skill-system.md` | Tools, Skills, Validations, MCP Adapter, Agent Skills |
| `acms.md` | Advanced Context Management System (ACMS), UKO, Context Resolution Protocol (CRP) |
| `invariant-system.md` | Invariants, Automation Profiles, Safety Profiles |
| `lsp-integration.md` | LSP Integration |
| `server-client-architecture.md` | A2A Protocol, Server Architecture, Client Architecture |
| `tui.md` | TUI Architecture, Persona System, Reference/Command System |
| `security.md` | Security, Sandbox, Checkpoints |
| `testing-strategy.md` | Testing Strategy, Quality Gates |
| `milestone-plan.md` | Milestone Plan (v3.2.0 through v3.9.0) |
### File Naming Conventions
- All filenames are lowercase kebab-case.
- Filenames reflect the primary domain concern, not the ADR numbers that informed the section.
- `index.md` is the canonical entry point; MkDocs renders it as the section landing page.
- No file should exceed ~3,000 lines after the initial split. If a section grows beyond that threshold in the future, it should be further subdivided (e.g., `tool-skill-system/tools.md`, `tool-skill-system/skills.md`).
### Cross-Reference Strategy
All internal links within the specification currently take the form:
```markdown
[Section Title](specification.md#anchor-slug)
```
After the split, links must be updated to:
```markdown
[Section Title](./target-file.md#anchor-slug)
```
or, from outside the `specification/` directory:
```markdown
[Section Title](specification/target-file.md#anchor-slug)
```
**Anchor preservation**: Every heading in the split files must retain the same slug it had in the monolithic file. Headings must not be renamed during the restructuring. This ensures that any external links (from ADRs, from the development docs, from the architecture page) that reference `specification.md#<anchor>` can be mechanically updated to `specification/<file>.md#<anchor>` without risk of broken anchors.
**Anchor mapping table**: As part of the migration, a mapping table must be produced that records every heading anchor in `specification.md` and its destination file in `docs/specification/`. This table is used to drive the link-update pass.
**Redirect stubs**: The original `docs/specification.md` must be replaced with a redirect stub that informs readers of the new location:
```markdown
# Specification
!!! note "Specification has moved"
The CleverAgents specification has been reorganized into a modular directory structure.
Please update your bookmarks.
- [Specification Overview](specification/index.md)
- [CLI Commands](specification/cli-commands.md)
- [Plan Lifecycle](specification/plan-lifecycle.md)
- *(see full navigation in the sidebar)*
```
This stub prevents hard 404s for any cached or external links to `specification.md` while the ecosystem updates.
### `mkdocs.yml` Navigation Update
The current single-entry navigation:
```yaml
nav:
- Specification: specification.md
```
Must be replaced with a structured sub-navigation:
```yaml
nav:
- Specification:
- Overview: specification/index.md
- CLI Commands: specification/cli-commands.md
- Plan Lifecycle: specification/plan-lifecycle.md
- Project & Resource Model: specification/project-resource-model.md
- Actor & Session Model: specification/actor-session-model.md
- Tool & Skill System: specification/tool-skill-system.md
- ACMS: specification/acms.md
- Invariant System: specification/invariant-system.md
- LSP Integration: specification/lsp-integration.md
- Server & Client Architecture: specification/server-client-architecture.md
- TUI: specification/tui.md
- Security: specification/security.md
- Testing Strategy: specification/testing-strategy.md
- Milestone Plan: specification/milestone-plan.md
```
### Migration Approach
The restructuring must be performed as a single atomic PR to avoid a period where the specification is partially split. The migration proceeds in the following steps:
1. **Audit**: Run a script to extract all heading anchors from `specification.md` and produce the anchor-to-file mapping table.
2. **Split**: Extract each section into its target file under `docs/specification/`. Use exact content copy — no edits, no reformatting, no reordering within sections.
3. **Link update pass**: Update all internal cross-references within the split files using the anchor mapping table. Update all references in `docs/adr/*.md`, `docs/architecture.md`, `docs/development/*.md`, and any other docs that link to `specification.md#<anchor>`.
4. **Redirect stub**: Replace `docs/specification.md` with the redirect stub described above.
5. **`mkdocs.yml` update**: Replace the single `Specification` nav entry with the structured sub-navigation.
6. **Build verification**: Run `mkdocs build --strict` to confirm no broken links, missing files, or navigation errors.
7. **Link check**: Run a link checker (e.g., `mkdocs-linkcheck` or `lychee`) against the built site to confirm all internal links resolve.
8. **Review**: The PR must be reviewed by at least one human architect before merge to confirm content fidelity (no content was lost or altered during the split).
### Tooling
The split and link-update passes should be automated via a migration script (`scripts/split_specification.py`) that:
- Parses `specification.md` using a Markdown AST parser (e.g., `mistletoe` or `markdown-it-py`).
- Identifies top-level section boundaries based on H1/H2 headings.
- Writes each section to its target file.
- Produces the anchor mapping table as a JSON artifact for the link-update pass.
- Updates all cross-references in the split files.
The script must be idempotent and must not modify content — only file boundaries and link paths.
## Constraints
- **No content changes during restructuring**: The split PR must contain zero substantive edits to specification content. Heading text, body text, code blocks, tables, and diagrams must be byte-identical to their source in `specification.md`. Any content corrections must be made in a separate PR before or after the restructuring.
- **All internal links must be updated**: No link in any documentation file may point to `specification.md#<anchor>` after the restructuring PR is merged. The redirect stub handles the root `specification.md` path, but anchor-specific links must be updated to their new targets.
- **`mkdocs.yml` must be updated atomically**: The navigation update and the file split must land in the same commit. A build with the new files but the old nav (or vice versa) must never be deployed.
- **Anchor slugs must be preserved**: Heading text must not be changed during the split. If a heading needs to be renamed for clarity, that is a separate content PR.
- **`docs/specification.md` must not be deleted immediately**: It must be replaced with the redirect stub and remain in place for at least one release cycle to allow external links to update gracefully. It may be deleted in a subsequent PR once the redirect period has elapsed.
- **Build must pass `--strict` mode**: `mkdocs build --strict` must succeed with zero warnings after the restructuring. Any broken link or missing file is a blocking defect.
## Consequences
### Positive
- **Navigability**: Readers can navigate directly to the relevant module without scrolling through tens of thousands of lines. MkDocs Material's sidebar and search index operate on focused pages.
- **Parallel editing**: Architecture workers can edit different specification files concurrently without merge conflicts. A change to `tui.md` does not conflict with a change to `acms.md`.
- **Scoped review**: Reviewers can approve a targeted change to one module without reading the entire specification. PR diffs are bounded to the relevant file.
- **Incremental builds**: MkDocs can rebuild only the changed page rather than re-processing the entire 3 MB file on every edit.
- **Tooling compatibility**: Editors, linters, and diff tools operate comfortably on files of a few hundred to a few thousand lines.
- **Cleaner git history**: `git log -- docs/specification/tui.md` shows only TUI-relevant changes; the history of each module is independently traceable.
- **Future growth**: New modules can be added as new files without inflating any existing file. The ~3,000-line-per-file guideline is enforceable at the file level.
### Negative
- **Migration effort**: The initial split requires careful automation and human review to confirm content fidelity. This is a one-time cost but is non-trivial given the file size.
- **Link update scope**: Every documentation file that links to `specification.md#<anchor>` must be updated. With 48+ ADRs and multiple development docs, this is a broad but mechanical change.
- **Redirect stub maintenance**: The `docs/specification.md` redirect stub must be maintained until external links have updated, adding a small ongoing maintenance burden.
- **`mkdocs.yml` complexity**: The navigation section grows from one line to fifteen lines. This is a minor increase in configuration complexity.
### Risks
- **Broken links if migration is incomplete**: If any link to `specification.md#<anchor>` is missed during the update pass, it will produce a 404 or a link to the redirect stub (which lacks the anchor). Mitigation: automated link checking in CI (`mkdocs build --strict` + link checker) must be run before merge and must pass with zero errors.
- **Content loss or corruption during split**: A manual or buggy automated split could silently drop or duplicate content. Mitigation: the migration script must produce a checksum of all content extracted from `specification.md` and verify it matches the concatenated content of all split files. Human review of the PR diff is required.
- **Anchor drift**: If headings are inadvertently renamed during the split, existing links will break. Mitigation: the constraint that heading text must be byte-identical to the source is enforced by the migration script's diff output.
- **Navigation regression**: If the `mkdocs.yml` update is incorrect, pages may become unreachable from the sidebar. Mitigation: `mkdocs build --strict` catches missing nav entries.
## Alternatives Considered
**Keep the single `docs/specification.md` file** — Continue adding content to the monolithic file. This avoids migration effort but compounds all of the problems described in Context: growing tooling friction, worsening merge conflicts, and increasingly poor navigation. The file is already 1,000× the recommended threshold. Rejected — the status quo is not sustainable.
**Split into multiple top-level files (e.g., `docs/specification-cli.md`, `docs/specification-tui.md`)** — Place split files at the `docs/` root rather than in a `docs/specification/` subdirectory. This avoids creating a new directory but pollutes the `docs/` root with many files and does not group the specification as a coherent navigable section in MkDocs. Rejected — a dedicated subdirectory provides better organization and a cleaner nav hierarchy.
**Split by ADR number rather than by module** — Create one file per ADR (e.g., `docs/specification/adr-006-plan-lifecycle.md`). This aligns the specification structure with the ADR history but produces files that are too granular (48+ files) and that do not map cleanly to the conceptual modules a reader would navigate. The specification synthesizes multiple ADRs into coherent module descriptions; splitting by ADR would fragment those syntheses. Rejected — module-based splitting is more reader-friendly.
**Use a documentation tool with native large-file support** — Switch to a tool that handles large Markdown files without friction (e.g., Docusaurus, GitBook). This would require migrating the entire documentation infrastructure and does not address the underlying organizational problem. Rejected — the issue is organizational, not tooling; MkDocs Material handles multi-file documentation well when files are appropriately sized.
**Lazy split — split only the largest sections** — Split only the sections that are individually very large (e.g., ACMS, Tool & Skill System) and leave the rest in `specification.md`. This reduces migration effort but leaves the file large and does not establish a clean, consistent structure. Future workers would face ambiguity about where to add new content. Rejected — a complete, consistent split is preferable to a partial one.
## Compliance
- **`mkdocs build --strict` in CI**: The CI pipeline must run `mkdocs build --strict` after the restructuring PR merges. Any broken link, missing file, or navigation error is a blocking failure.
- **Link checker in CI**: A link checker (e.g., `lychee` or `mkdocs-linkcheck`) must be run against the built site in CI. Zero broken internal links is required.
- **Content checksum verification**: The migration script must produce and log a SHA-256 checksum of all content extracted from `specification.md` and verify it matches the concatenated content of all files in `docs/specification/`. This checksum must be recorded in the PR description.
- **No direct references to `specification.md#<anchor>` in docs**: A CI lint rule (grep-based or ruff-based) must verify that no file in `docs/` (other than the redirect stub itself) contains a link of the form `specification.md#`. This rule must be added to the CI pipeline as part of the restructuring PR.
- **Human review required**: The restructuring PR must receive at least one approving review from a human architect before merge, confirming that no content was lost, altered, or duplicated during the split.
- **Navigation completeness**: Every heading in `docs/specification/index.md` that references a sub-page must correspond to an entry in `mkdocs.yml`. The CI build will catch missing entries via `--strict` mode.
---
**Automated by CleverAgents Bot**
Supervisor: Architecture | Agent: architecture-pool-supervisor
mkdocs.yml
+1
View File
@@ -94,6 +94,7 @@ nav:
- ADR-046 TUI Reference and Command System: adr/ADR-046-tui-reference-and-command-system.md
- ADR-047 A2A Standard Adoption: adr/ADR-047-acp-standard-adoption.md
- ADR-048 Server Application Architecture: adr/ADR-048-server-application-architecture.md
- ADR-049 Specification Restructuring: adr/ADR-049-specification-restructuring.md
theme:
name: material