initial commit
This commit is contained in:
6
.claude/skills/bmad-testarch-trace/SKILL.md
Normal file
6
.claude/skills/bmad-testarch-trace/SKILL.md
Normal file
@@ -0,0 +1,6 @@
|
||||
---
|
||||
name: bmad-testarch-trace
|
||||
description: 'Generate traceability matrix and quality gate decision. Use when the user says "lets create traceability matrix" or "I want to analyze test coverage"'
|
||||
---
|
||||
|
||||
Follow the instructions in [workflow.md](workflow.md).
|
||||
@@ -0,0 +1 @@
|
||||
type: skill
|
||||
647
.claude/skills/bmad-testarch-trace/checklist.md
Normal file
647
.claude/skills/bmad-testarch-trace/checklist.md
Normal file
@@ -0,0 +1,647 @@
|
||||
# Requirements Traceability & Gate Decision - Validation Checklist
|
||||
|
||||
**Workflow:** `testarch-trace`
|
||||
**Purpose:** Ensure complete traceability matrix with actionable gap analysis AND make deployment readiness decision (PASS/CONCERNS/FAIL/WAIVED)
|
||||
|
||||
This checklist covers **two sequential phases**:
|
||||
|
||||
- **PHASE 1**: Requirements Traceability (always executed)
|
||||
- **PHASE 2**: Quality Gate Decision (executed if `enable_gate_decision: true`)
|
||||
|
||||
---
|
||||
|
||||
# PHASE 1: REQUIREMENTS TRACEABILITY
|
||||
|
||||
## Prerequisites Validation
|
||||
|
||||
- [ ] Acceptance criteria are available (from story file OR inline)
|
||||
- [ ] Test suite exists (or gaps are acknowledged and documented)
|
||||
- [ ] If tests are missing, recommend `*atdd` (trace does not run it automatically)
|
||||
- [ ] Test directory path is correct (`test_dir` variable)
|
||||
- [ ] Story file is accessible (if using BMad mode)
|
||||
- [ ] Knowledge base is loaded (test-priorities, traceability, risk-governance)
|
||||
|
||||
---
|
||||
|
||||
## Context Loading
|
||||
|
||||
- [ ] Story file read successfully (if applicable)
|
||||
- [ ] Acceptance criteria extracted correctly
|
||||
- [ ] Story ID identified (e.g., 1.3)
|
||||
- [ ] `test-design.md` loaded (if available)
|
||||
- [ ] `tech-spec.md` loaded (if available)
|
||||
- [ ] `PRD.md` loaded (if available)
|
||||
- [ ] Relevant knowledge fragments loaded from `tea-index.csv`
|
||||
|
||||
---
|
||||
|
||||
## Test Discovery and Cataloging
|
||||
|
||||
- [ ] Tests auto-discovered using multiple strategies (test IDs, describe blocks, file paths)
|
||||
- [ ] Tests categorized by level (E2E, API, Component, Unit)
|
||||
- [ ] Test metadata extracted:
|
||||
- [ ] Test IDs (e.g., 1.3-E2E-001)
|
||||
- [ ] Describe/context blocks
|
||||
- [ ] It blocks (individual test cases)
|
||||
- [ ] Given-When-Then structure (if BDD)
|
||||
- [ ] Priority markers (P0/P1/P2/P3)
|
||||
- [ ] All relevant test files found (no tests missed due to naming conventions)
|
||||
|
||||
---
|
||||
|
||||
## Criteria-to-Test Mapping
|
||||
|
||||
- [ ] Each acceptance criterion mapped to tests (or marked as NONE)
|
||||
- [ ] Explicit references found (test IDs, describe blocks mentioning criterion)
|
||||
- [ ] Test level documented (E2E, API, Component, Unit)
|
||||
- [ ] Given-When-Then narrative verified for alignment
|
||||
- [ ] Traceability matrix table generated:
|
||||
- [ ] Criterion ID
|
||||
- [ ] Description
|
||||
- [ ] Test ID
|
||||
- [ ] Test File
|
||||
- [ ] Test Level
|
||||
- [ ] Coverage Status
|
||||
|
||||
---
|
||||
|
||||
## Coverage Classification
|
||||
|
||||
- [ ] Coverage status classified for each criterion:
|
||||
- [ ] **FULL** - All scenarios validated at appropriate level(s)
|
||||
- [ ] **PARTIAL** - Some coverage but missing edge cases or levels
|
||||
- [ ] **NONE** - No test coverage at any level
|
||||
- [ ] **UNIT-ONLY** - Only unit tests (missing integration/E2E validation)
|
||||
- [ ] **INTEGRATION-ONLY** - Only API/Component tests (missing unit confidence)
|
||||
- [ ] Classification justifications provided
|
||||
- [ ] Edge cases considered in FULL vs PARTIAL determination
|
||||
|
||||
---
|
||||
|
||||
## Duplicate Coverage Detection
|
||||
|
||||
- [ ] Duplicate coverage checked across test levels
|
||||
- [ ] Acceptable overlap identified (defense in depth for critical paths)
|
||||
- [ ] Unacceptable duplication flagged (same validation at multiple levels)
|
||||
- [ ] Recommendations provided for consolidation
|
||||
- [ ] Selective testing principles applied
|
||||
|
||||
---
|
||||
|
||||
## Gap Analysis
|
||||
|
||||
- [ ] Coverage gaps identified:
|
||||
- [ ] Criteria with NONE status
|
||||
- [ ] Criteria with PARTIAL status
|
||||
- [ ] Criteria with UNIT-ONLY status
|
||||
- [ ] Criteria with INTEGRATION-ONLY status
|
||||
- [ ] Coverage heuristics gaps identified:
|
||||
- [ ] Endpoints referenced in requirements but not covered by API tests
|
||||
- [ ] Auth/authz criteria missing denied/invalid path tests
|
||||
- [ ] Criteria with happy-path-only coverage (missing error scenarios)
|
||||
- [ ] Gaps prioritized by risk level using test-priorities framework:
|
||||
- [ ] **CRITICAL** - P0 criteria without FULL coverage (BLOCKER)
|
||||
- [ ] **HIGH** - P1 criteria without FULL coverage (PR blocker)
|
||||
- [ ] **MEDIUM** - P2 criteria without FULL coverage (nightly gap)
|
||||
- [ ] **LOW** - P3 criteria without FULL coverage (acceptable)
|
||||
- [ ] Specific test recommendations provided for each gap:
|
||||
- [ ] Suggested test level (E2E, API, Component, Unit)
|
||||
- [ ] Test description (Given-When-Then)
|
||||
- [ ] Recommended test ID (e.g., 1.3-E2E-004)
|
||||
- [ ] Explanation of why test is needed
|
||||
|
||||
---
|
||||
|
||||
## Coverage Metrics
|
||||
|
||||
- [ ] Overall coverage percentage calculated (FULL coverage / total criteria)
|
||||
- [ ] P0 coverage percentage calculated
|
||||
- [ ] P1 coverage percentage calculated
|
||||
- [ ] P2 coverage percentage calculated (if applicable)
|
||||
- [ ] Coverage by level calculated:
|
||||
- [ ] E2E coverage %
|
||||
- [ ] API coverage %
|
||||
- [ ] Component coverage %
|
||||
- [ ] Unit coverage %
|
||||
|
||||
---
|
||||
|
||||
## Test Quality Verification
|
||||
|
||||
For each mapped test, verify:
|
||||
|
||||
- [ ] Explicit assertions are present (not hidden in helpers)
|
||||
- [ ] Test follows Given-When-Then structure
|
||||
- [ ] No hard waits or sleeps (deterministic waiting only)
|
||||
- [ ] Self-cleaning (test cleans up its data)
|
||||
- [ ] File size < 300 lines
|
||||
- [ ] Test duration < 90 seconds
|
||||
|
||||
Quality issues flagged:
|
||||
|
||||
- [ ] **BLOCKER** issues identified (missing assertions, hard waits, flaky patterns)
|
||||
- [ ] **WARNING** issues identified (large files, slow tests, unclear structure)
|
||||
- [ ] **INFO** issues identified (style inconsistencies, missing documentation)
|
||||
|
||||
Knowledge fragments referenced:
|
||||
|
||||
- [ ] `test-quality.md` for Definition of Done
|
||||
- [ ] `fixture-architecture.md` for self-cleaning patterns
|
||||
- [ ] `network-first.md` for Playwright best practices
|
||||
- [ ] `data-factories.md` for test data patterns
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 Deliverables Generated
|
||||
|
||||
### Traceability Matrix Markdown
|
||||
|
||||
- [ ] File created at `{test_artifacts}/traceability-matrix.md`
|
||||
- [ ] Template from `trace-template.md` used
|
||||
- [ ] Full mapping table included
|
||||
- [ ] Coverage status section included
|
||||
- [ ] Gap analysis section included
|
||||
- [ ] Quality assessment section included
|
||||
- [ ] Recommendations section included
|
||||
|
||||
### Coverage Badge/Metric (if enabled)
|
||||
|
||||
- [ ] Badge markdown generated
|
||||
- [ ] Metrics exported to JSON for CI/CD integration
|
||||
|
||||
### Updated Story File (if enabled)
|
||||
|
||||
- [ ] "Traceability" section added to story markdown
|
||||
- [ ] Link to traceability matrix included
|
||||
- [ ] Coverage summary included
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 Quality Assurance
|
||||
|
||||
### Accuracy Checks
|
||||
|
||||
- [ ] All acceptance criteria accounted for (none skipped)
|
||||
- [ ] Test IDs correctly formatted (e.g., 1.3-E2E-001)
|
||||
- [ ] File paths are correct and accessible
|
||||
- [ ] Coverage percentages calculated correctly
|
||||
- [ ] No false positives (tests incorrectly mapped to criteria)
|
||||
- [ ] No false negatives (existing tests missed in mapping)
|
||||
|
||||
### Completeness Checks
|
||||
|
||||
- [ ] All test levels considered (E2E, API, Component, Unit)
|
||||
- [ ] All priorities considered (P0, P1, P2, P3)
|
||||
- [ ] All coverage statuses used appropriately (FULL, PARTIAL, NONE, UNIT-ONLY, INTEGRATION-ONLY)
|
||||
- [ ] All gaps have recommendations
|
||||
- [ ] All quality issues have severity and remediation guidance
|
||||
|
||||
### Actionability Checks
|
||||
|
||||
- [ ] Recommendations are specific (not generic)
|
||||
- [ ] Test IDs suggested for new tests
|
||||
- [ ] Given-When-Then provided for recommended tests
|
||||
- [ ] Impact explained for each gap
|
||||
- [ ] Priorities clear (CRITICAL, HIGH, MEDIUM, LOW)
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 Documentation
|
||||
|
||||
- [ ] Traceability matrix is readable and well-formatted
|
||||
- [ ] Tables render correctly in markdown
|
||||
- [ ] Code blocks have proper syntax highlighting
|
||||
- [ ] Links are valid and accessible
|
||||
- [ ] Recommendations are clear and prioritized
|
||||
|
||||
---
|
||||
|
||||
# PHASE 2: QUALITY GATE DECISION
|
||||
|
||||
**Note**: Phase 2 executes only if `enable_gate_decision: true` in workflow.yaml
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Evidence Gathering
|
||||
|
||||
- [ ] Test execution results obtained (CI/CD pipeline, test framework reports)
|
||||
- [ ] Story/epic/release file identified and read
|
||||
- [ ] Test design document discovered or explicitly provided (if available)
|
||||
- [ ] Traceability matrix discovered or explicitly provided (available from Phase 1)
|
||||
- [ ] NFR assessment discovered or explicitly provided (if available)
|
||||
- [ ] Code coverage report discovered or explicitly provided (if available)
|
||||
- [ ] Burn-in results discovered or explicitly provided (if available)
|
||||
|
||||
### Evidence Validation
|
||||
|
||||
- [ ] Evidence freshness validated (warn if >7 days old, recommend re-running workflows)
|
||||
- [ ] All required assessments available or user acknowledged gaps
|
||||
- [ ] Test results are complete (not partial or interrupted runs)
|
||||
- [ ] Test results match current codebase (not from outdated branch)
|
||||
|
||||
### Knowledge Base Loading
|
||||
|
||||
- [ ] `risk-governance.md` loaded successfully
|
||||
- [ ] `probability-impact.md` loaded successfully
|
||||
- [ ] `test-quality.md` loaded successfully
|
||||
- [ ] `test-priorities.md` loaded successfully
|
||||
- [ ] `ci-burn-in.md` loaded (if burn-in results available)
|
||||
|
||||
---
|
||||
|
||||
## Process Steps
|
||||
|
||||
### Step 1: Context Loading
|
||||
|
||||
- [ ] Gate type identified (story/epic/release/hotfix)
|
||||
- [ ] Target ID extracted (story_id, epic_num, or release_version)
|
||||
- [ ] Decision thresholds loaded from workflow variables
|
||||
- [ ] Risk tolerance configuration loaded
|
||||
- [ ] Waiver policy loaded
|
||||
|
||||
### Step 2: Evidence Parsing
|
||||
|
||||
**Test Results:**
|
||||
|
||||
- [ ] Total test count extracted
|
||||
- [ ] Passed test count extracted
|
||||
- [ ] Failed test count extracted
|
||||
- [ ] Skipped test count extracted
|
||||
- [ ] Test duration extracted
|
||||
- [ ] P0 test pass rate calculated
|
||||
- [ ] P1 test pass rate calculated
|
||||
- [ ] Overall test pass rate calculated
|
||||
|
||||
**Quality Assessments:**
|
||||
|
||||
- [ ] P0/P1/P2/P3 scenarios extracted from test-design.md (if available)
|
||||
- [ ] Risk scores extracted from test-design.md (if available)
|
||||
- [ ] Coverage percentages extracted from traceability-matrix.md (available from Phase 1)
|
||||
- [ ] Coverage gaps extracted from traceability-matrix.md (available from Phase 1)
|
||||
- [ ] NFR status extracted from nfr-assessment.md (if available)
|
||||
- [ ] Security issues count extracted from nfr-assessment.md (if available)
|
||||
|
||||
**Code Coverage:**
|
||||
|
||||
- [ ] Line coverage percentage extracted (if available)
|
||||
- [ ] Branch coverage percentage extracted (if available)
|
||||
- [ ] Function coverage percentage extracted (if available)
|
||||
- [ ] Critical path coverage validated (if available)
|
||||
|
||||
**Burn-in Results:**
|
||||
|
||||
- [ ] Burn-in iterations count extracted (if available)
|
||||
- [ ] Flaky tests count extracted (if available)
|
||||
- [ ] Stability score calculated (if available)
|
||||
|
||||
### Step 3: Decision Rules Application
|
||||
|
||||
**P0 Criteria Evaluation:**
|
||||
|
||||
- [ ] P0 test pass rate evaluated (must be 100%)
|
||||
- [ ] P0 acceptance criteria coverage evaluated (must be 100%)
|
||||
- [ ] Security issues count evaluated (must be 0)
|
||||
- [ ] Critical NFR failures evaluated (must be 0)
|
||||
- [ ] Flaky tests evaluated (must be 0 if burn-in enabled)
|
||||
- [ ] P0 decision recorded: PASS or FAIL
|
||||
|
||||
**P1 Criteria Evaluation:**
|
||||
|
||||
- [ ] P1 test pass rate evaluated (threshold: min_p1_pass_rate)
|
||||
- [ ] P1 acceptance criteria coverage evaluated (PASS >=90%, CONCERNS 80-89%, FAIL <80%)
|
||||
- [ ] Overall test pass rate evaluated (threshold: min_overall_pass_rate)
|
||||
- [ ] Overall requirements coverage evaluated (threshold: >=80%)
|
||||
- [ ] Code coverage considered if available (informational unless explicitly required by policy)
|
||||
- [ ] P1 decision recorded: PASS or CONCERNS
|
||||
|
||||
**P2/P3 Criteria Evaluation:**
|
||||
|
||||
- [ ] P2 failures tracked (informational, don't block if allow_p2_failures: true)
|
||||
- [ ] P3 failures tracked (informational, don't block if allow_p3_failures: true)
|
||||
- [ ] Residual risks documented
|
||||
|
||||
**Final Decision:**
|
||||
|
||||
- [ ] Decision determined: PASS / CONCERNS / FAIL / WAIVED
|
||||
- [ ] Decision rationale documented
|
||||
- [ ] Decision is deterministic (follows rules, not arbitrary)
|
||||
|
||||
### Step 4: Documentation
|
||||
|
||||
**Gate Decision Document Created:**
|
||||
|
||||
- [ ] Story/epic/release info section complete (ID, title, description, links)
|
||||
- [ ] Decision clearly stated (PASS / CONCERNS / FAIL / WAIVED)
|
||||
- [ ] Decision date recorded
|
||||
- [ ] Evaluator recorded (user or agent name)
|
||||
|
||||
**Evidence Summary Documented:**
|
||||
|
||||
- [ ] Test results summary complete (total, passed, failed, pass rates)
|
||||
- [ ] Coverage summary complete (P0/P1 criteria, code coverage)
|
||||
- [ ] NFR validation summary complete (security, performance, reliability, maintainability)
|
||||
- [ ] Flakiness summary complete (burn-in iterations, flaky test count)
|
||||
|
||||
**Rationale Documented:**
|
||||
|
||||
- [ ] Decision rationale clearly explained
|
||||
- [ ] Key evidence highlighted
|
||||
- [ ] Assumptions and caveats noted (if any)
|
||||
|
||||
**Residual Risks Documented (if CONCERNS or WAIVED):**
|
||||
|
||||
- [ ] Unresolved P1/P2 issues listed
|
||||
- [ ] Probability × impact estimated for each risk
|
||||
- [ ] Mitigations or workarounds described
|
||||
|
||||
**Waivers Documented (if WAIVED):**
|
||||
|
||||
- [ ] Waiver reason documented (business justification)
|
||||
- [ ] Waiver approver documented (name, role)
|
||||
- [ ] Waiver expiry date documented
|
||||
- [ ] Remediation plan documented (fix in next release, due date)
|
||||
- [ ] Monitoring plan documented
|
||||
|
||||
**Critical Issues Documented (if FAIL or CONCERNS):**
|
||||
|
||||
- [ ] Top 5-10 critical issues listed
|
||||
- [ ] Priority assigned to each issue (P0/P1/P2)
|
||||
- [ ] Owner assigned to each issue
|
||||
- [ ] Due date assigned to each issue
|
||||
|
||||
**Recommendations Documented:**
|
||||
|
||||
- [ ] Next steps clearly stated for decision type
|
||||
- [ ] Deployment recommendation provided
|
||||
- [ ] Monitoring recommendations provided (if applicable)
|
||||
- [ ] Remediation recommendations provided (if applicable)
|
||||
|
||||
### Step 5: Status Updates and Notifications
|
||||
|
||||
**Gate YAML Created:**
|
||||
|
||||
- [ ] Gate YAML snippet generated with decision and criteria
|
||||
- [ ] Evidence references included in YAML
|
||||
- [ ] Next steps included in YAML
|
||||
- [ ] YAML file saved to output folder
|
||||
|
||||
**Stakeholder Notification Generated:**
|
||||
|
||||
- [ ] Notification subject line created
|
||||
- [ ] Notification body created with summary
|
||||
- [ ] Recipients identified (PM, SM, DEV lead, stakeholders)
|
||||
- [ ] Notification ready for delivery (if notify_stakeholders: true)
|
||||
|
||||
**Outputs Saved:**
|
||||
|
||||
- [ ] Gate decision document saved to `{output_file}`
|
||||
- [ ] Gate YAML saved to `{test_artifacts}/gate-decision-{target}.yaml`
|
||||
- [ ] All outputs are valid and readable
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Output Validation
|
||||
|
||||
### Gate Decision Document
|
||||
|
||||
**Completeness:**
|
||||
|
||||
- [ ] All required sections present (info, decision, evidence, rationale, next steps)
|
||||
- [ ] No placeholder text or TODOs left in document
|
||||
- [ ] All evidence references are accurate and complete
|
||||
- [ ] All links to artifacts are valid
|
||||
|
||||
**Accuracy:**
|
||||
|
||||
- [ ] Decision matches applied criteria rules
|
||||
- [ ] Test results match CI/CD pipeline output
|
||||
- [ ] Coverage percentages match reports
|
||||
- [ ] NFR status matches assessment document
|
||||
- [ ] No contradictions or inconsistencies
|
||||
|
||||
**Clarity:**
|
||||
|
||||
- [ ] Decision rationale is clear and unambiguous
|
||||
- [ ] Technical jargon is explained or avoided
|
||||
- [ ] Stakeholders can understand next steps
|
||||
- [ ] Recommendations are actionable
|
||||
|
||||
### Gate YAML
|
||||
|
||||
**Format:**
|
||||
|
||||
- [ ] YAML is valid (no syntax errors)
|
||||
- [ ] All required fields present (target, decision, date, evaluator, criteria, evidence)
|
||||
- [ ] Field values are correct data types (numbers, strings, dates)
|
||||
|
||||
**Content:**
|
||||
|
||||
- [ ] Criteria values match decision document
|
||||
- [ ] Evidence references are accurate
|
||||
- [ ] Next steps align with decision type
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Quality Checks
|
||||
|
||||
### Decision Integrity
|
||||
|
||||
- [ ] Decision is deterministic (follows rules, not arbitrary)
|
||||
- [ ] P0 failures result in FAIL decision (unless waived)
|
||||
- [ ] Security issues result in FAIL decision (unless waived - but should never be waived)
|
||||
- [ ] Waivers have business justification and approver (if WAIVED)
|
||||
- [ ] Residual risks are documented (if CONCERNS or WAIVED)
|
||||
|
||||
### Evidence-Based
|
||||
|
||||
- [ ] Decision is based on actual test results (not guesses)
|
||||
- [ ] All claims are supported by evidence
|
||||
- [ ] No assumptions without documentation
|
||||
- [ ] Evidence sources are cited (CI run IDs, report URLs)
|
||||
|
||||
### Transparency
|
||||
|
||||
- [ ] Decision rationale is transparent and auditable
|
||||
- [ ] Criteria evaluation is documented step-by-step
|
||||
- [ ] Any deviations from standard process are explained
|
||||
- [ ] Waiver justifications are clear (if applicable)
|
||||
|
||||
### Consistency
|
||||
|
||||
- [ ] Decision aligns with risk-governance knowledge fragment
|
||||
- [ ] Priority framework (P0/P1/P2/P3) applied consistently
|
||||
- [ ] Terminology consistent with test-quality knowledge fragment
|
||||
- [ ] Decision matrix followed correctly
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Integration Points
|
||||
|
||||
### CI/CD Pipeline
|
||||
|
||||
- [ ] Gate YAML is CI/CD-compatible
|
||||
- [ ] YAML can be parsed by pipeline automation
|
||||
- [ ] Decision can be used to block/allow deployments
|
||||
- [ ] Evidence references are accessible to pipeline
|
||||
|
||||
### Stakeholders
|
||||
|
||||
- [ ] Notification message is clear and actionable
|
||||
- [ ] Decision is explained in non-technical terms
|
||||
- [ ] Next steps are specific and time-bound
|
||||
- [ ] Recipients are appropriate for decision type
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Compliance and Audit
|
||||
|
||||
### Audit Trail
|
||||
|
||||
- [ ] Decision date and time recorded
|
||||
- [ ] Evaluator identified (user or agent)
|
||||
- [ ] All evidence sources cited
|
||||
- [ ] Decision criteria documented
|
||||
- [ ] Rationale clearly explained
|
||||
|
||||
### Traceability
|
||||
|
||||
- [ ] Gate decision traceable to story/epic/release
|
||||
- [ ] Evidence traceable to specific test runs
|
||||
- [ ] Assessments traceable to workflows that created them
|
||||
- [ ] Waiver traceable to approver (if applicable)
|
||||
|
||||
### Compliance
|
||||
|
||||
- [ ] Security requirements validated (no unresolved vulnerabilities)
|
||||
- [ ] Quality standards met or waived with justification
|
||||
- [ ] Regulatory requirements addressed (if applicable)
|
||||
- [ ] Documentation sufficient for external audit
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 Edge Cases and Exceptions
|
||||
|
||||
### Missing Evidence
|
||||
|
||||
- [ ] If test-design.md missing, decision still possible with test results + trace
|
||||
- [ ] If traceability-matrix.md missing, decision still possible with test results (but Phase 1 should provide it)
|
||||
- [ ] If nfr-assessment.md missing, NFR validation marked as NOT ASSESSED
|
||||
- [ ] If code coverage missing, coverage criterion marked as NOT ASSESSED
|
||||
- [ ] User acknowledged gaps in evidence or provided alternative proof
|
||||
|
||||
### Stale Evidence
|
||||
|
||||
- [ ] Evidence freshness checked (if validate_evidence_freshness: true)
|
||||
- [ ] Warnings issued for assessments >7 days old
|
||||
- [ ] User acknowledged stale evidence or re-ran workflows
|
||||
- [ ] Decision document notes any stale evidence used
|
||||
|
||||
### Conflicting Evidence
|
||||
|
||||
- [ ] Conflicts between test results and assessments resolved
|
||||
- [ ] Most recent/authoritative source identified
|
||||
- [ ] Conflict resolution documented in decision rationale
|
||||
- [ ] User consulted if conflict cannot be resolved
|
||||
|
||||
### Waiver Scenarios
|
||||
|
||||
- [ ] Waiver only used for FAIL decision (not PASS or CONCERNS)
|
||||
- [ ] Waiver has business justification (not technical convenience)
|
||||
- [ ] Waiver has named approver with authority (VP/CTO/PO)
|
||||
- [ ] Waiver has expiry date (does NOT apply to future releases)
|
||||
- [ ] Waiver has remediation plan with concrete due date
|
||||
- [ ] Security vulnerabilities are NOT waived (enforced)
|
||||
|
||||
---
|
||||
|
||||
# FINAL VALIDATION (Both Phases)
|
||||
|
||||
## Non-Prescriptive Validation
|
||||
|
||||
- [ ] Traceability format adapted to team needs (not rigid template)
|
||||
- [ ] Examples are minimal and focused on patterns
|
||||
- [ ] Teams can extend with custom classifications
|
||||
- [ ] Integration with external systems supported (JIRA, Azure DevOps)
|
||||
- [ ] Compliance requirements considered (if applicable)
|
||||
|
||||
---
|
||||
|
||||
## Documentation and Communication
|
||||
|
||||
- [ ] All documents are readable and well-formatted
|
||||
- [ ] Tables render correctly in markdown
|
||||
- [ ] Code blocks have proper syntax highlighting
|
||||
- [ ] Links are valid and accessible
|
||||
- [ ] Recommendations are clear and prioritized
|
||||
- [ ] Gate decision is prominent and unambiguous (Phase 2)
|
||||
|
||||
---
|
||||
|
||||
## Final Validation
|
||||
|
||||
**Phase 1 (Traceability):**
|
||||
|
||||
- [ ] All prerequisites met
|
||||
- [ ] All acceptance criteria mapped or gaps documented
|
||||
- [ ] P0 coverage is 100% OR documented as BLOCKER
|
||||
- [ ] Gap analysis is complete and prioritized
|
||||
- [ ] Test quality issues identified and flagged
|
||||
- [ ] Deliverables generated and saved
|
||||
|
||||
**Phase 2 (Gate Decision):**
|
||||
|
||||
- [ ] All quality evidence gathered
|
||||
- [ ] Decision criteria applied correctly
|
||||
- [ ] Decision rationale documented
|
||||
- [ ] Gate YAML ready for CI/CD integration
|
||||
- [ ] Status file updated (if enabled)
|
||||
- [ ] Stakeholders notified (if enabled)
|
||||
|
||||
**Workflow Complete:**
|
||||
|
||||
- [ ] Phase 1 completed successfully
|
||||
- [ ] Phase 2 completed successfully (if enabled)
|
||||
- [ ] All outputs validated and saved
|
||||
- [ ] Ready to proceed based on gate decision
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off
|
||||
|
||||
**Phase 1 - Traceability Status:**
|
||||
|
||||
- [ ] ✅ PASS - All quality gates met, no critical gaps
|
||||
- [ ] ⚠️ WARN - P1 gaps exist, address before PR merge
|
||||
- [ ] ❌ FAIL - P0 gaps exist, BLOCKER for release
|
||||
|
||||
**Phase 2 - Gate Decision Status (if enabled):**
|
||||
|
||||
- [ ] ✅ PASS - Deploy to production
|
||||
- [ ] ⚠️ CONCERNS - Deploy with monitoring
|
||||
- [ ] ❌ FAIL - Block deployment, fix issues
|
||||
- [ ] 🔓 WAIVED - Deploy with business approval and remediation plan
|
||||
|
||||
**Next Actions:**
|
||||
|
||||
- If PASS (both phases): Proceed to deployment
|
||||
- If WARN/CONCERNS: Address gaps/issues, proceed with monitoring
|
||||
- If FAIL (either phase): Run `*atdd` for missing tests, fix issues, re-run `*trace`
|
||||
- If WAIVED: Deploy with approved waiver, schedule remediation
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
Record any issues, deviations, or important observations during workflow execution:
|
||||
|
||||
- **Phase 1 Issues**: [Note any traceability mapping challenges, missing tests, quality concerns]
|
||||
- **Phase 2 Issues**: [Note any missing, stale, or conflicting evidence]
|
||||
- **Decision Rationale**: [Document any nuanced reasoning or edge cases]
|
||||
- **Waiver Details**: [Document waiver negotiations or approvals]
|
||||
- **Follow-up Actions**: [List any actions required after gate decision]
|
||||
|
||||
---
|
||||
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
43
.claude/skills/bmad-testarch-trace/instructions.md
Normal file
43
.claude/skills/bmad-testarch-trace/instructions.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# Requirements Traceability & Quality Gate
|
||||
|
||||
**Workflow:** `bmad-testarch-trace`
|
||||
**Version:** 5.0 (Step-File Architecture)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Create a requirements-to-tests traceability matrix, analyze coverage gaps, and optionally make a gate decision (PASS/CONCERNS/FAIL/WAIVED) based on evidence.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **step-file architecture**:
|
||||
|
||||
- **Micro-file Design**: Each step is self-contained
|
||||
- **JIT Loading**: Only the current step file is in memory
|
||||
- **Sequential Enforcement**: Execute steps in order
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Configuration Loading
|
||||
|
||||
From `workflow.yaml`, resolve:
|
||||
|
||||
- `config_source`, `test_artifacts`, `user_name`, `communication_language`, `document_output_language`, `date`
|
||||
- `test_dir`, `source_dir`, `coverage_levels`, `gate_type`, `decision_mode`
|
||||
|
||||
### 2. First Step
|
||||
|
||||
Load, read completely, and execute:
|
||||
`./steps-c/step-01-load-context.md`
|
||||
|
||||
### 3. Resume Support
|
||||
|
||||
If the user selects **Resume** mode, load, read completely, and execute:
|
||||
`./steps-c/step-01b-resume.md`
|
||||
|
||||
This checks the output document for progress tracking frontmatter and routes to the next incomplete step.
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
name: 'step-01-load-context'
|
||||
description: 'Load requirements, knowledge base, and related artifacts'
|
||||
nextStepFile: './step-02-discover-tests.md'
|
||||
knowledgeIndex: '{project-root}/_bmad/tea/testarch/tea-index.csv'
|
||||
outputFile: '{test_artifacts}/traceability-report.md'
|
||||
---
|
||||
|
||||
# Step 1: Load Context & Knowledge Base
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Gather acceptance criteria, priorities, and supporting artifacts for traceability.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Acceptance criteria available (story or provided inline)
|
||||
- Tests exist OR gaps explicitly acknowledged
|
||||
|
||||
If acceptance criteria are missing, **HALT** and request them.
|
||||
|
||||
---
|
||||
|
||||
## 2. Load Knowledge Base
|
||||
|
||||
From `{knowledgeIndex}` load:
|
||||
|
||||
- `test-priorities-matrix.md`
|
||||
- `risk-governance.md`
|
||||
- `probability-impact.md`
|
||||
- `test-quality.md`
|
||||
- `selective-testing.md`
|
||||
|
||||
---
|
||||
|
||||
## 3. Load Artifacts
|
||||
|
||||
If available:
|
||||
|
||||
- Story file and acceptance criteria
|
||||
- Test design doc (priorities)
|
||||
- Tech spec / PRD
|
||||
|
||||
Summarize what was found.
|
||||
|
||||
---
|
||||
|
||||
### 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-01-load-context']
|
||||
lastStep: 'step-01-load-context'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-01-load-context'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-01-load-context'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
102
.claude/skills/bmad-testarch-trace/steps-c/step-01b-resume.md
Normal file
102
.claude/skills/bmad-testarch-trace/steps-c/step-01b-resume.md
Normal file
@@ -0,0 +1,102 @@
|
||||
---
|
||||
name: 'step-01b-resume'
|
||||
description: 'Resume interrupted workflow from last completed step'
|
||||
outputFile: '{test_artifacts}/traceability-report.md'
|
||||
---
|
||||
|
||||
# Step 1b: Resume Workflow
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Resume an interrupted workflow by loading the existing output document, displaying progress, and routing to the next incomplete step.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Output document with progress frontmatter
|
||||
- Focus: Load progress and route to next step
|
||||
- Limits: Do not re-execute completed steps
|
||||
- Dependencies: Output document must exist from a previous run
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
### 1. Load Output Document
|
||||
|
||||
Read `{outputFile}` and parse YAML frontmatter for:
|
||||
|
||||
- `stepsCompleted` — array of completed step names
|
||||
- `lastStep` — last completed step name
|
||||
- `lastSaved` — timestamp of last save
|
||||
|
||||
**If `{outputFile}` does not exist**, display:
|
||||
|
||||
"⚠️ **No previous progress found.** There is no output document to resume from. Please use **[C] Create** to start a fresh workflow run."
|
||||
|
||||
**THEN:** Halt. Do not proceed.
|
||||
|
||||
---
|
||||
|
||||
### 2. Display Progress Dashboard
|
||||
|
||||
Display:
|
||||
|
||||
"📋 **Workflow Resume — Requirements Traceability & Quality Gate**
|
||||
|
||||
**Last saved:** {lastSaved}
|
||||
**Steps completed:** {stepsCompleted.length} of 5
|
||||
|
||||
1. Load Context (step-01-load-context) — {✅ if in stepsCompleted, ⬜ otherwise}
|
||||
2. Discover Tests (step-02-discover-tests) — {✅ if in stepsCompleted, ⬜ otherwise}
|
||||
3. Map Criteria (step-03-map-criteria) — {✅ if in stepsCompleted, ⬜ otherwise}
|
||||
4. Analyze Gaps (step-04-analyze-gaps) — {✅ if in stepsCompleted, ⬜ otherwise}
|
||||
5. Gate Decision (step-05-gate-decision) — {✅ if in stepsCompleted, ⬜ otherwise}"
|
||||
|
||||
---
|
||||
|
||||
### 3. Route to Next Step
|
||||
|
||||
Based on `lastStep`, load the next incomplete step:
|
||||
|
||||
- `'step-01-load-context'` → Load `./step-02-discover-tests.md`
|
||||
- `'step-02-discover-tests'` → Load `./step-03-map-criteria.md`
|
||||
- `'step-03-map-criteria'` → Load `./step-04-analyze-gaps.md`
|
||||
- `'step-04-analyze-gaps'` → Load `./step-05-gate-decision.md`
|
||||
- `'step-05-gate-decision'` → **Workflow already complete.** Display: "✅ **All steps completed.** Use **[V] Validate** to review outputs or **[E] Edit** to make revisions." Then halt.
|
||||
|
||||
**If `lastStep` does not match any value above**, display: "⚠️ **Unknown progress state** (`lastStep`: {lastStep}). Please use **[C] Create** to start fresh." Then halt.
|
||||
|
||||
**Otherwise**, load the identified step file, read completely, and execute.
|
||||
|
||||
The existing content in `{outputFile}` provides context from previously completed steps. Use it as reference for remaining steps.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Output document loaded and parsed correctly
|
||||
- Progress dashboard displayed accurately
|
||||
- Routed to correct next step
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Not loading output document
|
||||
- Incorrect progress display
|
||||
- Routing to wrong step
|
||||
- Re-executing completed steps
|
||||
|
||||
**Master Rule:** Resume MUST route to the exact next incomplete step. Never re-execute completed steps.
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
name: 'step-02-discover-tests'
|
||||
description: 'Discover and catalog tests by level'
|
||||
nextStepFile: './step-03-map-criteria.md'
|
||||
outputFile: '{test_artifacts}/traceability-report.md'
|
||||
---
|
||||
|
||||
# Step 2: Discover & Catalog Tests
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Identify tests relevant to the requirements and classify by test level.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Discover Tests
|
||||
|
||||
Search `{test_dir}` for:
|
||||
|
||||
- Test IDs (e.g., `1.3-E2E-001`)
|
||||
- Feature name matches
|
||||
- Spec patterns (`*.spec.*`, `*.test.*`)
|
||||
|
||||
---
|
||||
|
||||
## 2. Categorize by Level
|
||||
|
||||
Classify as:
|
||||
|
||||
- E2E
|
||||
- API
|
||||
- Component
|
||||
- Unit
|
||||
|
||||
Record test IDs, describe blocks, and priority markers if present.
|
||||
|
||||
---
|
||||
|
||||
## 3. Build Coverage Heuristics Inventory
|
||||
|
||||
Capture explicit coverage signals so Phase 1 can detect common blind spots:
|
||||
|
||||
- API endpoint coverage
|
||||
- Inventory endpoints referenced by requirements/specs and endpoints exercised by API tests
|
||||
- Mark endpoints with no direct tests
|
||||
- Authentication/authorization coverage
|
||||
- Detect tests for login/session/token flows and permission-denied paths
|
||||
- Mark auth/authz requirements with missing negative-path tests
|
||||
- Error-path coverage
|
||||
- Detect validation, timeout, network-failure, and server-error scenarios
|
||||
- Mark criteria with happy-path-only tests
|
||||
|
||||
Record these findings in step output as `coverage_heuristics` for Step 3/4.
|
||||
|
||||
---
|
||||
|
||||
### 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-02-discover-tests']
|
||||
lastStep: 'step-02-discover-tests'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-02-discover-tests'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-02-discover-tests'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: 'step-03-map-criteria'
|
||||
description: 'Map acceptance criteria to tests and build traceability matrix'
|
||||
nextStepFile: './step-04-analyze-gaps.md'
|
||||
outputFile: '{test_artifacts}/traceability-report.md'
|
||||
---
|
||||
|
||||
# Step 3: Map Criteria to Tests
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Create the traceability matrix linking requirements to tests.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Build Matrix
|
||||
|
||||
For each acceptance criterion:
|
||||
|
||||
- Map to matching tests
|
||||
- Mark coverage status: FULL / PARTIAL / NONE / UNIT-ONLY / INTEGRATION-ONLY
|
||||
- Record test level and priority
|
||||
- Record heuristic signals:
|
||||
- Endpoint coverage present/missing (for API-impacting criteria)
|
||||
- Auth/authz coverage present/missing (positive and negative paths)
|
||||
- Error-path coverage present/missing (validation, timeout, network/server failures)
|
||||
|
||||
---
|
||||
|
||||
## 2. Validate Coverage Logic
|
||||
|
||||
Ensure:
|
||||
|
||||
- P0/P1 criteria have coverage
|
||||
- No duplicate coverage across levels without justification
|
||||
- Criteria are not happy-path-only when requirements imply error handling
|
||||
- API criteria are not marked FULL if endpoint-level checks are missing
|
||||
- Auth/authz criteria include at least one denied/invalid-path test where applicable
|
||||
|
||||
---
|
||||
|
||||
### 3. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-03-map-criteria']
|
||||
lastStep: 'step-03-map-criteria'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-03-map-criteria'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-03-map-criteria'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,421 @@
|
||||
---
|
||||
name: 'step-04-analyze-gaps'
|
||||
description: 'Complete Phase 1 with adaptive orchestration (agent-team, subagent, or sequential)'
|
||||
nextStepFile: './step-05-gate-decision.md'
|
||||
outputFile: '{test_artifacts}/traceability-report.md'
|
||||
tempOutputFile: '/tmp/tea-trace-coverage-matrix-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Step 4: Complete Phase 1 - Coverage Matrix Generation
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
**Phase 1 Final Step:** Analyze coverage gaps (including endpoint/auth/error-path blind spots), generate recommendations, and output complete coverage matrix to temp file for Phase 2 (gate decision).
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Output coverage matrix to temp file
|
||||
- ✅ Resolve execution mode from explicit user request first, then config
|
||||
- ✅ Apply fallback rules deterministically when requested mode is unsupported
|
||||
- ❌ Do NOT make gate decision (that's Phase 2 - Step 5)
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: requirements from Step 1, tests from Step 2, traceability matrix from Step 3
|
||||
- Focus: gap analysis and matrix completion
|
||||
- Limits: do not make gate decision (Phase 2 responsibility)
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
### 0. Resolve Execution Mode (User Override First)
|
||||
|
||||
```javascript
|
||||
const parseBooleanFlag = (value, defaultValue = true) => {
|
||||
if (typeof value === 'string') {
|
||||
const normalized = value.trim().toLowerCase();
|
||||
if (['false', '0', 'off', 'no'].includes(normalized)) return false;
|
||||
if (['true', '1', 'on', 'yes'].includes(normalized)) return true;
|
||||
}
|
||||
if (value === undefined || value === null) return defaultValue;
|
||||
return Boolean(value);
|
||||
};
|
||||
|
||||
const orchestrationContext = {
|
||||
config: {
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
},
|
||||
timestamp: new Date().toISOString().replace(/[:.]/g, '-'),
|
||||
};
|
||||
|
||||
const normalizeUserExecutionMode = (mode) => {
|
||||
if (typeof mode !== 'string') return null;
|
||||
const normalized = mode.trim().toLowerCase().replace(/[-_]/g, ' ').replace(/\s+/g, ' ');
|
||||
|
||||
if (normalized === 'auto') return 'auto';
|
||||
if (normalized === 'sequential') return 'sequential';
|
||||
if (normalized === 'subagent' || normalized === 'sub agent' || normalized === 'subagents' || normalized === 'sub agents') {
|
||||
return 'subagent';
|
||||
}
|
||||
if (normalized === 'agent team' || normalized === 'agent teams' || normalized === 'agentteam') {
|
||||
return 'agent-team';
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const normalizeConfigExecutionMode = (mode) => {
|
||||
if (mode === 'subagent') return 'subagent';
|
||||
if (mode === 'auto' || mode === 'sequential' || mode === 'subagent' || mode === 'agent-team') {
|
||||
return mode;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Explicit user instruction in the active run takes priority over config.
|
||||
const explicitModeFromUser = normalizeUserExecutionMode(runtime.getExplicitExecutionModeHint?.() || null);
|
||||
|
||||
const requestedMode = explicitModeFromUser || normalizeConfigExecutionMode(orchestrationContext.config.execution_mode) || 'auto';
|
||||
const probeEnabled = orchestrationContext.config.capability_probe;
|
||||
|
||||
const supports = { subagent: false, agentTeam: false };
|
||||
if (probeEnabled) {
|
||||
supports.subagent = runtime.canLaunchSubagents?.() === true;
|
||||
supports.agentTeam = runtime.canLaunchAgentTeams?.() === true;
|
||||
}
|
||||
|
||||
let resolvedMode = requestedMode;
|
||||
if (requestedMode === 'auto') {
|
||||
if (supports.agentTeam) resolvedMode = 'agent-team';
|
||||
else if (supports.subagent) resolvedMode = 'subagent';
|
||||
else resolvedMode = 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'agent-team' && !supports.agentTeam) {
|
||||
resolvedMode = supports.subagent ? 'subagent' : 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'subagent' && !supports.subagent) {
|
||||
resolvedMode = 'sequential';
|
||||
}
|
||||
```
|
||||
|
||||
Resolution precedence:
|
||||
|
||||
1. Explicit user request in this run (`agent team` => `agent-team`; `subagent` => `subagent`; `sequential`; `auto`)
|
||||
2. `tea_execution_mode` from config
|
||||
3. Runtime capability fallback (when probing enabled)
|
||||
|
||||
### 1. Gap Analysis
|
||||
|
||||
**Identify uncovered requirements:**
|
||||
|
||||
```javascript
|
||||
const uncoveredRequirements = traceabilityMatrix.filter((req) => req.coverage === 'NONE');
|
||||
const partialCoverage = traceabilityMatrix.filter((req) => req.coverage === 'PARTIAL');
|
||||
const unitOnlyCoverage = traceabilityMatrix.filter((req) => req.coverage === 'UNIT-ONLY');
|
||||
```
|
||||
|
||||
**Prioritize gaps by risk:**
|
||||
|
||||
```javascript
|
||||
const criticalGaps = uncoveredRequirements.filter((req) => req.priority === 'P0');
|
||||
const highGaps = uncoveredRequirements.filter((req) => req.priority === 'P1');
|
||||
const mediumGaps = uncoveredRequirements.filter((req) => req.priority === 'P2');
|
||||
const lowGaps = uncoveredRequirements.filter((req) => req.priority === 'P3');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Coverage Heuristics Checks
|
||||
|
||||
Use the heuristics inventory from Step 2 and mapped criteria from Step 3 to flag common coverage blind spots:
|
||||
|
||||
```javascript
|
||||
const endpointCoverageGaps = coverageHeuristics?.endpoints_without_tests || [];
|
||||
const authCoverageGaps = coverageHeuristics?.auth_missing_negative_paths || [];
|
||||
const errorPathGaps = coverageHeuristics?.criteria_happy_path_only || [];
|
||||
|
||||
const heuristicGapCounts = {
|
||||
endpoints_without_tests: endpointCoverageGaps.length,
|
||||
auth_missing_negative_paths: authCoverageGaps.length,
|
||||
happy_path_only_criteria: errorPathGaps.length,
|
||||
};
|
||||
```
|
||||
|
||||
Heuristics are advisory but must influence gap severity and recommendations, especially for P0/P1 criteria.
|
||||
|
||||
---
|
||||
|
||||
### 3. Generate Recommendations
|
||||
|
||||
**Based on gap analysis:**
|
||||
|
||||
```javascript
|
||||
const recommendations = [];
|
||||
|
||||
// Critical gaps (P0)
|
||||
if (criticalGaps.length > 0) {
|
||||
recommendations.push({
|
||||
priority: 'URGENT',
|
||||
action: `Run /bmad:tea:atdd for ${criticalGaps.length} P0 requirements`,
|
||||
requirements: criticalGaps.map((r) => r.id),
|
||||
});
|
||||
}
|
||||
|
||||
// High priority gaps (P1)
|
||||
if (highGaps.length > 0) {
|
||||
recommendations.push({
|
||||
priority: 'HIGH',
|
||||
action: `Run /bmad:tea:automate to expand coverage for ${highGaps.length} P1 requirements`,
|
||||
requirements: highGaps.map((r) => r.id),
|
||||
});
|
||||
}
|
||||
|
||||
// Partial coverage
|
||||
if (partialCoverage.length > 0) {
|
||||
recommendations.push({
|
||||
priority: 'MEDIUM',
|
||||
action: `Complete coverage for ${partialCoverage.length} partially covered requirements`,
|
||||
requirements: partialCoverage.map((r) => r.id),
|
||||
});
|
||||
}
|
||||
|
||||
if (endpointCoverageGaps.length > 0) {
|
||||
recommendations.push({
|
||||
priority: 'HIGH',
|
||||
action: `Add API tests for ${endpointCoverageGaps.length} uncovered endpoint(s)`,
|
||||
requirements: endpointCoverageGaps.map((r) => r.id || r.endpoint || 'unknown'),
|
||||
});
|
||||
}
|
||||
|
||||
if (authCoverageGaps.length > 0) {
|
||||
recommendations.push({
|
||||
priority: 'HIGH',
|
||||
action: `Add negative-path auth/authz tests for ${authCoverageGaps.length} requirement(s)`,
|
||||
requirements: authCoverageGaps.map((r) => r.id || 'unknown'),
|
||||
});
|
||||
}
|
||||
|
||||
if (errorPathGaps.length > 0) {
|
||||
recommendations.push({
|
||||
priority: 'MEDIUM',
|
||||
action: `Add error/edge scenario tests for ${errorPathGaps.length} happy-path-only criterion/criteria`,
|
||||
requirements: errorPathGaps.map((r) => r.id || 'unknown'),
|
||||
});
|
||||
}
|
||||
|
||||
// Quality issues
|
||||
recommendations.push({
|
||||
priority: 'LOW',
|
||||
action: 'Run /bmad:tea:test-review to assess test quality',
|
||||
requirements: [],
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Calculate Coverage Statistics
|
||||
|
||||
```javascript
|
||||
const totalRequirements = traceabilityMatrix.length;
|
||||
const coveredRequirements = traceabilityMatrix.filter((r) => r.coverage === 'FULL' || r.coverage === 'PARTIAL').length;
|
||||
const fullyCovered = traceabilityMatrix.filter((r) => r.coverage === 'FULL').length;
|
||||
|
||||
const safePct = (covered, total) => (total > 0 ? Math.round((covered / total) * 100) : 100);
|
||||
const coveragePercentage = safePct(fullyCovered, totalRequirements);
|
||||
|
||||
// Priority-specific coverage
|
||||
const p0Total = traceabilityMatrix.filter((r) => r.priority === 'P0').length;
|
||||
const p0Covered = traceabilityMatrix.filter((r) => r.priority === 'P0' && r.coverage === 'FULL').length;
|
||||
const p1Total = traceabilityMatrix.filter((r) => r.priority === 'P1').length;
|
||||
const p1Covered = traceabilityMatrix.filter((r) => r.priority === 'P1' && r.coverage === 'FULL').length;
|
||||
const p2Total = traceabilityMatrix.filter((r) => r.priority === 'P2').length;
|
||||
const p2Covered = traceabilityMatrix.filter((r) => r.priority === 'P2' && r.coverage === 'FULL').length;
|
||||
const p3Total = traceabilityMatrix.filter((r) => r.priority === 'P3').length;
|
||||
const p3Covered = traceabilityMatrix.filter((r) => r.priority === 'P3' && r.coverage === 'FULL').length;
|
||||
|
||||
const p0CoveragePercentage = safePct(p0Covered, p0Total);
|
||||
const p1CoveragePercentage = safePct(p1Covered, p1Total);
|
||||
const p2CoveragePercentage = safePct(p2Covered, p2Total);
|
||||
const p3CoveragePercentage = safePct(p3Covered, p3Total);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Generate Complete Coverage Matrix
|
||||
|
||||
**Compile all Phase 1 outputs:**
|
||||
|
||||
```javascript
|
||||
const coverageMatrix = {
|
||||
phase: 'PHASE_1_COMPLETE',
|
||||
generated_at: new Date().toISOString(),
|
||||
|
||||
requirements: traceabilityMatrix, // Full matrix from Step 3
|
||||
|
||||
coverage_statistics: {
|
||||
total_requirements: totalRequirements,
|
||||
fully_covered: fullyCovered,
|
||||
partially_covered: partialCoverage.length,
|
||||
uncovered: uncoveredRequirements.length,
|
||||
overall_coverage_percentage: coveragePercentage,
|
||||
|
||||
priority_breakdown: {
|
||||
P0: { total: p0Total, covered: p0Covered, percentage: p0CoveragePercentage },
|
||||
P1: { total: p1Total, covered: p1Covered, percentage: p1CoveragePercentage },
|
||||
P2: { total: p2Total, covered: p2Covered, percentage: p2CoveragePercentage },
|
||||
P3: { total: p3Total, covered: p3Covered, percentage: p3CoveragePercentage },
|
||||
},
|
||||
},
|
||||
|
||||
gap_analysis: {
|
||||
critical_gaps: criticalGaps,
|
||||
high_gaps: highGaps,
|
||||
medium_gaps: mediumGaps,
|
||||
low_gaps: lowGaps,
|
||||
partial_coverage_items: partialCoverage,
|
||||
unit_only_items: unitOnlyCoverage,
|
||||
},
|
||||
|
||||
coverage_heuristics: {
|
||||
endpoint_gaps: endpointCoverageGaps,
|
||||
auth_negative_path_gaps: authCoverageGaps,
|
||||
happy_path_only_gaps: errorPathGaps,
|
||||
counts: heuristicGapCounts,
|
||||
},
|
||||
|
||||
recommendations: recommendations,
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Output Coverage Matrix to Temp File
|
||||
|
||||
**Write to temp file for Phase 2:**
|
||||
|
||||
```javascript
|
||||
const outputPath = '{tempOutputFile}';
|
||||
fs.writeFileSync(outputPath, JSON.stringify(coverageMatrix, null, 2), 'utf8');
|
||||
|
||||
console.log(`✅ Phase 1 Complete: Coverage matrix saved to ${outputPath}`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Display Phase 1 Summary
|
||||
|
||||
```
|
||||
✅ Phase 1 Complete: Coverage Matrix Generated
|
||||
|
||||
📊 Coverage Statistics:
|
||||
- Total Requirements: {totalRequirements}
|
||||
- Fully Covered: {fullyCovered} ({coveragePercentage}%)
|
||||
- Partially Covered: {partialCoverage.length}
|
||||
- Uncovered: {uncoveredRequirements.length}
|
||||
|
||||
🎯 Priority Coverage:
|
||||
- P0: {p0Covered}/{p0Total} ({p0CoveragePercentage}%)
|
||||
- P1: {p1Covered}/{p1Total} ({p1CoveragePercentage}%)
|
||||
- P2: {p2Covered}/{p2Total} ({p2CoveragePercentage}%)
|
||||
- P3: {p3Covered}/{p3Total} ({p3CoveragePercentage}%)
|
||||
|
||||
⚠️ Gaps Identified:
|
||||
- Critical (P0): {criticalGaps.length}
|
||||
- High (P1): {highGaps.length}
|
||||
- Medium (P2): {mediumGaps.length}
|
||||
- Low (P3): {lowGaps.length}
|
||||
|
||||
🔍 Coverage Heuristics:
|
||||
- Endpoints without tests: {endpointCoverageGaps.length}
|
||||
- Auth negative-path gaps: {authCoverageGaps.length}
|
||||
- Happy-path-only criteria: {errorPathGaps.length}
|
||||
|
||||
📝 Recommendations: {recommendations.length}
|
||||
|
||||
🔄 Phase 2: Gate decision (next step)
|
||||
```
|
||||
|
||||
### Orchestration Notes for This Step
|
||||
|
||||
When `resolvedMode` is `agent-team` or `subagent`, parallelize only dependency-safe sections:
|
||||
|
||||
- Worker A: gap classification (section 1)
|
||||
- Worker B: heuristics gap extraction (section 2)
|
||||
- Worker C: coverage statistics (section 4)
|
||||
|
||||
Section 3 (recommendation synthesis) depends on outputs from sections 1 and 2, so run it only after Workers A and B complete.
|
||||
|
||||
Section 5 remains the deterministic merge point after sections 1-4 are finished.
|
||||
|
||||
If `resolvedMode` is `sequential`, execute sections 1→7 in order.
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
**PHASE 1 COMPLETE when:**
|
||||
|
||||
- ✅ Gap analysis complete
|
||||
- ✅ Recommendations generated
|
||||
- ✅ Coverage statistics calculated
|
||||
- ✅ Coverage matrix saved to temp file
|
||||
- ✅ Summary displayed
|
||||
|
||||
**Proceed to Phase 2 (Step 5: Gate Decision)**
|
||||
|
||||
---
|
||||
|
||||
### 8. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-04-analyze-gaps']
|
||||
lastStep: 'step-04-analyze-gaps'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-04-analyze-gaps'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-04-analyze-gaps'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
---
|
||||
|
||||
## 🚨 PHASE 1 SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Coverage matrix complete and accurate
|
||||
- All gaps identified and prioritized
|
||||
- Recommendations actionable
|
||||
- Temp file output valid JSON
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Coverage matrix incomplete
|
||||
- Gap analysis missing
|
||||
- Invalid JSON output
|
||||
|
||||
**Master Rule:** Phase 1 MUST output complete coverage matrix to temp file before Phase 2 can proceed.
|
||||
@@ -0,0 +1,266 @@
|
||||
---
|
||||
name: 'step-05-gate-decision'
|
||||
description: 'Phase 2: Apply gate decision logic and generate outputs'
|
||||
outputFile: '{test_artifacts}/traceability-report.md'
|
||||
---
|
||||
|
||||
# Step 5: Phase 2 - Gate Decision
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
**Phase 2:** Read coverage matrix from Phase 1, apply deterministic gate decision logic, and generate traceability report.
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Read coverage matrix from Phase 1 temp file
|
||||
- ✅ Apply gate decision logic
|
||||
- ❌ Do NOT regenerate coverage matrix (use Phase 1 output)
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 This is the FINAL step
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Coverage matrix from Phase 1 temp file
|
||||
- Focus: gate decision logic only
|
||||
- Dependencies: Phase 1 complete (coverage matrix exists)
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
### 1. Read Phase 1 Coverage Matrix
|
||||
|
||||
```javascript
|
||||
const matrixPath = '/tmp/tea-trace-coverage-matrix-{{timestamp}}.json';
|
||||
const coverageMatrix = JSON.parse(fs.readFileSync(matrixPath, 'utf8'));
|
||||
|
||||
console.log('✅ Phase 1 coverage matrix loaded');
|
||||
```
|
||||
|
||||
**Verify Phase 1 complete:**
|
||||
|
||||
```javascript
|
||||
if (coverageMatrix.phase !== 'PHASE_1_COMPLETE') {
|
||||
throw new Error('Phase 1 not complete - cannot proceed to gate decision');
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Apply Gate Decision Logic
|
||||
|
||||
**Decision Tree:**
|
||||
|
||||
```javascript
|
||||
const stats = coverageMatrix.coverage_statistics;
|
||||
const p0Coverage = stats.priority_breakdown.P0.percentage;
|
||||
const p1Coverage = stats.priority_breakdown.P1.percentage;
|
||||
const hasP1Requirements = (stats.priority_breakdown.P1.total || 0) > 0;
|
||||
const effectiveP1Coverage = hasP1Requirements ? p1Coverage : 100;
|
||||
const overallCoverage = stats.overall_coverage_percentage;
|
||||
const criticalGaps = coverageMatrix.gap_analysis.critical_gaps.length;
|
||||
|
||||
let gateDecision;
|
||||
let rationale;
|
||||
|
||||
// Rule 1: P0 coverage must be 100%
|
||||
if (p0Coverage < 100) {
|
||||
gateDecision = 'FAIL';
|
||||
rationale = `P0 coverage is ${p0Coverage}% (required: 100%). ${criticalGaps} critical requirements uncovered.`;
|
||||
}
|
||||
// Rule 2: Overall coverage must be >= 80%
|
||||
else if (overallCoverage < 80) {
|
||||
gateDecision = 'FAIL';
|
||||
rationale = `Overall coverage is ${overallCoverage}% (minimum: 80%). Significant gaps exist.`;
|
||||
}
|
||||
// Rule 3: P1 coverage < 80% → FAIL
|
||||
else if (effectiveP1Coverage < 80) {
|
||||
gateDecision = 'FAIL';
|
||||
rationale = hasP1Requirements
|
||||
? `P1 coverage is ${effectiveP1Coverage}% (minimum: 80%). High-priority gaps must be addressed.`
|
||||
: `P1 requirements are not present; continuing with remaining gate criteria.`;
|
||||
}
|
||||
// Rule 4: P1 coverage >= 90% and overall >= 80% with P0 at 100% → PASS
|
||||
else if (effectiveP1Coverage >= 90) {
|
||||
gateDecision = 'PASS';
|
||||
rationale = hasP1Requirements
|
||||
? `P0 coverage is 100%, P1 coverage is ${effectiveP1Coverage}% (target: 90%), and overall coverage is ${overallCoverage}% (minimum: 80%).`
|
||||
: `P0 coverage is 100% and overall coverage is ${overallCoverage}% (minimum: 80%). No P1 requirements detected.`;
|
||||
}
|
||||
// Rule 5: P1 coverage 80-89% with P0 at 100% and overall >= 80% → CONCERNS
|
||||
else if (effectiveP1Coverage >= 80) {
|
||||
gateDecision = 'CONCERNS';
|
||||
rationale = hasP1Requirements
|
||||
? `P0 coverage is 100% and overall coverage is ${overallCoverage}% (minimum: 80%), but P1 coverage is ${effectiveP1Coverage}% (target: 90%).`
|
||||
: `P0 coverage is 100% and overall coverage is ${overallCoverage}% (minimum: 80%), but additional non-P1 gaps need mitigation.`;
|
||||
}
|
||||
|
||||
// Rule 6: Manual waiver option
|
||||
const manualWaiver = false; // Can be set via config or user input
|
||||
if (manualWaiver) {
|
||||
gateDecision = 'WAIVED';
|
||||
rationale += ' Manual waiver applied by stakeholder.';
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Generate Gate Report
|
||||
|
||||
```javascript
|
||||
const gateReport = {
|
||||
decision: gateDecision,
|
||||
rationale: rationale,
|
||||
decision_date: new Date().toISOString(),
|
||||
|
||||
coverage_matrix: coverageMatrix,
|
||||
|
||||
gate_criteria: {
|
||||
p0_coverage_required: '100%',
|
||||
p0_coverage_actual: `${p0Coverage}%`,
|
||||
p0_status: p0Coverage === 100 ? 'MET' : 'NOT MET',
|
||||
|
||||
p1_coverage_target_pass: '90%',
|
||||
p1_coverage_minimum: '80%',
|
||||
p1_coverage_actual: `${effectiveP1Coverage}%`,
|
||||
p1_status: effectiveP1Coverage >= 90 ? 'MET' : effectiveP1Coverage >= 80 ? 'PARTIAL' : 'NOT MET',
|
||||
|
||||
overall_coverage_minimum: '80%',
|
||||
overall_coverage_actual: `${overallCoverage}%`,
|
||||
overall_status: overallCoverage >= 80 ? 'MET' : 'NOT MET',
|
||||
},
|
||||
|
||||
uncovered_requirements: coverageMatrix.gap_analysis.critical_gaps.concat(coverageMatrix.gap_analysis.high_gaps),
|
||||
|
||||
recommendations: coverageMatrix.recommendations,
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Generate Traceability Report
|
||||
|
||||
**Use trace-template.md to generate:**
|
||||
|
||||
```markdown
|
||||
# Traceability Report
|
||||
|
||||
## Gate Decision: {gateDecision}
|
||||
|
||||
**Rationale:** {rationale}
|
||||
|
||||
## Coverage Summary
|
||||
|
||||
- Total Requirements: {totalRequirements}
|
||||
- Covered: {fullyCovered} ({coveragePercentage}%)
|
||||
- P0 Coverage: {p0CoveragePercentage}%
|
||||
|
||||
## Traceability Matrix
|
||||
|
||||
[Full matrix with requirement → test mappings]
|
||||
|
||||
## Gaps & Recommendations
|
||||
|
||||
[List of uncovered requirements with recommended actions]
|
||||
|
||||
## Next Actions
|
||||
|
||||
{recommendations}
|
||||
```
|
||||
|
||||
**Save to:**
|
||||
|
||||
```javascript
|
||||
fs.writeFileSync('{outputFile}', reportContent, 'utf8');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Display Gate Decision
|
||||
|
||||
```
|
||||
🚨 GATE DECISION: {gateDecision}
|
||||
|
||||
📊 Coverage Analysis:
|
||||
- P0 Coverage: {p0Coverage}% (Required: 100%) → {p0_status}
|
||||
- P1 Coverage: {effectiveP1Coverage}% (PASS target: 90%, minimum: 80%) → {p1_status}
|
||||
- Overall Coverage: {overallCoverage}% (Minimum: 80%) → {overall_status}
|
||||
|
||||
✅ Decision Rationale:
|
||||
{rationale}
|
||||
|
||||
⚠️ Critical Gaps: {criticalGaps.length}
|
||||
|
||||
📝 Recommended Actions:
|
||||
{list top 3 recommendations}
|
||||
|
||||
📂 Full Report: {outputFile}
|
||||
|
||||
{if FAIL}
|
||||
🚫 GATE: FAIL - Release BLOCKED until coverage improves
|
||||
{endif}
|
||||
|
||||
{if CONCERNS}
|
||||
⚠️ GATE: CONCERNS - Proceed with caution, address gaps soon
|
||||
{endif}
|
||||
|
||||
{if PASS}
|
||||
✅ GATE: PASS - Release approved, coverage meets standards
|
||||
{endif}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Save Progress
|
||||
|
||||
**Update the YAML frontmatter in `{outputFile}` to mark this final step complete.**
|
||||
|
||||
Since step 4 (Generate Traceability Report) already wrote the report content to `{outputFile}`, do NOT overwrite it. Instead, update only the frontmatter at the top of the existing file:
|
||||
|
||||
- Add `'step-05-gate-decision'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-05-gate-decision'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
|
||||
Then append the gate decision summary (from section 5 above) to the end of the existing report content.
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
**WORKFLOW COMPLETE when:**
|
||||
|
||||
- ✅ Phase 1 coverage matrix read successfully
|
||||
- ✅ Gate decision logic applied
|
||||
- ✅ Traceability report generated
|
||||
- ✅ Gate decision displayed
|
||||
|
||||
**Workflow terminates here.**
|
||||
|
||||
---
|
||||
|
||||
## 🚨 PHASE 2 SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Coverage matrix read from Phase 1
|
||||
- Gate decision made with clear rationale
|
||||
- Report generated and saved
|
||||
- Decision communicated clearly
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Could not read Phase 1 matrix
|
||||
- Gate decision logic incorrect
|
||||
- Report missing or incomplete
|
||||
|
||||
**Master Rule:** Gate decision MUST be deterministic based on clear criteria (P0 100%, P1 90/80, overall >=80).
|
||||
65
.claude/skills/bmad-testarch-trace/steps-e/step-01-assess.md
Normal file
65
.claude/skills/bmad-testarch-trace/steps-e/step-01-assess.md
Normal file
@@ -0,0 +1,65 @@
|
||||
---
|
||||
name: 'step-01-assess'
|
||||
description: 'Load an existing output for editing'
|
||||
nextStepFile: './step-02-apply-edit.md'
|
||||
---
|
||||
|
||||
# Step 1: Assess Edit Target
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Identify which output should be edited and load it.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Ask the user which output file to edit
|
||||
- 🚫 Do not edit until target is confirmed
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: existing outputs
|
||||
- Focus: select edit target
|
||||
- Limits: no edits yet
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Identify Target
|
||||
|
||||
Ask the user to provide the output file path or select from known outputs.
|
||||
|
||||
### 2. Load Target
|
||||
|
||||
Read the provided output file in full.
|
||||
|
||||
### 3. Confirm
|
||||
|
||||
Confirm the target and proceed to edit.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Target identified and loaded
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Proceeding without a confirmed target
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: 'step-02-apply-edit'
|
||||
description: 'Apply edits to the selected output'
|
||||
---
|
||||
|
||||
# Step 2: Apply Edits
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Apply the requested edits to the selected output and confirm changes.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Only apply edits explicitly requested by the user
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: selected output and user changes
|
||||
- Focus: apply edits only
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Confirm Requested Changes
|
||||
|
||||
Restate what will be changed and confirm.
|
||||
|
||||
### 2. Apply Changes
|
||||
|
||||
Update the output file accordingly.
|
||||
|
||||
### 3. Report
|
||||
|
||||
Summarize the edits applied.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Changes applied and confirmed
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Unconfirmed edits or missing update
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
name: 'step-01-validate'
|
||||
description: 'Validate workflow outputs against checklist'
|
||||
outputFile: '{test_artifacts}/trace-validation-report.md'
|
||||
validationChecklist: '../checklist.md'
|
||||
---
|
||||
|
||||
# Step 1: Validate Outputs
|
||||
|
||||
## STEP GOAL:
|
||||
|
||||
Validate outputs using the workflow checklist and record findings.
|
||||
|
||||
## MANDATORY EXECUTION RULES (READ FIRST):
|
||||
|
||||
### Universal Rules:
|
||||
|
||||
- 📖 Read the complete step file before taking any action
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
### Role Reinforcement:
|
||||
|
||||
- ✅ You are the Master Test Architect
|
||||
|
||||
### Step-Specific Rules:
|
||||
|
||||
- 🎯 Validate against `{validationChecklist}`
|
||||
- 🚫 Do not skip checks
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Write findings to `{outputFile}`
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: workflow outputs and checklist
|
||||
- Focus: validation only
|
||||
- Limits: do not modify outputs in this step
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Checklist
|
||||
|
||||
Read `{validationChecklist}` and list all criteria.
|
||||
|
||||
### 2. Validate Outputs
|
||||
|
||||
Evaluate outputs against each checklist item.
|
||||
|
||||
### 3. Write Report
|
||||
|
||||
Write a validation report to `{outputFile}` with PASS/WARN/FAIL per section.
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Validation report written
|
||||
- All checklist items evaluated
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped checklist items
|
||||
- No report produced
|
||||
708
.claude/skills/bmad-testarch-trace/trace-template.md
Normal file
708
.claude/skills/bmad-testarch-trace/trace-template.md
Normal file
@@ -0,0 +1,708 @@
|
||||
---
|
||||
stepsCompleted: []
|
||||
lastStep: ''
|
||||
lastSaved: ''
|
||||
workflowType: 'testarch-trace'
|
||||
inputDocuments: []
|
||||
---
|
||||
|
||||
# Traceability Matrix & Gate Decision - Story {STORY_ID}
|
||||
|
||||
**Story:** {STORY_TITLE}
|
||||
**Date:** {DATE}
|
||||
**Evaluator:** {user_name or TEA Agent}
|
||||
|
||||
---
|
||||
|
||||
Note: This workflow does not generate tests. If gaps exist, run `*atdd` or `*automate` to create coverage.
|
||||
|
||||
## PHASE 1: REQUIREMENTS TRACEABILITY
|
||||
|
||||
### Coverage Summary
|
||||
|
||||
| Priority | Total Criteria | FULL Coverage | Coverage % | Status |
|
||||
| --------- | -------------- | ------------- | ---------- | ------------ |
|
||||
| P0 | {P0_TOTAL} | {P0_FULL} | {P0_PCT}% | {P0_STATUS} |
|
||||
| P1 | {P1_TOTAL} | {P1_FULL} | {P1_PCT}% | {P1_STATUS} |
|
||||
| P2 | {P2_TOTAL} | {P2_FULL} | {P2_PCT}% | {P2_STATUS} |
|
||||
| P3 | {P3_TOTAL} | {P3_FULL} | {P3_PCT}% | {P3_STATUS} |
|
||||
| **Total** | **{TOTAL}** | **{FULL}** | **{PCT}%** | **{STATUS}** |
|
||||
|
||||
**Legend:**
|
||||
|
||||
- ✅ PASS - Coverage meets quality gate threshold
|
||||
- ⚠️ WARN - Coverage below threshold but not critical
|
||||
- ❌ FAIL - Coverage below minimum threshold (blocker)
|
||||
|
||||
---
|
||||
|
||||
### Detailed Mapping
|
||||
|
||||
#### {CRITERION_ID}: {CRITERION_DESCRIPTION} ({PRIORITY})
|
||||
|
||||
- **Coverage:** {COVERAGE_STATUS} {STATUS_ICON}
|
||||
- **Tests:**
|
||||
- `{TEST_ID}` - {TEST_FILE}:{LINE}
|
||||
- **Given:** {GIVEN}
|
||||
- **When:** {WHEN}
|
||||
- **Then:** {THEN}
|
||||
- `{TEST_ID_2}` - {TEST_FILE_2}:{LINE}
|
||||
- **Given:** {GIVEN_2}
|
||||
- **When:** {WHEN_2}
|
||||
- **Then:** {THEN_2}
|
||||
|
||||
- **Gaps:** (if PARTIAL or UNIT-ONLY or INTEGRATION-ONLY)
|
||||
- Missing: {MISSING_SCENARIO_1}
|
||||
- Missing: {MISSING_SCENARIO_2}
|
||||
|
||||
- **Recommendation:** {RECOMMENDATION_TEXT}
|
||||
|
||||
---
|
||||
|
||||
#### Example: AC-1: User can login with email and password (P0)
|
||||
|
||||
- **Coverage:** FULL ✅
|
||||
- **Tests:**
|
||||
- `1.3-E2E-001` - tests/e2e/auth.spec.ts:12
|
||||
- **Given:** User has valid credentials
|
||||
- **When:** User submits login form
|
||||
- **Then:** User is redirected to dashboard
|
||||
- `1.3-UNIT-001` - tests/unit/auth-service.spec.ts:8
|
||||
- **Given:** Valid email and password hash
|
||||
- **When:** validateCredentials is called
|
||||
- **Then:** Returns user object
|
||||
|
||||
---
|
||||
|
||||
#### Example: AC-3: User can reset password via email (P1)
|
||||
|
||||
- **Coverage:** PARTIAL ⚠️
|
||||
- **Tests:**
|
||||
- `1.3-E2E-003` - tests/e2e/auth.spec.ts:44
|
||||
- **Given:** User requests password reset
|
||||
- **When:** User clicks reset link in email
|
||||
- **Then:** User can set new password
|
||||
|
||||
- **Gaps:**
|
||||
- Missing: Email delivery validation
|
||||
- Missing: Expired token handling (error path)
|
||||
- Missing: Invalid token handling (security test)
|
||||
- Missing: Unit test for token generation logic
|
||||
|
||||
- **Recommendation:** Add `1.3-API-001` for email service integration testing and `1.3-UNIT-003` for token generation logic. Add `1.3-E2E-004` for error path validation (expired/invalid tokens).
|
||||
|
||||
---
|
||||
|
||||
### Gap Analysis
|
||||
|
||||
#### Critical Gaps (BLOCKER) ❌
|
||||
|
||||
{CRITICAL_GAP_COUNT} gaps found. **Do not release until resolved.**
|
||||
|
||||
1. **{CRITERION_ID}: {CRITERION_DESCRIPTION}** (P0)
|
||||
- Current Coverage: {COVERAGE_STATUS}
|
||||
- Missing Tests: {MISSING_TEST_DESCRIPTION}
|
||||
- Recommend: {RECOMMENDED_TEST_ID} ({RECOMMENDED_TEST_LEVEL})
|
||||
- Impact: {IMPACT_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
#### High Priority Gaps (PR BLOCKER) ⚠️
|
||||
|
||||
{HIGH_GAP_COUNT} gaps found. **Address before PR merge.**
|
||||
|
||||
1. **{CRITERION_ID}: {CRITERION_DESCRIPTION}** (P1)
|
||||
- Current Coverage: {COVERAGE_STATUS}
|
||||
- Missing Tests: {MISSING_TEST_DESCRIPTION}
|
||||
- Recommend: {RECOMMENDED_TEST_ID} ({RECOMMENDED_TEST_LEVEL})
|
||||
- Impact: {IMPACT_DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
#### Medium Priority Gaps (Nightly) ⚠️
|
||||
|
||||
{MEDIUM_GAP_COUNT} gaps found. **Address in nightly test improvements.**
|
||||
|
||||
1. **{CRITERION_ID}: {CRITERION_DESCRIPTION}** (P2)
|
||||
- Current Coverage: {COVERAGE_STATUS}
|
||||
- Recommend: {RECOMMENDED_TEST_ID} ({RECOMMENDED_TEST_LEVEL})
|
||||
|
||||
---
|
||||
|
||||
#### Low Priority Gaps (Optional) ℹ️
|
||||
|
||||
{LOW_GAP_COUNT} gaps found. **Optional - add if time permits.**
|
||||
|
||||
1. **{CRITERION_ID}: {CRITERION_DESCRIPTION}** (P3)
|
||||
- Current Coverage: {COVERAGE_STATUS}
|
||||
|
||||
---
|
||||
|
||||
### Coverage Heuristics Findings
|
||||
|
||||
#### Endpoint Coverage Gaps
|
||||
|
||||
- Endpoints without direct API tests: {endpoint_gap_count}
|
||||
- Examples:
|
||||
- {endpoint_gap_1}
|
||||
- {endpoint_gap_2}
|
||||
|
||||
#### Auth/Authz Negative-Path Gaps
|
||||
|
||||
- Criteria missing denied/invalid-path tests: {auth_negative_gap_count}
|
||||
- Examples:
|
||||
- {auth_gap_1}
|
||||
- {auth_gap_2}
|
||||
|
||||
#### Happy-Path-Only Criteria
|
||||
|
||||
- Criteria missing error/edge scenarios: {happy_path_only_gap_count}
|
||||
- Examples:
|
||||
- {happy_path_gap_1}
|
||||
- {happy_path_gap_2}
|
||||
|
||||
---
|
||||
|
||||
### Quality Assessment
|
||||
|
||||
#### Tests with Issues
|
||||
|
||||
**BLOCKER Issues** ❌
|
||||
|
||||
- `{TEST_ID}` - {ISSUE_DESCRIPTION} - {REMEDIATION}
|
||||
|
||||
**WARNING Issues** ⚠️
|
||||
|
||||
- `{TEST_ID}` - {ISSUE_DESCRIPTION} - {REMEDIATION}
|
||||
|
||||
**INFO Issues** ℹ️
|
||||
|
||||
- `{TEST_ID}` - {ISSUE_DESCRIPTION} - {REMEDIATION}
|
||||
|
||||
---
|
||||
|
||||
#### Example Quality Issues
|
||||
|
||||
**WARNING Issues** ⚠️
|
||||
|
||||
- `1.3-E2E-001` - 145 seconds (exceeds 90s target) - Optimize fixture setup to reduce test duration
|
||||
- `1.3-UNIT-005` - 320 lines (exceeds 300 line limit) - Split into multiple focused test files
|
||||
|
||||
**INFO Issues** ℹ️
|
||||
|
||||
- `1.3-E2E-002` - Missing Given-When-Then structure - Refactor describe block to use BDD format
|
||||
|
||||
---
|
||||
|
||||
#### Tests Passing Quality Gates
|
||||
|
||||
**{PASSING_TEST_COUNT}/{TOTAL_TEST_COUNT} tests ({PASSING_PCT}%) meet all quality criteria** ✅
|
||||
|
||||
---
|
||||
|
||||
### Duplicate Coverage Analysis
|
||||
|
||||
#### Acceptable Overlap (Defense in Depth)
|
||||
|
||||
- {CRITERION_ID}: Tested at unit (business logic) and E2E (user journey) ✅
|
||||
|
||||
#### Unacceptable Duplication ⚠️
|
||||
|
||||
- {CRITERION_ID}: Same validation at E2E and Component level
|
||||
- Recommendation: Remove {TEST_ID} or consolidate with {OTHER_TEST_ID}
|
||||
|
||||
---
|
||||
|
||||
### Coverage by Test Level
|
||||
|
||||
| Test Level | Tests | Criteria Covered | Coverage % |
|
||||
| ---------- | ----------------- | -------------------- | ---------------- |
|
||||
| E2E | {E2E_COUNT} | {E2E_CRITERIA} | {E2E_PCT}% |
|
||||
| API | {API_COUNT} | {API_CRITERIA} | {API_PCT}% |
|
||||
| Component | {COMP_COUNT} | {COMP_CRITERIA} | {COMP_PCT}% |
|
||||
| Unit | {UNIT_COUNT} | {UNIT_CRITERIA} | {UNIT_PCT}% |
|
||||
| **Total** | **{TOTAL_TESTS}** | **{TOTAL_CRITERIA}** | **{TOTAL_PCT}%** |
|
||||
|
||||
---
|
||||
|
||||
### Traceability Recommendations
|
||||
|
||||
#### Immediate Actions (Before PR Merge)
|
||||
|
||||
1. **{ACTION_1}** - {DESCRIPTION}
|
||||
2. **{ACTION_2}** - {DESCRIPTION}
|
||||
|
||||
#### Short-term Actions (This Milestone)
|
||||
|
||||
1. **{ACTION_1}** - {DESCRIPTION}
|
||||
2. **{ACTION_2}** - {DESCRIPTION}
|
||||
|
||||
#### Long-term Actions (Backlog)
|
||||
|
||||
1. **{ACTION_1}** - {DESCRIPTION}
|
||||
|
||||
---
|
||||
|
||||
#### Example Recommendations
|
||||
|
||||
**Immediate Actions (Before PR Merge)**
|
||||
|
||||
1. **Add P1 Password Reset Tests** - Implement `1.3-API-001` for email service integration and `1.3-E2E-004` for error path validation. P1 coverage currently at 80%, target is 90%.
|
||||
2. **Optimize Slow E2E Test** - Refactor `1.3-E2E-001` to use faster fixture setup. Currently 145s, target is <90s.
|
||||
|
||||
**Short-term Actions (This Milestone)**
|
||||
|
||||
1. **Enhance P2 Coverage** - Add E2E validation for session timeout (`1.3-E2E-005`). Currently UNIT-ONLY coverage.
|
||||
2. **Split Large Test File** - Break `1.3-UNIT-005` (320 lines) into multiple focused test files (<300 lines each).
|
||||
|
||||
**Long-term Actions (Backlog)**
|
||||
|
||||
1. **Enrich P3 Coverage** - Add tests for edge cases in P3 criteria if time permits.
|
||||
|
||||
---
|
||||
|
||||
## PHASE 2: QUALITY GATE DECISION
|
||||
|
||||
**Gate Type:** {story | epic | release | hotfix}
|
||||
**Decision Mode:** {deterministic | manual}
|
||||
|
||||
---
|
||||
|
||||
### Evidence Summary
|
||||
|
||||
#### Test Execution Results
|
||||
|
||||
- **Total Tests**: {total_count}
|
||||
- **Passed**: {passed_count} ({pass_percentage}%)
|
||||
- **Failed**: {failed_count} ({fail_percentage}%)
|
||||
- **Skipped**: {skipped_count} ({skip_percentage}%)
|
||||
- **Duration**: {total_duration}
|
||||
|
||||
**Priority Breakdown:**
|
||||
|
||||
- **P0 Tests**: {p0_passed}/{p0_total} passed ({p0_pass_rate}%) {✅ | ❌}
|
||||
- **P1 Tests**: {p1_passed}/{p1_total} passed ({p1_pass_rate}%) {✅ | ⚠️ | ❌}
|
||||
- **P2 Tests**: {p2_passed}/{p2_total} passed ({p2_pass_rate}%) {informational}
|
||||
- **P3 Tests**: {p3_passed}/{p3_total} passed ({p3_pass_rate}%) {informational}
|
||||
|
||||
**Overall Pass Rate**: {overall_pass_rate}% {✅ | ⚠️ | ❌}
|
||||
|
||||
**Test Results Source**: {CI_run_id | test_report_url | local_run}
|
||||
|
||||
---
|
||||
|
||||
#### Coverage Summary (from Phase 1)
|
||||
|
||||
**Requirements Coverage:**
|
||||
|
||||
- **P0 Acceptance Criteria**: {p0_covered}/{p0_total} covered ({p0_coverage}%) {✅ | ❌}
|
||||
- **P1 Acceptance Criteria**: {p1_covered}/{p1_total} covered ({p1_coverage}%) {✅ | ⚠️ | ❌}
|
||||
- **P2 Acceptance Criteria**: {p2_covered}/{p2_total} covered ({p2_coverage}%) {informational}
|
||||
- **Overall Coverage**: {overall_coverage}%
|
||||
|
||||
**Code Coverage** (if available):
|
||||
|
||||
- **Line Coverage**: {line_coverage}% {✅ | ⚠️ | ❌}
|
||||
- **Branch Coverage**: {branch_coverage}% {✅ | ⚠️ | ❌}
|
||||
- **Function Coverage**: {function_coverage}% {✅ | ⚠️ | ❌}
|
||||
|
||||
**Coverage Source**: {coverage_report_url | coverage_file_path}
|
||||
|
||||
---
|
||||
|
||||
#### Non-Functional Requirements (NFRs)
|
||||
|
||||
**Security**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
|
||||
|
||||
- Security Issues: {security_issue_count}
|
||||
- {details_if_issues}
|
||||
|
||||
**Performance**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
|
||||
|
||||
- {performance_metrics_summary}
|
||||
|
||||
**Reliability**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
|
||||
|
||||
- {reliability_metrics_summary}
|
||||
|
||||
**Maintainability**: {PASS | CONCERNS | FAIL | NOT_ASSESSED} {✅ | ⚠️ | ❌}
|
||||
|
||||
- {maintainability_metrics_summary}
|
||||
|
||||
**NFR Source**: {nfr_assessment_file_path | not_assessed}
|
||||
|
||||
---
|
||||
|
||||
#### Flakiness Validation
|
||||
|
||||
**Burn-in Results** (if available):
|
||||
|
||||
- **Burn-in Iterations**: {iteration_count} (e.g., 10)
|
||||
- **Flaky Tests Detected**: {flaky_test_count} {✅ if 0 | ❌ if >0}
|
||||
- **Stability Score**: {stability_percentage}%
|
||||
|
||||
**Flaky Tests List** (if any):
|
||||
|
||||
- {flaky_test_1_name} - {failure_rate}
|
||||
- {flaky_test_2_name} - {failure_rate}
|
||||
|
||||
**Burn-in Source**: {CI_burn_in_run_id | not_available}
|
||||
|
||||
---
|
||||
|
||||
### Decision Criteria Evaluation
|
||||
|
||||
#### P0 Criteria (Must ALL Pass)
|
||||
|
||||
| Criterion | Threshold | Actual | Status |
|
||||
| --------------------- | --------- | ------------------------- | -------- | -------- |
|
||||
| P0 Coverage | 100% | {p0_coverage}% | {✅ PASS | ❌ FAIL} |
|
||||
| P0 Test Pass Rate | 100% | {p0_pass_rate}% | {✅ PASS | ❌ FAIL} |
|
||||
| Security Issues | 0 | {security_issue_count} | {✅ PASS | ❌ FAIL} |
|
||||
| Critical NFR Failures | 0 | {critical_nfr_fail_count} | {✅ PASS | ❌ FAIL} |
|
||||
| Flaky Tests | 0 | {flaky_test_count} | {✅ PASS | ❌ FAIL} |
|
||||
|
||||
**P0 Evaluation**: {✅ ALL PASS | ❌ ONE OR MORE FAILED}
|
||||
|
||||
---
|
||||
|
||||
#### P1 Criteria (Required for PASS, May Accept for CONCERNS)
|
||||
|
||||
| Criterion | Threshold | Actual | Status |
|
||||
| ---------------------- | ------------------------- | -------------------- | -------- | ----------- | -------- |
|
||||
| P1 Coverage | ≥{min_p1_coverage}% | {p1_coverage}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
|
||||
| P1 Test Pass Rate | ≥{min_p1_pass_rate}% | {p1_pass_rate}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
|
||||
| Overall Test Pass Rate | ≥{min_overall_pass_rate}% | {overall_pass_rate}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
|
||||
| Overall Coverage | ≥{min_coverage}% | {overall_coverage}% | {✅ PASS | ⚠️ CONCERNS | ❌ FAIL} |
|
||||
|
||||
**P1 Evaluation**: {✅ ALL PASS | ⚠️ SOME CONCERNS | ❌ FAILED}
|
||||
|
||||
---
|
||||
|
||||
#### P2/P3 Criteria (Informational, Don't Block)
|
||||
|
||||
| Criterion | Actual | Notes |
|
||||
| ----------------- | --------------- | ------------------------------------------------------------ |
|
||||
| P2 Test Pass Rate | {p2_pass_rate}% | {allow_p2_failures ? "Tracked, doesn't block" : "Evaluated"} |
|
||||
| P3 Test Pass Rate | {p3_pass_rate}% | {allow_p3_failures ? "Tracked, doesn't block" : "Evaluated"} |
|
||||
|
||||
---
|
||||
|
||||
### GATE DECISION: {PASS | CONCERNS | FAIL | WAIVED}
|
||||
|
||||
---
|
||||
|
||||
### Rationale
|
||||
|
||||
{Explain decision based on criteria evaluation}
|
||||
|
||||
{Highlight key evidence that drove decision}
|
||||
|
||||
{Note any assumptions or caveats}
|
||||
|
||||
**Example (PASS):**
|
||||
|
||||
> All P0 criteria met with 100% coverage and pass rates across critical tests. All P1 criteria exceeded thresholds with 98% overall pass rate and 92% coverage. No security issues detected. No flaky tests in validation. Feature is ready for production deployment with standard monitoring.
|
||||
|
||||
**Example (CONCERNS):**
|
||||
|
||||
> All P0 criteria met, ensuring critical user journeys are protected. However, P1 coverage (88%) falls below threshold (90%) due to missing E2E test for AC-5 edge case. Overall pass rate (96%) is excellent. Issues are non-critical and have acceptable workarounds. Risk is low enough to deploy with enhanced monitoring.
|
||||
|
||||
**Example (FAIL):**
|
||||
|
||||
> CRITICAL BLOCKERS DETECTED:
|
||||
>
|
||||
> 1. P0 coverage incomplete (80%) - AC-2 security validation missing
|
||||
> 2. P0 test failures (75% pass rate) in core search functionality
|
||||
> 3. Unresolved SQL injection vulnerability in search filter (CRITICAL)
|
||||
>
|
||||
> Release MUST BE BLOCKED until P0 issues are resolved. Security vulnerability cannot be waived.
|
||||
|
||||
**Example (WAIVED):**
|
||||
|
||||
> Original decision was FAIL due to P0 test failure in legacy Excel 2007 export module (affects <1% of users). However, release contains critical GDPR compliance features required by regulatory deadline (Oct 15). Business has approved waiver given:
|
||||
>
|
||||
> - Regulatory priority overrides legacy module risk
|
||||
> - Workaround available (use Excel 2010+)
|
||||
> - Issue will be fixed in v2.4.1 hotfix (due Oct 20)
|
||||
> - Enhanced monitoring in place
|
||||
|
||||
---
|
||||
|
||||
### {Section: Delete if not applicable}
|
||||
|
||||
#### Residual Risks (For CONCERNS or WAIVED)
|
||||
|
||||
List unresolved P1/P2 issues that don't block release but should be tracked:
|
||||
|
||||
1. **{Risk Description}**
|
||||
- **Priority**: P1 | P2
|
||||
- **Probability**: Low | Medium | High
|
||||
- **Impact**: Low | Medium | High
|
||||
- **Risk Score**: {probability × impact}
|
||||
- **Mitigation**: {workaround or monitoring plan}
|
||||
- **Remediation**: {fix in next milestone/release}
|
||||
|
||||
**Overall Residual Risk**: {LOW | MEDIUM | HIGH}
|
||||
|
||||
---
|
||||
|
||||
#### Waiver Details (For WAIVED only)
|
||||
|
||||
**Original Decision**: ❌ FAIL
|
||||
|
||||
**Reason for Failure**:
|
||||
|
||||
- {list_of_blocking_issues}
|
||||
|
||||
**Waiver Information**:
|
||||
|
||||
- **Waiver Reason**: {business_justification}
|
||||
- **Waiver Approver**: {name}, {role} (e.g., Jane Doe, VP Engineering)
|
||||
- **Approval Date**: {YYYY-MM-DD}
|
||||
- **Waiver Expiry**: {YYYY-MM-DD} (**NOTE**: Does NOT apply to next release)
|
||||
|
||||
**Monitoring Plan**:
|
||||
|
||||
- {enhanced_monitoring_1}
|
||||
- {enhanced_monitoring_2}
|
||||
- {escalation_criteria}
|
||||
|
||||
**Remediation Plan**:
|
||||
|
||||
- **Fix Target**: {next_release_version} (e.g., v2.4.1 hotfix)
|
||||
- **Due Date**: {YYYY-MM-DD}
|
||||
- **Owner**: {team_or_person}
|
||||
- **Verification**: {how_fix_will_be_verified}
|
||||
|
||||
**Business Justification**:
|
||||
{detailed_explanation_of_why_waiver_is_acceptable}
|
||||
|
||||
---
|
||||
|
||||
#### Critical Issues (For FAIL or CONCERNS)
|
||||
|
||||
Top blockers requiring immediate attention:
|
||||
|
||||
| Priority | Issue | Description | Owner | Due Date | Status |
|
||||
| -------- | ------------- | ------------------- | ------------ | ------------ | ------------------ |
|
||||
| P0 | {issue_title} | {brief_description} | {owner_name} | {YYYY-MM-DD} | {OPEN/IN_PROGRESS} |
|
||||
| P0 | {issue_title} | {brief_description} | {owner_name} | {YYYY-MM-DD} | {OPEN/IN_PROGRESS} |
|
||||
| P1 | {issue_title} | {brief_description} | {owner_name} | {YYYY-MM-DD} | {OPEN/IN_PROGRESS} |
|
||||
|
||||
**Blocking Issues Count**: {p0_blocker_count} P0 blockers, {p1_blocker_count} P1 issues
|
||||
|
||||
---
|
||||
|
||||
### Gate Recommendations
|
||||
|
||||
#### For PASS Decision ✅
|
||||
|
||||
1. **Proceed to deployment**
|
||||
- Deploy to staging environment
|
||||
- Validate with smoke tests
|
||||
- Monitor key metrics for 24-48 hours
|
||||
- Deploy to production with standard monitoring
|
||||
|
||||
2. **Post-Deployment Monitoring**
|
||||
- {metric_1_to_monitor}
|
||||
- {metric_2_to_monitor}
|
||||
- {alert_thresholds}
|
||||
|
||||
3. **Success Criteria**
|
||||
- {success_criterion_1}
|
||||
- {success_criterion_2}
|
||||
|
||||
---
|
||||
|
||||
#### For CONCERNS Decision ⚠️
|
||||
|
||||
1. **Deploy with Enhanced Monitoring**
|
||||
- Deploy to staging with extended validation period
|
||||
- Enable enhanced logging/monitoring for known risk areas:
|
||||
- {risk_area_1}
|
||||
- {risk_area_2}
|
||||
- Set aggressive alerts for potential issues
|
||||
- Deploy to production with caution
|
||||
|
||||
2. **Create Remediation Backlog**
|
||||
- Create story: "{fix_title_1}" (Priority: {priority})
|
||||
- Create story: "{fix_title_2}" (Priority: {priority})
|
||||
- Target milestone: {next_milestone}
|
||||
|
||||
3. **Post-Deployment Actions**
|
||||
- Monitor {specific_areas} closely for {time_period}
|
||||
- Weekly status updates on remediation progress
|
||||
- Re-assess after fixes deployed
|
||||
|
||||
---
|
||||
|
||||
#### For FAIL Decision ❌
|
||||
|
||||
1. **Block Deployment Immediately**
|
||||
- Do NOT deploy to any environment
|
||||
- Notify stakeholders of blocking issues
|
||||
- Escalate to tech lead and PM
|
||||
|
||||
2. **Fix Critical Issues**
|
||||
- Address P0 blockers listed in Critical Issues section
|
||||
- Owner assignments confirmed
|
||||
- Due dates agreed upon
|
||||
- Daily standup on blocker resolution
|
||||
|
||||
3. **Re-Run Gate After Fixes**
|
||||
- Re-run full test suite after fixes
|
||||
- Re-run `bmad tea *trace` workflow
|
||||
- Verify decision is PASS before deploying
|
||||
|
||||
---
|
||||
|
||||
#### For WAIVED Decision 🔓
|
||||
|
||||
1. **Deploy with Business Approval**
|
||||
- Confirm waiver approver has signed off
|
||||
- Document waiver in release notes
|
||||
- Notify all stakeholders of waived risks
|
||||
|
||||
2. **Aggressive Monitoring**
|
||||
- {enhanced_monitoring_plan}
|
||||
- {escalation_procedures}
|
||||
- Daily checks on waived risk areas
|
||||
|
||||
3. **Mandatory Remediation**
|
||||
- Fix MUST be completed by {due_date}
|
||||
- Issue CANNOT be waived in next release
|
||||
- Track remediation progress weekly
|
||||
- Verify fix in next gate
|
||||
|
||||
---
|
||||
|
||||
### Next Steps
|
||||
|
||||
**Immediate Actions** (next 24-48 hours):
|
||||
|
||||
1. {action_1}
|
||||
2. {action_2}
|
||||
3. {action_3}
|
||||
|
||||
**Follow-up Actions** (next milestone/release):
|
||||
|
||||
1. {action_1}
|
||||
2. {action_2}
|
||||
3. {action_3}
|
||||
|
||||
**Stakeholder Communication**:
|
||||
|
||||
- Notify PM: {decision_summary}
|
||||
- Notify SM: {decision_summary}
|
||||
- Notify DEV lead: {decision_summary}
|
||||
|
||||
---
|
||||
|
||||
## Integrated YAML Snippet (CI/CD)
|
||||
|
||||
```yaml
|
||||
traceability_and_gate:
|
||||
# Phase 1: Traceability
|
||||
traceability:
|
||||
story_id: "{STORY_ID}"
|
||||
date: "{DATE}"
|
||||
coverage:
|
||||
overall: {OVERALL_PCT}%
|
||||
p0: {P0_PCT}%
|
||||
p1: {P1_PCT}%
|
||||
p2: {P2_PCT}%
|
||||
p3: {P3_PCT}%
|
||||
gaps:
|
||||
critical: {CRITICAL_COUNT}
|
||||
high: {HIGH_COUNT}
|
||||
medium: {MEDIUM_COUNT}
|
||||
low: {LOW_COUNT}
|
||||
quality:
|
||||
passing_tests: {PASSING_COUNT}
|
||||
total_tests: {TOTAL_TESTS}
|
||||
blocker_issues: {BLOCKER_COUNT}
|
||||
warning_issues: {WARNING_COUNT}
|
||||
recommendations:
|
||||
- "{RECOMMENDATION_1}"
|
||||
- "{RECOMMENDATION_2}"
|
||||
|
||||
# Phase 2: Gate Decision
|
||||
gate_decision:
|
||||
decision: "{PASS | CONCERNS | FAIL | WAIVED}"
|
||||
gate_type: "{story | epic | release | hotfix}"
|
||||
decision_mode: "{deterministic | manual}"
|
||||
criteria:
|
||||
p0_coverage: {p0_coverage}%
|
||||
p0_pass_rate: {p0_pass_rate}%
|
||||
p1_coverage: {p1_coverage}%
|
||||
p1_pass_rate: {p1_pass_rate}%
|
||||
overall_pass_rate: {overall_pass_rate}%
|
||||
overall_coverage: {overall_coverage}%
|
||||
security_issues: {security_issue_count}
|
||||
critical_nfrs_fail: {critical_nfr_fail_count}
|
||||
flaky_tests: {flaky_test_count}
|
||||
thresholds:
|
||||
min_p0_coverage: 100
|
||||
min_p0_pass_rate: 100
|
||||
min_p1_coverage: {min_p1_coverage}
|
||||
min_p1_pass_rate: {min_p1_pass_rate}
|
||||
min_overall_pass_rate: {min_overall_pass_rate}
|
||||
min_coverage: {min_coverage}
|
||||
evidence:
|
||||
test_results: "{CI_run_id | test_report_url}"
|
||||
traceability: "{trace_file_path}"
|
||||
nfr_assessment: "{nfr_file_path}"
|
||||
code_coverage: "{coverage_report_url}"
|
||||
next_steps: "{brief_summary_of_recommendations}"
|
||||
waiver: # Only if WAIVED
|
||||
reason: "{business_justification}"
|
||||
approver: "{name}, {role}"
|
||||
expiry: "{YYYY-MM-DD}"
|
||||
remediation_due: "{YYYY-MM-DD}"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related Artifacts
|
||||
|
||||
- **Story File:** {STORY_FILE_PATH}
|
||||
- **Test Design:** {TEST_DESIGN_PATH} (if available)
|
||||
- **Tech Spec:** {TECH_SPEC_PATH} (if available)
|
||||
- **Test Results:** {TEST_RESULTS_PATH}
|
||||
- **NFR Assessment:** {NFR_FILE_PATH} (if available)
|
||||
- **Test Files:** {TEST_DIR_PATH}
|
||||
|
||||
---
|
||||
|
||||
## Sign-Off
|
||||
|
||||
**Phase 1 - Traceability Assessment:**
|
||||
|
||||
- Overall Coverage: {OVERALL_PCT}%
|
||||
- P0 Coverage: {P0_PCT}% {P0_STATUS}
|
||||
- P1 Coverage: {P1_PCT}% {P1_STATUS}
|
||||
- Critical Gaps: {CRITICAL_COUNT}
|
||||
- High Priority Gaps: {HIGH_COUNT}
|
||||
|
||||
**Phase 2 - Gate Decision:**
|
||||
|
||||
- **Decision**: {PASS | CONCERNS | FAIL | WAIVED} {STATUS_ICON}
|
||||
- **P0 Evaluation**: {✅ ALL PASS | ❌ ONE OR MORE FAILED}
|
||||
- **P1 Evaluation**: {✅ ALL PASS | ⚠️ SOME CONCERNS | ❌ FAILED}
|
||||
|
||||
**Overall Status:** {STATUS} {STATUS_ICON}
|
||||
|
||||
**Next Steps:**
|
||||
|
||||
- If PASS ✅: Proceed to deployment
|
||||
- If CONCERNS ⚠️: Deploy with monitoring, create remediation backlog
|
||||
- If FAIL ❌: Block deployment, fix critical issues, re-run workflow
|
||||
- If WAIVED 🔓: Deploy with business approval and aggressive monitoring
|
||||
|
||||
**Generated:** {DATE}
|
||||
**Workflow:** testarch-trace v4.0 (Enhanced with Gate Decision)
|
||||
|
||||
---
|
||||
|
||||
<!-- Powered by BMAD-CORE™ -->
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-trace
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-trace
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:03:10
|
||||
---
|
||||
|
||||
# Validation Report: testarch-trace
|
||||
|
||||
**Validation Started:** 2026-01-27 09:50:21
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-load-context.md: 73 lines [GOOD]
|
||||
- steps-c/step-02-discover-tests.md: 62 lines [GOOD]
|
||||
- steps-c/step-03-map-criteria.md: 58 lines [GOOD]
|
||||
- steps-c/step-04-analyze-gaps.md: 57 lines [GOOD]
|
||||
- steps-c/step-05-gate-decision.md: 66 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 51 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 46 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 53 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
- No dead relative links detected
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- Last step steps-v/step-01-validate.md has no nextStepFile (final step OK)
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: trace-template.md
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-05-gate-decision.md
|
||||
- steps-v/step-01-validate.md
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- checklist.md present: YES
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- All steps include STEP GOAL, MANDATORY EXECUTION RULES, EXECUTION PROTOCOLS, CONTEXT BOUNDARIES, and SUCCESS/FAILURE metrics
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:03:10
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
validationDate: 2026-01-27
|
||||
workflowName: testarch-trace
|
||||
workflowPath: {project-root}/src/workflows/testarch/bmad-testarch-trace
|
||||
validationStatus: COMPLETE
|
||||
completionDate: 2026-01-27 10:24:01
|
||||
---
|
||||
|
||||
# Validation Report: testarch-trace
|
||||
|
||||
**Validation Started:** 2026-01-27 10:24:01
|
||||
**Validator:** BMAD Workflow Validation System (Codex)
|
||||
**Standards Version:** BMAD Workflow Standards
|
||||
|
||||
## File Structure & Size
|
||||
|
||||
- workflow.md present: YES
|
||||
- instructions.md present: YES
|
||||
- workflow.yaml present: YES
|
||||
- step files found: 8
|
||||
|
||||
**Step File Sizes:**
|
||||
|
||||
- steps-c/step-01-load-context.md: 72 lines [GOOD]
|
||||
- steps-c/step-02-discover-tests.md: 61 lines [GOOD]
|
||||
- steps-c/step-03-map-criteria.md: 57 lines [GOOD]
|
||||
- steps-c/step-04-analyze-gaps.md: 56 lines [GOOD]
|
||||
- steps-c/step-05-gate-decision.md: 65 lines [GOOD]
|
||||
- steps-e/step-01-assess.md: 50 lines [GOOD]
|
||||
- steps-e/step-02-apply-edit.md: 45 lines [GOOD]
|
||||
- steps-v/step-01-validate.md: 52 lines [GOOD]
|
||||
- workflow-plan.md present: YES
|
||||
|
||||
## Frontmatter Validation
|
||||
|
||||
- No frontmatter violations found
|
||||
|
||||
## Critical Path Violations
|
||||
|
||||
### Config Variables (Exceptions)
|
||||
|
||||
Standard BMAD config variables treated as valid exceptions: bmb_creations_output_folder, communication_language, document_output_language, output_folder, planning_artifacts, project-root, project_name, test_artifacts, user_name
|
||||
|
||||
- No {project-root} hardcoded paths detected in body
|
||||
|
||||
- No dead relative links detected
|
||||
|
||||
- No module path assumptions detected
|
||||
|
||||
**Status:** ✅ PASS - No critical violations
|
||||
|
||||
## Menu Handling Validation
|
||||
|
||||
- No menu structures detected (linear step flow) [N/A]
|
||||
|
||||
## Step Type Validation
|
||||
|
||||
- steps-c/step-01-load-context.md: Init [PASS]
|
||||
- steps-c/step-02-discover-tests.md: Middle [PASS]
|
||||
- steps-c/step-03-map-criteria.md: Middle [PASS]
|
||||
- steps-c/step-04-analyze-gaps.md: Middle [PASS]
|
||||
- steps-c/step-05-gate-decision.md: Final [PASS]
|
||||
- Step type validation assumes linear sequence (no branching/menu). Workflow-plan.md present for reference. [INFO]
|
||||
|
||||
## Output Format Validation
|
||||
|
||||
- Templates present: trace-template.md
|
||||
- Steps with outputFile in frontmatter:
|
||||
- steps-c/step-05-gate-decision.md
|
||||
- steps-v/step-01-validate.md
|
||||
- checklist.md present: YES
|
||||
|
||||
## Validation Design Check
|
||||
|
||||
- Validation steps folder (steps-v) present: YES
|
||||
- Validation step(s) present: step-01-validate.md
|
||||
- Validation steps reference checklist data and auto-proceed
|
||||
|
||||
## Instruction Style Check
|
||||
|
||||
- Instruction style: Prescriptive (appropriate for TEA quality/compliance workflows)
|
||||
- Steps emphasize mandatory sequence, explicit success/failure metrics, and risk-based guidance
|
||||
|
||||
## Collaborative Experience Check
|
||||
|
||||
- Overall facilitation quality: GOOD
|
||||
- Steps use progressive prompts and clear role reinforcement; no laundry-list interrogation detected
|
||||
- Flow progression is clear and aligned to workflow goals
|
||||
|
||||
## Subagent Optimization Opportunities
|
||||
|
||||
- No high-priority subagent optimizations identified; workflow already uses step-file architecture
|
||||
- Pattern 1 (grep/regex): N/A for most steps
|
||||
- Pattern 2 (per-file analysis): already aligned to validation structure
|
||||
- Pattern 3 (data ops): minimal data file loads
|
||||
- Pattern 4 (parallel): optional for validation only
|
||||
|
||||
## Cohesive Review
|
||||
|
||||
- Overall assessment: GOOD
|
||||
- Flow is linear, goals are clear, and outputs map to TEA artifacts
|
||||
- Voice and tone consistent with Test Architect persona
|
||||
- Recommendation: READY (minor refinements optional)
|
||||
|
||||
## Plan Quality Validation
|
||||
|
||||
- Plan file present: workflow-plan.md
|
||||
- Planned steps found: 8 (all implemented)
|
||||
- Plan implementation status: Fully Implemented
|
||||
|
||||
## Summary
|
||||
|
||||
- Validation completed: 2026-01-27 10:24:01
|
||||
- Critical issues: 0
|
||||
- Warnings: 0 (informational notes only)
|
||||
- Readiness: READY (manual review optional)
|
||||
21
.claude/skills/bmad-testarch-trace/workflow-plan.md
Normal file
21
.claude/skills/bmad-testarch-trace/workflow-plan.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# Workflow Plan: testarch-trace
|
||||
|
||||
## Create Mode (steps-c)
|
||||
- step-01-load-context.md
|
||||
|
||||
- step-02-discover-tests.md
|
||||
- step-03-map-criteria.md
|
||||
- step-04-analyze-gaps.md
|
||||
- step-05-gate-decision.md
|
||||
|
||||
## Validate Mode (steps-v)
|
||||
- step-01-validate.md
|
||||
|
||||
## Edit Mode (steps-e)
|
||||
- step-01-assess.md
|
||||
- step-02-apply-edit.md
|
||||
|
||||
## Outputs
|
||||
- {test_artifacts}/traceability-matrix.md
|
||||
|
||||
- Gate decision summary (if evidence available)
|
||||
41
.claude/skills/bmad-testarch-trace/workflow.md
Normal file
41
.claude/skills/bmad-testarch-trace/workflow.md
Normal file
@@ -0,0 +1,41 @@
|
||||
---
|
||||
name: bmad-testarch-trace
|
||||
description: Generate traceability matrix and quality gate decision. Use when user says 'lets create traceability matrix' or 'I want to analyze test coverage'
|
||||
web_bundle: true
|
||||
---
|
||||
|
||||
# Requirements Traceability & Quality Gate
|
||||
|
||||
**Goal:** Generate requirements-to-tests traceability matrix, analyze coverage, and make quality gate decision (PASS/CONCERNS/FAIL/WAIVED)
|
||||
|
||||
**Role:** You are the Master Test Architect.
|
||||
|
||||
---
|
||||
|
||||
## WORKFLOW ARCHITECTURE
|
||||
|
||||
This workflow uses **tri-modal step-file architecture**:
|
||||
|
||||
- **Create mode (steps-c/)**: primary execution flow
|
||||
- **Validate mode (steps-v/)**: validation against checklist
|
||||
- **Edit mode (steps-e/)**: revise existing outputs
|
||||
|
||||
---
|
||||
|
||||
## INITIALIZATION SEQUENCE
|
||||
|
||||
### 1. Mode Determination
|
||||
|
||||
"Welcome to the workflow. What would you like to do?"
|
||||
|
||||
- **[C] Create** — Run the workflow
|
||||
- **[R] Resume** — Resume an interrupted workflow
|
||||
- **[V] Validate** — Validate existing outputs
|
||||
- **[E] Edit** — Edit existing outputs
|
||||
|
||||
### 2. Route to First Step
|
||||
|
||||
- **If C:** Load `steps-c/step-01-load-context.md`
|
||||
- **If R:** Load `steps-c/step-01b-resume.md`
|
||||
- **If V:** Load `steps-v/step-01-validate.md`
|
||||
- **If E:** Load `steps-e/step-01-assess.md`
|
||||
56
.claude/skills/bmad-testarch-trace/workflow.yaml
Normal file
56
.claude/skills/bmad-testarch-trace/workflow.yaml
Normal file
@@ -0,0 +1,56 @@
|
||||
# Test Architect workflow: bmad-testarch-trace
|
||||
name: bmad-testarch-trace
|
||||
# prettier-ignore
|
||||
description: 'Generate traceability matrix and quality gate decision. Use when the user says "lets create traceability matrix" or "I want to analyze test coverage"'
|
||||
|
||||
# Critical variables from config
|
||||
config_source: "{project-root}/_bmad/tea/config.yaml"
|
||||
output_folder: "{config_source}:output_folder"
|
||||
test_artifacts: "{config_source}:test_artifacts"
|
||||
user_name: "{config_source}:user_name"
|
||||
communication_language: "{config_source}:communication_language"
|
||||
document_output_language: "{config_source}:document_output_language"
|
||||
date: system-generated
|
||||
|
||||
# Workflow components
|
||||
installed_path: "."
|
||||
instructions: "./instructions.md"
|
||||
validation: "./checklist.md"
|
||||
template: "./trace-template.md"
|
||||
|
||||
# Variables and inputs
|
||||
variables:
|
||||
# Directory paths
|
||||
test_dir: "{project-root}/tests" # Root test directory
|
||||
source_dir: "{project-root}" # Source code directory (customize if needed, e.g., {project-root}/src or {project-root}/lib)
|
||||
|
||||
# Workflow behavior
|
||||
coverage_levels: "e2e,api,component,unit" # Which test levels to trace
|
||||
gate_type: "story" # story | epic | release | hotfix - determines gate scope
|
||||
decision_mode: "deterministic" # deterministic (rule-based) | manual (team decision)
|
||||
|
||||
# Output configuration
|
||||
default_output_file: "{test_artifacts}/traceability-matrix.md"
|
||||
|
||||
# Required tools
|
||||
required_tools:
|
||||
- read_file # Read story, test files, BMad artifacts
|
||||
- write_file # Create traceability matrix, gate YAML
|
||||
- list_files # Discover test files
|
||||
- search_repo # Find tests by test ID, describe blocks
|
||||
- glob # Find test files matching patterns
|
||||
|
||||
tags:
|
||||
- qa
|
||||
- traceability
|
||||
- test-architect
|
||||
- coverage
|
||||
- requirements
|
||||
- gate
|
||||
- decision
|
||||
- release
|
||||
|
||||
execution_hints:
|
||||
interactive: false # Minimize prompts
|
||||
autonomous: true # Proceed without user input unless blocked
|
||||
iterative: true
|
||||
Reference in New Issue
Block a user