initial commit
This commit is contained in:
138
.claude/skills/bmad-testarch-nfr/steps-c/step-01-load-context.md
Normal file
138
.claude/skills/bmad-testarch-nfr/steps-c/step-01-load-context.md
Normal file
@@ -0,0 +1,138 @@
|
||||
---
|
||||
name: 'step-01-load-context'
|
||||
description: 'Load NFR requirements, evidence sources, and knowledge base'
|
||||
nextStepFile: './step-02-define-thresholds.md'
|
||||
knowledgeIndex: '{project-root}/_bmad/tea/testarch/tea-index.csv'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 1: Load Context & Knowledge Base
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Gather NFR requirements, evidence sources, and knowledge fragments needed for assessment.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Halt if implementation or evidence is unavailable
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Implementation accessible for evaluation
|
||||
- Evidence sources available (test results, metrics, logs)
|
||||
|
||||
If missing: **HALT** and request the missing inputs.
|
||||
|
||||
---
|
||||
|
||||
## 2. Load Configuration
|
||||
|
||||
From `{config_source}`:
|
||||
|
||||
- Read `tea_browser_automation`
|
||||
|
||||
---
|
||||
|
||||
### Tiered Knowledge Loading
|
||||
|
||||
Load fragments based on their `tier` classification in `tea-index.csv`:
|
||||
|
||||
1. **Core tier** (always load): Foundational fragments required for this workflow
|
||||
2. **Extended tier** (load on-demand): Load when deeper analysis is needed or when the user's context requires it
|
||||
3. **Specialized tier** (load only when relevant): Load only when the specific use case matches (e.g., contract-testing only for microservices, email-auth only for email flows)
|
||||
|
||||
> **Context Efficiency**: Loading only core fragments reduces context usage by 40-50% compared to loading all fragments.
|
||||
|
||||
## 3. Load Knowledge Base Fragments
|
||||
|
||||
From `{knowledgeIndex}` load:
|
||||
|
||||
- `adr-quality-readiness-checklist.md`
|
||||
- `ci-burn-in.md`
|
||||
- `test-quality.md`
|
||||
- `playwright-config.md`
|
||||
- `error-handling.md`
|
||||
|
||||
**Playwright CLI (if `tea_browser_automation` is "cli" or "auto"):**
|
||||
|
||||
- `playwright-cli.md`
|
||||
|
||||
**MCP Patterns (if `tea_browser_automation` is "mcp" or "auto"):**
|
||||
|
||||
- (existing MCP-related fragments, if any are added in future)
|
||||
|
||||
---
|
||||
|
||||
## 4. Load Artifacts
|
||||
|
||||
If available, read:
|
||||
|
||||
- `tech-spec.md` (primary NFRs)
|
||||
- `PRD.md` (product-level NFRs)
|
||||
- `story` or `test-design` docs (feature-level NFRs)
|
||||
|
||||
---
|
||||
|
||||
## 5. Confirm Inputs
|
||||
|
||||
Summarize loaded NFR sources and evidence availability.
|
||||
|
||||
---
|
||||
|
||||
## 6. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-01-load-context']
|
||||
lastStep: 'step-01-load-context'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-01-load-context'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-01-load-context'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
**Update `inputDocuments`**: Set `inputDocuments` in the output template frontmatter to the list of artifact paths loaded in this step (e.g., knowledge fragments, test design documents, configuration files).
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
106
.claude/skills/bmad-testarch-nfr/steps-c/step-01b-resume.md
Normal file
106
.claude/skills/bmad-testarch-nfr/steps-c/step-01b-resume.md
Normal file
@@ -0,0 +1,106 @@
|
||||
---
|
||||
name: 'step-01b-resume'
|
||||
description: 'Resume interrupted workflow from last completed step'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 1b: Resume Workflow
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Resume an interrupted workflow by loading the existing output document, displaying progress, and routing to the next incomplete step.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- Read the entire step file before acting
|
||||
- Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- Follow the MANDATORY SEQUENCE exactly
|
||||
- Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: Output document with progress frontmatter
|
||||
- Focus: Load progress and route to next step
|
||||
- Limits: Do not re-execute completed steps
|
||||
- Dependencies: Output document must exist from a previous run
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly.
|
||||
|
||||
### 1. Load Output Document
|
||||
|
||||
Read `{outputFile}` and parse YAML frontmatter for:
|
||||
|
||||
- `stepsCompleted` -- array of completed step names
|
||||
- `lastStep` -- last completed step name
|
||||
- `lastSaved` -- timestamp of last save
|
||||
|
||||
**If `{outputFile}` does not exist**, display:
|
||||
|
||||
"No previous progress found. There is no output document to resume from. Please use **[C] Create** to start a fresh workflow run."
|
||||
|
||||
**THEN:** Halt. Do not proceed.
|
||||
|
||||
---
|
||||
|
||||
### 2. Display Progress Dashboard
|
||||
|
||||
Display progress with checkmark/empty indicators:
|
||||
|
||||
```
|
||||
NFR Assessment - Resume Progress:
|
||||
|
||||
1. Load Context (step-01-load-context) [completed/pending]
|
||||
2. Define Thresholds (step-02-define-thresholds) [completed/pending]
|
||||
3. Gather Evidence (step-03-gather-evidence) [completed/pending]
|
||||
4. Evaluate & Aggregate (step-04e-aggregate-nfr) [completed/pending]
|
||||
5. Generate Report (step-05-generate-report) [completed/pending]
|
||||
|
||||
Last saved: {lastSaved}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Route to Next Step
|
||||
|
||||
Based on `lastStep`, load the next incomplete step:
|
||||
|
||||
| lastStep | Next Step File |
|
||||
| --------------------------- | --------------------------------- |
|
||||
| `step-01-load-context` | `./step-02-define-thresholds.md` |
|
||||
| `step-02-define-thresholds` | `./step-03-gather-evidence.md` |
|
||||
| `step-03-gather-evidence` | `./step-04-evaluate-and-score.md` |
|
||||
| `step-04e-aggregate-nfr` | `./step-05-generate-report.md` |
|
||||
| `step-05-generate-report` | **Workflow already complete.** |
|
||||
|
||||
**If `lastStep` is the final step** (`step-05-generate-report`), display: "All steps completed. Use **[C] Create** to start fresh, **[V] Validate** to review outputs, or **[E] Edit** to make revisions." Then halt.
|
||||
|
||||
**If `lastStep` does not match any value above**, display: "Unknown progress state (`lastStep`: {lastStep}). Please use **[C] Create** to start fresh." Then halt.
|
||||
|
||||
**Otherwise**, load the identified step file, read completely, and execute.
|
||||
|
||||
The existing content in `{outputFile}` provides context from previously completed steps.
|
||||
|
||||
---
|
||||
|
||||
## SYSTEM SUCCESS/FAILURE METRICS
|
||||
|
||||
### SUCCESS:
|
||||
|
||||
- Output document loaded and parsed correctly
|
||||
- Progress dashboard displayed accurately
|
||||
- Routed to correct next step
|
||||
|
||||
### FAILURE:
|
||||
|
||||
- Not loading output document
|
||||
- Incorrect progress display
|
||||
- Routing to wrong step
|
||||
|
||||
**Master Rule:** Resume MUST route to the exact next incomplete step. Never re-execute completed steps.
|
||||
@@ -0,0 +1,107 @@
|
||||
---
|
||||
name: 'step-02-define-thresholds'
|
||||
description: 'Identify NFR categories and thresholds'
|
||||
nextStepFile: './step-03-gather-evidence.md'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 2: Define NFR Categories & Thresholds
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Establish the NFR categories to assess and the thresholds used for validation.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- 🚫 Never guess thresholds
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Select Categories
|
||||
|
||||
Use the ADR Quality Readiness Checklist (8 categories):
|
||||
|
||||
1. Testability & Automation
|
||||
2. Test Data Strategy
|
||||
3. Scalability & Availability
|
||||
4. Disaster Recovery
|
||||
5. Security
|
||||
6. Monitorability/Debuggability/Manageability
|
||||
7. QoS/QoE
|
||||
8. Deployability
|
||||
|
||||
Add any `custom_nfr_categories` if provided.
|
||||
|
||||
---
|
||||
|
||||
## 2. Define Thresholds
|
||||
|
||||
For each category, extract thresholds from:
|
||||
|
||||
- tech-spec (primary)
|
||||
- PRD (secondary)
|
||||
- story or test-design (feature-specific)
|
||||
|
||||
If a threshold is unknown, mark it **UNKNOWN** and plan to report **CONCERNS**.
|
||||
|
||||
---
|
||||
|
||||
## 3. Confirm NFR Matrix
|
||||
|
||||
List each NFR category with its threshold or UNKNOWN status.
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-02-define-thresholds']
|
||||
lastStep: 'step-02-define-thresholds'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-02-define-thresholds'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-02-define-thresholds'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: 'step-03-gather-evidence'
|
||||
description: 'Collect evidence for each NFR category'
|
||||
nextStepFile: './step-04-evaluate-and-score.md'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 3: Gather Evidence
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Collect measurable evidence to evaluate each NFR category.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Evidence Sources
|
||||
|
||||
Collect evidence for:
|
||||
|
||||
- **Performance**: load tests, metrics, response time data
|
||||
- **Security**: scans, auth tests, vuln reports
|
||||
- **Reliability**: error rates, burn-in runs, failover tests
|
||||
- **Maintainability**: test quality, code health signals
|
||||
- **Other categories**: logs, monitoring, DR drills, deployability checks
|
||||
|
||||
---
|
||||
|
||||
## 2. Browser-Based Evidence Collection (if `tea_browser_automation` is `cli` or `auto`)
|
||||
|
||||
> **Fallback:** If CLI is not installed, fall back to MCP (if available) or skip browser-based evidence collection.
|
||||
|
||||
For performance and security categories, CLI can gather live evidence:
|
||||
|
||||
**Performance evidence (page load, response times):**
|
||||
|
||||
1. `playwright-cli -s=tea-nfr open <target_url>`
|
||||
2. `playwright-cli -s=tea-nfr network` → capture response times and payload sizes
|
||||
3. `playwright-cli -s=tea-nfr screenshot --filename={test_artifacts}/nfr/perf-<page>.png`
|
||||
4. `playwright-cli -s=tea-nfr close`
|
||||
|
||||
> **Session Hygiene:** Always close sessions using `playwright-cli -s=tea-nfr close`. Do NOT use `close-all` — it kills every session on the machine and breaks parallel execution.
|
||||
|
||||
Store artifacts under `{test_artifacts}/nfr/`
|
||||
|
||||
---
|
||||
|
||||
## 3. Evidence Gaps
|
||||
|
||||
If evidence is missing for a category, mark that category as **CONCERNS**.
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-03-gather-evidence']
|
||||
lastStep: 'step-03-gather-evidence'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-03-gather-evidence'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-03-gather-evidence'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
@@ -0,0 +1,254 @@
|
||||
---
|
||||
name: 'step-04-evaluate-and-score'
|
||||
description: 'Orchestrate adaptive NFR domain assessments (agent-team, subagent, or sequential)'
|
||||
nextStepFile: './step-04e-aggregate-nfr.md'
|
||||
---
|
||||
|
||||
# Step 4: Orchestrate Adaptive NFR Assessment
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Select execution mode deterministically, then assess NFR domains using agent-team, subagent, or sequential execution while preserving output contracts.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Resolve execution mode from config (`tea_execution_mode`, `tea_capability_probe`)
|
||||
- ✅ Apply fallback rules deterministically when requested mode is unsupported
|
||||
- ✅ Wait for required worker steps to complete
|
||||
- ❌ Do NOT skip capability checks when probing is enabled
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Wait for subagent outputs
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
### 1. Prepare Execution Context
|
||||
|
||||
**Generate unique timestamp:**
|
||||
|
||||
```javascript
|
||||
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
|
||||
```
|
||||
|
||||
**Prepare context:**
|
||||
|
||||
```javascript
|
||||
const parseBooleanFlag = (value, defaultValue = true) => {
|
||||
if (typeof value === 'string') {
|
||||
const normalized = value.trim().toLowerCase();
|
||||
if (['false', '0', 'off', 'no'].includes(normalized)) return false;
|
||||
if (['true', '1', 'on', 'yes'].includes(normalized)) return true;
|
||||
}
|
||||
if (value === undefined || value === null) return defaultValue;
|
||||
return Boolean(value);
|
||||
};
|
||||
|
||||
const subagentContext = {
|
||||
system_context: /* from Step 1 */,
|
||||
nfr_thresholds: /* from Step 2 */,
|
||||
evidence_gathered: /* from Step 3 */,
|
||||
config: {
|
||||
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
||||
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
||||
},
|
||||
timestamp: timestamp
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Resolve Execution Mode with Capability Probe
|
||||
|
||||
```javascript
|
||||
const normalizeUserExecutionMode = (mode) => {
|
||||
if (typeof mode !== 'string') return null;
|
||||
const normalized = mode.trim().toLowerCase().replace(/[-_]/g, ' ').replace(/\s+/g, ' ');
|
||||
|
||||
if (normalized === 'auto') return 'auto';
|
||||
if (normalized === 'sequential') return 'sequential';
|
||||
if (normalized === 'subagent' || normalized === 'sub agent' || normalized === 'subagents' || normalized === 'sub agents') {
|
||||
return 'subagent';
|
||||
}
|
||||
if (normalized === 'agent team' || normalized === 'agent teams' || normalized === 'agentteam') {
|
||||
return 'agent-team';
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
const normalizeConfigExecutionMode = (mode) => {
|
||||
if (mode === 'subagent') return 'subagent';
|
||||
if (mode === 'auto' || mode === 'sequential' || mode === 'subagent' || mode === 'agent-team') {
|
||||
return mode;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
// Explicit user instruction in the active run takes priority over config.
|
||||
const explicitModeFromUser = normalizeUserExecutionMode(runtime.getExplicitExecutionModeHint?.() || null);
|
||||
|
||||
const requestedMode = explicitModeFromUser || normalizeConfigExecutionMode(subagentContext.config.execution_mode) || 'auto';
|
||||
const probeEnabled = subagentContext.config.capability_probe;
|
||||
|
||||
const supports = {
|
||||
subagent: false,
|
||||
agentTeam: false,
|
||||
};
|
||||
|
||||
if (probeEnabled) {
|
||||
supports.subagent = runtime.canLaunchSubagents?.() === true;
|
||||
supports.agentTeam = runtime.canLaunchAgentTeams?.() === true;
|
||||
}
|
||||
|
||||
let resolvedMode = requestedMode;
|
||||
|
||||
if (requestedMode === 'auto') {
|
||||
if (supports.agentTeam) resolvedMode = 'agent-team';
|
||||
else if (supports.subagent) resolvedMode = 'subagent';
|
||||
else resolvedMode = 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'agent-team' && !supports.agentTeam) {
|
||||
resolvedMode = supports.subagent ? 'subagent' : 'sequential';
|
||||
} else if (probeEnabled && requestedMode === 'subagent' && !supports.subagent) {
|
||||
resolvedMode = 'sequential';
|
||||
}
|
||||
|
||||
subagentContext.execution = {
|
||||
requestedMode,
|
||||
resolvedMode,
|
||||
probeEnabled,
|
||||
supports,
|
||||
};
|
||||
```
|
||||
|
||||
Resolution precedence:
|
||||
|
||||
1. Explicit user request in this run (`agent team` => `agent-team`; `subagent` => `subagent`; `sequential`; `auto`)
|
||||
2. `tea_execution_mode` from config
|
||||
3. Runtime capability fallback (when probing enabled)
|
||||
|
||||
If probing is disabled, honor the requested mode strictly. If that mode cannot be executed at runtime, fail with explicit error instead of silent fallback.
|
||||
|
||||
---
|
||||
|
||||
### 3. Dispatch 4 NFR Workers
|
||||
|
||||
**Subagent A: Security Assessment**
|
||||
|
||||
- File: `./step-04a-subagent-security.md`
|
||||
- Output: `/tmp/tea-nfr-security-${timestamp}.json`
|
||||
- Execution:
|
||||
- `agent-team` or `subagent`: launch non-blocking
|
||||
- `sequential`: run blocking and wait
|
||||
- Status: Running... ⟳
|
||||
|
||||
**Subagent B: Performance Assessment**
|
||||
|
||||
- File: `./step-04b-subagent-performance.md`
|
||||
- Output: `/tmp/tea-nfr-performance-${timestamp}.json`
|
||||
- Status: Running... ⟳
|
||||
|
||||
**Subagent C: Reliability Assessment**
|
||||
|
||||
- File: `./step-04c-subagent-reliability.md`
|
||||
- Output: `/tmp/tea-nfr-reliability-${timestamp}.json`
|
||||
- Status: Running... ⟳
|
||||
|
||||
**Subagent D: Scalability Assessment**
|
||||
|
||||
- File: `./step-04d-subagent-scalability.md`
|
||||
- Output: `/tmp/tea-nfr-scalability-${timestamp}.json`
|
||||
- Status: Running... ⟳
|
||||
|
||||
In `agent-team` and `subagent` modes, runtime decides worker scheduling and concurrency.
|
||||
|
||||
---
|
||||
|
||||
### 4. Wait for Expected Worker Completion
|
||||
|
||||
**If `resolvedMode` is `agent-team` or `subagent`:**
|
||||
|
||||
```
|
||||
⏳ Waiting for 4 NFR subagents to complete...
|
||||
├── Subagent A (Security): Running... ⟳
|
||||
├── Subagent B (Performance): Running... ⟳
|
||||
├── Subagent C (Reliability): Running... ⟳
|
||||
└── Subagent D (Scalability): Running... ⟳
|
||||
|
||||
[... time passes ...]
|
||||
|
||||
✅ All 4 NFR subagents completed!
|
||||
```
|
||||
|
||||
**If `resolvedMode` is `sequential`:**
|
||||
|
||||
```
|
||||
✅ Sequential mode: each worker already completed during dispatch.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Verify All Outputs Exist
|
||||
|
||||
```javascript
|
||||
const outputs = ['security', 'performance', 'reliability', 'scalability'].map((domain) => `/tmp/tea-nfr-${domain}-${timestamp}.json`);
|
||||
|
||||
outputs.forEach((output) => {
|
||||
if (!fs.existsSync(output)) {
|
||||
throw new Error(`Subagent output missing: ${output}`);
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Execution Report
|
||||
|
||||
```
|
||||
🚀 Performance Report:
|
||||
- Execution Mode: {resolvedMode}
|
||||
- Total Elapsed: ~mode-dependent
|
||||
- Parallel Gain: ~67% faster when mode is subagent/agent-team
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Proceed to Aggregation
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
The aggregation step will:
|
||||
|
||||
- Read all 4 NFR domain outputs
|
||||
- Calculate overall risk level
|
||||
- Aggregate compliance status
|
||||
- Identify cross-domain risks
|
||||
- Generate executive summary
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed when all 4 required worker steps completed and outputs exist.
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All required worker steps completed
|
||||
- Fallback behavior respected configuration and capability probe rules
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- One or more subagents failed
|
||||
- Unsupported requested mode with probing disabled
|
||||
@@ -0,0 +1,138 @@
|
||||
---
|
||||
name: 'step-04a-subagent-security'
|
||||
description: 'Subagent: Security NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-security-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4A: Security NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess SECURITY NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- ✅ Assess SECURITY only (not performance, reliability, scalability)
|
||||
- ✅ Output structured JSON to temp file
|
||||
- ❌ Do NOT assess other NFR domains
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Security Assessment Categories
|
||||
|
||||
**Assess the following security dimensions:**
|
||||
|
||||
**A) Authentication & Authorization:**
|
||||
|
||||
- OAuth2/JWT implementation
|
||||
- Session management
|
||||
- Multi-factor authentication
|
||||
- Role-based access control (RBAC)
|
||||
|
||||
**B) Data Protection:**
|
||||
|
||||
- Encryption at rest
|
||||
- Encryption in transit (HTTPS/TLS)
|
||||
- Sensitive data handling (PII, passwords)
|
||||
- Database encryption
|
||||
|
||||
**C) Input Validation:**
|
||||
|
||||
- SQL injection prevention
|
||||
- XSS prevention
|
||||
- CSRF protection
|
||||
- Input sanitization
|
||||
|
||||
**D) API Security:**
|
||||
|
||||
- Rate limiting
|
||||
- API authentication
|
||||
- CORS configuration
|
||||
- Security headers
|
||||
|
||||
**E) Secrets Management:**
|
||||
|
||||
- Environment variables for secrets
|
||||
- No hardcoded credentials
|
||||
- Secret rotation policies
|
||||
- Key management systems
|
||||
|
||||
### 2. Risk Assessment
|
||||
|
||||
For each category, determine status:
|
||||
|
||||
- **PASS**: Properly implemented
|
||||
- **CONCERN**: Partially implemented or weak
|
||||
- **FAIL**: Not implemented or critical vulnerability
|
||||
- **N/A**: Not applicable to this system
|
||||
|
||||
### 3. Compliance Check
|
||||
|
||||
**Common compliance standards:**
|
||||
|
||||
- SOC2
|
||||
- GDPR
|
||||
- HIPAA
|
||||
- PCI-DSS
|
||||
- ISO 27001
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "security",
|
||||
"risk_level": "MEDIUM",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Authentication",
|
||||
"status": "PASS",
|
||||
"description": "OAuth2 with JWT tokens implemented",
|
||||
"evidence": ["src/auth/oauth.ts", "JWT refresh token rotation"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Data Encryption",
|
||||
"status": "CONCERN",
|
||||
"description": "Database encryption at rest not enabled",
|
||||
"evidence": ["Database config shows no encryption"],
|
||||
"recommendations": ["Enable database encryption at rest", "Use AWS RDS encryption or equivalent", "Implement key rotation policy"]
|
||||
},
|
||||
{
|
||||
"category": "Input Validation",
|
||||
"status": "FAIL",
|
||||
"description": "SQL injection vulnerability in search endpoint",
|
||||
"evidence": ["src/api/search.ts:42 - direct SQL concatenation"],
|
||||
"recommendations": ["URGENT: Use parameterized queries", "Add input sanitization library", "Implement WAF rules"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"SOC2": "PARTIAL",
|
||||
"GDPR": "PASS",
|
||||
"HIPAA": "N/A",
|
||||
"PCI-DSS": "FAIL"
|
||||
},
|
||||
"priority_actions": [
|
||||
"Fix SQL injection vulnerability (URGENT)",
|
||||
"Enable database encryption within 30 days",
|
||||
"Implement rate limiting for all APIs"
|
||||
],
|
||||
"summary": "Security posture is MEDIUM risk with 1 critical vulnerability requiring immediate attention"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
|
||||
**Subagent terminates here.**
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
name: 'step-04b-subagent-performance'
|
||||
description: 'Subagent: Performance NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-performance-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4B: Performance NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess PERFORMANCE NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Performance Assessment Categories
|
||||
|
||||
**A) Response Times:**
|
||||
|
||||
- API response times (<200ms target)
|
||||
- Page load times (<2s target)
|
||||
- Time to interactive (<3s target)
|
||||
|
||||
**B) Throughput:**
|
||||
|
||||
- Requests per second capacity
|
||||
- Concurrent user support
|
||||
- Database query performance
|
||||
|
||||
**C) Resource Usage:**
|
||||
|
||||
- Memory consumption
|
||||
- CPU utilization
|
||||
- Database connection pooling
|
||||
|
||||
**D) Optimization:**
|
||||
|
||||
- Caching strategies
|
||||
- CDN usage
|
||||
- Code splitting/lazy loading
|
||||
- Database indexing
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "performance",
|
||||
"risk_level": "LOW",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Response Times",
|
||||
"status": "PASS",
|
||||
"description": "API endpoints respond in <150ms (P95)",
|
||||
"evidence": ["Load testing results show 140ms P95"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Caching",
|
||||
"status": "CONCERN",
|
||||
"description": "No CDN for static assets",
|
||||
"evidence": ["Static files served from origin"],
|
||||
"recommendations": ["Implement CDN (CloudFront/Cloudflare)", "Cache static assets for 1 year"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"SLA_99.9": "PASS",
|
||||
"SLA_99.99": "CONCERN"
|
||||
},
|
||||
"priority_actions": ["Implement CDN for static assets", "Add database query caching for frequent reads"],
|
||||
"summary": "Performance is acceptable with minor optimization opportunities"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
name: 'step-04c-subagent-reliability'
|
||||
description: 'Subagent: Reliability NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-reliability-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4C: Reliability NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess RELIABILITY NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Reliability Assessment Categories
|
||||
|
||||
**A) Error Handling:**
|
||||
|
||||
- Try-catch blocks for critical operations
|
||||
- Graceful degradation
|
||||
- Circuit breakers
|
||||
- Retry mechanisms
|
||||
|
||||
**B) Monitoring & Observability:**
|
||||
|
||||
- Logging implementation
|
||||
- Error tracking (Sentry/Datadog)
|
||||
- Health check endpoints
|
||||
- Alerting systems
|
||||
|
||||
**C) Fault Tolerance:**
|
||||
|
||||
- Database failover
|
||||
- Service redundancy
|
||||
- Backup strategies
|
||||
- Disaster recovery plan
|
||||
|
||||
**D) Uptime & Availability:**
|
||||
|
||||
- SLA targets
|
||||
- Historical uptime
|
||||
- Incident response
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "reliability",
|
||||
"risk_level": "LOW",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Error Handling",
|
||||
"status": "PASS",
|
||||
"description": "Comprehensive error handling with circuit breakers",
|
||||
"evidence": ["Circuit breaker pattern in src/services/", "Retry logic implemented"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Monitoring",
|
||||
"status": "CONCERN",
|
||||
"description": "No APM (Application Performance Monitoring) tool",
|
||||
"evidence": ["Logging present but no distributed tracing"],
|
||||
"recommendations": ["Implement APM (Datadog/New Relic)", "Add distributed tracing"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"SLA_99.9": "PASS"
|
||||
},
|
||||
"priority_actions": ["Implement APM for better observability"],
|
||||
"summary": "Reliability is good with minor monitoring gaps"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
name: 'step-04d-subagent-scalability'
|
||||
description: 'Subagent: Scalability NFR assessment'
|
||||
subagent: true
|
||||
outputFile: '/tmp/tea-nfr-scalability-{{timestamp}}.json'
|
||||
---
|
||||
|
||||
# Subagent 4D: Scalability NFR Assessment
|
||||
|
||||
## SUBAGENT CONTEXT
|
||||
|
||||
This is an **isolated subagent** running in parallel with other NFR domain assessments.
|
||||
|
||||
**Your task:** Assess SCALABILITY NFR domain only.
|
||||
|
||||
---
|
||||
|
||||
## SUBAGENT TASK
|
||||
|
||||
### 1. Scalability Assessment Categories
|
||||
|
||||
**A) Horizontal Scaling:**
|
||||
|
||||
- Stateless architecture
|
||||
- Load balancer configuration
|
||||
- Container orchestration (K8s)
|
||||
- Auto-scaling policies
|
||||
|
||||
**B) Vertical Scaling:**
|
||||
|
||||
- Resource allocation
|
||||
- Database size limits
|
||||
- Memory management
|
||||
- CPU optimization
|
||||
|
||||
**C) Data Scaling:**
|
||||
|
||||
- Database partitioning/sharding
|
||||
- Read replicas
|
||||
- Caching layers
|
||||
- Data archival strategy
|
||||
|
||||
**D) Traffic Handling:**
|
||||
|
||||
- CDN for static assets
|
||||
- Rate limiting
|
||||
- Queue systems for async work
|
||||
- WebSocket scaling
|
||||
|
||||
---
|
||||
|
||||
## OUTPUT FORMAT
|
||||
|
||||
```json
|
||||
{
|
||||
"domain": "scalability",
|
||||
"risk_level": "MEDIUM",
|
||||
"findings": [
|
||||
{
|
||||
"category": "Horizontal Scaling",
|
||||
"status": "PASS",
|
||||
"description": "Stateless architecture with container orchestration",
|
||||
"evidence": ["Docker + Kubernetes setup", "Auto-scaling configured"],
|
||||
"recommendations": []
|
||||
},
|
||||
{
|
||||
"category": "Data Scaling",
|
||||
"status": "CONCERN",
|
||||
"description": "No database sharding strategy for large data growth",
|
||||
"evidence": ["Single database instance", "No partitioning"],
|
||||
"recommendations": ["Plan database sharding strategy", "Implement read replicas", "Consider database clustering"]
|
||||
}
|
||||
],
|
||||
"compliance": {
|
||||
"1M_users": "PASS",
|
||||
"10M_users": "CONCERN",
|
||||
"100M_users": "FAIL"
|
||||
},
|
||||
"priority_actions": ["Design database sharding strategy for future growth", "Implement read replicas for read-heavy workloads"],
|
||||
"summary": "Scalability is good up to 1M users, concerns for 10M+ users"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Subagent completes when JSON output written to temp file.
|
||||
@@ -0,0 +1,264 @@
|
||||
---
|
||||
name: 'step-04e-aggregate-nfr'
|
||||
description: 'Aggregate NFR domain assessments into executive summary'
|
||||
nextStepFile: './step-05-generate-report.md'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 4E: Aggregate NFR Assessment Results
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Read outputs from 4 parallel NFR subagents, calculate overall risk level, aggregate compliance status, and identify cross-domain risks.
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
- ✅ Read all 4 subagent outputs
|
||||
- ✅ Calculate overall risk level
|
||||
- ❌ Do NOT re-assess NFRs (use subagent outputs)
|
||||
|
||||
---
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
### 1. Read All Subagent Outputs
|
||||
|
||||
```javascript
|
||||
const domains = ['security', 'performance', 'reliability', 'scalability'];
|
||||
const assessments = {};
|
||||
|
||||
domains.forEach((domain) => {
|
||||
const outputPath = `/tmp/tea-nfr-${domain}-{{timestamp}}.json`;
|
||||
assessments[domain] = JSON.parse(fs.readFileSync(outputPath, 'utf8'));
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Calculate Overall Risk Level
|
||||
|
||||
**Risk hierarchy:** HIGH > MEDIUM > LOW > NONE
|
||||
|
||||
```javascript
|
||||
const riskLevels = { HIGH: 3, MEDIUM: 2, LOW: 1, NONE: 0 };
|
||||
const domainRisks = domains.map((d) => assessments[d].risk_level);
|
||||
const maxRiskValue = Math.max(...domainRisks.map((r) => riskLevels[r]));
|
||||
const overallRisk = Object.keys(riskLevels).find((k) => riskLevels[k] === maxRiskValue);
|
||||
```
|
||||
|
||||
**Risk assessment:**
|
||||
|
||||
- If ANY domain is HIGH → overall is HIGH
|
||||
- If ANY domain is MEDIUM (and none HIGH) → overall is MEDIUM
|
||||
- If ALL domains are LOW/NONE → overall is LOW
|
||||
|
||||
---
|
||||
|
||||
### 3. Aggregate Compliance Status
|
||||
|
||||
```javascript
|
||||
const allCompliance = {};
|
||||
|
||||
domains.forEach((domain) => {
|
||||
const compliance = assessments[domain].compliance;
|
||||
Object.entries(compliance).forEach(([standard, status]) => {
|
||||
if (!allCompliance[standard]) {
|
||||
allCompliance[standard] = [];
|
||||
}
|
||||
allCompliance[standard].push({ domain, status });
|
||||
});
|
||||
});
|
||||
|
||||
// Determine overall compliance per standard
|
||||
const complianceSummary = {};
|
||||
Object.entries(allCompliance).forEach(([standard, statuses]) => {
|
||||
const hasFail = statuses.some((s) => s.status === 'FAIL');
|
||||
const hasPartial = statuses.some((s) => s.status === 'PARTIAL' || s.status === 'CONCERN');
|
||||
|
||||
complianceSummary[standard] = hasFail ? 'FAIL' : hasPartial ? 'PARTIAL' : 'PASS';
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Identify Cross-Domain Risks
|
||||
|
||||
**Look for risks that span multiple domains:**
|
||||
|
||||
```javascript
|
||||
const crossDomainRisks = [];
|
||||
|
||||
// Example: Performance + Scalability issue
|
||||
const perfConcerns = assessments.performance.findings.filter((f) => f.status !== 'PASS');
|
||||
const scaleConcerns = assessments.scalability.findings.filter((f) => f.status !== 'PASS');
|
||||
if (perfConcerns.length > 0 && scaleConcerns.length > 0) {
|
||||
crossDomainRisks.push({
|
||||
domains: ['performance', 'scalability'],
|
||||
description: 'Performance issues may worsen under scale',
|
||||
impact: 'HIGH',
|
||||
});
|
||||
}
|
||||
|
||||
// Example: Security + Reliability issue
|
||||
const securityFails = assessments.security.findings.filter((f) => f.status === 'FAIL');
|
||||
const reliabilityConcerns = assessments.reliability.findings.filter((f) => f.status !== 'PASS');
|
||||
if (securityFails.length > 0 && reliabilityConcerns.length > 0) {
|
||||
crossDomainRisks.push({
|
||||
domains: ['security', 'reliability'],
|
||||
description: 'Security vulnerabilities may cause reliability incidents',
|
||||
impact: 'CRITICAL',
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 5. Aggregate Priority Actions
|
||||
|
||||
```javascript
|
||||
const allPriorityActions = domains.flatMap((domain) =>
|
||||
assessments[domain].priority_actions.map((action) => ({
|
||||
domain,
|
||||
action,
|
||||
urgency: assessments[domain].risk_level === 'HIGH' ? 'URGENT' : 'NORMAL',
|
||||
})),
|
||||
);
|
||||
|
||||
// Sort by urgency
|
||||
const prioritizedActions = allPriorityActions.sort((a, b) => (a.urgency === 'URGENT' ? -1 : 1));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6. Generate Executive Summary
|
||||
|
||||
```javascript
|
||||
const resolvedMode = subagentContext?.execution?.resolvedMode ?? 'unknown';
|
||||
const subagentExecutionLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'SEQUENTIAL (4 NFR domains)'
|
||||
: resolvedMode === 'agent-team'
|
||||
? 'AGENT-TEAM (4 NFR domains)'
|
||||
: resolvedMode === 'subagent'
|
||||
? 'SUBAGENT (4 NFR domains)'
|
||||
: 'MODE-DEPENDENT (4 NFR domains)';
|
||||
|
||||
const performanceGainLabel =
|
||||
resolvedMode === 'sequential'
|
||||
? 'baseline (no parallel speedup)'
|
||||
: resolvedMode === 'agent-team' || resolvedMode === 'subagent'
|
||||
? '~67% faster than sequential'
|
||||
: 'mode-dependent';
|
||||
|
||||
const executiveSummary = {
|
||||
overall_risk: overallRisk,
|
||||
assessment_date: new Date().toISOString(),
|
||||
|
||||
domain_assessments: assessments,
|
||||
|
||||
compliance_summary: complianceSummary,
|
||||
|
||||
cross_domain_risks: crossDomainRisks,
|
||||
|
||||
priority_actions: prioritizedActions,
|
||||
|
||||
risk_breakdown: {
|
||||
security: assessments.security.risk_level,
|
||||
performance: assessments.performance.risk_level,
|
||||
reliability: assessments.reliability.risk_level,
|
||||
scalability: assessments.scalability.risk_level,
|
||||
},
|
||||
|
||||
subagent_execution: subagentExecutionLabel,
|
||||
performance_gain: performanceGainLabel,
|
||||
};
|
||||
|
||||
// Save for Step 5 (report generation)
|
||||
fs.writeFileSync('/tmp/tea-nfr-summary-{{timestamp}}.json', JSON.stringify(executiveSummary, null, 2), 'utf8');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 7. Display Summary to User
|
||||
|
||||
```
|
||||
✅ NFR Assessment Complete ({subagentExecutionLabel})
|
||||
|
||||
🎯 Overall Risk Level: {overallRisk}
|
||||
|
||||
📊 Domain Risk Breakdown:
|
||||
- Security: {security_risk}
|
||||
- Performance: {performance_risk}
|
||||
- Reliability: {reliability_risk}
|
||||
- Scalability: {scalability_risk}
|
||||
|
||||
✅ Compliance Summary:
|
||||
{list standards with PASS/PARTIAL/FAIL}
|
||||
|
||||
⚠️ Cross-Domain Risks: {cross_domain_risk_count}
|
||||
|
||||
🎯 Priority Actions: {priority_action_count}
|
||||
|
||||
🚀 Performance: {performanceGainLabel}
|
||||
|
||||
✅ Ready for report generation (Step 5)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
### 8. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-04e-aggregate-nfr']
|
||||
lastStep: 'step-04e-aggregate-nfr'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-04e-aggregate-nfr'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-04e-aggregate-nfr'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
---
|
||||
|
||||
## EXIT CONDITION
|
||||
|
||||
Proceed to Step 5 when:
|
||||
|
||||
- ✅ All subagent outputs read
|
||||
- ✅ Overall risk calculated
|
||||
- ✅ Compliance aggregated
|
||||
- ✅ Summary saved
|
||||
- ✅ Progress saved to output document
|
||||
|
||||
Load next step: `{nextStepFile}`
|
||||
|
||||
---
|
||||
|
||||
## 🚨 SYSTEM SUCCESS METRICS
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- All 4 NFR domains aggregated correctly
|
||||
- Overall risk level determined
|
||||
- Executive summary complete
|
||||
|
||||
### ❌ FAILURE:
|
||||
|
||||
- Failed to read subagent outputs
|
||||
- Risk calculation incorrect
|
||||
@@ -0,0 +1,108 @@
|
||||
---
|
||||
name: 'step-05-generate-report'
|
||||
description: 'Create NFR report and validation summary'
|
||||
outputFile: '{test_artifacts}/nfr-assessment.md'
|
||||
---
|
||||
|
||||
# Step 5: Generate Report & Validate
|
||||
|
||||
## STEP GOAL
|
||||
|
||||
Produce the NFR assessment report and validate completeness.
|
||||
|
||||
## MANDATORY EXECUTION RULES
|
||||
|
||||
- 📖 Read the entire step file before acting
|
||||
- ✅ Speak in `{communication_language}`
|
||||
|
||||
---
|
||||
|
||||
## EXECUTION PROTOCOLS:
|
||||
|
||||
- 🎯 Follow the MANDATORY SEQUENCE exactly
|
||||
- 💾 Record outputs before proceeding
|
||||
- 📖 Load the next step only when instructed
|
||||
|
||||
## CONTEXT BOUNDARIES:
|
||||
|
||||
- Available context: config, loaded artifacts, and knowledge fragments
|
||||
- Focus: this step's goal only
|
||||
- Limits: do not execute future steps
|
||||
- Dependencies: prior steps' outputs (if any)
|
||||
|
||||
## MANDATORY SEQUENCE
|
||||
|
||||
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
||||
|
||||
## 1. Report Generation
|
||||
|
||||
Use `nfr-report-template.md` to produce `{outputFile}` containing:
|
||||
|
||||
- Category results (PASS/CONCERNS/FAIL)
|
||||
- Evidence summary
|
||||
- Remediation actions
|
||||
- Gate-ready YAML snippet (if applicable)
|
||||
|
||||
---
|
||||
|
||||
## 2. Polish Output
|
||||
|
||||
Before finalizing, review the complete output document for quality:
|
||||
|
||||
1. **Remove duplication**: Progressive-append workflow may have created repeated sections — consolidate
|
||||
2. **Verify consistency**: Ensure terminology, risk scores, and references are consistent throughout
|
||||
3. **Check completeness**: All template sections should be populated or explicitly marked N/A
|
||||
4. **Format cleanup**: Ensure markdown formatting is clean (tables aligned, headers consistent, no orphaned references)
|
||||
|
||||
---
|
||||
|
||||
## 3. Validation
|
||||
|
||||
Validate against `checklist.md` and fix gaps.
|
||||
|
||||
- [ ] CLI sessions cleaned up (no orphaned browsers)
|
||||
|
||||
---
|
||||
|
||||
## 4. Save Progress
|
||||
|
||||
**Save this step's accumulated work to `{outputFile}`.**
|
||||
|
||||
- **If `{outputFile}` does not exist** (first save), create it using the workflow template (if available) with YAML frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
stepsCompleted: ['step-05-generate-report']
|
||||
lastStep: 'step-05-generate-report'
|
||||
lastSaved: '{date}'
|
||||
---
|
||||
```
|
||||
|
||||
Then write this step's output below the frontmatter.
|
||||
|
||||
- **If `{outputFile}` already exists**, update:
|
||||
- Add `'step-05-generate-report'` to `stepsCompleted` array (only if not already present)
|
||||
- Set `lastStep: 'step-05-generate-report'`
|
||||
- Set `lastSaved: '{date}'`
|
||||
- Append this step's output to the appropriate section of the document.
|
||||
|
||||
---
|
||||
|
||||
## 5. Completion Summary
|
||||
|
||||
Report:
|
||||
|
||||
- Overall NFR status
|
||||
- Critical blockers or waivers needed
|
||||
- Next recommended workflow (`trace` or release gate)
|
||||
|
||||
## 🚨 SYSTEM SUCCESS/FAILURE METRICS:
|
||||
|
||||
### ✅ SUCCESS:
|
||||
|
||||
- Step completed in full with required outputs
|
||||
|
||||
### ❌ SYSTEM FAILURE:
|
||||
|
||||
- Skipped sequence steps or missing outputs
|
||||
**Master Rule:** Skipping steps is FORBIDDEN.
|
||||
Reference in New Issue
Block a user