initial commit

This commit is contained in:
2026-03-16 19:54:53 -04:00
commit bfe0e01254
3341 changed files with 483939 additions and 0 deletions

View File

@@ -0,0 +1,78 @@
# Semantic Splitting Strategy
When the source content is large (exceeds ~15,000 tokens) or a token_budget requires it, split the distillate into semantically coherent sections rather than arbitrary size breaks.
## Why Semantic Over Size-Based
Arbitrary splits (every N tokens) break coherence. A downstream workflow loading "part 2 of 4" gets context fragments. Semantic splits produce self-contained topic clusters that a workflow can load selectively — "give me just the technical decisions section" — which is more useful and more token-efficient for the consumer.
## Splitting Process
### 1. Identify Natural Boundaries
After the initial extraction and deduplication (Steps 1-2 of the compression process), look for natural semantic boundaries:
- Distinct problem domains or functional areas
- Different stakeholder perspectives (users, technical, business)
- Temporal boundaries (current state vs future vision)
- Scope boundaries (in-scope vs out-of-scope vs deferred)
- Phase boundaries (analysis, design, implementation)
Choose boundaries that produce sections a downstream workflow might load independently.
### 2. Assign Items to Sections
For each extracted item, assign it to the most relevant section. Items that span multiple sections go in the root distillate.
Cross-cutting items (items relevant to multiple sections):
- Constraints that affect all areas → root distillate
- Decisions with broad impact → root distillate
- Section-specific decisions → section distillate
### 3. Produce Root Distillate
The root distillate contains:
- **Orientation** (3-5 bullets): what was distilled, from what sources, for what consumer, how many sections
- **Cross-references**: list of section distillates with 1-line descriptions
- **Cross-cutting items**: facts, decisions, and constraints that span multiple sections
- **Scope summary**: high-level in/out/deferred if applicable
### 4. Produce Section Distillates
Each section distillate must be self-sufficient — a reader loading only one section should understand it without the others.
Each section includes:
- **Context header** (1 line): "This section covers [topic]. Part N of M from [source document names]."
- **Section content**: thematically-grouped bullets following the same compression rules as a single distillate
- **Cross-references** (if needed): pointers to other sections for related content
### 5. Output Structure
Create a folder `{base-name}-distillate/` containing:
```
{base-name}-distillate/
├── _index.md # Root distillate: orientation, cross-cutting items, section manifest
├── 01-{topic-slug}.md # Self-contained section
├── 02-{topic-slug}.md
└── 03-{topic-slug}.md
```
Example:
```
product-brief-distillate/
├── _index.md
├── 01-problem-solution.md
├── 02-technical-decisions.md
└── 03-users-market.md
```
## Size Targets
When a token_budget is specified:
- Root distillate: ~20% of budget (orientation + cross-cutting items)
- Remaining budget split proportionally across sections based on content density
- If a section exceeds its proportional share, compress more aggressively or sub-split
When no token_budget but splitting is needed:
- Aim for sections of 3,000-5,000 tokens each
- Root distillate as small as possible while remaining useful standalone