# Semantic Splitting Strategy

When the source content is large (exceeds ~15,000 tokens) or a token_budget requires it, split the distillate into semantically coherent sections rather than arbitrary size breaks.

## Why Semantic Over Size-Based

Arbitrary splits (every N tokens) break coherence. A downstream workflow loading "part 2 of 4" gets context fragments. Semantic splits produce self-contained topic clusters that a workflow can load selectively — "give me just the technical decisions section" — which is more useful and more token-efficient for the consumer.

## Splitting Process

### 1. Identify Natural Boundaries

After the initial extraction and deduplication (Steps 1-2 of the compression process), look for natural semantic boundaries:
- Distinct problem domains or functional areas
- Different stakeholder perspectives (users, technical, business)
- Temporal boundaries (current state vs future vision)
- Scope boundaries (in-scope vs out-of-scope vs deferred)
- Phase boundaries (analysis, design, implementation)

Choose boundaries that produce sections a downstream workflow might load independently.

### 2. Assign Items to Sections

For each extracted item, assign it to the most relevant section. Items that span multiple sections go in the root distillate.

Cross-cutting items (items relevant to multiple sections):
- Constraints that affect all areas → root distillate
- Decisions with broad impact → root distillate
- Section-specific decisions → section distillate

### 3. Produce Root Distillate

The root distillate contains:
- **Orientation** (3-5 bullets): what was distilled, from what sources, for what consumer, how many sections
- **Cross-references**: list of section distillates with 1-line descriptions
- **Cross-cutting items**: facts, decisions, and constraints that span multiple sections
- **Scope summary**: high-level in/out/deferred if applicable

### 4. Produce Section Distillates

Each section distillate must be self-sufficient — a reader loading only one section should understand it without the others.

Each section includes:
- **Context header** (1 line): "This section covers [topic]. Part N of M from [source document names]."
- **Section content**: thematically-grouped bullets following the same compression rules as a single distillate
- **Cross-references** (if needed): pointers to other sections for related content

### 5. Output Structure

Create a folder `{base-name}-distillate/` containing:

```
{base-name}-distillate/
├── _index.md           # Root distillate: orientation, cross-cutting items, section manifest
├── 01-{topic-slug}.md  # Self-contained section
├── 02-{topic-slug}.md
└── 03-{topic-slug}.md
```

Example:
```
product-brief-distillate/
├── _index.md
├── 01-problem-solution.md
├── 02-technical-decisions.md
└── 03-users-market.md
```

## Size Targets

When a token_budget is specified:
- Root distillate: ~20% of budget (orientation + cross-cutting items)
- Remaining budget split proportionally across sections based on content density
- If a section exceeds its proportional share, compress more aggressively or sub-split

When no token_budget but splitting is needed:
- Aim for sections of 3,000-5,000 tokens each
- Root distillate as small as possible while remaining useful standalone