# Stage 11: Captions - Implementation Plan

**Date**: 2026-02-08
**Context**: Completing the V5 orchestrator pipeline for water-walking-dog story

## Design Decisions

1. **Manual trigger** (like extract-frames) - `generate-captions` command, with support to add captions for newly added videos
2. **No speaker labels** - Just show text without "DAVE:" prefix
3. **Global style** - Same caption style for whole story
4. **Auto line splitting** - Detect long lines and split intelligently
5. **Action captions** - Support `[knocking]`, `[dog barking]` etc. for understanding without audio

## Current State

### V3 Approach (generate_captions_v3.py)
- Generates captions globally for all scenes at once
- Saves `captions_overlay` array into `08-script-adapted/script.json`
- No per-scene iteration/refinement capability
- Timing calculated from video metadata

### What We Need for V5
- Per-scene caption management (like frames and videos)
- Iteration-based refinement per scene
- Auto-generate when video is selected
- Re-use captions when video re-selected
- Support for manual text editing

## Data Model

### Pipeline.json Structure

```json
{
  "10-videos": {
    "01-scene": "007",
    "02-scene": "006",
    ...
  },
  "11-captions": {
    "01-scene": null,      // No dialogue in scene 1
    "02-scene": "001",     // Caption iteration selected
    "03-scene": null,      // No dialogue
    "04-scene": null,
    "05-scene": "002",     // Refined caption
    ...
  },
  "12-assembly": null
}
```

### Directory Structure

```
11-captions/
├── 02-scene/
│   ├── 001/
│   │   ├── caption.json     # Caption data
│   │   ├── metadata.json    # V5 metadata (based_on, refined_from)
│   │   └── feedback.txt     # Optional feedback
│   └── 002/
│       ├── caption.json
│       ├── metadata.json
│       └── feedback.txt
├── 05-scene/
│   └── 001/
│       ├── caption.json
│       └── metadata.json
...
```

### caption.json Format

```json
{
  "scene": 2,
  "character": "dave",
  "text": "Good dog, fast dog, very good dog!",
  "delivery": "proud, warm, impressed",
  "lines": [
    {
      "text": "Good dog, fast dog,",
      "start": 1.5,
      "end": 3.2,
      "position": "bottom"
    },
    {
      "text": "very good dog!",
      "start": 3.2,
      "end": 4.8,
      "position": "bottom"
    }
  ],
  "video_duration": 5.041,
  "font_size": 42,
  "font_color": "white",
  "bg_color": "black@0.7"
}
```

## Auto-Generation Logic

### When to Auto-Generate Captions

1. **On video selection** (`select-video SCENE ITER`):
   - Check if scene has dialogue in adapted script
   - If no dialogue → set `11-captions.SCENE = null`
   - If dialogue exists:
     - Check if caption already exists for this video iteration
     - If exists → auto-select it
     - If not exists → generate new caption iteration

2. **On new video generation** (`new-video` / `refine-video`):
   - Don't auto-generate captions
   - Captions generated when video is selected

### Dependency Tracking

Caption `based_on` tracks:
```json
{
  "based_on": {
    "10-videos": "007",           // Which video iteration
    "08-script-adapted": "004"    // Which script version
  }
}
```

When video changes, check if existing caption matches new video:
- If `based_on["10-videos"]` matches → re-use caption
- If different → generate new caption for new video

## Commands

### Automatic (triggered by select-video)

```bash
# User selects video → caption auto-generated if needed
orchestrator_v5.py STORY select-video 02-scene 006
# → Checks if 02-scene has dialogue
# → Generates caption if needed
# → Updates pipeline.json: "11-captions": {"02-scene": "001"}
```

### Manual Commands

```bash
# Refine caption text/timing
orchestrator_v5.py STORY refine-caption 02-scene 001 --feedback "Split into 3 lines instead of 2"

# Edit caption directly (for simple text fixes)
orchestrator_v5.py STORY edit-caption 02-scene 001 --text "New caption text"

# Regenerate caption (e.g., if video changed)
orchestrator_v5.py STORY new-caption 02-scene

# Select specific caption iteration
orchestrator_v5.py STORY select-caption 02-scene 002

# Status
orchestrator_v5.py STORY status-captions
```

## Implementation

### stage_11_captions.py

```python
def generate_caption(
    take_path: Path,
    scene: str,
    video_iteration: str,
    feedback: Optional[str] = None
) -> dict:
    """
    Generate caption for a scene.

    1. Read dialogue from adapted script
    2. Get video duration from video metadata
    3. Split dialogue into lines (max 45 chars)
    4. Calculate timing based on video duration
    5. Save caption.json and metadata.json
    """

def refine_caption(
    take_path: Path,
    scene: str,
    iteration: str,
    feedback: str
) -> dict:
    """
    Refine existing caption with feedback.

    Uses Claude to modify:
    - Line splits
    - Timing adjustments
    - Position changes
    - Text edits
    """

def select_caption(take_path: Path, scene: str, iteration: str) -> dict:
    """Select caption iteration for scene."""

def auto_generate_captions_for_video(
    take_path: Path,
    scene: str,
    video_iteration: str
) -> Optional[str]:
    """
    Auto-generate or find existing caption for video.

    Returns caption iteration or None if no dialogue.
    Called by select-video command.
    """
```

### Integration with select-video

In `stage_10_videos.py::select_video()`:

```python
def select_video(take_path, scene, iteration):
    # ... existing selection logic ...

    # Auto-generate caption
    from stages.stage_11_captions import auto_generate_captions_for_video
    caption_iter = auto_generate_captions_for_video(take_path, scene, iteration)

    # Update pipeline
    if caption_iter is not None:
        pipeline["11-captions"][scene] = caption_iter
    else:
        pipeline["11-captions"][scene] = None  # No dialogue
```

## Caption Styling

### Defaults (configurable per story)

```python
DEFAULT_CAPTION_STYLE = {
    "font_size": 42,
    "font_color": "white",
    "bg_color": "black@0.7",
    "position": "bottom",
    "max_chars_per_line": 45,
    "box_border_width": 15
}
```

### Position Rules

- `"bottom"` - Default for most dialogue
- `"top"` - For wide/establishing shots where action is at bottom
- `"middle"` - Rarely used, for emphasis

## Assembly Integration

Stage 12 reads captions from `11-captions/XX-scene/{selected}/caption.json`:

```python
def get_captions_for_assembly(take_path: Path) -> dict:
    """
    Build caption timing map for all scenes.

    Returns: {scene_num: [caption_lines]}
    """
    pipeline = load_pipeline(take_path)
    captions_selections = pipeline.get("11-captions", {})

    caption_map = {}
    for scene, iteration in captions_selections.items():
        if iteration is None:
            continue

        caption_path = take_path / "11-captions" / scene / iteration / "caption.json"
        if caption_path.exists():
            with open(caption_path) as f:
                caption_data = json.load(f)
            caption_map[caption_data["scene"]] = caption_data["lines"]

    return caption_map
```

## Scenes with/without Dialogue (water-walking-dog)

| Scene | Has Dialogue | Character | Text |
|-------|--------------|-----------|------|
| 01 | No | - | - |
| 02 | Yes | Dave | "Good dog, fast dog, very good dog!" |
| 03 | No | - | - |
| 04 | No | - | - |
| 05 | Yes | Dave | "That's incredible! Did you see that?!" + Paul: "Yes, I saw" |
| 06 | No | - | - |
| 07 | Yes | Dave | "Did you notice anything unusual about my new dog?" |
| 08 | Yes | Paul + Dave | "Yeah." [pause] "He can't swim." + Dave: "What?!" |
| 09 | No | - | - |

**Scenes with captions: 02, 05, 07, 08** (4 of 9)

## Implementation Priority

1. **Core functions** (stage_11_captions.py)
   - `generate_caption()` - from script dialogue + video duration
   - `select_caption()` - update pipeline
   - `auto_generate_captions_for_video()` - called by select-video

2. **Integration with select-video**
   - Modify stage_10_videos.py to trigger caption generation

3. **Manual refinement**
   - `refine_caption()` with Claude for timing/text adjustments
   - `edit_caption()` for direct text edits

4. **Orchestrator commands**
   - Add caption commands to orchestrator_v5.py

5. **Assembly integration**
   - Modify stage_12_assembly.py to read from new structure

## Questions for User

1. **Auto-generation trigger**: Should captions auto-generate on `select-video`, or should there be a separate `generate-captions` command?

2. **Multi-character scenes**: Scene 05 and 08 have multiple speakers. Should each speaker get a separate caption line with their name shown?

3. **Caption style per scene**: Should caption position/style be configurable per scene, or global for the story?

4. **Feedback for captions**: What kind of feedback would you want to give? Examples:
   - "Make text shorter"
   - "Show earlier in the scene"
   - "Split into more lines"
   - "Change exact text to: ..."
