# Microdrama V3 - Refined Workflow Plan

## Goal
Create a **repeatable workflow** for generating short-form video content (45-60 seconds) from jokes/stories, with iterative human feedback at each stage.

---

## Key Principles

### From External Research (10,000+ generations)

1. **Volume over Perfection** - Generate multiple variations, select the best
2. **ONE Action Per Scene** - "Walking while talking while eating" = chaos
3. **3-Second Hook** - First 3 seconds must create immediate emotional response
4. **Front-load important elements** - AI weights early words more heavily
5. **Negative prompts always** - Prevent common failures (watermarks, distorted hands, text)

### From Our Airport-Reaper Experience (V2 Grid Test)

6. **Grid technique works** - Character consistency significantly better than individual frame generation
7. **AI interprets literally, not perceptually** - "green poncho APPEARS as black cloak through fog" generates actual black cloak, not visual illusion
8. **Compound actions fail** - "airplane moving while man waves" gives weird results - choose ONE action
9. **Explicit direction required** - "looking OUT through the airplane window, eyes fixed on window" works; implicit "reacting to what they see outside" doesn't
10. **Reference images help for cultural specifics** - Traditional scythe usage needed visual reference
11. **Special effect frames need separate generation** - Generate frame 8 (horror POV) separately, not in grid
12. **PIL crop sufficient** - No need for NanoBanana extraction; simple crop gives good quality at 4K resolution

### From V1 Microdrama Pipeline (Greenland, Gagarin-Lebedev)

13. **Veo3 duration parameter DOESN'T WORK** - Always generates 8 seconds regardless of parameter. Must trim with FFmpeg if shorter needed.
14. **Prominent people filter** - Veo3 rejects:
    - Real historical figure names in prompts
    - Images resembling famous people (even character sheets based on them)
    - Text captions with real names visible
    - **Workaround:** Use generic descriptions: "fictional elderly diplomat, bald, white beard"
15. **Sound effects work** - Can request diegetic sounds (gasps, rustling, impacts). NO speech capability, NO background music (explicitly request "no music")
16. **Frame descriptions must be DRAMATICALLY different** - For visible motion between start/end frames, the descriptions need significant change, not subtle
17. **Character sheets help consistency** - 7-panel turnaround (4 full-body + 3 portrait) in 16:9 format
18. **First frame → Last frame continuity** - Use first frame as reference when generating last frame for same scene

---

## Workflow Stages

### Stage 1: Source Material
**Input:** Joke, anecdote, short article
**Output:** `story.md`

Save the raw source text. No processing yet.

---

### Stage 2: Story Understanding
**Input:** `story.md`
**Output:** `understanding.md`

Claude analyzes and writes:
1. **Retelling** - the story in own words
2. **Meaning** - what's the point/punchline
3. **What makes it funny/interesting** - the comedic/dramatic mechanism
4. **Visual potential** - what can be shown vs what's just words
5. **What WON'T work** - complex transformations, perceptual illusions, compound actions

**User feedback:** Accept or correct interpretation before proceeding.

---

### Stage 3: Script Draft
**Input:** `understanding.md`
**Output:** `script/001/script.json`

Create script with:
- **Up to 16 scenes** (even number for start/end frame pairs)
- **3-4 seconds per scene**
- **ONE action per scene** - no compound actions!
- **Hook in first 3 seconds** - something visually engaging immediately
- **Simple, achievable visuals** - avoid perceptual tricks AI can't do

**Critical lesson from airport-reaper:**
> Don't write "same man but LOOKS like Death through fog" - AI will generate actual Death.
> Instead: either show the actual man, or show actual Death - not the perceptual transformation.

Script structure:
```json
{
  "title": "Story title",
  "style": {
    "visual": "cinematic-realistic | cartoon-pixar | anime | documentary | noir",
    "genre": "comedy | horror | drama | thriller | absurdist",
    "tone": "dark-comedy | wholesome | tense | surreal"
  },
  "hook_description": "What happens in first 3 seconds to grab attention",
  "scenes": [
    {
      "number": 1,
      "duration": 3,
      "action": "Single clear action - ONE verb only",
      "what_is_shown": "Literal description of what viewer sees",
      "shot_type": "wide | medium | close-up | extreme-close-up | POV",
      "camera": "static | slow-push | slow-pull | orbit | handheld",
      "audio_cues": "ambient sounds, effects (no speech)"
    }
  ],
  "special_frames": ["List of frames needing separate generation (e.g., POV, transformations)"],
  "total_duration": 48
}
```

**User feedback:** Accept, modify scenes, or change style. Iterate until satisfied.

---

### Stage 4: Assets Identification
**Input:** `script/current/script.json`
**Output:** `assets.json` + reference searches

Identify all needed assets:
- **Characters** - name, age, appearance, clothing, key features
- **Locations** - where scenes take place
- **Objects** - props that appear (especially cultural-specific items!)

**Lesson from airport-reaper:** Traditional Russian scythe (коса-литовка) needed reference image because AI defaulted to Grim Reaper scythe. Always search for references when objects have cultural/regional specifics.

```json
{
  "characters": [
    {
      "id": "worker",
      "name": "Semyonych",
      "age": "45-50",
      "appearance": "weathered face, stubble, tired eyes",
      "clothing": "dark green military rain poncho with hood",
      "appears_in_scenes": [1, 2, 4, 5, 9],
      "needs_consistency": true
    }
  ],
  "locations": [...],
  "objects": [
    {
      "id": "scythe",
      "description": "traditional grass mowing scythe - long handle, curved blade held PARALLEL to ground",
      "NOT": "Grim Reaper scythe (vertical)",
      "reference_needed": true,
      "reference_search": "коса литовка косить траву"
    }
  ]
}
```

**User feedback:** Accept references or provide alternatives.

---

### Stage 5: Characteristic Shot (NEW - Critical Validation Gate)
**Input:** Script + approved references
**Output:** `characteristic/001/shot.jpg`

Generate ONE image with NanoBanana that includes:
- Main character in main location
- Key objects from story
- The visual style defined in script

**Purpose:** Validate the entire visual direction BEFORE committing to full grid.

**Why this matters:** In airport-reaper, we iterated the grid prompt 5+ times. If we had validated one characteristic shot first, we would have caught issues earlier:
- Scythe held wrong
- Poncho color/style
- Location atmosphere

**Prompt formula:**
```
A [STYLE] scene showing [MAIN CHARACTER with full description] in [MAIN LOCATION].
[Key objects visible in scene].
[Lighting and atmosphere from script].
Style: [visual style], [technical specs: 4K, realistic textures, etc.]
```

**User feedback:** This is the GATE. If characteristic shot doesn't work:
- Return to Stage 4 (change character/location/object design)
- Return to Stage 3 (change style)
- Iterate until visual direction is approved

---

### Stage 6: Grid Generation (4x4)
**Input:** Approved characteristic shot + script
**Output:** `grids/001/grid.jpg`

Generate 4x4 grid (16 frames) at 4K resolution, 9:16 aspect ratio.

**Grid prompt formula (tested with airport-reaper):**
```
A cinematic 4x4 storyboard in 9:16 vertical format. Each frame is 9:16 portrait.
Label each frame with 'frame 1, frame 2' etc in small text on top.

[Attach characteristic shot as reference for visual consistency]

Frame 1 (row 1, col 1): [Shot type]. [SINGLE action]. [Composition]. Vertical 9:16.
Frame 2 (row 1, col 2): SAME [character] [doing single action]. Vertical 9:16.
...
Frame 16 (row 4, col 4): [Final scene]. Vertical 9:16.

CRITICAL: [MAIN CHARACTER] must remain visually consistent across frames [list].
Style: [style], cinematic lighting, ultra-realistic, 4K resolution.
NO TEXT OR SPEECH BUBBLES IN ANY FRAME.
```

**Lessons from airport-reaper:**
- Always say "SAME worker Semyonych" not just "the worker"
- Add "Vertical 9:16 composition" to EVERY frame description
- List specific frames for character consistency
- Exclude special effect frames from grid (generate separately)

If script has fewer than 16 scenes, remaining cells can be:
- Variations of key scenes (for selection - volume over perfection!)
- Alternative camera angles

**User feedback:** Accept grid or request regeneration.

---

### Stage 7: Special Frame Generation (if needed)
**Input:** Grid + script's `special_frames` list
**Output:** `frames/special/frame_XX.jpg`

For frames that can't be in grid (POV shots, transformations, etc.):
- Generate separately with appropriate reference
- Use image-to-image if transforming existing frame

**Example from airport-reaper - Frame 8 (horror POV):**
The grid couldn't generate "same man through dirty window looks like Death" because AI interprets literally. Solution: generate horror POV separately, accepting it will be stylized.

---

### Stage 8: Frame Extraction
**Input:** `grids/001/grid.jpg`
**Output:** `frames/001/frame_01.jpg` ... `frame_16.jpg`

Extract frames using PIL crop (not NanoBanana extraction).

**Why PIL crop is sufficient:**
- 4K grid (3072x5504 for 9:16) gives 768x918 per frame in 4x4
- Actually: we should use 4K with 9:16 for GRID, but individual frames come out ~768x1376 each
- This is adequate for video generation

**Calculation for 4x4 grid at 4K (9:16):**
- Grid: 2160 × 3840 (4K portrait)
- Each frame: 540 × 960
- May need upscaling for best video quality

**Alternative: Generate grid at max resolution:**
- NanoBanana 4K: 3072 × 5504 (9:16)
- Each frame: 768 × 1376
- Better starting quality

**Future enhancement:** NanoBanana "extract frame N and upscale" if quality insufficient.

---

### Stage 9: Script Adaptation
**Input:** Extracted frames + original script
**Output:** `script/002/script.json` (adapted)

**Critical step - lesson from airport-reaper:**

AI interprets prompts differently than intended. The man in frame 5 might be holding scythe at different angle than described. Instead of fighting this:

1. Review each extracted frame
2. Describe what's ACTUALLY shown (not what we wanted)
3. Adapt video prompts to match actual visuals
4. Adjust actions if the frame shows something different

This ensures video prompts describe what's in the frame, not what we wished was there.

---

### Stage 10: Video Generation (Two Scenes Per Clip, Start+End Frame)
**Input:** Extracted frames + adapted script
**Output:** `videos/clip_01/video.mp4` ... (8 clips for 16 frames)

**Method (from V1 Gagarin-Lebedev):** Each video clip covers TWO consecutive scenes with start AND end frame:
- Clip 1: frame_01 (start) → frame_02 (end) = scenes 1→2
- Clip 2: frame_03 (start) → frame_04 (end) = scenes 3→4
- ...
- Clip 8: frame_15 (start) → frame_16 (end) = scenes 15→16

**Why start+end frame matters:**
1. **Avoids hallucinations** - AI must reach the specified end frame, can't drift into random content
2. **More consistency** - Both start and end are from our grid, ensuring visual coherence
3. **Shorter per-scene segments** - Each scene gets ~4 seconds (half of 8s clip), more controlled motion
4. **Predictable results** - Know exactly what each clip will look like at start and end

This gives **8 video clips** from 16 frames (16 scenes, 2 per clip).

**Video prompt formula (from V1 Gagarin-Lebedev):**
```
Generate an 8-second video clip.

VISUAL STYLE: [style from script]

ACTION (what happens in this 8-second clip):
[scene N frame description] → [transition] → [scene N+1 frame description]

STARTING FRAME: [scene N frame - describe what's in the actual image]
ENDING FRAME: [scene N+1 frame - describe what's in the actual image]

TRANSITION: [cut/zoom_in/zoom_out/dissolve]

CAMERA: [movement type]

AUDIO: Include natural sound effects:
- [scene N sound hints], [scene N+1 sound hints]
- Character reactions: sighs, exhales, gasps (NO speech, NO dialogue)
- NO background music
- Only diegetic sounds

IMPORTANT:
- Photorealistic, natural movements
- Continuous smooth motion throughout 8 seconds
- Subtle expressions and gestures, not over-dramatic

NO text, NO speech, NO watermarks, NO distorted hands.
```

**Models comparison (from our testing):**

| Model | Duration | Quality | Speed | Cost | Notes |
|-------|----------|---------|-------|------|-------|
| Veo3 fast | **Always 8s** | Good | ~30s | ~$0.15 | Duration param IGNORED |
| Kling O1 | 5s fixed | Good | ~60s | ~$0.15 | Predictable |

**CRITICAL:** Veo3 always outputs 8 seconds - perfect for "two scenes per clip" method where each clip needs enough time for transition between scenes.

**Kling consideration:** 5 seconds may be tight for 2-scene transitions. Test needed.

**Audio in Veo3:**
- Can request: gasps, rustling, impacts, ambient sounds
- Cannot: speech, dialogue, background music
- Explicitly add "no music" to avoid unwanted soundtracks

**Total video duration:** 8 clips × 8 seconds = 64 seconds (trim if needed for 45-60 sec target)

---

### Stage 11: Assembly
**Input:** All scene videos
**Output:** `assembly/001/final.mp4`

FFmpeg assembly:
1. Optional caption overlay (tested with airport-reaper)
2. Concatenate all scenes
3. Optional audio track

**Caption command (tested):**
```bash
ffmpeg -i input.mp4 -vf "drawtext=text='Caption':fontsize=36:fontcolor=white:x=(w-tw)/2:y=h-th-40:box=1:boxcolor=black@0.6:boxborderw=10:fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf" output.mp4
```

---

## Directory Structure

```
/mnt/video/microdrama-v3/{story-name}/
├── CLAUDE.md                    # Instructions for Claude sessions
├── story.md                     # Original source (Stage 1)
├── understanding.md             # Analysis (Stage 2)
├── assets.json                  # Assets list (Stage 4)
├── script/
│   ├── 001/script.json          # Initial script (Stage 3)
│   ├── 002/script.json          # Adapted to frames (Stage 9)
│   └── current → 002
├── references/
│   ├── character_{name}/
│   │   ├── search/              # Yandex results
│   │   └── selected.jpg
│   ├── location_{name}/
│   └── object_{name}/
├── characteristic/
│   ├── 001/
│   │   ├── shot.jpg             # Validation shot (Stage 5)
│   │   └── prompt.txt
│   └── current → 001
├── grids/
│   ├── 001/
│   │   ├── grid.jpg             # 4x4 grid (Stage 6)
│   │   └── prompt.txt
│   └── current → 001
├── frames/
│   ├── 001/
│   │   ├── frame_01.jpg ... frame_16.jpg  # Extracted (Stage 8)
│   ├── special/
│   │   └── frame_08.jpg         # Special frames (Stage 7)
│   └── current → 001
├── videos/
│   ├── scene_01/
│   │   ├── 001/video.mp4
│   │   └── current → 001
│   └── ...
└── assembly/
    ├── 001/final.mp4
    └── current → 001
```

---

## Iteration Rules

1. **Never delete** - always create new iteration
2. **Symlink `current`** - points to latest approved version
3. **Save prompts** - every generation saves prompt.txt
4. **Can go back** - return to any previous stage with feedback

---

## What We Learned NOT to Do

### From Airport-Reaper (V2)

1. **Don't try perceptual transformations**
   - BAD: "green poncho APPEARS as black cloak through fog"
   - GOOD: Either show green poncho OR show black cloak, not the illusion

2. **Don't combine actions in one scene**
   - BAD: "airplane taxiing while worker waves"
   - GOOD: Scene 1: airplane taxiing. Scene 2: worker waves.

3. **Don't assume AI understands implicit relationships**
   - BAD: "passengers react to what they see outside"
   - GOOD: "passengers looking OUT through airplane window, eyes fixed on window, faces frozen in terror"

4. **Don't skip reference images for cultural specifics**
   - Russian scythe ≠ Grim Reaper scythe
   - Always search and provide visual reference

5. **Don't put special effect frames in grid**
   - POV shots, horror transformations, unusual perspectives - generate separately

### From V1 Microdrama Pipeline

6. **Don't use real names for historical/famous people**
   - BAD: "Prince Gorchakov signs the document"
   - GOOD: "fictional elderly diplomat in military uniform signs the document"
   - Veo3 has aggressive prominent people filter

7. **Don't expect Veo3 duration to work**
   - BAD: Planning 4-second clips with duration=4
   - GOOD: Plan for 8-second clips, trim in post if needed

8. **Don't make subtle frame differences for video**
   - BAD: start="man standing" end="man standing slightly moved"
   - GOOD: start="man standing" end="man walking away" - DRAMATIC change needed for visible motion

9. **Don't request speech or dialogue**
   - Veo3 can only do exclamations ("Wow!", gasps, reactions)
   - For dialogue: use captions overlay instead

10. **Don't put text captions in generated frames**
    - May trigger content filters if text resembles real names
    - Add captions with FFmpeg in post-production

---

## Style Library

### Visual Styles
- cinematic-realistic (Fincher, Villeneuve)
- documentary
- cartoon-pixar
- noir
- vintage-film
- horror-atmospheric

### Genres
- comedy (dark, absurdist, wholesome)
- horror
- thriller
- drama

### Camera Movements (reliable)
- static (highest quality)
- slow push-in
- slow pull-out
- orbit around subject
- handheld (adds energy)

**Avoid:** Complex combinations ("pan while zooming")

---

## Negative Prompts (always include)
```
NO watermark, NO text, NO speech bubbles, NO subtitles, NO warped faces, NO floating limbs, NO distorted hands, NO blurry edges
```

---

## Cost Estimate

| Stage | Tool | Cost |
|-------|------|------|
| Characteristic shot | NanoBanana | ~$0.05 |
| 4x4 Grid | NanoBanana | ~$0.10 |
| 8 videos (5 sec each) | Veo3/Kling | ~$1.20 |
| **Total** | | **~$1.35** |

With iterations: budget $3-5 per story.

**Comparison:** Airport-reaper cost ~$1.50 for 9 scenes (3x3 grid). 4x4 grid with 16 frames should be similar, maybe slightly more for videos.

---

## Implementation Order

### Phase 1: Manual Test (this session)
1. User provides English joke
2. Walk through stages 1-11 manually
3. Document what works / what doesn't

### Phase 2: Automation Scripts
4. `extract_frames_4x4.py` - PIL crop for 4x4 grid
5. `generate_videos_startend.py` - Start+end frame video gen
6. Update `assemble_video.py` for new workflow

### Phase 3: CLAUDE.md Template
7. Template for story CLAUDE.md
8. Entry points per stage

---

## Decisions Made

1. **Grid size:** 4x4 (16 frames)
2. **Video method:** Two scenes per clip (frame N → frame N+1 transition), resulting in 8 clips
3. **Aspect ratio:** 9:16 vertical (TikTok/Reels/Shorts)
4. **Test scope:** Full pipeline (all 11 stages)

---

## Test Story

User will provide an English joke after plan approval.

---

## Verification Checklist

After completing the test:
- [ ] Characteristic shot validated visual direction before grid
- [ ] 4x4 grid generated with consistent character
- [ ] 16 frames extracted successfully
- [ ] 8 video clips generated (each covering 2 scenes)
- [ ] Final assembly 45-60 seconds
- [ ] Document what worked and what didn't
