# Student Guide: AI Video Generation System

This repository contains two production systems for automated video generation using AI:

1. **WB Video Bot** — A Telegram bot that generates 8-second promotional videos for marketplace products
2. **Microdrama Pipeline** — A 13-stage pipeline for producing 30-90 second short videos (sketches, story adaptations)

Both systems demonstrate how to orchestrate multiple AI models (Claude, Veo3, NanoBanana/Gemini) into automated production pipelines.

---

## Architecture Overview

```
┌─────────────────────────────────────────────────────────────────┐
│                    WB VIDEO BOT                                 │
│                                                                 │
│  Telegram Bot ← User sends product URL                         │
│       ↓                                                         │
│  Stage 1: Data Extraction (Sellematics API / Playwright)        │
│       ↓                                                         │
│  Stage 2: Story Vision (Claude) → archetype + style + story     │
│       ↓                                                         │
│  Stage 3: Frame Design (Claude + NanoBanana) → start frames     │
│       ↓                                                         │
│  Stage 4: Video Generation (Veo3) → 8-sec promo video           │
│       ↓                                                         │
│  User receives video in Telegram                                │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                 MICRODRAMA PIPELINE                              │
│                                                                 │
│  Story/Script Input                                             │
│       ↓                                                         │
│  01: Story → 02: Understanding → 03: Script → 04: Assets        │
│       ↓                                                         │
│  05: References → 06: Characteristic Shot → 07: Grid Generation │
│       ↓                                                         │
│  08: Frame Extraction → 09: Script Adaptation → 10: Video Gen   │
│       ↓                                                         │
│  11: Titles → 12: Captions → 13: Assembly (FFmpeg)              │
│       ↓                                                         │
│  Final video with subtitles and title cards                     │
└─────────────────────────────────────────────────────────────────┘
```

---

## WB Video Bot

### How It Works

The bot receives a Wildberries product URL from a user via Telegram and generates a short promotional video through 4 automated stages.

### Stage 1: Data Extraction

**Files:** `scripts/sellematics_fetcher.py`, `scripts/wb_fetcher_enhanced.py`

Extracts product data: name, description, characteristics, images, reviews, rating. Two data sources:
- **Sellematics API** — rich structured data (requires SSH tunnel)
- **Playwright scraper** — fallback, scrapes the WB product page directly

### Stage 2: Story Vision

**Files:** `scripts/full_pipeline_v2.py`, `prompts/stages/stage2_story_vision.md`

Claude analyzes the product data and creates a video concept:
- Selects an **archetype** (problem/solution, before/after, epic hero, gift, etc.)
- Picks a **visual style** from 360 combinations (20 styles × 18 video kinds)
- Writes a 2-act story structure
- Incorporates user feedback from previous iterations

### Stage 3: Frame Design

**Files:** `scripts/full_pipeline_v2.py`, `prompts/stages/stage3_frame_design.md`

Claude analyzes the product photos (via Vision API) and generates detailed image prompts. NanoBanana (Gemini-based image generator) creates the starting frames for each act.

Key technique: Act 2's frame uses Act 1's frame as a reference image for visual consistency.

### Stage 4: Video Generation

**Files:** `scripts/full_pipeline_v2.py`, `prompts/stages/stage4_video_prompt.md`

The pipeline sends the starting frame + motion prompt to Veo3, which generates an 8-second video. For the WB card format (3:4), Veo3 generates 9:16 and the pipeline crops it.

### Generation Modes

| Mode | Aspect Ratio | Resolution | Use Case |
|------|-------------|------------|----------|
| **wbcard** | 3:4 | 720×960 | WB product cards (default) |
| vertical | 9:16 | 720×1280 | Stories, Reels |
| cinematic | 16:9 | 1280×720 | YouTube, websites |

### Token System

Users pay with tokens (1 token = 1 generation). Key files:
- `scripts/database.py` — SQLite schema and CRUD operations
- `scripts/token_manager.py` — Balance operations with audit trail
- `scripts/payment_handlers.py` — YooKassa integration via Telegram Payments

### Feedback Loop

Users can iterate on results:
- "Refine" — adjusts the current direction
- "Explore" — tries a completely different approach
- Voice messages are transcribed and incorporated

---

## Microdrama Pipeline

### How It Works

A 13-stage pipeline that produces 30-90 second videos from scripts, jokes, or story adaptations. Each stage is automated but allows human review.

**Files:** `microdrama-v2/scripts-v4/orchestrator_v5.py`

### Key Stages

| # | Stage | What It Does | Tool |
|---|-------|-------------|------|
| 01 | Story | Input text / scenario | — |
| 02 | Understanding | Analyze key moments, characters, style | Claude |
| 03 | Script | Scene-by-scene script with dialogue | Claude |
| 04 | Assets | Character & location descriptions | Claude |
| 05 | References | Visual reference search | Yandex Search |
| 06 | Characteristic | One frame with all characters (style validation) | NanoBanana |
| 07 | **Grid Generation** | **All frames in one image (3×3 grid)** | NanoBanana |
| 08 | Frames | Extract and refine individual frames from grid | NanoBanana |
| 09 | Script Adaptation | Adjust script to match actual frames | Claude |
| 10 | Video | Generate video for each scene | Kling / Veo3 |
| 11 | Titles | Opening and closing title cards | FFmpeg |
| 12 | Captions | Subtitle timing and text | Claude |
| 13 | Assembly | Concatenation, subtitles, final render | FFmpeg |

### Grid Generation (Key Innovation)

Instead of generating 9 frames separately, all frames are created in a single 3×3 grid image. This ensures:
- **Character consistency** across all scenes
- **9× cost reduction** (one API call instead of nine)
- **Unified color palette** and lighting

### Story Adaptation Pipeline

For multi-episode adaptations of longer stories: `microdrama-v2/scripts-v4/story_pipeline.py`

Story-level stages (S1-S5) handle ingestion, bible creation, style, characters, and episode splitting. Then each episode goes through the 13-stage orchestrator.

---

## Key Concepts

### Iteration System

Both pipelines support iterative refinement:
- Each generation saves all prompts and results to disk
- Previous feedback is fed into the next attempt
- Downstream invalidation: changing stage N marks stages N+1...13 for regeneration

### pipeline.json

The microdrama pipeline stores all state in a `pipeline.json` file per project:
- Current stage status (pending/in_progress/done)
- All generated data per stage
- Iteration history

### Video Providers

The system supports multiple video generation providers with automatic fallback:
- **Veo 3.1** (Google) — highest quality, 8-sec limit
- **Kling 3.0** (Kuaishou) — good quality, longer clips, character consistency
- **Multiple proxy endpoints** for reliability

### Prompt Engineering

Prompts are stored as markdown templates in `prompts/stages/`. They are filled with product data, feedback history, and style parameters at runtime. Key patterns:
- **Negative prompts** — explicit lists of what to avoid (text, morphing, children)
- **Two-act structure** — exposition → action, with fade transition
- **Archetype selection** — matches product type to storytelling pattern

---

## File Reference

### WB Video Bot (`scripts/`)

| File | Purpose |
|------|---------|
| `telegram_bot.py` | Telegram bot with commands, payments, feedback |
| `full_pipeline_v2.py` | Main 4-stage video generation pipeline |
| `sellematics_fetcher.py` | Product data from Sellematics API |
| `wb_fetcher_enhanced.py` | Playwright-based WB scraper (fallback) |
| `laozhang_proxy.py` | API wrapper for NanoBanana and Veo3 |
| `creative_styles.py` | 20 visual styles + 18 video kinds (360 combos) |
| `generation_modes.py` | Output formats: wbcard, vertical, cinematic |
| `feedback_parser.py` | Parse and structure user feedback |
| `database.py` | SQLite schema, migrations, CRUD |
| `token_manager.py` | Token balance operations with audit trail |
| `payment_handlers.py` | YooKassa via Telegram Payments API |
| `pollo_proxy.py` | Pollo.ai API client for Veo 3.1 |

### Microdrama Pipeline (`microdrama-v2/scripts-v4/`)

| File | Purpose |
|------|---------|
| `orchestrator_v5.py` | Main 13-stage orchestrator |
| `story_pipeline.py` | Story-level pipeline (multi-episode) |
| `stages/stage_01_story.py` | Stage implementations (01-13) |
| `...stage_13_assembly.py` | |
| `story_stages/stage_s1_ingest.py` | Story-level stages (S1-S5) |

### Admin Bot (`viewer/`)

| File | Purpose |
|------|---------|
| `viewer_bot.py` | Browse videos + admin token management |
| `data_reader.py` | Read generation data from disk |
| `search.py` | Claude-powered semantic search |

### Prompts (`prompts/`)

| Directory | Purpose |
|-----------|---------|
| `stages/` | Claude prompt templates for each pipeline stage |
| `categories/` | Product category → style mappings (JSON) |
| `negatives/` | Negative prompt library (what to avoid) |

### Infrastructure (`systemd/`)

| File | Purpose |
|------|---------|
| `videobot-test.service` | Dev bot systemd unit |
| `videobot-prod.service` | Production bot systemd unit |
| `sellematics-tunnel.service` | SSH tunnel for API access |
| `nginx-videobot.conf` | Nginx config for serving videos |

---

## How to Run

### Prerequisites

- Python 3.11+
- FFmpeg (for video processing)
- Playwright + Chromium (for WB scraping)
- API keys: LaoZhang (Claude proxy), Pollo.ai (Veo3), Telegram Bot Token

### Setup

```bash
# 1. Clone and set up virtual environment
git clone <repository-url>
cd project
python3 -m venv venv
source venv/bin/activate

# 2. Install dependencies
pip install -r scripts/requirements.txt
playwright install chromium

# 3. Configure environment
cp .env.example .env
# Edit .env with your API keys and settings

# 4. Create storage directory
mkdir -p /mnt/video

# 5. Run the bot
cd scripts
python telegram_bot.py
```

### Running the Pipeline Manually

```bash
# Generate video for a WB product
cd scripts
python full_pipeline_v2.py "https://www.wildberries.ru/catalog/PRODUCT_ID/detail.aspx"
```

### Running the Microdrama Pipeline

```bash
# Initialize a new project
cd microdrama-v2/scripts-v4
./orchestrator_v5.py /mnt/video/microdrama-v3/my-project init --brief "A comedy sketch about..."

# Run through stages interactively
./orchestrator_v5.py /mnt/video/microdrama-v3/my-project run
```

---

## Technology Stack

| Component | Technology | Purpose |
|-----------|-----------|---------|
| Language | Python 3.11 | All pipeline code |
| AI - Text | Claude (Opus/Sonnet) via LaoZhang API | Scriptwriting, prompt generation, analysis |
| AI - Images | NanoBanana (Gemini 3 Pro) | Frame generation, image editing |
| AI - Video | Veo 3.1 (Google), Kling 3.0 | Video generation from frames |
| Bot Framework | python-telegram-bot | Telegram bot interface |
| Database | SQLite | Users, tokens, transactions, showcase |
| Video Processing | FFmpeg | Cropping, concatenation, subtitles |
| Web Scraping | Playwright | WB product data extraction |
| Web Server | Nginx | Serving generated videos |
| Process Manager | systemd | Bot and tunnel services |
| Payments | YooKassa via Telegram Payments | Token purchases |

---

## Important Limitations

- **Veo3** cannot generate meaningful speech (only exclamations like "Wow!", "Mmm!")
- **Veo3** struggles with complex objects (multi-section pans, mechanisms)
- **NanoBanana** maintains scene consistency via reference images
- **Children** are prohibited in all generated scenes (models aged 25-45 only)
- **WB scraping** may be blocked by CAPTCHA — Sellematics API is preferred

---

## Notes

- All API keys in `.env.example` are placeholders — you need your own keys
- The `systemd/` configs use placeholder hostnames — update for your infrastructure
- User IDs in scripts are fake examples — replace with your own Telegram IDs
