diff --git a/.planning/.continue-here.md b/.planning/.continue-here.md new file mode 100644 index 0000000..f27ae84 --- /dev/null +++ b/.planning/.continue-here.md @@ -0,0 +1,72 @@ +--- +context: pre-planning +phase: 05-deployment +task: 0 +total_tasks: 0 +status: ready-to-plan +last_updated: 2026-04-27T18:44:51.832Z +--- + + +Phase 04 полностью завершена и закоммичена на ветке `feat/matrix-direct-agent-prototype` (135 тестов зелёные). Этот сеанс был посвящён архитектуре деплоя — изучили платформенные репозитории и обсудили топологию с командой платформы. Вся информация о деплое зафиксирована в `docs/deploy-architecture.md`. Phase 05 не спланирована, следующий шаг — `/gsd-plan-phase`. + + + + +- Изучены актуальные версии platform-agent, platform-agent_api, platform-master +- Уточнена топология деплоя с платформой (схема с reverse proxy и shared volume) +- Созданы `docs/deploy-architecture.md` — полное summary архитектуры деплоя + + + + +- Смержить `feat/matrix-direct-agent-prototype` → `main` +- Спланировать Phase 05 (деплой) +- Выполнить Phase 05: + - Обновить `config/matrix-agents.yaml` (добавить `base_url`, `workspace_path`, `user_agents`) + - Обновить `sdk/real.py` (AgentApi конструктор, file transfer) + - Обработка `MsgEventSendFile` в Matrix адаптере (скачать файл из volume, отправить пользователю) + - Обработка входящих файлов от Matrix пользователей (сохранить в workspace, передать в attachments) + - Написать `docker-compose.yml` для деплоя + + + + +- **Топология**: один инстанс Matrix-бота, один агент-контейнер на пользователя, reverse proxy на `lambda.coredump.ru:7000` роутит по пути `/agent_N/` +- **Файлы**: через shared volume `/agents/`. Surface пишет файл в `/agents/{N}/`, передаёт относительный путь в `attachments=["file.txt"]`. При `MsgEventSendFile(path)` — читает файл из `/agents/{N}/{path}` и шлёт в Matrix. +- **Agent API**: используем master (`attachments` и `MsgEventSendFile` есть). Ветку `#9-clientside-tool-call` игнорируем — она в разработке и убирает нужные фичи. +- **Конфиг**: два словаря — `user_id → agent_id` и `agent_id → {base_url, workspace_path}` +- **Master**: не используем для MVP. Статический конфиг. При готовности Master — мигрируем. +- **chat_id**: пока `chat_id=0` (один контекст на пользователя) + + + + +- **AGENT_ID + COMPOSIO_API_KEY**: Composio смержен в main platform-agent, теперь обязателен. Значения нужны от Азамата перед деплоем. +- **agent_api #9**: убирает `attachments` и `MsgEventSendFile` — если смержат до деплоя, сломает наш file transfer. Нужно уточнить сроки. + + +## Required Reading (in order) + +1. `docs/deploy-architecture.md` — полная архитектура деплоя, топология, API, файловый обмен, конфиг +2. `adapter/matrix/routed_platform.py` — текущий RoutedPlatformClient +3. `sdk/real.py` — текущий AgentApi wrapper +4. `config/matrix-agents.yaml` и `config/matrix-agents.example.yaml` — текущий формат конфига (нужно расширить) + +## Infrastructure State + +- Ветка: `feat/matrix-direct-agent-prototype` — готова к merge, 135 тестов зелёные +- `config/matrix-agents.yaml` — незакоммичен (live config, добавить в `.gitignore`) +- `docs/deploy-architecture.md` — незакоммичен (новый файл этого сеанса) +- platform-agent main: Composio уже смержен (требует `AGENT_ID`, `COMPOSIO_API_KEY` в env) + + +Архитектура деплоя полностью прояснена. Нет неизвестных блокеров (кроме env-переменных от платформы). Phase 05 — чисто инженерная задача: обновить конфиг, sdk, Matrix адаптер, написать compose. Всё что нужно знать — в docs/deploy-architecture.md. + + + +1. /clear +2. /gsd-resume-work — прочитает этот файл и предложит план Phase 05 +3. Прочитать docs/deploy-architecture.md +4. /gsd-plan-phase 05 + diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md index d90b47e..9c859f8 100644 --- a/.planning/PROJECT.md +++ b/.planning/PROJECT.md @@ -2,44 +2,56 @@ ## What This Is -Surfaces (поверхности) — это тонкие адаптеры-клиенты, соединяющие мессенджеры с агентами платформы Lambda. -Текущая и главная реализация — **Matrix MVP**. Бот работает как stateless-прослойка: преобразует события Matrix во внутренний протокол `core/` и маршрутизирует их на внешние контейнеры агентов (через `AgentApi` по WebSocket). +Telegram и Matrix боты для взаимодействия пользователя с AI-агентом Lambda. Каждый бот — тонкий адаптер поверх общего ядра (`core/`), изолирующего бизнес-логику от транспорта. Платформа подключается через `sdk/interface.py` Protocol; сейчас используется `MockPlatformClient`. ## Core Value -Пользователь может бесшовно взаимодействовать с изолированными AI-агентами через нативные интерфейсы мессенджеров (с поддержкой пересылки файлов и работы в комнатах), в то время как сама платформа агентов не зависит от транспорта. +Пользователь может вести диалог с Lambda-агентом через любой из поддерживаемых мессенджеров без изменения ядра системы. ## Requirements ### Validated -- ✓ `core/` — унифицированный протокол событий, EventDispatcher, StateStore, ChatManager. -- ✓ `adapter/matrix/` — Space+rooms адаптер. Прием инвайтов, автосоздание иерархии комнат, команды `!new`, `!archive`, `!clear`, `!yes`/`!no`. -- ✓ `sdk/real.py` — интеграция с AgentApi. Поддержка WebSocket для обмена сообщениями и передачи вложений в обе стороны. -- ✓ Shared Volume — прямая передача файлов в локальные рабочие папки агентов (`/agents/`). -- ✓ Dynamic Routing — маршрутизация чатов к агентам на основе `config/matrix-agents.yaml`. -- ✓ Deployment — Разделение окружений на `docker-compose.prod.yml` (только бот) и `docker-compose.fullstack.yml` (бот + локальный агент для E2E). +- ✓ core/ — унифицированный протокол событий, EventDispatcher, StateStore, ChatManager, AuthManager, SettingsManager — existing +- ✓ adapter/telegram/ — forum-first адаптер (Threaded Mode), `/start`, `/new`, `/archive`, `/rename`, `/settings`, стриминг ответов — existing, QA passed +- ✓ adapter/matrix/ — Space+rooms адаптер, invite flow, `!new`, `!archive`, `!rename`, `!settings`, room-per-chat — existing +- ✓ sdk/mock.py — MockPlatformClient: `stream_message`, `get_or_create_user`, `get_settings`, `update_settings` — existing -### Out of Scope / Deferred +### Active -- E2EE для Matrix (отложено из-за сложностей сборки `python-olm` на кросс-платформенных средах). -- Интеграция с Master-сервисом платформы (временно используется прямое соединение с `platform-agent` через AgentApi). -- Telegram-адаптер (вынесен в легаси ветку `feat/telegram-adapter`, MVP фокусируется на Matrix). +- [ ] Matrix QA — ручное тестирование Matrix адаптера, фиксация багов +- [ ] SDK integration — заменить MockPlatformClient реальным Lambda SDK (когда платформа готова) +- [ ] Production hardening — конфиг для деплоя, логирование, мониторинг + +### Out of Scope + +- E2EE для Matrix (python-olm не собирается на macOS/ARM) — инфраструктурная задача, отдельный трек +- Supergroup forum mode для Telegram — заменён Threaded Mode как основным режимом +- Telegram DM-first режим — заменён forum-first (Threaded Mode) ## Context -- Стек: Python 3.11+, `matrix-nio`, `uv`, `pydantic`. -- Бот хранит только локальную привязку (`room_id` <-> `platform_chat_id`) в SQLite. Вся долговременная память и история диалогов хранятся на стороне агента. -- Жизненный цикл контейнеров агентов управляется платформой, а не ботом. +- Python 3.11+, aiogram 3.4+, matrix-nio 0.21+, SQLite, pytest-asyncio +- Threaded Mode — Bot API 9.3, Mac клиент имеет известные баги (новые топики не сразу видны в сайдбаре) +- Lambda platform SDK ещё не готов, всё работает через MockPlatformClient +- Архитектура: Hexagonal / Ports-and-Adapters; `core/` не зависит от транспорта + +## Constraints + +- **Tech stack**: aiogram 3.x для Telegram, matrix-nio для Matrix — не менять без обсуждения +- **Platform**: SDK подключается только через `sdk/interface.py` Protocol — core/ и adapters не трогаются при смене реализации +- **Telegram**: Threaded Mode — единственный поддерживаемый режим; `closeForumTopic`/`deleteForumTopic` не работают в personal chat forums +- **E2EE**: python-olm не собирается на текущей среде — Matrix работает только без шифрования ## Key Decisions | Decision | Rationale | Outcome | |----------|-----------|---------| -| Space+rooms для Matrix | Room-based UX и явные чаты (по одному на тред) удобнее, чем DM-каша | ✓ Good | -| Прямая интеграция AgentApi | Master API не был готов, прямое WebSocket соединение позволяет передавать стейт и файлы | ✓ Good | -| Shared Volume для файлов | Избавляет от необходимости гонять base64 по сети, быстрый прямой доступ к файлам | ✓ Good | -| Stateless бот | Бот легко перезапускать и масштабировать, память изолирована в агентах | ✓ Good | +| Forum-first (Threaded Mode) для Telegram | Bot API 9.3 позволяет личный чат как форум — чище, без суперпруппы | ✓ Good | +| (user_id, thread_id) как PK в chats | Изоляция контекстов по топику | ✓ Good | +| MockPlatformClient через sdk/interface.py | Не ждать SDK, разрабатывать независимо | ✓ Good | +| Space+rooms для Matrix | Room-based UX и явные чаты важнее DM-first упрощений | ✓ Good | +| Отказ от E2EE в Matrix | python-olm не собирается на macOS/ARM | — Pending | ## Evolution @@ -49,5 +61,10 @@ Surfaces (поверхности) — это тонкие адаптеры-кл 3. New requirements emerged? → Add to Active 4. Decisions to log? → Add to Key Decisions +**After each milestone:** +1. Full review of all sections +2. Core Value check — still the right priority? +3. Update Context with current state + --- -*Last updated: 2026-05-03 after codebase consolidation* +*Last updated: 2026-04-02 after initialization* diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index ffd6801..4e8799b 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -1,32 +1,101 @@ # Roadmap — v1.0 -## Milestone: v1.0 — Production-ready Matrix MVP +## Milestone: v1.0 — Production-ready surfaces + +### Phase 1: Matrix QA & Polish + +**Goal:** Переработать Matrix адаптер с DM-first на Space+rooms, убрать реакции в пользу !yes/!no, довести до уровня "приемлемо работает" как Telegram. + +**Depends on:** Telegram QA complete + +**Plans:** 6 plans + +Plans: +- [x] 01-01-PLAN.md — Space+rooms infrastructure (store helpers, handle_invite rewrite, room_router) +- [x] 01-02-PLAN.md — Chat command handlers (!new, !archive, !rename) Space-aware +- [x] 01-03-PLAN.md — Reaction removal + !yes/!no confirmation + settings dashboard +- [x] 01-04-PLAN.md — Test suite (fix 4 broken + 12 new MAT-01..MAT-12) +- [x] 01-05-PLAN.md — Gap closure for Matrix `!yes` / `!no` pending-confirm scope +- [x] 01-06-PLAN.md — Remaining Phase 01 gap closure work (completed 2026-04-03) -### Phase 01: Matrix QA & Polish -**Goal:** Переработать Matrix адаптер с DM-first на Space+rooms, убрать реакции в пользу `!yes`/`!no`. -**Status:** Completed **Deliverables:** - Space+rooms architecture for Matrix adapter -- !yes/!no text-based confirmation -- Test suite green - -### Phase 04: Matrix MVP: Agent Integration -**Goal:** Подключить реального агента через `AgentApi`, добавить команды управления контекстом (`!clear`). -**Status:** Completed -**Deliverables:** -- `sdk/real.py` — реализация `PlatformClient` через реальный SDK (`AgentApi`). -- Поддержка WebSocket стриминга. -- Команды управления контекстом. -- Обертка в Docker. - -### Phase 05: MVP Deployment -**Goal:** Подготовить Matrix-бот к реальному деплою на lambda.coredump.ru с маршрутизацией по агентам и передачей файлов. -**Status:** Completed -**Deliverables:** -- Загрузка `matrix-agents.yaml` для маппинга пользователей к агентам. -- Per-room `platform_chat_id` routing. -- File transfer через shared `/agents/` volume. -- Разделение `docker-compose.prod.yml` и `docker-compose.fullstack.yml`. +- !yes/!no text-based confirmation (no reactions) +- Read-only !settings dashboard +- 96+ tests green --- -*Note: Легаси-фазы, связанные с Telegram, прототипами и Mock-платформой, были удалены из Roadmap после закрепления архитектуры MVP в ветке main.* + +### Phase 01.1: Matrix restart reconciliation and dev reset workflow (INSERTED) + +**Goal:** Сделать Matrix-адаптер пригодным для повторяемого локального рестарта и ручного QA: бот восстанавливает минимальный local state из существующих Space/rooms и даёт явный dev reset workflow вместо ручного ritual reset. +**Requirements**: none explicitly mapped +**Depends on:** Phase 1 +**Plans:** 3 plans + +Plans: +- [ ] 01.1-01-PLAN.md — Non-destructive Matrix reconciliation module and tests +- [ ] 01.1-02-PLAN.md — Wire startup/bootstrap recovery into the Matrix runtime +- [ ] 01.1-03-PLAN.md — Dev reset CLI and updated Matrix restart runbook + +### Phase 2: SDK Integration + +**Goal:** Заменить MockPlatformClient реальным Lambda SDK — бот начинает работать с настоящим AI-агентом. + +**Depends on:** Phase 1, Lambda platform SDK готов + +**Deliverables:** +- `sdk/real.py` — реализация PlatformClient через реальный SDK +- `bot.py` для обоих адаптеров переключается на реальный клиент через env var +- `stream_message` работает с реальным стримингом +- Интеграционные тесты с реальным SDK (или staging) + +### Phase 4: Matrix MVP: shared agent context and context management commands + +**Goal:** Привести Matrix-бот к рабочему состоянию для MVP-деплоя: заменить AgentSessionClient на AgentApi, добавить !save/!load/!reset/!context команды управления контекстом агента, упаковать в Docker. +**Requirements**: Replace AgentSessionClient with AgentApi; Wire AgentApi lifecycle; Implement !save, !load, !reset, !context commands; Dockerfile + docker-compose +**Depends on:** Phase 1 (Matrix adapter complete) +**Plans:** 3 plans + +Plans: +- [x] 04-01-PLAN.md — Replace AgentSessionClient with AgentApi; update sdk/real.py, bot.py, broken tests +- [x] 04-02-PLAN.md — !save, !load, !reset, !context handlers; PrototypeStateStore extensions; numeric interception +- [x] 04-03-PLAN.md — Dockerfile + docker-compose.yml + .env.example update + +--- + +### Phase 05: MVP Deployment + +**Goal:** Подготовить Matrix-бот к реальному деплою на lambda.coredump.ru без потери Space+rooms UX: закрепить per-room `platform_chat_id`, реальный `!clear`, reconciliation, file transfer через shared volume и разделение prod/fullstack compose. + +**Depends on:** Phase 4 + +**Plans:** 4/4 plans complete + +Plans: +- [x] 05-01-PLAN.md — Startup reconciliation from authoritative Matrix Space topology before live sync +- [x] 05-02-PLAN.md — Room-local `platform_chat_id` routing and real `!clear` semantics +- [x] 05-03-PLAN.md — Shared-volume attachment path hardening for `/agents` deployment +- [x] 05-04-PLAN.md — Split bot-only prod compose from internal fullstack compose and update docs + +**Deliverables:** +- Space+rooms onboarding remains the primary Matrix UX +- Per-room `platform_chat_id` provides true context isolation and `!clear` +- Reconciliation restores room metadata and routing after restart +- File transfer uses shared `/agents/` volume with room-safe behavior +- `docker-compose.prod.yml` is bot-only handoff; `docker-compose.fullstack.yml` is for internal E2E testing + +--- + +### Phase 3: Production Hardening + +**Goal:** Подготовить боты к реальному деплою — конфиг, логирование, мониторинг, обработка ошибок. + +**Depends on:** Phase 2 + +**Deliverables:** +- Docker / systemd конфиг для деплоя +- Структурированное логирование в production формате +- Health-check endpoint (если нужен) +- Rate limiting и защита от спама +- Graceful shutdown diff --git a/.planning/STATE.md b/.planning/STATE.md index 47a860b..eb05f42 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,12 +2,12 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: — Production-ready surfaces -status: MVP Deployed -last_updated: "2026-05-03T23:00:00Z" +status: Phase 05 Paused +last_updated: "2026-04-29T08:49:04Z" progress: - total_phases: 3 + total_phases: 6 completed_phases: 3 - total_plans: 13 + total_plans: 16 completed_plans: 13 --- @@ -15,35 +15,115 @@ progress: ## Project Reference -See: `.planning/PROJECT.md` (updated 2026-05-03) +See: .planning/PROJECT.md (updated 2026-04-02) -**Core value:** Пользователь бесшовно взаимодействует с изолированными агентами через нативные интерфейсы (Matrix), в то время как платформа агентов не зависит от транспорта. -**Current focus:** Итерационное развитие текущей архитектуры, добавление новых фич и поверхностей (по мере необходимости). +**Core value:** Пользователь ведёт диалог с Lambda через любой мессенджер без изменения ядра +**Current focus:** Phase 05 paused — latest file-contract change needs a new image build before platform redeploy ## Current Phase -Текущий MVP успешно завершен. Все базовые механизмы внедрены и работают: -- Маршрутизация к `AgentApi` -- Shared Volume файловый обмен (`/agents/`) -- Dynamic config через `matrix-agents.yaml` -- Изоляция контекстов через `platform_chat_id` +**Phase 05** paused: MVP deployment hardening is in place, but the latest attachment workspace-root change is not yet published -Проект находится в чистом состоянии для начала нового планирования. Неактуальные легаси фазы и прототипы (Telegram, MockPlatformClient) удалены из Roadmap и трекинга. +Deployment handoff follow-up is external. The last published image predates the latest file-handling change; the next step is to rebuild and publish a fresh image, then ask the platform to redeploy Matrix with the shared `/agents` volumes and `config/matrix-agents.yaml`. + +Plan `05-01` is complete. Matrix startup now reconciles managed Space rooms from synced topology before live traffic, restoring local metadata and deterministic legacy `platform_chat_id` bindings on restart. + +- `a75b26a` — failing restart reconciliation regressions for recovery, idempotence, startup ordering, and legacy backfill +- `8a80d00` — startup reconciliation module and pre-sync wiring in the Matrix runtime + +Verified with `MATRIX_AGENT_REGISTRY_PATH='' MATRIX_PLATFORM_BACKEND='' UV_CACHE_DIR=/tmp/uv-cache-surfaces uv run pytest tests/adapter/matrix/test_invite_space.py tests/adapter/matrix/test_chat_space.py tests/adapter/matrix/test_reconciliation.py tests/adapter/matrix/test_restart_persistence.py tests/adapter/matrix/test_dispatcher.py -v`. + +Plan `05-02` is complete. Matrix room-local context commands now rely on repaired per-room `platform_chat_id` bindings, and `!clear` rotates only the active room's upstream context when prototype room state is available. + +- `ae37476` — failing regressions for clear registration, room-local rotation, and strict routed-platform metadata requirements +- `85e2fda` — room-local clear semantics, compatibility alias wiring, and strict context resolution without shared chat fallbacks + +Verified with `MATRIX_AGENT_REGISTRY_PATH='' MATRIX_PLATFORM_BACKEND='' UV_CACHE_DIR=/tmp/uv-cache-surfaces uv run pytest tests/adapter/matrix/test_context_commands.py tests/adapter/matrix/test_routed_platform.py tests/adapter/matrix/test_dispatcher.py -v`. + +Plan `05-03` is complete. Shared-volume attachment handling now preserves relative agent paths while tolerating both `/workspace` and `/agents` absolute prefixes during normalization and Matrix file rendering. + +- `7a12a71` — failing regressions for shared-volume path normalization and room-safe attachment handling +- `5eddf16` — `/agents` deployment path hardening for Matrix files and routed platform attachments + +Verified with `uv run pytest tests/adapter/matrix/test_files.py tests/platform/test_real.py tests/adapter/matrix/test_send_outgoing.py -v`. + +Plan `05-04` is complete. Production handoff now uses `docker-compose.prod.yml` for a bot-only runtime, while internal end-to-end verification uses `docker-compose.fullstack.yml` with shared `/agents` volume guidance and health-gated startup. + +- `df6d8bf` — split prod and full-stack compose artifacts with the shared `/agents` contract +- `22a3a2b` — operator and deployment docs aligned to the split compose artifacts + +Verified with `docker compose -f docker-compose.prod.yml config`, `docker compose -f docker-compose.fullstack.yml config`, and docs grep checks for `docker-compose.prod.yml`, `docker-compose.fullstack.yml`, and `/agents`. ## Decisions -- **Space+rooms для Matrix**: Разделение тредов на отдельные Matrix-комнаты, собранные в едином Space пользователя. -- **AgentApi**: Прямая интеграция с локальным агентом без Master-прослойки по WebSocket. -- **Shared Volume**: Файлы кладутся напрямую в рабочую папку агента, избавляя от необходимости гонять их по сети в Base64. -- **Статическая маршрутизация**: На данном этапе пользователи маппятся на агентов жестко через YAML. +- Продолжаем с Threaded Mode несмотря на баги Mac клиента (2026-04-02) +- Invite flow Matrix переведён на idempotent-проверку через `user_meta.space_id`, а не через invite-room metadata (2026-04-02) +- Неизвестные Matrix rooms больше не auto-register в роутере; используется явный fallback `unregistered:{room_id}` с warning-логом (2026-04-02) +- [Phase 01]: Use ChatContext.surface_ref as the Matrix room identifier for !rename updates. +- [Phase 01]: Keep !archive limited to core archive state in Phase 1; Space child removal remains deferred. +- [Phase 01]: Matrix OutgoingUI no longer emits reactions; confirmation state is persisted and resumed via `!yes` / `!no`. +- [Phase 01]: `!settings` now renders a dashboard snapshot instead of advertising mutable subcommands. +- [Phase 01]: Split Matrix regression coverage into dedicated invite/chat/send_outgoing/confirm test modules. +- [Phase 01]: Kept 01-04 scoped to test coverage without widening into production-code changes. +- [Phase 01]: Matrix command callbacks now include room_id in payload for !yes and !no so confirm handlers can resolve runtime state without changing core protocol types. +- [Phase 01]: Pending confirmations are stored under the D-08 composite key of matrix user id plus room id, with a narrow legacy fallback only for callers that omit room context. +- [Phase 01]: Removed Matrix reaction conversion entirely and kept command callbacks limited to !yes/!no. +- [Phase 01]: Kept !settings as a pure snapshot surface while preserving mutable subcommands outside the dashboard. +- [Phase 01]: Seeded invite and dispatcher tests with explicit next_chat_index and room ids instead of treating C1 as Matrix transport identity. +- [Phase 04]: Replaced AgentSessionClient with AgentApiWrapper and persistent agent connection lifecycle in Matrix runtime. +- [Phase 04]: Added !save, !load, !reset, and !context commands with pending-state interception and local prototype session metadata. +- [Phase 04]: Added Matrix-only Docker packaging for MVP deployment; platform services remain external to this compose setup. +- [Phase 04]: Replaced the Matrix prod path again with direct upstream `AgentApi` per request; removed the local runtime wrapper from the prod flow. +- [Phase 04]: Adopted `AGENT_BASE_URL` as the primary runtime contract and kept `AGENT_WS_URL` only as backward-compatible env fallback. +- [Phase 04 follow-up]: Kept shared PlatformClient unchanged; introduced Matrix-specific RoutedPlatformClient to avoid breaking Telegram adapter. +- [Phase 04 follow-up]: agent_routing_enabled flag on MatrixRuntime activates stale-room check only in real multi-agent mode (RoutedPlatformClient). +- [Phase 04 follow-up]: !new binds agent_id at room creation time using selected_agent_id from user metadata. +- [Phase 04 follow-up]: platform_chat_seq (PLATFORM_CHAT_SEQ_KEY) is stored in SQLiteStore and survives restart — confirmed by test. +- [Phase 05 reset]: Discard the single-chat / DM-first deployment direction. Replan around Space+rooms, per-room `platform_chat_id`, real `!clear`, reconciliation, and split prod/fullstack compose artifacts. +- [Phase 05]: Keep adapter/matrix/files.py as the sole path builder; sdk/real.py only normalizes shared-volume attachment references. +- [Phase 05]: Normalize /workspace and /agents absolute file paths back to relative workspace_path values before agent transport and Matrix file rendering. +- [Phase 05]: Treat synced Matrix topology as authoritative for startup recovery; keep SQLite rebuildable. +- [Phase 05]: Backfill missing platform_chat_id values during startup reconciliation before routed handling begins. +- [Phase 05]: Expose `clear` only when prototype room-context support is available, while keeping `reset` as a compatibility alias. +- [Phase 05]: Require recovered `platform_chat_id` for save/context/clear flows instead of falling back to shared local chat ids. +- [Phase 05]: Split Compose artifacts by runtime intent: bot-only prod handoff vs internal full-stack verification. +- [Phase 05]: Document /agents as the bot-side shared volume root while internal platform-agent keeps /workspace on the same named volume. ## Blockers -- Отсутствуют. Проект готов к деплою (см. `docker-compose.prod.yml`). +- Lambda platform SDK не готов — Phase 2 заблокирована до готовности платформы +- Full production verification depends on the platform team's real multi-agent orchestration, production Matrix credentials, `config/matrix-agents.yaml`, and shared `/agents/N` volume mounts. ## Accumulated Context ### Roadmap Evolution -- Изначальный Roadmap включал множество ответвлений (прототипы Telegram, локальный mock-клиент). После закрепления MVP в `main` Roadmap был очищен, чтобы отражать только актуальный путь продукта. -- Следующие фазы будут добавляться по мере возникновения новых задач (например, переход от YAML-конфига к БД для реестра агентов). +- Phase 01.1 inserted after Phase 01: Matrix restart reconciliation and dev reset workflow (URGENT) +- Phase 4 added: Matrix MVP: shared agent context and context management command +- Phase 04 follow-up added inline: multi-agent routing (RoutedPlatformClient, !agent, stale room blocking, restart persistence) +- Phase 05 reset on 2026-04-28: erroneous single-chat deployment artifacts were removed before fresh planning. + +## Performance Metrics + +| Phase | Plan | Duration | Tasks | Files | Recorded | +| --- | --- | --- | --- | --- | --- | +| 01 | 01 | 1 min | 3 | 3 | 2026-04-02T19:50:50Z | +| 01 | 02 | 1 min | 2 | 2 | 2026-04-02 | +| 01 | 03 | 3 min | 2 | 5 | 2026-04-02T19:57:34Z | +| 01 | 04 | 3 min | 2 | 7 | 2026-04-02T20:03:38Z | +| 01 | 05 | 2 min | 2 | 7 | 2026-04-03T09:28:47Z | +| 01 | 06 | 4 min | 2 | 7 | 2026-04-03T09:35:47Z | +| 04 | 01 | 1 session | 1 wave | 8 | 2026-04-17 | +| 04 | 02 | 1 session | 2 commits + summary | 8 | 2026-04-17 | +| 04 | 03 | 1 session | 1 commit + summary | 4 | 2026-04-17 | +| 04 | follow-up | 1 session | 5 tasks | 10+ | 2026-04-24 | +| 05 | 03 | 3 min | 2 | 3 | 2026-04-27T22:06:43Z | +| 05 | 01 | 8 min | 2 | 4 | 2026-04-27T22:09:28Z | +| 05 | 02 | 16 min | 2 | 4 | 2026-04-27T22:15:58Z | +| 05 | 04 | 3 min | 2 | 5 | 2026-04-27T22:17:10Z | + +## Session + +- Last session: 2026-04-29T08:49:04Z +- Stopped at: Handoff updated after attachment workspace-root change; waiting for image rebuild and platform redeploy +- Resume file: .planning/phases/05-mvp-deployment/.continue-here.md diff --git a/.planning/codebase/ARCHITECTURE.md b/.planning/codebase/ARCHITECTURE.md index 05f7a7f..0cc6c4c 100644 --- a/.planning/codebase/ARCHITECTURE.md +++ b/.planning/codebase/ARCHITECTURE.md @@ -1,14 +1,134 @@ -# Архитектура (ARCHITECTURE.md) +# Architecture -## Паттерн "Thin Adapter" (Тонкая поверхность) +**Analysis Date:** 2026-04-01 -Система разделена на три логических слоя: -1. **Транспортный слой (Adapter)**: Подключается к внешней платформе (Matrix). Занимается конвертацией нативных событий (`room.message`) во внутренние структуры (`IncomingMessage`). -2. **Ядро (Core)**: Предоставляет единый протокол (`core/protocol.py`), не зависящий от конкретной реализации (Matrix, Telegram и т.д.). -3. **Платформенный слой (SDK)**: `RealPlatformClient` инкапсулирует подключение по WebSocket к реальным агентам (AgentApi). +## Pattern Overview -## Routing & Registry -Бот может обслуживать множество агентов (multi-tenant). Маршрутизация настраивается статически через `config/matrix-agents.yaml`. Каждый пользователь (`@user:server`) привязан к конкретному `agent_id`, у которого есть свой HTTP URL и свой изолированный `workspace_path` (например, `/agents/1/`). +**Overall:** Hexagonal / Ports-and-Adapters -## Файловый контракт -Файлы не передаются агенту в base64. Бот сохраняет вложение напрямую в локальную директорию (общий volume), и передает агенту только относительный путь (`workspace_path`). +**Key Characteristics:** +- A platform-neutral `core/` defines all business logic and unified event types +- Adapters (`adapter/telegram/`, `adapter/matrix/`) translate platform-specific events into core types and back +- The AI platform SDK is hidden behind a `PlatformClient` Protocol; the current implementation (`sdk/mock.py`) is swappable without touching core or adapters +- All state is stored through a `StateStore` Protocol, with `InMemoryStore` for tests and `SQLiteStore` for production + +## Layers + +**Protocol Layer:** +- Purpose: Defines every data structure crossing layer boundaries +- Location: `core/protocol.py` +- Contains: `IncomingMessage`, `IncomingCommand`, `IncomingCallback`, `OutgoingMessage`, `OutgoingUI`, `OutgoingNotification`, `OutgoingTyping`, `ChatContext`, `AuthFlow`, `SettingsAction`, type aliases `IncomingEvent` and `OutgoingEvent` +- Depends on: Python stdlib only +- Used by: All other layers + +**Core / Business Logic Layer:** +- Purpose: Handles all domain logic independent of any platform +- Location: `core/` +- Contains: + - `core/handler.py` — `EventDispatcher`: routes `IncomingEvent` to registered handler functions; returns `list[OutgoingEvent]` + - `core/handlers/` — one module per event category (`start`, `message`, `chat`, `settings`, `callback`) + - `core/store.py` — `StateStore` Protocol + `InMemoryStore` + `SQLiteStore` + - `core/chat.py` — `ChatManager`: creates/renames/archives chat workspaces (C1/C2/C3); persists via `StateStore` + - `core/auth.py` — `AuthManager`: tracks auth flow state (`pending` → `confirmed`); persists via `StateStore` + - `core/settings.py` — `SettingsManager`: fetches/caches user settings from SDK; invalidates on write +- Depends on: `core/protocol.py`, `sdk/interface.py` (Protocol only), `core/store.py` +- Used by: Adapters + +**SDK / Platform Layer:** +- Purpose: Wraps the external Lambda AI platform; isolated behind a Protocol +- Location: `sdk/` +- Contains: + - `sdk/interface.py` — `PlatformClient` Protocol: `get_or_create_user`, `send_message`, `stream_message`, `get_settings`, `update_settings`; also `WebhookReceiver` Protocol, Pydantic models (`User`, `MessageResponse`, `MessageChunk`, `UserSettings`, `AgentEvent`) + - `sdk/mock.py` — `MockPlatformClient`: full in-memory implementation with simulated latency; supports both sync (`send_message`) and streaming (`stream_message`, currently returns single chunk); includes webhook simulation via `simulate_agent_event()` +- Depends on: `sdk/interface.py` +- Used by: `core/` managers, adapters during bot startup + +**Adapter Layer:** +- Purpose: Translates platform-native events into `IncomingEvent` and `OutgoingEvent` back to platform-native calls +- Location: `adapter/matrix/`, adapter/telegram/ (in `.worktrees/telegram/`) +- Contains per adapter: `bot.py` (entry point + send logic), `converter.py` (native event → protocol), `handlers/` (adapter-specific handler overrides registered on top of core handlers), optional `store.py` / `room_router.py` / `reactions.py` for adapter state +- Depends on: `core/`, `sdk/`, platform SDK (aiogram or matrix-nio) +- Used by: `__main__` / `asyncio.run(main())` + +## Data Flow + +**Incoming Message (Matrix example):** + +1. `matrix-nio` fires `RoomMessageText` callback → `MatrixBot.on_room_message()` in `adapter/matrix/bot.py` +2. `resolve_chat_id()` in `adapter/matrix/room_router.py` maps `room_id` → logical `chat_id` (e.g. `C1`), persisted in `StateStore` +3. `from_room_event()` in `adapter/matrix/converter.py` converts the nio event to `IncomingMessage` or `IncomingCommand` +4. `EventDispatcher.dispatch(incoming)` in `core/handler.py` selects the handler by routing key (command name, callback action, or `"*"` for messages) +5. Handler (e.g. `core/handlers/message.py:handle_message`) calls `platform.send_message()` on `MockPlatformClient`, receives `MessageResponse` +6. Handler returns `list[OutgoingEvent]` (e.g. `[OutgoingTyping(..., False), OutgoingMessage(...)]`) +7. `MatrixBot._send_all()` iterates the list; `send_outgoing()` converts each to a `client.room_send()` / `client.room_typing()` call + +**Incoming Reaction (Matrix):** + +1. `ReactionEvent` callback → `MatrixBot.on_reaction()` +2. `from_reaction()` maps emoji key to `IncomingCallback` with `action="confirm"`, `"cancel"`, or `"toggle_skill"` +3. Dispatch → `core/handlers/callback.py` + +**Command Routing:** + +The `EventDispatcher` uses a routing key per event type: +- `IncomingCommand` → `event.command` (e.g. `"start"`, `"new"`, `"settings"`) +- `IncomingCallback` → `event.action` (e.g. `"confirm"`, `"toggle_skill"`) +- `IncomingMessage` → `"*"` (catch-all), or `event.attachments[0].type` if attachments present + +Adapters call `register_all(dispatcher)` first (core handlers), then `register_matrix_handlers(dispatcher, ...)` to override or add platform-specific variants (e.g. `!new` creates a real Matrix room via the nio client). + +**State Management:** +- All persistent state goes through `StateStore` (key-value, async interface) +- Key namespaces: `chat:{user_id}:{chat_id}`, `auth:{user_id}`, `settings:{user_id}`, `matrix_room:{room_id}`, `matrix_user:{matrix_user_id}`, `matrix_state:{room_id}`, `matrix_skills_msg:{room_id}` +- Production uses `SQLiteStore` (row-per-key, JSON-serialised values); tests use `InMemoryStore` + +## Key Abstractions + +**EventDispatcher (`core/handler.py`):** +- Purpose: Single dispatch table for all event types; decouples handler logic from transport +- Pattern: Registry (map of `event_type → {key → HandlerFn}`); wildcard `"*"` as fallback +- Handler signature: `async def handler(event, chat_mgr, auth_mgr, settings_mgr, platform) → list[OutgoingEvent]` + +**StateStore Protocol (`core/store.py`):** +- Purpose: Pluggable persistence behind a minimal `get/set/delete/keys` interface +- Implementations: `InMemoryStore` (tests/dev), `SQLiteStore` (production) +- Key pattern: `"{namespace}:{discriminator}"` + +**PlatformClient Protocol (`sdk/interface.py`):** +- Purpose: Contracts the entire surface of the Lambda AI SDK +- Current implementation: `MockPlatformClient` in `sdk/mock.py` +- Swap path: Replace `sdk/mock.py` with a real SDK client; no changes needed elsewhere + +**Converter functions (`adapter/matrix/converter.py`):** +- Purpose: One-way transformation from platform-native event to `IncomingEvent` +- Always produce canonical protocol types; adapters never pass raw library objects to core + +## Entry Points + +**Matrix Bot:** +- Location: `adapter/matrix/bot.py:main()` +- Run: `python -m adapter.matrix.bot` +- Startup sequence: load `.env` → build `AsyncClient` → `build_runtime()` → register callbacks → `client.sync_forever()` + +**Telegram Bot:** +- Location: `.worktrees/telegram/adapter/telegram/bot.py` (feature branch, not merged to main yet) +- Run: `python -m adapter.telegram.bot` + +## Error Handling + +**Strategy:** Errors propagate up to the adapter's event callback. The adapter logs and drops the event; the bot keeps running. + +**Patterns:** +- `EventDispatcher.dispatch()` returns `[]` (empty list) when no handler is found and logs a warning +- `AuthManager` and `ChatManager` raise `ValueError` for not-found entities; callers are responsible for catching +- `MockPlatformClient` raises `PlatformError` (defined in `sdk/interface.py`) on unexpected states + +## Cross-Cutting Concerns + +**Logging:** `structlog` throughout; all managers and the dispatcher use `structlog.get_logger(__name__)` +**Validation:** Pydantic models in `sdk/interface.py` for SDK responses; plain dataclasses in `core/protocol.py` for internal events +**Authentication:** `AuthManager.is_authenticated()` is checked in `handle_message` before forwarding to platform; unauthenticated users receive a prompt to run `!start` / `/start` + +--- + +*Architecture analysis: 2026-04-01* diff --git a/.planning/codebase/CONCERNS.md b/.planning/codebase/CONCERNS.md index 5848135..473d257 100644 --- a/.planning/codebase/CONCERNS.md +++ b/.planning/codebase/CONCERNS.md @@ -1,6 +1,235 @@ -# Известные проблемы (CONCERNS.md) +# Codebase Concerns -- **Отсутствие E2E шифрования в Matrix**: На данный момент бот не поддерживает зашифрованные комнаты, так как библиотека `matrix-nio` требует нативной сборки `python-olm`, что усложняет кросс-платформенный деплой. -- **Потеря стейта агентов**: Так как текущий `platform-agent` часто работает с `MemorySaver`, его стейт теряется при перезапусках. Это проблема внешнего агента, но она напрямую влияет на UX поверхности. -- **Общий том (Shared Volume)**: Контракт обязывает бота и агента запускаться на одном физическом хосте (или иметь распределенный сетевой диск), что может стать бутылочным горлышком при сильном масштабировании. -- **Hardcoded роутинг**: `matrix-agents.yaml` требует ручного редактирования и перезапуска бота при добавлении новых агентов. Желательно вынести этот процесс в динамическую БД или API. +**Analysis Date:** 2026-04-01 + +--- + +## Tech Debt + +### Telegram adapter not merged to main + +- Issue: The entire `adapter/telegram/` directory exists only in the `feat/telegram-adapter` branch (worktree at `.worktrees/telegram/`). `main` has no Telegram adapter at all. +- Files: `.worktrees/telegram/adapter/telegram/` and remote branch `origin/feat/telegram-adapter` +- Impact: Running `python -m adapter.telegram.bot` from `main` fails with ImportError. Tests referencing `adapter/telegram/` (e.g., `tests/adapter/test_forum_db.py`) only exist in the worktree and are absent from `main`. +- Fix approach: Merge `feat/telegram-adapter` into `main` after final manual QA pass. The branch is ahead of main by 5 commits (`a1b7a14` being the most recent). + +### Divergent core/handlers between main and feat/telegram-adapter + +- Issue: `feat/telegram-adapter` removed platform-awareness from `core/handlers/chat.py` and `core/handlers/message.py` — the `_command()` and `_start_command()` helpers that format Matrix `!cmd` vs Telegram `/cmd` prompts were deleted. The branch hardcodes `/start` everywhere. +- Files: `core/handlers/chat.py`, `core/handlers/message.py` (differ between branches) +- Impact: If the Matrix adapter relies on these platform-aware helpers being in `main`'s version of core, merging `feat/telegram-adapter` will break Matrix `!start` prompt text for unauthenticated users. +- Fix approach: Before merging, decide which version of `core/handlers/` is canonical. The Matrix adapter in `main` currently passes because `main` still has the platform-aware helpers. + +### SQLiteStore uses blocking I/O in async context + +- Issue: `core/store.py` `SQLiteStore` methods are declared `async` but perform synchronous blocking `sqlite3.connect()` calls without `asyncio.to_thread` or `aiosqlite`. +- Files: `core/store.py` lines 46–73 +- Impact: Each database call blocks the asyncio event loop. Under any concurrent load (e.g., two Matrix users sending messages simultaneously) this will cause visible latency spikes and potential event loop starvation. +- Fix approach: Replace `sqlite3` calls with `aiosqlite` or wrap each call in `asyncio.to_thread()`. + +### Telegram adapter has its own separate SQLite database layer + +- Issue: `adapter/telegram/db.py` is a fully independent SQLite database (file: `lambda_bot.db`) with its own schema (`tg_users`, `chats`). Meanwhile, `core/store.py` has `SQLiteStore` with a KV schema (`lambda_matrix.db` for Matrix). The two stores are incompatible and do not share data. +- Files: `.worktrees/telegram/adapter/telegram/db.py`, `core/store.py` +- Impact: There is no unified storage layer. Chat state is split across two databases. A user's Telegram chats cannot be seen from Matrix and vice versa (even conceptually). Violates the "single core" architecture principle from CLAUDE.md. +- Fix approach: This is a fundamental design gap. Either extend `StateStore` to support the Telegram-specific data model, or accept separate stores as intentional for the prototype stage and document the constraint. + +### MockPlatformClient hardcoded throughout — no production path wired + +- Issue: Both `adapter/matrix/bot.py` and `.worktrees/telegram/adapter/telegram/bot.py` instantiate `MockPlatformClient()` directly. `PLATFORM_MODE` is defined in `.env.example` but is never read or acted upon anywhere in the codebase. +- Files: `adapter/matrix/bot.py` line 71, `sdk/mock.py`, `sdk/interface.py` +- Impact: There is no runtime switch to connect a real SDK. Switching to production requires code changes, not configuration. +- Fix approach: Add a factory function in `sdk/` that reads `PLATFORM_MODE` and returns either `MockPlatformClient` or a real `PlatformClient`. Both bot entrypoints should use this factory. + +### MatrixRuntime type annotation leaks MockPlatformClient + +- Issue: `adapter/matrix/bot.py` `MatrixRuntime.platform` is typed as `MockPlatformClient` (not `PlatformClient`). `build_event_dispatcher` and `build_runtime` signatures also use `MockPlatformClient` as the parameter type. +- Files: `adapter/matrix/bot.py` lines 46, 54, 67 +- Impact: The isolation promise ("replace only `sdk/mock.py` when real SDK arrives") is broken — the bot layer is coupled to the mock concrete type, not the Protocol. +- Fix approach: Change type annotations to `PlatformClient` from `sdk.interface`. + +--- + +## Known Bugs / Open Issues + +### Telegram forum: global commands visible inside topic context + +- Issue: Telegram shows the full bot command menu (including `/chats`, `/new`, `/settings`) even when the user is inside a forum topic. The code blocks `switch` and `new_chat` callbacks inside topics but the commands themselves still appear in the UI. +- Files: `.worktrees/telegram/adapter/telegram/handlers/forum.py`, `.worktrees/telegram/adapter/telegram/bot.py` +- Impact: Users can tap `/settings` or `/chats` inside a topic and get confusing behavior. +- Tracked: Issue `#15` — `Telegram forum topics: remaining UX and synchronization gaps` + +### Telegram forum: `/new ` inside linked topic does not rename the Telegram topic + +- Issue: Running `/new ` inside a forum topic that is already linked to a chat renames the internal chat record but does not call `edit_forum_topic` to rename the actual Telegram topic. +- Files: `.worktrees/telegram/adapter/telegram/handlers/forum.py` +- Impact: Topic name in Telegram goes out of sync with internal chat name. +- Tracked: Issue `#15` + +### Matrix: `handle_invite` hardcodes `chat_id = "C1"` for all new rooms + +- Issue: `adapter/matrix/handlers/auth.py` `handle_invite()` always assigns `chat_id = "C1"` regardless of how many rooms the user already has. If a user invites the bot into a second room before using `!new`, both rooms get `C1`. +- Files: `adapter/matrix/handlers/auth.py` line 26 +- Impact: Two rooms mapped to the same `chat_id` causes routing collisions. +- Fix approach: Call `next_chat_id(store, user_id)` here instead of hardcoding `"C1"`. + +### Matrix: `remove_reaction` uses non-standard `undo` field + +- Issue: `adapter/matrix/reactions.py` `remove_reaction()` sends a `"undo": True` field in the reaction event body. This is not part of the Matrix spec for reaction redaction. The correct approach is to redact the original reaction event via `client.room_redact()`. +- Files: `adapter/matrix/reactions.py` lines 56–68 +- Impact: Reaction "undo" will silently fail on compliant homeservers. + +### Matrix: E2EE not supported (blocked by `python-olm`) + +- Issue: `matrix-nio` E2EE requires `python-olm`, which fails to build on macOS/ARM. No encrypted DM support. +- Files: `adapter/matrix/bot.py` +- Impact: The bot cannot operate in encrypted rooms. Users who have DM encryption enforced cannot use the Matrix bot. +- Status: Documented as a known infrastructure constraint in `docs/reports/2026-04-01-surfaces-progress-report.md`. Needs a separate infrastructure task. + +--- + +## Security Considerations + +### SQLite database files not in .gitignore + +- Risk: `lambda_bot.db` and `lambda_matrix.db` are present in the working tree (shown in `git status`) but not listed in `.gitignore`. These files may contain user data including chat content and display names. +- Files: `lambda_bot.db`, `lambda_matrix.db`, `.gitignore` +- Current mitigation: Files are currently untracked (not yet staged) but nothing prevents them from being accidentally committed. +- Recommendation: Add `*.db` or specific filenames to `.gitignore` immediately. + +### Auth flow is auto-confirmed in mock — no real validation exists + +- Issue: `core/auth.py` `confirm()` automatically sets `state = "confirmed"` and generates a fake `platform_user_id`. There is no real verification step, no code exchange, no token validation. +- Files: `core/auth.py` lines 39–48 +- Impact: The auth layer is decorative for the prototype. Any user who sends `!start` or `/start` is immediately authenticated. If the real SDK auth requires a different flow (e.g., OAuth, code), the current `AuthManager` interface may not match. +- Current mitigation: Acceptable for mock stage. Must be re-evaluated before production use. + +### Matrix room metadata stored without access control + +- Issue: `adapter/matrix/store.py` stores room metadata keyed by `room_id`. Any call that can supply an arbitrary `room_id` can read or overwrite another user's room metadata. +- Files: `adapter/matrix/store.py`, `adapter/matrix/room_router.py` +- Impact: In the current single-process bot this is not exploitable. If the store is ever shared across processes or users, room metadata can be poisoned. + +--- + +## Fragile Areas + +### `core/chat.py` scan-by-suffix fallback is O(N) and collision-prone + +- Issue: `ChatManager.get()` when called without `user_id` scans all `chat:*` keys and matches by suffix (e.g., `":C1"`). If two users both have a chat named `C1` (which is always the case), this returns the first one found, non-deterministically. +- Files: `core/chat.py` lines 76–82 +- Impact: Functions like `rename` and `archive` that call `chat_mgr.get(chat_id)` without `user_id` will operate on the wrong user's chat in a multi-user scenario. +- Fix approach: Audit all callers and always pass `user_id`. The scan-by-suffix fallback should be removed or explicitly guarded. + +### `adapter/matrix/handlers/chat.py` chat_id counter races under concurrency + +- Issue: `make_handle_new_chat` calls `chat_mgr.list_active()` and uses `len(chats) + 1` to compute a new `chat_id`. This is not atomic. Two concurrent `!new` commands from the same user can produce the same `chat_id`. +- Files: `adapter/matrix/handlers/chat.py` line 17 +- Impact: Duplicate `chat_id` values (`C2`, `C2`) for the same user, leading to state corruption. +- Fix approach: Use `next_chat_id()` from `adapter/matrix/store.py` which increments an atomic counter in the store. The `next_chat_id()` function already exists but is not used here. + +### `conftest.py` contains a fragile stdlib `platform` module workaround + +- Issue: `conftest.py` patches `sys.modules` to remove the Python stdlib `platform` module so local `platform/` (which no longer exists — renamed to `sdk/`) doesn't shadow it. The comment still refers to `platform/` but the directory was renamed to `sdk/` in commit `41660fe`. +- Files: `conftest.py` lines 1–13 +- Impact: The workaround is now a no-op (there is no `platform/` package to shadow) but adds confusion. The comment is incorrect. If someone creates a `platform/` directory again, unexpected behavior can return. +- Fix approach: Remove the `sys.modules` patching entirely since `sdk/` does not conflict with stdlib. Update the comment. + +### Forum onboarding `chat_shared` constructs a fake `Chat` object + +- Issue: `adapter/telegram/handlers/forum.py` handles `chat_shared` by constructing `Chat(id=..., type="supergroup", is_forum=True)` and passing it to `_complete_group_link()`. The `is_forum=True` is hardcoded — the real value from Telegram is not verified. This means the check `if getattr(forwarded_chat, "is_forum", None) is False` in the forwarding fallback path is bypassed entirely. +- Files: `.worktrees/telegram/adapter/telegram/handlers/forum.py` lines 162–168 +- Impact: A user could link a regular supergroup (without Topics enabled) via `chat_shared`, which would succeed in linking but fail when the bot tries to create forum topics. + +--- + +## Gaps Between CLAUDE.md and Actual Code + +### CLAUDE.md says `platform/` — code uses `sdk/` + +- CLAUDE.md architecture diagram shows `platform/interface.py` and `platform/mock.py` +- Actual code uses `sdk/interface.py` and `sdk/mock.py` (renamed in commit `41660fe`) +- Files: `CLAUDE.md` (project instructions), `sdk/interface.py`, `sdk/mock.py` +- Also: Agent config files at `.claude/agents/core-developer.md` still reference `platform/` throughout +- Impact: New contributors reading CLAUDE.md will look for a `platform/` directory that does not exist. + +### CLAUDE.md lists `core/handlers/` sub-handlers that partially do not exist + +- CLAUDE.md lists handler modules but the actual `core/handlers/` only has: `start.py`, `message.py`, `chat.py`, `settings.py`, `callback.py` +- No `voice.py` handler exists; voice is handled as a fallback inside `core/handlers/message.py` (returns stub response) +- No `payment.py` handler exists; `PaymentRequired` dataclass is defined in `core/protocol.py` but never dispatched +- Files: `core/protocol.py` (PaymentRequired defined), `core/handlers/` (no payment or voice handlers) + +### CLAUDE.md workflow describes `@reviewer` agent but agent file references old patterns + +- `.claude/agents/core-developer.md` still says "Твоя зона — `core/` и `platform/`" +- The old Haiku/Sonnet researcher-developer workflow is captured in `docs/workflow-backup-2026-04-01.md`, but `.claude/agents/` configs were not updated to match + +### `tests/adapter/test_forum_db.py` is untracked on main + +- This test file exists in the working tree (visible in `git status`) but is not committed to `main`. It tests `adapter/telegram/db.py` which also does not exist on `main`. +- Files: `tests/adapter/test_forum_db.py` +- Impact: Running `pytest tests/` from main currently includes this test, which imports `adapter.telegram.db`. This import succeeds only because the test auto-reloads the module from an untracked file. This is fragile — if the file is deleted, tests silently pass with fewer tests counted. + +--- + +## Missing Critical Features + +### No streaming response support in adapters + +- Both adapters use `platform.send_message()` (sync) not `platform.stream_message()` (streaming) +- `sdk/interface.py` defines `stream_message` returning `AsyncIterator[MessageChunk]` +- No adapter sends a typing indicator before the response arrives and then streams chunks +- Impact: User experience with slow AI responses will show nothing until the full response is ready +- Files: `core/handlers/message.py` line 28, `sdk/interface.py` lines 83–88 + +### No webhook/push notification handling + +- `sdk/interface.py` defines `WebhookReceiver` Protocol with `on_agent_event()` +- `sdk/mock.py` has `register_webhook_receiver()` and `simulate_agent_event()` +- Neither bot entrypoint registers a `WebhookReceiver` +- Impact: Push notifications from the platform (task completions, background jobs) cannot reach the user +- Files: `sdk/interface.py` lines 95–97, `adapter/matrix/bot.py`, no registration present + +### Telegram adapter uses InMemoryStore for core state + +- `.worktrees/telegram/adapter/telegram/bot.py` calls `InMemoryStore()` for the `EventDispatcher`'s state +- All `core/` state (auth, chat metadata in the KV layer) is lost on bot restart +- `adapter/telegram/db.py` SQLite is used only for Telegram-specific data +- Impact: On restart, authenticated users are logged out; core chat context is wiped +- Files: `.worktrees/telegram/adapter/telegram/bot.py` line 46 + +### No multi-user isolation in Matrix store + +- `adapter/matrix/store.py` keys are global (`matrix_room:ROOMID`, `matrix_user:USERID`) +- There is no namespace or tenant isolation +- Impact: At scale, any key collision would corrupt state. For a single-user prototype this is acceptable, but it is an architectural constraint to document before expanding scope. + +--- + +## Test Coverage Gaps + +### No tests for `adapter/telegram/` in main test suite + +- `tests/adapter/` on main only contains `matrix/` tests and the untracked `test_forum_db.py` +- All Telegram adapter tests live in the worktree at `.worktrees/telegram/tests/` +- Files: `tests/adapter/` (missing `telegram/` subdirectory on main) +- Risk: Merging `feat/telegram-adapter` without also merging its tests leaves Telegram untested on main +- Priority: High + +### No tests for `core/handlers/callback.py` confirm/cancel real behavior + +- `core/handlers/callback.py` `handle_confirm` and `handle_cancel` return stub text with `action_id` +- No test verifies that a real confirmation flow (dispatch → confirm → side effect) works end to end +- Files: `core/handlers/callback.py`, `tests/core/test_dispatcher.py` +- Priority: Medium + +### No tests for `adapter/matrix/handlers/auth.py` multi-room invite scenario + +- The hardcoded `C1` bug (see Known Bugs section) is not caught by any test +- Files: `adapter/matrix/handlers/auth.py`, `tests/adapter/matrix/test_dispatcher.py` +- Priority: Medium + +--- + +*Concerns audit: 2026-04-01* diff --git a/.planning/codebase/CONVENTIONS.md b/.planning/codebase/CONVENTIONS.md index 36a4ed5..04c7f6a 100644 --- a/.planning/codebase/CONVENTIONS.md +++ b/.planning/codebase/CONVENTIONS.md @@ -1,7 +1,195 @@ -# Конвенции (CONVENTIONS.md) +# Coding Conventions -- **Асинхронность**: Весь код бота асинхронный (`asyncio`). Вызовы SDK и Matrix-клиента выполняются через `await`. Блокирующие вызовы (если они есть) должны выноситься в тредпул. -- **Обработка ошибок**: Бот не должен падать из-за ошибок отдельного агента. Ошибки SDK (например, `PlatformError`) отлавливаются в боте и возвращаются пользователю в виде системных сообщений или уведомлений. -- **Стейтлесс-подход**: Поверхность хранит минимальный стейт (только локальный SQLite для связки `room_id` <-> `platform_chat_id`). Вся история сообщений и память лежат на стороне агентов. -- **Переменные окружения**: Бот полностью конфигурируется через `.env` (префиксы `MATRIX_` и `SURFACES_`). -- **Добавление новой поверхности**: Новая поверхность должна быть самостоятельной папкой в `adapter/`, реализовывать `converter.py`, и переиспользовать `sdk/real.py` и `core/protocol.py`. +**Analysis Date:** 2026-04-01 + +## Linting and Formatting + +**Tool:** ruff (configured in `pyproject.toml`) + +**Settings:** +- Line length: 100 characters +- Target: Python 3.11 +- Active rule sets: `E` (pycodestyle errors), `F` (pyflakes), `I` (isort), `UP` (pyupgrade), `B` (bugbear) + +**Type checking:** mypy (available as dev dependency; not enforced in CI at this time) + +Run linting: +```bash +ruff check . +ruff format . +``` + +## File Naming + +- Module files: `snake_case.py` (e.g., `room_router.py`, `test_dispatcher.py`) +- Each module starts with a comment declaring its path: `# core/handler.py` +- Test files: `test_.py` (e.g., `test_store.py`, `test_converter.py`) +- No index/barrel files except `__init__.py` for package registration + +## Class Naming + +- `PascalCase` for all classes (e.g., `EventDispatcher`, `MockPlatformClient`, `MatrixBot`) +- Protocol/interface classes named after the capability: `StateStore`, `PlatformClient`, `WebhookReceiver` +- Manager classes suffixed with `Manager`: `ChatManager`, `AuthManager`, `SettingsManager` +- Dataclasses follow the same `PascalCase` rule: `IncomingMessage`, `OutgoingUI`, `MatrixRuntime` + +## Function and Method Naming + +- `snake_case` for all functions and methods +- Private helpers prefixed with single underscore: `_to_dict`, `_from_dict`, `_routing_key`, `_latency` +- Handler functions named `handle_`: `handle_start`, `handle_message`, `handle_new_chat` +- Builder functions named `build_`: `build_runtime`, `build_event_dispatcher`, `build_skills_text` +- Converter functions named `from_`: `from_room_event`, `from_command`, `from_reaction` +- Predicate functions named `is_`: `is_authenticated`, `is_new` + +## Variable Naming + +- `snake_case` for all variables and parameters +- Internal state attributes prefixed with `_`: `self._store`, `self._platform`, `self._handlers` +- Store key prefixes are module-level constants in `UPPER_SNAKE_CASE`: + ```python + ROOM_META_PREFIX = "matrix_room:" + USER_META_PREFIX = "matrix_user:" + ``` +- Constants for reaction strings are module-level: `CONFIRM_REACTION = "👍"`, `PLATFORM = "matrix"` + +## Type Annotations + +All files use `from __future__ import annotations` at the top for deferred evaluation. + +**Annotation style:** +- Use built-in generics (`list[str]`, `dict[str, Any]`) — not `List`, `Dict` from `typing` +- Union types written with `|`: `str | None`, `IncomingCallback | None` +- Type aliases at module level: `IncomingEvent = IncomingMessage | IncomingCommand | IncomingCallback` +- Callable types use `typing.Callable` and `typing.Awaitable`: + ```python + HandlerFn = Callable[..., Awaitable[list[OutgoingEvent]]] + ``` +- Handler functions use loose `list` return type without generics (consistent across `core/handlers/`) +- Protocol classes use `...` as body for abstract methods: + ```python + async def get(self, key: str) -> dict | None: ... + ``` + +**Pydantic vs dataclasses:** +- `core/protocol.py` — plain `@dataclass` with `field(default_factory=...)` for mutable defaults +- `sdk/interface.py` — Pydantic `BaseModel` for all SDK-facing models (`User`, `MessageResponse`, `UserSettings`) +- Choose `@dataclass` for internal protocol structs, `BaseModel` for SDK boundary models + +## Import Organization + +Order (enforced by ruff `I` rules): +1. `from __future__ import annotations` +2. Standard library imports (grouped) +3. Third-party imports (grouped) +4. Local imports from project packages (grouped) + +Example from `adapter/matrix/bot.py`: +```python +from __future__ import annotations + +import asyncio +import os +from dataclasses import dataclass +from pathlib import Path + +import structlog +from nio import AsyncClient, ... +from dotenv import load_dotenv + +from adapter.matrix.converter import from_reaction, from_room_event +from core.auth import AuthManager +from core.protocol import OutgoingEvent, ... +from sdk.mock import MockPlatformClient +``` + +No relative imports; all imports use absolute package paths from the project root. + +## Async Patterns + +All I/O methods are `async def`. There are no sync wrappers around async code. + +**Handler signature pattern** (used uniformly across `core/handlers/`): +```python +async def handle_(event: IncomingEvent, auth_mgr, platform, chat_mgr, settings_mgr) -> list: +``` +Note: manager parameters are untyped in handler signatures (accepted as `**kwargs` at call site in `EventDispatcher.dispatch`). + +**Awaiting store calls:** +```python +stored = await self._store.get(f"auth:{user_id}") +await self._store.set(f"auth:{user_id}", _to_dict(flow)) +``` + +**SQLiteStore uses sync sqlite3** inside `async def` methods — blocking I/O is not off-loaded to a thread executor. This is a known limitation (see CONCERNS.md). + +**Mock latency simulation:** +```python +await self._latency(200, 600) # min_ms, max_ms +``` + +## Logging + +**Library:** `structlog` + +**Pattern:** +```python +import structlog +logger = structlog.get_logger(__name__) + +logger.info("Chat created", chat_id=chat_id, user_id=user_id) +logger.warning("No handler registered", event_type=event_type.__name__, key=key) +``` + +- Always pass structured keyword arguments — never use f-strings in log calls +- Logger created at module level with `structlog.get_logger(__name__)` + +## Error Handling + +- Raise `ValueError` for invalid domain state (e.g., chat not found in `ChatManager.rename`) +- `sdk/interface.py` defines `PlatformError(Exception)` with a `code` field for SDK-level errors +- Handler functions never raise — they return `[]` or a fallback `OutgoingMessage` +- No `try/except` blocks in core handlers; errors from the platform are expected to propagate + +## Comments + +- Module-level comment declaring file path at top: `# core/handler.py` +- Docstrings for classes with non-obvious behavior: + ```python + class MockPlatformClient: + """ + Заглушка SDK платформы Lambda. + ... + """ + ``` +- Inline comments for non-obvious blocks: + ```python + # Scan by chat_id suffix when user_id unknown (slower) + ``` +- Comments in Russian are normal and acceptable throughout the codebase + +## Serialization Pattern + +Dataclasses are serialized/deserialized via private module-level functions, not class methods: + +```python +def _to_dict(ctx: ChatContext) -> dict: + return { "chat_id": ctx.chat_id, ... } + +def _from_dict(d: dict) -> ChatContext: + return ChatContext(chat_id=d["chat_id"], ...) +``` + +This pattern is used in `core/auth.py` and `core/chat.py`. Follow this pattern for any new manager that persists to `StateStore`. + +## Module Design + +- No barrel `__init__.py` exports except `core/handlers/__init__.py` which exposes `register_all` +- Manager classes take `(platform, store)` as constructor args; `platform` is often stored as `object` or not stored at all if unused +- `@dataclass` is preferred for plain data containers, not NamedTuple or TypedDict +- Store key namespacing follows `::` pattern: + `"chat:u1:C1"`, `"auth:u1"`, `"matrix_room:!r:m.org"` + +--- + +*Convention analysis: 2026-04-01* diff --git a/.planning/codebase/INTEGRATIONS.md b/.planning/codebase/INTEGRATIONS.md index cd771d1..3cdae98 100644 --- a/.planning/codebase/INTEGRATIONS.md +++ b/.planning/codebase/INTEGRATIONS.md @@ -1,15 +1,173 @@ -# Интеграции (INTEGRATIONS.md) +# External Integrations -## Platform Agent API -- **Тип**: WebSocket (через `AgentApi` SDK) -- **Назначение**: Связь между Matrix-адаптером и внешней LLM-платформой. -- **Контракт**: Surface выступает "тупым" клиентом. Он отправляет `platform_chat_id` и `user_id` вместе с сообщениями. Платформа/Агент отвечает текстом и вложениями. Контейнерами агентов бот не управляет. +**Analysis Date:** 2026-04-01 -## Matrix Homeserver -- **Тип**: HTTP/HTTPS API (via `matrix-nio`) -- **Назначение**: Пользовательский интерфейс и транспорт сообщений для бота. -- **Ограничения**: Поддерживается только нешифрованное (unencrypted) взаимодействие. +## Bot Platform APIs -## Файловая система (Shared Volume) -- **Тип**: Docker Shared Volume (`/agents/`) -- **Назначение**: Прямая передача файлов между поверхностью и агентами в обход сети. Поверхность пишет файлы в поддиректорию конкретного агента, агент их читает, и наоборот. +**Telegram Bot API:** +- Purpose: Primary messaging surface for user ↔ Lambda agent interaction +- Client library: `aiogram` 3.26.0 (async, wraps Telegram Bot API v7+) +- Authentication: Bot token via `TELEGRAM_BOT_TOKEN` +- Entry point: `adapter/telegram/bot.py` (planned; aiogram worktree branch `feat/telegram-adapter`) +- Transport: Long-polling or webhook (aiogram supports both; mode not yet locked in) +- Bot API docs: https://core.telegram.org/bots/api + +**Matrix Client-Server API:** +- Purpose: Secondary messaging surface (Matrix/Element clients) +- Client library: `matrix-nio` 0.25.2 (async) +- Authentication: password login or pre-existing access token (`MATRIX_ACCESS_TOKEN`) +- Login flow in `adapter/matrix/bot.py` `main()`: + - If `MATRIX_ACCESS_TOKEN` is set → assigned directly to `client.access_token` + - Else if `MATRIX_PASSWORD` is set → `client.login(password=..., device_name="surfaces-bot")` +- Sync method: `client.sync_forever(timeout=30000)` (30-second long-poll) +- E2EE store: nio file-based store at path from `MATRIX_STORE_PATH` (default: `"matrix_store"`) +- Matrix C-S API docs: https://spec.matrix.org/latest/client-server-api/ + +### Matrix Room Model + +Rooms are mapped to Lambda chat slots (C1, C2, C3…) via `adapter/matrix/room_router.py`: +- First message in a room → assigns next chat ID (C1, C2, …) and persists mapping to store +- Room metadata stored under key `matrix_room:` in `StateStore` +- User metadata (next chat index) stored under `matrix_user:` + +### Matrix Event Types Handled + +| nio Event Class | Handler | Action | +|--------------------|-----------------------------|-------------------------------| +| `RoomMessageText` | `MatrixBot.on_room_message` | Dispatch to `EventDispatcher` | +| `ReactionEvent` | `MatrixBot.on_reaction` | Button confirmation / skill toggle | +| `InviteMemberEvent`| `MatrixBot.on_member` | Accept room invite | +| `RoomMemberEvent` | `MatrixBot.on_member` | Membership change handling | + +## Lambda Platform (Internal SDK) + +**Purpose:** AI agent backend — processes user messages, manages user accounts, returns responses + +**Interface:** `sdk/interface.py` — `PlatformClient` Protocol + +**Current Implementation:** `sdk/mock.py` — `MockPlatformClient` +- Simulates network latency (10–80 ms default, 200–600 ms for message calls) +- In-process in-memory state (users, messages, settings dicts) +- Supports webhook simulation via `simulate_agent_event()` + +**Production Integration (future):** +- URL: `LAMBDA_PLATFORM_URL` (default: `http://localhost:8000`) +- Auth: `LAMBDA_SERVICE_TOKEN` (bearer token) +- Mode switch: `PLATFORM_MODE=mock` vs `PLATFORM_MODE=production` +- Swap path: replace `sdk/mock.py` only; no changes to `core/` or `adapter/` + +**Platform API Methods (from `sdk/interface.py`):** + +```python +async def get_or_create_user(external_id, platform, display_name) -> User +async def send_message(user_id, chat_id, text, attachments) -> MessageResponse +async def stream_message(user_id, chat_id, text, attachments) -> AsyncIterator[MessageChunk] +async def get_settings(user_id) -> UserSettings +async def update_settings(user_id, action) -> None +``` + +**Webhook / Push (outbound from platform → bot):** +- Interface: `WebhookReceiver` Protocol (`sdk/interface.py`) +- Registration: `MockPlatformClient.register_webhook_receiver(receiver)` +- Event types: `task_done`, `task_error`, `task_progress` (modelled in `AgentEvent`) +- Production implementation not yet wired; mock supports `simulate_agent_event()` for testing + +## Data Storage + +**Databases:** + +*SQLite (primary persistence):* +- Client: stdlib `sqlite3` (synchronous, called from async code without `asyncio.to_thread`) +- Schema: single key-value table: `kv (key TEXT PRIMARY KEY, value TEXT NOT NULL)` +- JSON serialization for values (`json.dumps` / `json.loads`) +- Matrix bot DB path: `MATRIX_DB_PATH` (default: `"lambda_matrix.db"`) +- Telegram bot DB path: implicit `"lambda_bot.db"` (file present in repo root — development artifact) +- Implementation: `core/store.py` `SQLiteStore` + +*In-Memory (testing / development):* +- `InMemoryStore` — plain Python dict, no persistence across restarts +- `MockPlatformClient` internal state — also in-memory dicts + +**File Storage:** +- Matrix nio E2EE store: local filesystem directory at `MATRIX_STORE_PATH` (default: `"matrix_store/"`) +- No object storage (S3/GCS/etc.) currently; mock client has `attachment_mode` flag (`"url"` | `"binary"` | `"s3"`) reserved for future real SDK + +**Caching:** +- None — no Redis or external cache layer + +## Authentication & Identity + +**Telegram Auth:** +- Bot token → passed to aiogram dispatcher at startup +- User identity: Telegram user ID mapped to platform `external_id` + +**Matrix Auth:** +- Password or access token (see above) +- User identity: Matrix user ID (e.g. `@user:matrix.org`) mapped to platform `external_id` + +**Lambda Platform User Identity:** +- `get_or_create_user(external_id, platform)` → returns `User` with internal `user_id` +- External IDs are platform-prefixed in mock: `"{platform}:{external_id}"` + +## Monitoring & Observability + +**Logging:** +- `structlog` 25.5.0 — structured logging (key=value pairs) +- Logger instantiation: `structlog.get_logger(__name__)` in each module +- Log calls use keyword arguments: `logger.info("event_name", key=value, ...)` +- No log shipping / aggregation configured (local stdout only) + +**Error Tracking:** +- None — no Sentry, Datadog, or similar integration + +**Metrics:** +- None — `MockPlatformClient.get_stats()` returns basic in-memory counters (not exported) + +## CI/CD & Deployment + +**Hosting:** +- Not specified — no Dockerfile, docker-compose, or cloud config files present + +**CI Pipeline:** +- None detected — no `.github/workflows/`, `.gitlab-ci.yml`, etc. + +## Environment Configuration + +**Required variables (from `.env.example`):** + +| Variable | Required | Default | Purpose | +|-----------------------|----------|--------------------|--------------------------------------| +| `TELEGRAM_BOT_TOKEN` | Yes* | — | Telegram Bot API token | +| `MATRIX_HOMESERVER` | Yes* | — | Matrix homeserver URL (e.g. `https://matrix.org`) | +| `MATRIX_USER_ID` | Yes* | — | Bot's Matrix user ID | +| `MATRIX_PASSWORD` | Cond. | — | Login password (if no access token) | +| `MATRIX_ACCESS_TOKEN` | Cond. | — | Pre-issued access token (preferred) | +| `MATRIX_DEVICE_ID` | No | `""` | Matrix device ID | +| `MATRIX_DB_PATH` | No | `"lambda_matrix.db"` | SQLite DB file path (Matrix bot) | +| `MATRIX_STORE_PATH` | No | `"matrix_store"` | nio E2EE store directory | +| `LAMBDA_PLATFORM_URL` | No** | `http://localhost:8000` | Lambda platform base URL | +| `LAMBDA_SERVICE_TOKEN`| No** | — | Service auth token for Lambda API | +| `PLATFORM_MODE` | No | `"mock"` | `"mock"` or `"production"` | + +\* Required for the respective bot to function. +\*\* Only required when `PLATFORM_MODE=production`. + +**Secrets location:** +- `.env` file (gitignored) +- Never committed — `.env.example` provides template +- Loaded via `python-dotenv` at module import in each `bot.py` entry point + +## Webhooks & Callbacks + +**Incoming (platform → bot):** +- `WebhookReceiver.on_agent_event(event: AgentEvent)` — receives async task completion notifications +- Not yet wired to an HTTP endpoint; `MockPlatformClient.simulate_agent_event()` used for testing + +**Outgoing (bot → external):** +- Telegram: all via `aiogram` polling or webhook (no direct outbound HTTP beyond Telegram API) +- Matrix: all via `matrix-nio` `AsyncClient.room_send()`, `room_typing()`, etc. +- Platform: via `PlatformClient` send/stream methods + +--- + +*Integration audit: 2026-04-01* diff --git a/.planning/codebase/STACK.md b/.planning/codebase/STACK.md index b40772d..708a4bf 100644 --- a/.planning/codebase/STACK.md +++ b/.planning/codebase/STACK.md @@ -1,14 +1,113 @@ -# Технологический стек (STACK.md) +# Technology Stack -## Язык и Runtime -- **Python**: 3.11-slim (используется в Docker-образах) -- **Пакетный менеджер**: `uv` (используется для быстрой и строгой установки зависимостей, frozen lockfiles). +**Analysis Date:** 2026-04-01 -## Ключевые библиотеки -- **matrix-nio**: Асинхронный клиент для Matrix (события, синхронизация, отправка). -- **pydantic**: Для валидации структур данных (события из AgentApi). -- **structlog**: Структурированное логирование (json/console). +## Languages -## Инфраструктура -- **Docker / Docker Compose**: Используется для локального (fullstack) и продакшн развертывания. -- **SQLite**: Легковесная локальная база данных для хранения маппингов пользователей/комнат (`adapter/matrix/store.py`). +**Primary:** +- Python 3.11+ — all application code (enforced via `pyproject.toml` `requires-python = ">=3.11"`) + +**Type Annotations:** +- Full `from __future__ import annotations` usage throughout +- `typing.Protocol` used for dependency inversion (`core/store.py`, `sdk/interface.py`) + +## Runtime + +**Environment:** +- CPython — runtime (development host currently runs 3.14.3) +- Minimum: Python 3.11 (uses `match`-compatible union syntax, `Self`, `X | Y` type hints) + +**Package Manager:** +- `uv` 0.9.30 (Homebrew) +- Lockfile: `uv.lock` present and committed +- Install: `uv sync` + +## Frameworks + +**Telegram Bot:** +- `aiogram` 3.26.0 — async Telegram Bot API framework + - Used in `adapter/telegram/` (planned; directory not yet present in main branch) + - Brings in `aiohttp` 3.13.3 as its HTTP transport + +**Matrix Bot:** +- `matrix-nio` 0.25.2 — async Matrix Client-Server API client + - Used in `adapter/matrix/bot.py` + - Key classes: `AsyncClient`, `AsyncClientConfig`, `RoomMessageText`, `ReactionEvent`, `InviteMemberEvent`, `RoomMemberEvent`, `MatrixRoom` + - Long-polling via `client.sync_forever(timeout=30000)` + +**Data Validation:** +- `pydantic` 2.12.5 — data models in `sdk/interface.py` + - `User`, `Attachment`, `MessageResponse`, `MessageChunk`, `UserSettings`, `AgentEvent`, `PlatformError` + - Core protocol structs (`core/protocol.py`) use plain `dataclasses` instead + +**Build/Dev:** +- `setuptools` ≥68 + `setuptools-scm` + `wheel` — build backend (`pyproject.toml`) +- `ruff` 0.15.8 — linting and import sorting (`line-length = 100`, `target-version = "py311"`, rules: E, F, I, UP, B) +- `mypy` 1.19.1 — static type checking + +## Key Dependencies + +**Critical:** +- `aiogram>=3.4,<4` (resolved: 3.26.0) — Telegram adapter; pin avoids breaking v4 API +- `matrix-nio>=0.21` (resolved: 0.25.2) — Matrix adapter; async-only client +- `pydantic>=2.5` (resolved: 2.12.5) — SDK interface models; v2 required (v1 incompatible) + +**Infrastructure:** +- `structlog` 25.5.0 — structured logging throughout; used via `structlog.get_logger(__name__)` +- `python-dotenv` 1.2.2 — loads `.env` at bot startup (`load_dotenv(Path(...) / ".env")`) +- `httpx` 0.28.1 — available for HTTP calls (future SDK integration, not yet used in core logic) + +**Async I/O:** +- `aiohttp` 3.13.3 — transitive via aiogram; provides HTTP session to Telegram Bot API +- `asyncio` — stdlib; used directly in `sdk/mock.py` (`asyncio.sleep` for latency simulation) and all bot entry points (`asyncio.run(main())`) + +## Testing + +**Runner:** +- `pytest` 9.0.2 +- `pytest-asyncio` 1.3.0 — `asyncio_mode = "auto"` (set in `pyproject.toml`) +- `pytest-cov` 7.1.0 — coverage reporting + +**Configuration:** +- `pyproject.toml` `[tool.pytest.ini_options]`: `testpaths = ["tests"]`, `pythonpath = ["."]` +- `conftest.py` at project root + +## Internal Module Structure + +**Core (no external deps except stdlib + pydantic via sdk):** +- `core/protocol.py` — `dataclasses`-based unified event types +- `core/store.py` — `StateStore` Protocol + `InMemoryStore` (dict) + `SQLiteStore` (stdlib `sqlite3`) +- `core/handler.py` — `EventDispatcher` +- `core/auth.py`, `core/chat.py`, `core/settings.py` — domain managers + +**SDK Layer:** +- `sdk/interface.py` — `PlatformClient` Protocol (pydantic models) +- `sdk/mock.py` — `MockPlatformClient` in-process stub; simulates latency via `asyncio.sleep` + +**Adapters:** +- `adapter/matrix/` — matrix-nio integration (active) +- `adapter/telegram/` — aiogram integration (referenced in deps, worktree branch exists) + +## Configuration + +**Environment:** +- Loaded from `.env` via `python-dotenv` at startup +- See `INTEGRATIONS.md` for full variable list + +**Build:** +- `pyproject.toml` — single source of truth for deps, build, lint, test config + +## Platform Requirements + +**Development:** +- Python ≥3.11 +- `uv` for dependency management + +**Production:** +- Any environment with Python ≥3.11 +- Matrix bot: requires writable filesystem path for `matrix_store/` (nio E2EE store) and SQLite DB +- Telegram bot: stateless beyond env vars (or optionally SQLite for persistence) + +--- + +*Stack analysis: 2026-04-01* diff --git a/.planning/codebase/STRUCTURE.md b/.planning/codebase/STRUCTURE.md index 9ea8a18..08896a5 100644 --- a/.planning/codebase/STRUCTURE.md +++ b/.planning/codebase/STRUCTURE.md @@ -1,18 +1,210 @@ -# Структура (STRUCTURE.md) +# Codebase Structure -- `core/`: - - `protocol.py` — Унифицированные структуры данных (сообщения, файлы, UI). -- `adapter/matrix/`: - - `bot.py` — Главный event-loop Matrix. - - `converter.py` — Конвертация событий Matrix-nio ⇄ `core/protocol.py`. - - `agent_registry.py` — Парсинг `matrix-agents.yaml`. - - `files.py` — Работа с вложениями и shared volume. - - `store.py` — SQLite база для маппинга чатов Matrix и `platform_chat_id`. - - `routed_platform.py` — Динамический роутинг вызовов к нужным агентам на лету. -- `sdk/`: - - `interface.py` — Интерфейс PlatformClient. - - `real.py` — Имплементация WebSocket клиента (`AgentApi`). - - `mock.py` — Мок-клиент для E2E тестов без платформы. -- `config/`: Конфиги маршрутизации (YAML). -- `docs/`: Актуальная документация по развертыванию и архитектуре. -- `docker-compose*.yml`: Продакшн и локальные манифесты для сборки. +**Analysis Date:** 2026-04-01 + +## Directory Layout + +``` +surfaces-bot/ +├── adapter/ +│ ├── __init__.py +│ └── matrix/ # matrix-nio adapter (merged to main) +│ ├── __init__.py +│ ├── bot.py # Entry point, MatrixBot class, send_outgoing() +│ ├── converter.py # nio Event → IncomingEvent +│ ├── reactions.py # Emoji constants, skills text builder +│ ├── room_router.py # room_id → chat_id resolution +│ ├── store.py # Matrix-specific StateStore helpers (room meta, user meta) +│ └── handlers/ +│ ├── __init__.py # register_matrix_handlers() +│ ├── auth.py # handle_invite (invite member event) +│ ├── chat.py # Chat creation (creates real Matrix rooms) +│ ├── confirm.py # Confirmation flow callbacks +│ └── settings.py # Settings sub-commands and toggle_skill +├── core/ +│ ├── auth.py # AuthManager: start_flow, confirm, is_authenticated +│ ├── chat.py # ChatManager: get_or_create, list_active, rename, archive +│ ├── handler.py # EventDispatcher: register, dispatch, _routing_key +│ ├── protocol.py # All shared dataclasses and type aliases +│ ├── settings.py # SettingsManager: get (cached), apply (invalidates cache) +│ ├── store.py # StateStore Protocol, InMemoryStore, SQLiteStore +│ └── handlers/ +│ ├── __init__.py # register_all() — binds all core handlers to dispatcher +│ ├── callback.py # handle_confirm, handle_cancel, handle_toggle_skill +│ ├── chat.py # handle_new_chat, handle_rename, handle_archive, handle_list_chats +│ ├── message.py # handle_message — auth guard + platform.send_message +│ ├── settings.py # handle_settings — displays settings menu +│ └── start.py # handle_start — get_or_create_user + welcome message +├── sdk/ +│ ├── __init__.py +│ ├── interface.py # PlatformClient Protocol, WebhookReceiver Protocol, Pydantic models +│ └── mock.py # MockPlatformClient — full in-memory implementation +├── tests/ +│ ├── __init__.py +│ ├── conftest.py # (root conftest — sys.path fix for local sdk/ shadowing stdlib) +│ ├── adapter/ +│ │ ├── __init__.py +│ │ ├── matrix/ +│ │ │ ├── __init__.py +│ │ │ ├── test_converter.py +│ │ │ ├── test_dispatcher.py +│ │ │ ├── test_reactions.py +│ │ │ └── test_store.py +│ │ └── test_forum_db.py # untracked — forum DB exploration +│ ├── core/ +│ │ ├── test_auth.py +│ │ ├── test_chat.py +│ │ ├── test_dispatcher.py +│ │ ├── test_integration.py +│ │ ├── test_protocol.py +│ │ ├── test_settings.py +│ │ ├── test_store.py +│ │ └── test_voice_slot.py +│ └── platform/ +│ └── test_mock.py +├── docs/ # All human documentation +├── .planning/ # GSD planning artefacts +│ └── codebase/ # Codebase map documents (this directory) +├── .claude/ +│ └── agents/ # Agent configuration files +├── .worktrees/ +│ └── telegram/ # Telegram adapter on feat/telegram-adapter branch +│ └── ... # Mirrors main layout; merged separately +├── conftest.py # Root pytest conftest: sys.path hack for local sdk/ +├── pyproject.toml # Project metadata, dependencies, ruff + pytest config +├── uv.lock # Lockfile (uv) +├── lambda_matrix.db # SQLite DB written by Matrix bot (gitignored) +└── .env.example # Environment variable template +``` + +## Directory Purposes + +**`core/`:** +- Purpose: Platform-neutral business logic. Never imports from `adapter/`. +- Key files: `protocol.py` (all shared types), `handler.py` (dispatcher), `store.py` (persistence interface) +- Add new domain logic here; keep it free of aiogram/matrix-nio imports + +**`core/handlers/`:** +- Purpose: One async function per command/callback/message type. Each returns `list[OutgoingEvent]`. +- Registration: `register_all()` in `core/handlers/__init__.py` binds them to the dispatcher +- Adapters can override any key by calling `dispatcher.register(event_type, key, fn)` after `register_all()` + +**`sdk/`:** +- Purpose: Contract (`interface.py`) and mock (`mock.py`) for the Lambda AI platform SDK +- Note: The directory is named `sdk/` in actual code (not `platform/` as CLAUDE.md describes); `handler.py` imports from `sdk.interface` +- When real SDK arrives: replace `sdk/mock.py` only; `sdk/interface.py` must not change unless the contract changes + +**`adapter/matrix/`:** +- Purpose: Everything matrix-nio-specific. Translates between nio and core protocol. +- `bot.py` owns `MatrixBot`, `build_runtime()`, `send_outgoing()`, and `main()` +- `store.py` provides key-namespaced helpers on top of `StateStore` (not a separate store implementation) +- `room_router.py` maintains the `room_id → chat_id` mapping persisted in `StateStore` + +**`adapter/telegram/`:** +- Purpose: aiogram 3.x adapter. Lives in `.worktrees/telegram/` on `feat/telegram-adapter` branch. +- Uses aiogram FSM states (`states.py`) and inline keyboards (`keyboards/`) +- Not yet merged to `main` + +**`tests/`:** +- Purpose: pytest test suite mirroring the source tree +- `tests/core/` — unit tests for each core module +- `tests/adapter/matrix/` — Matrix adapter tests (converter, dispatcher, reactions, store) +- `tests/platform/` — MockPlatformClient tests + +**`docs/`:** +- Purpose: Human-readable design documents; not consumed by code +- Key docs: `docs/surface-protocol.md` (unification rationale), `docs/api-contract.md` (SDK contract), `docs/telegram-prototype.md`, `docs/matrix-prototype.md` + +## Key File Locations + +**Entry Points:** +- `adapter/matrix/bot.py` — Matrix bot `main()`, run via `python -m adapter.matrix.bot` +- `.worktrees/telegram/adapter/telegram/bot.py` — Telegram bot entry (feature branch) + +**Shared Protocol:** +- `core/protocol.py` — single source of truth for all inter-layer data types + +**SDK Contract:** +- `sdk/interface.py` — `PlatformClient` Protocol; defines the API surface for the real SDK +- `sdk/mock.py` — `MockPlatformClient`; current runtime implementation + +**Dispatcher Registration:** +- `core/handlers/__init__.py` — `register_all()` for platform-agnostic handlers +- `adapter/matrix/handlers/__init__.py` — `register_matrix_handlers()` for Matrix overrides + +**Persistence:** +- `core/store.py` — `StateStore` Protocol, `InMemoryStore`, `SQLiteStore` +- `adapter/matrix/store.py` — Matrix-specific store helper functions (not a store implementation) + +**Configuration:** +- `pyproject.toml` — dependencies, pytest config (`asyncio_mode = "auto"`, `pythonpath = ["."]`), ruff config +- `conftest.py` — `sys.path` insert so local `sdk/` shadows stdlib `platform` module + +## Naming Conventions + +**Files:** +- Modules: `snake_case.py` +- Entry points: `bot.py` per adapter +- Converter: `converter.py` per adapter +- Handlers directory: `handlers/` per layer + +**Classes:** +- Managers: `{Domain}Manager` (e.g. `ChatManager`, `AuthManager`, `SettingsManager`) +- Bot runtime: `{Platform}Bot` (e.g. `MatrixBot`) +- Protocol types: PascalCase dataclasses (e.g. `IncomingMessage`, `OutgoingUI`) +- SDK types: PascalCase Pydantic models (e.g. `MessageResponse`, `UserSettings`) + +**Handler functions:** +- `handle_{command}` for command handlers (e.g. `handle_start`, `handle_new_chat`) +- `make_handle_{command}` for factory functions that close over adapter state (e.g. `make_handle_new_chat(client, store)`) + +**State keys:** +- `"{namespace}:{discriminator}"` — always use the prefix constants defined in `adapter/matrix/store.py` + +## Where to Add New Code + +**New core command handler:** +1. Add `async def handle_{cmd}(event, chat_mgr, auth_mgr, settings_mgr, platform) -> list` in `core/handlers/{category}.py` +2. Register it in `core/handlers/__init__.py:register_all()` with `dispatcher.register(IncomingCommand, "{cmd}", handle_{cmd})` +3. Write tests in `tests/core/test_dispatcher.py` or a dedicated `tests/core/test_{category}.py` + +**New Matrix-specific handler (needs nio client or matrix store):** +1. Add handler in `adapter/matrix/handlers/{category}.py` +2. Register in `adapter/matrix/handlers/__init__.py:register_matrix_handlers()` — this overrides the core handler for that key + +**New protocol type:** +- Add dataclass to `core/protocol.py`; update `IncomingEvent` or `OutgoingEvent` union aliases if it crosses layer boundaries +- Update `EventDispatcher._routing_key()` if it requires a new dispatch strategy + +**New StateStore key namespace:** +- Add prefix constant and helper functions in `adapter/matrix/store.py` (for Matrix-specific state) or directly in the relevant manager (for core state) + +**New test:** +- Unit tests for core logic: `tests/core/test_{module}.py` +- Adapter tests: `tests/adapter/matrix/test_{module}.py` +- Use `InMemoryStore` as the store; use `MockPlatformClient` as the platform client + +## Special Directories + +**`.worktrees/telegram/`:** +- Purpose: Git worktree for `feat/telegram-adapter` branch; full copy of the repo root +- Generated: Yes (via `git worktree add`) +- Committed: No (worktrees are local) + +**`.planning/`:** +- Purpose: GSD planning artefacts — phase plans and codebase maps +- Generated: Yes (by `/gsd:` commands) +- Committed: Yes (tracked with the repo) + +**`.claude/agents/`:** +- Purpose: Agent role configuration files for the multi-agent workflow +- Committed: Yes + +**`src/`:** +- Purpose: Contains only `surfaces_bot.egg-info/` (setuptools build artefact); no source code +- Generated: Yes +- Committed: No + +--- + +*Structure analysis: 2026-04-01* diff --git a/.planning/codebase/TESTING.md b/.planning/codebase/TESTING.md index 07311dc..f685abc 100644 --- a/.planning/codebase/TESTING.md +++ b/.planning/codebase/TESTING.md @@ -1,17 +1,210 @@ -# Тестирование (TESTING.md) +# Testing Patterns -## Unit-тесты -Расположены в `tests/`. Покрытие сфокусировано на логике Matrix адаптера (пока он является основной поверхностью): -- Файловый контракт (`test_files.py`) -- Диспетчер и конвертация (`test_dispatcher.py`) -- Взаимодействие с PlatformClient (`test_routed_platform.py`) -- Работа с контекстными командами бота (`test_context_commands.py`) +**Analysis Date:** 2026-04-01 -## E2E тестирование -Локально тестируется через запуск контейнеров из `docker-compose.fullstack.yml`, который поднимает один инстанс бота и один локальный `platform-agent`. Это позволяет имитировать полную цепочку взаимодействия (Matrix -> Бот -> Агент) с общим каталогом для файлов. +## Test Framework -## Запуск тестов -```bash -# Запуск юнит-тестов (только для Matrix адаптера) -pytest tests/adapter/matrix/ -v +**Runner:** pytest 8.x +**Config:** `pyproject.toml` `[tool.pytest.ini_options]` + +```toml +[tool.pytest.ini_options] +asyncio_mode = "auto" +testpaths = ["tests"] +pythonpath = ["."] ``` + +**Async support:** pytest-asyncio with `asyncio_mode = "auto"` — all `async def` test functions run automatically without decorators. + +**Coverage:** pytest-cov (available but no minimum threshold configured) + +**Run commands:** +```bash +pytest tests/ -v # all tests +pytest tests/core/ -v # core layer only +pytest tests/adapter/telegram/ -v # telegram adapter only +pytest tests/adapter/matrix/ -v # matrix adapter only +pytest tests/ --cov=. --cov-report=term # with coverage report +``` + +## Test Directory Structure + +``` +tests/ +├── __init__.py +├── core/ +│ ├── test_auth.py — AuthManager unit tests +│ ├── test_chat.py — ChatManager unit tests +│ ├── test_dispatcher.py — EventDispatcher routing tests +│ ├── test_integration.py — full flow smoke tests (dispatcher + managers + mock) +│ ├── test_protocol.py — dataclass defaults and construction +│ ├── test_settings.py — SettingsManager unit tests +│ ├── test_store.py — InMemoryStore + SQLiteStore tests +│ └── test_voice_slot.py — handle_message() handler unit tests +├── adapter/ +│ ├── __init__.py +│ ├── test_forum_db.py — Telegram SQLite DB helpers (untracked, new) +│ └── matrix/ +│ ├── __init__.py +│ ├── test_converter.py — matrix-nio event → IncomingEvent converter +│ ├── test_dispatcher.py — full Matrix bot integration (build_runtime) +│ ├── test_reactions.py — reaction text builders and emoji mapping +│ └── test_store.py — Matrix store helper functions +└── platform/ + └── test_mock.py — MockPlatformClient behavior +``` + +Tests mirror the source tree. New tests for `adapter/telegram/` go in `tests/adapter/telegram/` (directory exists in `.worktrees/telegram` branch, not yet merged to main). + +## conftest.py + +`conftest.py` at project root (`/Users/a/MAI/sem2/lambda/surfaces-bot/conftest.py`) handles a sys.path conflict: the project has a local `platform/` (now `sdk/`) package that shadows Python's stdlib `platform` module. It inserts the project root at `sys.path[0]` and removes the cached stdlib `platform` module. + +No shared fixtures are defined in `conftest.py`. All fixtures are local to test files. + +## Test Structure + +**Fixture pattern — local to each test file:** +```python +@pytest.fixture +def mgr(): + return AuthManager(MockPlatformClient(), InMemoryStore()) + +@pytest.fixture +def store() -> InMemoryStore: + return InMemoryStore() +``` + +**Async tests require no decorator** (asyncio_mode = "auto"): +```python +async def test_not_authenticated_initially(mgr): + assert await mgr.is_authenticated("u1") is False +``` + +**Sync tests** are used for pure-function tests (protocol dataclass construction, reaction text builders): +```python +def test_incoming_message_defaults(): + msg = IncomingMessage(user_id="u1", platform="telegram", chat_id="C1", text="hi") + assert msg.attachments == [] +``` + +**Integration fixture pattern** — builds full runtime in-process: +```python +@pytest.fixture +def dispatcher(): + platform = MockPlatformClient() + store = InMemoryStore() + d = EventDispatcher( + platform=platform, + chat_mgr=ChatManager(platform, store), + auth_mgr=AuthManager(platform, store), + settings_mgr=SettingsManager(platform, store), + ) + register_all(d) + return d +``` + +## Mocking Strategy + +**Primary mock: `MockPlatformClient`** from `sdk/mock.py` + +All tests use `MockPlatformClient()` directly — it is the real mock for the SDK layer. No unittest.mock patching of `MockPlatformClient` is needed. + +**`unittest.mock.AsyncMock`** is used only when testing integration with external clients (matrix-nio `AsyncClient`): +```python +from unittest.mock import AsyncMock + +client = SimpleNamespace( + room_create=AsyncMock(return_value=SimpleNamespace(room_id="!r2:example")) +) +``` + +**`types.SimpleNamespace`** is used to fabricate matrix-nio event objects without importing the full nio library: +```python +def text_event(body: str, sender: str = "@a:m.org", event_id: str = "$e1"): + return SimpleNamespace( + sender=sender, body=body, event_id=event_id, msgtype="m.text", replyto_event_id=None + ) +``` +This is the pattern for all matrix converter tests — define factory functions at module level that return `SimpleNamespace` objects. + +**`tmp_path` pytest fixture** is used for SQLiteStore tests to get a throwaway database file: +```python +async def test_sqlite_set_and_get(tmp_path): + store = SQLiteStore(str(tmp_path / "test.db")) +``` + +**`monkeypatch.setenv`** is used in `tests/adapter/test_forum_db.py` to inject `DB_PATH` env var and reload the module with a fresh database: +```python +@pytest.fixture(autouse=True) +def fresh_db(tmp_path, monkeypatch): + db_file = str(tmp_path / "test.db") + monkeypatch.setenv("DB_PATH", db_file) + import importlib + import adapter.telegram.db as db_mod + importlib.reload(db_mod) + db_mod.init_db() + return db_mod +``` + +**What NOT to mock:** +- `InMemoryStore` — use it directly; it's a real in-memory implementation +- `MockPlatformClient` — use it directly; patching it defeats the purpose +- Core manager classes (`AuthManager`, `ChatManager`, `SettingsManager`) — always instantiate real ones + +## Test Data Patterns + +**User IDs:** short strings like `"u1"`, `"u2"`, `"tg_123"`, `"@alice:m.org"` + +**Chat IDs:** `"C1"`, `"C2"`, `"C3"` — matches the workspace slot naming + +**Platform strings:** literal `"telegram"` or `"matrix"` + +**Room IDs:** `"!r:m.org"`, `"!dm:example.org"` — valid Matrix room ID format + +No shared factories or fixtures files. Test data is constructed inline or via simple factory functions local to the test module. + +## What Is Tested + +| Area | Status | +|------|--------| +| `core/protocol.py` — dataclass defaults | Covered (`test_protocol.py`) | +| `core/store.py` — InMemoryStore + SQLiteStore | Covered (`test_store.py`) | +| `core/auth.py` — AuthManager | Covered (`test_auth.py`) | +| `core/chat.py` — ChatManager | Covered (`test_chat.py`) | +| `core/settings.py` — SettingsManager | Covered (`test_settings.py`) | +| `core/handler.py` — EventDispatcher routing | Covered (`test_dispatcher.py`) | +| `core/handlers/message.py` — handle_message | Covered (`test_voice_slot.py`) | +| Full dispatcher + all core handlers integration | Covered (`test_integration.py`) | +| `sdk/mock.py` — MockPlatformClient | Covered (`test_mock.py`) | +| `adapter/matrix/converter.py` — event parsing | Covered (`test_converter.py`) | +| `adapter/matrix/store.py` — store helpers | Covered (`test_store.py`) | +| `adapter/matrix/reactions.py` — text builders | Covered (`test_reactions.py`) | +| `adapter/matrix/bot.py` — MatrixBot + build_runtime | Covered (`test_dispatcher.py`) | +| `adapter/telegram/db.py` — SQLite helpers | Covered (`test_forum_db.py`, untracked) | + +## Coverage Gaps + +**Telegram adapter handlers** — `adapter/telegram/handlers/` (`auth.py`, `chat.py`, `confirm.py`, `forum.py`, `settings.py`) have no tests in `main`. Tests exist only in `.worktrees/telegram` branch (not yet merged). + +**Telegram converter** — `adapter/telegram/converter.py` has no tests in `main`. + +**`core/handlers/callback.py` and `core/handlers/settings.py`** — tested indirectly through integration tests but lack dedicated unit tests. + +**`adapter/matrix/room_router.py`** — `resolve_chat_id` has no direct unit tests; exercised only through `MatrixBot.on_room_message` integration path. + +**`adapter/matrix/handlers/`** — individual handler files (`auth.py`, `chat.py`, `confirm.py`, `settings.py`) are tested only via `test_dispatcher.py` integration; no isolated unit tests. + +**`sdk/mock.py` streaming** — `stream_message` is not tested; only `send_message` is covered. + +**Error paths** — `ChatManager.rename` raises `ValueError` when chat not found; no test exercises this path. Same for `ChatManager.archive`. + +## Naming Conventions + +- Test functions: `test_` — descriptive, no abbreviations +- Fixture names match the object they create: `mgr`, `store`, `dispatcher`, `deps` +- Factory functions in converter tests: `text_event()`, `file_event()`, `image_event()`, `reaction_event()` + +--- + +*Testing analysis: 2026-04-01* diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.continue-here.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.continue-here.md new file mode 100644 index 0000000..6de8f62 --- /dev/null +++ b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.continue-here.md @@ -0,0 +1,63 @@ +--- +phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow +task: 1 +total_tasks: 2 +status: paused +last_updated: 2026-04-07T21:29:48.982Z +--- + + +Formally, the most recently active execution artifact inside the roadmap is still `01.1-03-PLAN.md`, which has not been implemented yet. In parallel, the platform-integration track has moved forward: the direct-agent Matrix prototype design is now approved, the implementation plan is written, and the next useful session should evaluate that spec/plan pair against the live platform repos before starting execution. + + + + +- Re-analysed live platform repos on 2026-04-07 by cloning `platform/agent`, `platform/agent_api`, `platform/master`, and `platform/docs`. +- Confirmed `master` is still only a thin HTTP skeleton with `/health` and `/users/{user_id}`, not a chat/session/settings backend for surfaces. +- Confirmed `agent` exposes a working `/agent_ws/` WebSocket and `agent_api` provides enough protocol/client code to stream model output. +- Identified the real technical gap for a prototype: `agent` currently uses a singleton service with a fixed `thread_id="default"`, so all conversations would share memory unless that is parameterized. +- Derived and got approval for the prototype path: keep Matrix adapter logic largely intact, add `sdk/agent_session.py`, `sdk/prototype_state.py`, and `sdk/real.py`, keep settings local, and use the direct `agent` WebSocket for real messaging. +- Resolved the repo-placement question: the prototype stays in this repo on its own branch, not in a separate prototype repo. +- Resolved the platform-change minimization question: prefer patching only `platform/agent`, not `platform/agent_api`, and use a tiny local WebSocket client in this repo. +- Wrote and committed the approved design spec: `docs/superpowers/specs/2026-04-08-matrix-direct-agent-prototype-design.md`. +- Wrote the implementation plan: `docs/superpowers/plans/2026-04-08-matrix-direct-agent-prototype.md`. + + + + +- Task 1: Implement `adapter.matrix.reset` with `local-only`, `server-leave-forget`, and `--dry-run`, plus tests. +- Task 2: Update `README.md` so restart vs explicit reset workflow is documented and the old manual reset ritual is removed. +- Prototype evaluation follow-up: review the approved spec and plan against the platform repos before starting execution. +- Future prototype work: introduce `sdk/real.py` plus a narrow compatibility boundary that keeps Matrix adapter logic stable while allowing later expansion toward a fuller platform split. + + + + +- Do not integrate with `master` yet; it is still not the backend surfaces needs. +- Use the direct `agent` WebSocket as the only realistic path for a working prototype right now. +- Keep consumer-facing Matrix logic as intact as possible and absorb backend differences inside a shim under `sdk/`. +- Treat future platform evolution as likely to split into at least two concerns: control-plane access and direct agent session streaming. +- Keep the prototype in this repo on its own branch. +- Minimize platform-side changes by patching only `platform/agent` if possible. + + + +- Phase 01.1 itself is not blocked; it is simply paused. +- Prototype blocker: the `agent` repo currently hardcodes a shared `thread_id`, so per-user/per-chat conversation isolation requires either a small upstream change or a careful workaround. +- Platform contract blocker remains for the longer-term Phase 02 direction: `master` still lacks stable user/chat/session/settings APIs for surfaces. + + + +The important mental model is now stable enough to execute. Full SDK integration through `master` is still premature, but a working Matrix prototype can be built now by talking directly to the `agent` WebSocket and hiding the split backend reality behind `sdk/real.py`. The approved design keeps the prototype in this repo, keeps settings local, and minimizes platform changes by preferring a tiny `platform/agent` patch over broader protocol churn. For evaluation and implementation context, inspect: +- local spec: `docs/superpowers/specs/2026-04-08-matrix-direct-agent-prototype-design.md` +- local plan: `docs/superpowers/plans/2026-04-08-matrix-direct-agent-prototype.md` +- remote repos: `https://git.lambda.coredump.ru/platform/agent`, `https://git.lambda.coredump.ru/platform/master`, `https://git.lambda.coredump.ru/platform/agent_api` +- local clones: `/tmp/platform-agent`, `/tmp/platform-master`, `/tmp/platform-agent_api` + + + +Resume with one of these depending on priority: +1. Evaluate the approved prototype spec and implementation plan against the live platform repos and decide whether to start in this repo or patch `platform/agent` first. +2. If staying on roadmap execution, implement `01.1-03-PLAN.md` Task 1 (`adapter.matrix.reset`) first. +3. If starting prototype execution immediately, begin with Task 1 of `docs/superpowers/plans/2026-04-08-matrix-direct-agent-prototype.md`. + diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.gitkeep b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md new file mode 100644 index 0000000..187baa9 --- /dev/null +++ b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md @@ -0,0 +1,157 @@ +--- +phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - adapter/matrix/reconcile.py + - tests/adapter/matrix/test_reconcile.py +autonomous: true +requirements: [] + +must_haves: + truths: + - "A normal Matrix restart can rebuild missing local metadata from already joined Space/chat rooms instead of requiring a destructive reset." + - "Reconciliation restores the minimal local state needed for routing and chat operations: `matrix_user:*`, `matrix_room:*`, and missing `chat:{user}:{chat_id}` rows." + - "Reconciliation never provisions new Matrix rooms or Spaces while repairing local state." + - "Recovered users get `next_chat_index` advanced past the highest recovered `C*` chat id." + artifacts: + - path: "adapter/matrix/reconcile.py" + provides: "Matrix bootstrap reconciliation helpers and structured report objects." + - path: "tests/adapter/matrix/test_reconcile.py" + provides: "Regression coverage for startup and single-room reconciliation behavior." + key_links: + - from: "adapter/matrix/reconcile.py" + to: "adapter/matrix/store.py" + via: "set_user_meta and set_room_meta restore Matrix metadata" + pattern: "set_(user|room)_meta" + - from: "adapter/matrix/reconcile.py" + to: "core/chat.py" + via: "chat_mgr.get_or_create repairs missing `chat:*` rows" + pattern: "chat_mgr\\.get_or_create" +--- + + +Create the non-destructive Matrix reconciliation layer that Phase 01.1 depends on. + +Purpose: Per D-01 through D-07, the adapter must stop treating local SQLite state as the only truth in dev. Startup and recovery code need a single helper module that can rebuild local metadata from the homeserver room graph without creating duplicate Spaces or chats. +Output: `adapter/matrix/reconcile.py` with full-run and single-room recovery helpers, plus targeted pytest coverage. + + + +@/Users/a/.codex/get-shit-done/workflows/execute-plan.md +@/Users/a/.codex/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md +@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md +@.planning/phases/01-matrix-qa-polish/01-01-SUMMARY.md +@adapter/matrix/store.py +@adapter/matrix/handlers/auth.py +@core/chat.py +@tests/adapter/matrix/test_invite_space.py + + +From `adapter/matrix/store.py`: + +```python +async def get_room_meta(store: StateStore, room_id: str) -> dict | None +async def set_room_meta(store: StateStore, room_id: str, meta: dict) -> None +async def get_user_meta(store: StateStore, matrix_user_id: str) -> dict | None +async def set_user_meta(store: StateStore, matrix_user_id: str, meta: dict) -> None +``` + +From `core/chat.py`: + +```python +async def get_or_create( + self, + user_id: str, + chat_id: str, + platform: str, + surface_ref: str, + name: str | None = None, +) -> ChatContext +``` + +From Phase 01 room metadata shape: + +```python +{ + "room_type": "chat", + "chat_id": "C4", + "display_name": "Чат 4", + "matrix_user_id": "@alice:example.org", + "space_id": "!space:example.org", +} +``` + + + + + + + Task 1: Add reconciliation module for startup and single-room recovery + adapter/matrix/reconcile.py, tests/adapter/matrix/test_reconcile.py + adapter/matrix/store.py, adapter/matrix/handlers/auth.py, core/chat.py, tests/adapter/matrix/test_invite_space.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md + + - Test 1: `reconcile_matrix_state(...)` recreates missing `matrix_user:*`, `matrix_room:*`, and `chat:*` entries from joined Matrix rooms without calling `room_create`. + - Test 2: `reconcile_matrix_state(...)` leaves already-correct local metadata intact and reports restored vs skipped/conflicting rooms. + - Test 3: `reconcile_single_room(...)` can repair one `unregistered:{room_id}` chat room on demand and recompute `next_chat_index` for that user. + - Test 4: Space rooms or unrelated joined rooms are skipped, not converted into chat rows. + + +Create `adapter/matrix/reconcile.py` as the authoritative recovery module for this phase. Implement a small, explicit API that Plan 02 can wire directly: + +```python +async def reconcile_matrix_state(client: Any, store: StateStore, chat_mgr: ChatManager) -> dict: ... +async def reconcile_single_room( + client: Any, store: StateStore, chat_mgr: ChatManager, room_id: str, matrix_user_id: str +) -> dict: ... +``` + +Inside this module, add focused private helpers as needed for room classification, extracting room names, parsing `C` ids, and recomputing `next_chat_index`. Keep the logic non-destructive per D-04: +- never call `room_create`, `room_invite`, or provisioning code from `handlers/auth.py` +- prefer already-hydrated room data from the post-sync client object, and only fall back to explicit room-state fetches if required for room classification +- rebuild only the minimal metadata required by D-03: `matrix_user:*`, `matrix_room:*`, and missing `chat:{user}:{chat_id}` records +- if `chat:*` exists but points at the wrong `surface_ref`, repair it from Matrix room metadata and include the fix in the returned report +- derive `next_chat_index` from the highest recovered `C` for that user instead of trusting stale local counters + +Return a structured reconciliation report with stable keys such as: +`joined_rooms`, `restored_user_meta`, `restored_room_meta`, `restored_chat_rows`, `repaired_chat_rows`, `skipped_rooms`, and `conflicts`. + +Write `tests/adapter/matrix/test_reconcile.py` with lightweight `SimpleNamespace`/fake-client fixtures following the existing Matrix test style. Cover both full startup reconciliation and `reconcile_single_room(...)`. Assert that no provisioning calls are made during reconciliation, because D-04 forbids creating new Space/room topology while recovering local state. + + + cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_reconcile.py -q + + + - `adapter/matrix/reconcile.py` exports `reconcile_matrix_state` and `reconcile_single_room`. + - Reconciliation restores missing `matrix_user:*`, `matrix_room:*`, and `chat:*` entries for already-joined Matrix chat rooms per D-02 and D-03. + - Reconciliation does not call `room_create` or otherwise provision new server-side rooms per D-04. + - The report returned by reconciliation clearly distinguishes restored items, skipped rooms, and conflicts. + - `tests/adapter/matrix/test_reconcile.py` proves `next_chat_index` is recomputed from recovered chat ids rather than stale local state. + + The repository has an executable, tested reconciliation layer that can rebuild local Matrix metadata after dev-state loss without duplicating server-side rooms. + + + + + +Run `pytest tests/adapter/matrix/test_reconcile.py -q` and confirm startup and single-room reconciliation paths are covered. + + + +- Matrix recovery logic exists as a dedicated module instead of being scattered through handlers. +- Reconciliation is idempotent, non-destructive, and sufficient to restore routing/chat metadata from existing Matrix rooms. +- Plan 02 can wire startup and first-access recovery by calling exported functions rather than inventing new recovery logic. + + + +After completion, create `.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-SUMMARY.md` + diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-PLAN.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-PLAN.md new file mode 100644 index 0000000..bdfdaf8 --- /dev/null +++ b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-PLAN.md @@ -0,0 +1,167 @@ +--- +phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow +plan: 02 +type: execute +wave: 2 +depends_on: ["01.1-01"] +files_modified: + - adapter/matrix/bot.py + - tests/adapter/matrix/test_dispatcher.py +autonomous: true +requirements: [] + +must_haves: + truths: + - "The Matrix bot performs an initial sync and reconciliation before entering steady-state `sync_forever()`." + - "If a room still arrives as `unregistered:{room_id}` after startup, the bot makes one targeted recovery attempt before dispatching or failing." + - "When reconciliation cannot repair a room, the bot logs a clear diagnostic reason instead of crashing on downstream commands like `!rename`." + artifacts: + - path: "adapter/matrix/bot.py" + provides: "Startup bootstrap flow with initial sync, reconciliation, and targeted runtime retry." + - path: "tests/adapter/matrix/test_dispatcher.py" + provides: "Matrix runtime coverage for pre-sync reconcile and on-message recovery behavior." + key_links: + - from: "adapter/matrix/bot.py" + to: "adapter/matrix/reconcile.py" + via: "startup bootstrap and single-room recovery calls" + pattern: "reconcile_(matrix_state|single_room)" + - from: "adapter/matrix/bot.py" + to: "adapter/matrix/room_router.py" + via: "unregistered room detection before dispatch" + pattern: "unregistered:" +--- + + +Wire the new reconciliation layer into the actual Matrix runtime. + +Purpose: D-05 through D-07 require restart recovery to be the default developer path. The bot must bootstrap itself from existing Matrix rooms on startup and make one on-demand repair attempt before routing an unknown room through the dispatcher. +Output: `adapter/matrix/bot.py` performs initial sync + reconciliation before `sync_forever()`, and runtime tests prove the bot recovers or logs clearly instead of blindly dispatching broken state. + + + +@/Users/a/.codex/get-shit-done/workflows/execute-plan.md +@/Users/a/.codex/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md +@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md +@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md +@adapter/matrix/bot.py +@adapter/matrix/room_router.py +@adapter/matrix/reconcile.py +@tests/adapter/matrix/test_dispatcher.py + + +From `adapter/matrix/bot.py`: + +```python +class MatrixBot: + async def on_room_message(self, room: MatrixRoom, event: RoomMessageText) -> None + +async def main() -> None +``` + +From `adapter/matrix/reconcile.py`: + +```python +async def reconcile_matrix_state(client: Any, store: StateStore, chat_mgr: ChatManager) -> dict +async def reconcile_single_room( + client: Any, store: StateStore, chat_mgr: ChatManager, room_id: str, matrix_user_id: str +) -> dict +``` + +From `adapter/matrix/room_router.py`: + +```python +async def resolve_chat_id(store: StateStore, room_id: str, matrix_user_id: str) -> str +``` + + + + + + + Task 1: Run initial sync and reconciliation before the long-poll loop + adapter/matrix/bot.py, tests/adapter/matrix/test_dispatcher.py + adapter/matrix/bot.py, adapter/matrix/reconcile.py, tests/adapter/matrix/test_dispatcher.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md + + - Test 1: `main()` performs `client.sync(timeout=0, full_state=True)` before `sync_forever()`. + - Test 2: `main()` calls `reconcile_matrix_state(...)` after the initial sync and logs the returned report. + - Test 3: startup still reaches `sync_forever()` when reconciliation reports recoverable skips/conflicts instead of fatal failure. + + +Modify `adapter/matrix/bot.py` so normal startup follows the two-phase bootstrap recommended in research: +1. build client and runtime +2. authenticate +3. register callbacks +4. run `await client.sync(timeout=0, full_state=True)` +5. run `await reconcile_matrix_state(client, runtime.store, runtime.chat_mgr)` +6. log a structured `matrix_reconcile_complete` event with the report fields +7. enter `await client.sync_forever(timeout=30000)` + +Do not move provisioning logic into startup. The startup step only rehydrates local state from server-side rooms per D-02 through D-04. + +Update or add focused tests in `tests/adapter/matrix/test_dispatcher.py` using `monkeypatch`/fake-client patterns already used in the repo so the verify command proves the call order and logging-safe behavior. The test should fail if `sync_forever()` starts before reconciliation. + + + cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_dispatcher.py -q + + + - `adapter/matrix/bot.py` runs an initial full-state sync before steady-state polling. + - `adapter/matrix/bot.py` invokes `reconcile_matrix_state(...)` exactly once during startup. + - Startup logs a structured reconciliation summary instead of silently skipping the recovery step. + - `tests/adapter/matrix/test_dispatcher.py` asserts the bootstrap order explicitly. + + Normal Matrix bot startup now includes a recovery pass before the event loop begins handling user traffic. + + + + Task 2: Retry unknown-room routing once before dispatching broken state + adapter/matrix/bot.py, tests/adapter/matrix/test_dispatcher.py + adapter/matrix/bot.py, adapter/matrix/room_router.py, adapter/matrix/reconcile.py, tests/adapter/matrix/test_dispatcher.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md + + - Test 1: `MatrixBot.on_room_message(...)` detects `unregistered:{room_id}`, runs `reconcile_single_room(...)`, then retries `resolve_chat_id(...)`. + - Test 2: if retry succeeds, the event is dispatched against the recovered logical chat id. + - Test 3: if retry still fails, the bot does not crash; it logs a clear warning and sends a user-facing diagnostic message to that room. + + +Extend `MatrixBot.on_room_message(...)` so D-07 is satisfied even when startup could not repair a room yet. Keep `resolve_chat_id(...)` as the room-router source of truth, but treat `unregistered:{room_id}` as a recovery trigger rather than a stable runtime identity: +- first call `resolve_chat_id(...)` +- if the result starts with `unregistered:`, call `reconcile_single_room(client, runtime.store, runtime.chat_mgr, room.room_id, event.sender)` +- immediately retry `resolve_chat_id(...)` +- only dispatch once a concrete logical chat id exists +- if the retry still returns `unregistered:{room_id}`, log a structured warning with room id, matrix user id, and reconciliation report, then send a short `OutgoingMessage`-equivalent Matrix text explaining that local state could not be restored automatically and a dev reset/restart may be required + +Do not invent a new fallback chat id and do not auto-create rooms here; that would violate D-04. Keep this change inside `adapter/matrix/bot.py` so file ownership stays isolated for this plan. + + + cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_dispatcher.py -q + + + - Unknown Matrix rooms trigger one targeted reconciliation attempt before dispatch. + - Successful targeted recovery leads to normal dispatch with a real logical `chat_id`. + - Failed targeted recovery logs a clear diagnostic and avoids a handler crash on missing chat state per D-06. + - No code path in this task provisions new Matrix rooms or Spaces. + + The runtime treats unknown rooms as recoverable state drift first, not as a silent routing failure or crash path. + + + + + +Run `pytest tests/adapter/matrix/test_dispatcher.py -q` and confirm both startup-bootstrap and first-access recovery behaviors are covered. + + + +- A standard Matrix restart now attempts recovery before the bot starts processing live events. +- Unknown-room events are diagnosable and recoverable instead of falling straight into broken command handling. +- The runtime never provisions new server-side rooms during restart reconciliation. + + + +After completion, create `.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-SUMMARY.md` + diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-PLAN.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-PLAN.md new file mode 100644 index 0000000..bd78891 --- /dev/null +++ b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-PLAN.md @@ -0,0 +1,149 @@ +--- +phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow +plan: 03 +type: execute +wave: 1 +depends_on: [] +files_modified: + - adapter/matrix/reset.py + - tests/adapter/matrix/test_reset.py + - README.md +autonomous: true +requirements: [] + +must_haves: + truths: + - "Developers have an explicit dev-only reset command instead of relying on memory or ad hoc shell history." + - "The default reset mode clears only local Matrix state and explains the manual Matrix-client cleanup that may still be needed." + - "Optional server cleanup is clearly named around leave/forget semantics and supports dry-run output." + artifacts: + - path: "adapter/matrix/reset.py" + provides: "Dev reset CLI for local-only, server-leave-forget, and dry-run workflows." + - path: "tests/adapter/matrix/test_reset.py" + provides: "CLI coverage for local reset behavior and printed operator guidance." + - path: "README.md" + provides: "Updated developer instructions for normal restart vs explicit reset." + key_links: + - from: "adapter/matrix/reset.py" + to: "README.md" + via: "documented invocation and manual Matrix cleanup guidance" + pattern: "adapter\\.matrix\\.reset" +--- + + +Ship the dev reset workflow that complements normal restart reconciliation. + +Purpose: D-08 through D-10 require a repeatable, explicit reset path for clean-room QA without making destructive cleanup the default restart flow. This plan creates the tool and updates the runbook developers actually use. +Output: `adapter/matrix/reset.py`, pytest coverage, and README instructions that replace the old `rm -f lambda_matrix.db` ritual. + + + +@/Users/a/.codex/get-shit-done/workflows/execute-plan.md +@/Users/a/.codex/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/ROADMAP.md +@.planning/STATE.md +@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md +@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md +@README.md +@adapter/matrix/bot.py +@core/store.py + + +From `adapter/matrix/bot.py` env usage: + +```python +db_path = os.environ.get("MATRIX_DB_PATH", "lambda_matrix.db") +store_path = os.environ.get("MATRIX_STORE_PATH", "matrix_store") +homeserver = os.environ.get("MATRIX_HOMESERVER") +user_id = os.environ.get("MATRIX_USER_ID") +``` + +From `core/store.py`: + +```python +class SQLiteStore: + def __init__(self, db_path: str) -> None: ... +``` + + + + + + + Task 1: Add a dev-only Matrix reset CLI with explicit modes + adapter/matrix/reset.py, tests/adapter/matrix/test_reset.py + adapter/matrix/bot.py, core/store.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md + + - Test 1: `--mode local-only` deletes the configured local DB/store paths or reports what would be deleted in dry-run mode. + - Test 2: `--mode server-leave-forget --dry-run` prints the exact rooms it would leave/forget and does not mutate local files. + - Test 3: when server cleanup is not executed, the command prints the manual Matrix-client steps required by D-10. + + +Create `adapter/matrix/reset.py` as a CLI entrypoint runnable via `uv run python -m adapter.matrix.reset`. Use `argparse` and keep the tool explicitly dev-only in its help text and logs. + +Implement the following modes from research and locked decisions: +- `local-only` (default destructive mode for local QA): remove `MATRIX_DB_PATH` and `MATRIX_STORE_PATH` if they exist; if not, report that they were already absent +- `server-leave-forget`: for the bot account only, log in using the same Matrix env vars as `adapter/matrix/bot.py`, inspect joined rooms, and call `room_leave()` + `room_forget()` for each joined room; support `--dry-run` so the operator can inspect the target set before mutation +- `--dry-run` must work with both modes and print a structured summary instead of mutating files or Matrix membership + +Always print a post-run summary that distinguishes: +- what local files/directories were deleted or would be deleted +- what server-side leave/forget actions were executed or would be executed +- the manual Matrix client steps still required for a true clean-room QA rerun (leave/archive old rooms or Space in Element, accept fresh invites, etc.) when those actions are outside this phase + +Write `tests/adapter/matrix/test_reset.py` to cover local-only deletion, dry-run output, and server-leave-forget dry-run behavior with fake clients/temporary directories. Follow the repo’s existing lightweight async test style. + + + cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_reset.py -q + + + - `adapter/matrix/reset.py` supports `local-only`, `server-leave-forget`, and `--dry-run`. + - `local-only` reset targets both `lambda_matrix.db` and `matrix_store` via env-aware paths per D-09. + - The tool never claims to globally delete Matrix rooms; it uses leave/forget semantics or prints manual cleanup instructions per D-10. + - `tests/adapter/matrix/test_reset.py` proves dry-run mode is non-destructive. + + The repository contains a repeatable dev reset tool that replaces the undocumented shell ritual and names server-side cleanup honestly. + + + + Task 2: Replace the README reset ritual with the new restart and reset workflow + README.md + README.md, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md + +Update `README.md` so Matrix development instructions reflect Phase 01.1 instead of the old destructive reset ritual. Replace the current manual QA block that tells developers to `rm -f lambda_matrix.db` with a short, explicit split: +- normal restart: `PYTHONPATH=. uv run python -m adapter.matrix.bot` now performs reconciliation automatically +- explicit clean-room reset: `PYTHONPATH=. uv run python -m adapter.matrix.reset --mode local-only` +- optional server cleanup preview: `PYTHONPATH=. uv run python -m adapter.matrix.reset --mode server-leave-forget --dry-run` + +State clearly that normal restart is the default path per D-05, and that full server-side cleanup may still require manual steps in the Matrix client. Keep the README concise; do not add production guidance or Phase 2 SDK content. + + + cd /Users/a/MAI/sem2/lambda/surfaces-bot && python -m adapter.matrix.reset --help >/tmp/matrix-reset-help.txt && rg -n "adapter.matrix.reset|local-only|server-leave-forget|reconciliation" README.md /tmp/matrix-reset-help.txt + + + - `README.md` no longer recommends raw `rm -f lambda_matrix.db` as the default Matrix restart workflow. + - `README.md` documents the normal restart path and the explicit reset path separately. + - The documented reset commands match the CLI implemented in `adapter/matrix/reset.py`. + + Developers can follow a repeatable README workflow for ordinary restart and clean-room QA reset without relying on tribal knowledge. + + + + + +Run `pytest tests/adapter/matrix/test_reset.py -q` and `python -m adapter.matrix.reset --help`, then confirm the README commands and help text stay aligned. + + + +- Dev reset is an explicit tool, not a remembered shell sequence. +- Local-only reset is automated and documented. +- Server cleanup semantics are honest, dry-runnable, and accompanied by manual Matrix-client guidance where needed. + + + +After completion, create `.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-SUMMARY.md` + diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md new file mode 100644 index 0000000..665061e --- /dev/null +++ b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md @@ -0,0 +1,121 @@ +# Phase 01.1: Matrix restart reconciliation and dev reset workflow - Context + +**Gathered:** 2026-04-03 +**Status:** Ready for planning + + +## Phase Boundary + +Сделать Matrix-адаптер пригодным для повторяемой локальной разработки и ручного QA без ручного “ритуала” удаления БД, выхода из всех комнат и пересоздания пользователя. + +В scope этой фазы: +- безопасный restart flow для Matrix-бота после потери локального state +- reconciliation локального store с уже существующими Matrix rooms / Space +- отдельный dev reset workflow для controlled clean-room QA +- диагностируемое поведение при несогласованности local state и server-side Matrix state + +Вне scope: +- реальный Lambda SDK +- новые пользовательские Matrix features +- E2EE +- production-grade multi-user migration framework + + + + +## Implementation Decisions + +### Matrix state lifecycle + +- **D-01:** Локальный SQLite store больше не должен считаться единственной точкой истины для Matrix runtime в dev workflow. +- **D-02:** При старте бот должен пытаться восстановить минимально необходимое локальное состояние из уже существующих Matrix rooms / Space, а не требовать full reset. +- **D-03:** Reconciliation должен восстанавливать как минимум `matrix_user:*`, `matrix_room:*` и missing `chat:{user}:{chat_id}` записи, если серверные комнаты уже существуют. +- **D-04:** Reconciliation не должен создавать новые Space/rooms, если задача — именно восстановление локального state после рестарта. + +### Dev restart behavior + +- **D-05:** Обычный restart бота должен быть основным путём для разработки; удаление `lambda_matrix.db` и `matrix_store` не должно быть обязательным для проверки workflow. +- **D-06:** Если local state неполон, бот должен либо восстановить его, либо логировать понятную причину, а не падать на командах вроде `!rename`. +- **D-07:** Несогласованность между `room_meta` и `ChatManager` должна обнаруживаться и устраняться автоматически на startup или при первом обращении. + +### Dev reset workflow + +- **D-08:** Нужен отдельный dev-only reset tool/script для controlled QA, вместо ручного набора shell-команд. +- **D-09:** Reset workflow должен как минимум поддерживать `local-only` reset: удаление `lambda_matrix.db` и `matrix_store` с понятной инструкцией, что делать с server-side Matrix rooms. +- **D-10:** Если full server-side cleanup не автоматизируется в этой фазе, tool должен явно печатать, какие ручные шаги обязательны в Matrix client. + +### The agent's Discretion + +- Точное место вызова reconciliation в startup flow +- Внутренняя структура helper-модуля (`bootstrap.py`, `reconcile.py` или аналог) +- Формат dev reset script и уровень автоматизации server-side cleanup +- Детали debug-logging и dry-run режима, если они помогают без раздувания scope + + + + +## Specific Ideas + +- Главный критерий: после обычного restart бот не должен ломаться только потому, что local DB была сброшена или частично потеряна. +- Reset workflow должен быть явным и repeatable, а не завязанным на память разработчика. +- Нужно различать две ситуации: + - broken because code is wrong + - broken because local dev state was deliberately reset and requires reconciliation + + + + +## Canonical References + +**Downstream agents MUST read these before planning or implementing.** + +### Matrix phase artifacts +- `.planning/phases/01-matrix-qa-polish/01-CONTEXT.md` — locked Matrix decisions from Phase 1 +- `.planning/phases/01-matrix-qa-polish/01-VERIFICATION.md` — what Phase 1 already validated and what manual QA still expects +- `.planning/phases/01-matrix-qa-polish/01-HUMAN-UAT.md` — remaining real-client Matrix checks + +### Current Matrix runtime +- `adapter/matrix/bot.py` — startup flow, sync loop, runtime wiring, DB/store env vars +- `adapter/matrix/store.py` — persisted Matrix metadata and pending confirmation keys +- `adapter/matrix/room_router.py` — room_id to chat_id resolution and current unregistered fallback +- `adapter/matrix/handlers/auth.py` — invite bootstrap that creates Space and first chat room +- `core/chat.py` — `ChatManager` persistence contract that currently breaks when local state is missing + +### Supporting docs +- `docs/matrix-prototype.md` — intended Matrix UX and architecture direction +- `README.md` — current run instructions and existing manual QA/reset habits + + + + +## Existing Code Insights + +### Reusable Assets +- `adapter/matrix/store.py` already persists room/user metadata and is the obvious place to anchor reconciliation inputs. +- `adapter/matrix/room_router.py` already detects unknown rooms via `unregistered:{room_id}` fallback; this is a useful reconciliation trigger point. +- `core/chat.py` already has `get_or_create`, `rename`, `archive`, `list_active`; missing chat records can be rebuilt through this API instead of inventing a parallel format. + +### Established Patterns +- Matrix runtime uses `SQLiteStore` for adapter-local metadata and `matrix-nio` room callbacks for transport events. +- Phase 1 already moved Matrix to Space+rooms and command-only confirmations, so this phase must preserve that model rather than reverting to DM-first simplifications. + +### Integration Points +- Startup path in `adapter/matrix/bot.py:main()` is the natural place to run reconciliation before `sync_forever`. +- Invite/bootstrap path in `adapter/matrix/handlers/auth.py` is the existing source of truth for what metadata a healthy first room should have. +- `ChatManager` records and `matrix_room:*` metadata must stay consistent enough that commands like `!rename`, `!archive`, and `!chats` work after restart. + + + + +## Deferred Ideas + +- Full production-grade migration of historical Matrix state across schema versions +- Automatic server-side deletion/leave for all Matrix rooms and Space during reset, if it requires broader admin semantics +- Any Phase 2 SDK integration work + + + +--- + +*Phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow* +*Context gathered: 2026-04-03* diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md new file mode 100644 index 0000000..792031d --- /dev/null +++ b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md @@ -0,0 +1,350 @@ +# Phase 01.1: Matrix restart reconciliation and dev reset workflow - Research + +**Researched:** 2026-04-03 +**Domain:** Matrix adapter restart reconciliation, local state recovery, dev reset workflow +**Confidence:** HIGH + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions +- **D-01:** Локальный SQLite store больше не должен считаться единственной точкой истины для Matrix runtime в dev workflow. +- **D-02:** При старте бот должен пытаться восстановить минимально необходимое локальное состояние из уже существующих Matrix rooms / Space, а не требовать full reset. +- **D-03:** Reconciliation должен восстанавливать как минимум `matrix_user:*`, `matrix_room:*` и missing `chat:{user}:{chat_id}` записи, если серверные комнаты уже существуют. +- **D-04:** Reconciliation не должен создавать новые Space/rooms, если задача — именно восстановление локального state после рестарта. +- **D-05:** Обычный restart бота должен быть основным путём для разработки; удаление `lambda_matrix.db` и `matrix_store` не должно быть обязательным для проверки workflow. +- **D-06:** Если local state неполон, бот должен либо восстановить его, либо логировать понятную причину, а не падать на командах вроде `!rename`. +- **D-07:** Несогласованность между `room_meta` и `ChatManager` должна обнаруживаться и устраняться автоматически на startup или при первом обращении. +- **D-08:** Нужен отдельный dev-only reset tool/script для controlled QA, вместо ручного набора shell-команд. +- **D-09:** Reset workflow должен как минимум поддерживать `local-only` reset: удаление `lambda_matrix.db` и `matrix_store` с понятной инструкцией, что делать с server-side Matrix rooms. +- **D-10:** Если full server-side cleanup не автоматизируется в этой фазе, tool должен явно печатать, какие ручные шаги обязательны в Matrix client. + +### Claude's Discretion +- Точное место вызова reconciliation в startup flow +- Внутренняя структура helper-модуля (`bootstrap.py`, `reconcile.py` или аналог) +- Формат dev reset script и уровень автоматизации server-side cleanup +- Детали debug-logging и dry-run режима, если они помогают без раздувания scope + +### Deferred Ideas (OUT OF SCOPE) +- Full production-grade migration of historical Matrix state across schema versions +- Automatic server-side deletion/leave for all Matrix rooms and Space during reset, if it requires broader admin semantics +- Any Phase 2 SDK integration work + + +## Summary + +Phase 01.1 should be planned as a bootstrap/recovery phase, not as another chat-feature phase. The current Matrix adapter has no startup reconciliation path: `adapter/matrix/bot.py` logs in and goes directly to `sync_forever()`, while routing and command handlers assume `matrix_room:*`, `matrix_user:*`, and `chat:*` keys already exist. That means local DB loss currently produces logical corruption, not just missing cache. + +The safe standard approach is: perform a first sync that hydrates joined-room state, inspect the bot's current joined rooms and room state from the homeserver, rebuild the minimal local metadata needed for command routing, and only then enter the long-running sync loop. Reconciliation should be non-destructive and idempotent: if local keys already exist and match server state, leave them alone; if they are missing, recreate them; if they conflict, prefer the server room topology for Matrix-specific metadata and recreate missing `ChatManager` rows from that. + +For reset, separate two workflows explicitly. `local-only` reset is the default and should be automated. Optional server-side cleanup may leave/forget rooms for the bot account, but it cannot promise global deletion of Matrix rooms for all members; if that is not automated, the tool must print the exact manual steps for the Matrix client. + +**Primary recommendation:** Add a startup `reconcile_matrix_state()` step before `sync_forever()`, and ship a dev-only reset CLI with `local-only`, `server-leave-forget`, and `dry-run` modes. + +## Project Constraints (from CLAUDE.md) + +- Do not treat missing Lambda SDK as a blocker. +- Keep all platform calls behind `platform/interface.py`. +- Current runtime implementation is `platform/mock.py`; recommendations must work with that. +- Prefer architecture changes in adapters and core without coupling to future SDK internals. +- Use pytest-based verification. +- Do not recommend committing `.env`. +- Respect dependency order: `core/` first, then `platform/`, then adapters. + +## Standard Stack + +### Core +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| Python | 3.14.3 installed | Runtime for bot and scripts | Already available locally; codebase targets `>=3.11`. | +| `matrix-nio` | 0.25.2, published 2024-10-04 | Matrix client, sync, room membership/state APIs | Already installed; exposes the exact bootstrap/reset APIs this phase needs. | +| `SQLiteStore` (repo) | local | Adapter/core KV persistence | Existing persistence contract for `matrix_user:*`, `matrix_room:*`, and `chat:*`. | +| Matrix Client-Server API | spec latest | Authoritative room membership/state semantics | Needed to reason about restart recovery and leave/forget behavior correctly. | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| `pytest` | 9.0.2, published 2025-12-06 | Test runner | For targeted adapter/bootstrap regression tests. | +| `pytest-asyncio` | 1.3.0, published 2025-11-10 | Async test execution | For async reconciliation/reset flows. | +| `structlog` | 25.5.0, published 2025-10-27 | Diagnostics | For reconciliation summaries and conflict logging. | +| `python-dotenv` | 1.2.2, published 2026-03-01 | Env loading | Already used by `adapter/matrix/bot.py` for Matrix config. | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| Startup reconciliation from joined rooms + state | Force developers to wipe local DB and recreate rooms | Simpler code, but directly violates D-01, D-02, D-05. | +| Non-destructive local rebuild | Full auto-recreate of Space/rooms on missing local state | Easier to implement, but causes duplicate Matrix rooms and breaks D-04. | +| Dev reset script | README-only manual ritual | Lower code cost, but not repeatable and fails D-08..D-10. | + +**Installation:** +```bash +uv sync +``` + +**Version verification:** Verified via installed environment and PyPI metadata on 2026-04-03: +- `matrix-nio` `0.25.2` - 2024-10-04 +- `pytest` `9.0.2` - 2025-12-06 +- `pytest-asyncio` `1.3.0` - 2025-11-10 +- `structlog` `25.5.0` - 2025-10-27 +- `python-dotenv` `1.2.2` - 2026-03-01 + +## Architecture Patterns + +### Recommended Project Structure +```text +adapter/matrix/ +├── bot.py # startup flow calls reconciliation before sync loop +├── reconcile.py # bootstrap/rebuild logic from Matrix server state +├── reset.py # dev-only reset CLI / entrypoint +├── room_router.py # room_id -> chat_id with recovery hook +├── store.py # metadata helpers, prefix scans, derived counters +└── handlers/ + ├── auth.py # first-time provisioning only + └── chat.py # uses recovered state, no provisioning fallback +``` + +### Pattern 1: Two-Phase Startup Bootstrap +**What:** Split startup into `login -> initial sync/full_state -> reconcile -> steady-state sync_forever`. +**When to use:** Always for Matrix bot startup when local DB may be missing or stale. +**Example:** +```python +# Source: matrix-nio AsyncClient docs/source + repo startup flow +client = AsyncClient(...) +runtime = build_runtime(store=SQLiteStore(db_path), client=client) + +await login_or_restore_session(client) +await client.sync(timeout=0, full_state=True) +report = await reconcile_matrix_state(client, runtime.store, runtime.chat_mgr) +logger.info("matrix_reconcile_complete", **report) +await client.sync_forever(timeout=30000) +``` + +### Pattern 2: Rebuild Local Metadata From Joined Rooms +**What:** Enumerate joined rooms, inspect local hydrated room objects or room state, and recreate missing `matrix_room:*`, `matrix_user:*`, and `chat:*` records. +**When to use:** On startup and optionally on `unregistered:{room_id}` fallback at runtime. +**Example:** +```python +# Source: matrix-nio AsyncClient.joined_rooms/room_get_state + repo store contracts +joined = await client.joined_rooms() +for room_id in joined.rooms: + state = await client.room_get_state(room_id) + # detect: space room vs chat room, owner user, child relationship, display name + # rebuild matrix_room:{room_id} + # rebuild chat:{matrix_user_id}:{chat_id} if absent +``` + +### Pattern 3: Non-Destructive Reconciliation Report +**What:** Return a structured report: scanned rooms, restored rooms, restored chats, conflicts, skipped rooms. +**When to use:** Every reconciliation run, including dry-run. +**Example:** +```python +{ + "joined_rooms": 4, + "restored_user_meta": 1, + "restored_room_meta": 3, + "restored_chat_rows": 3, + "conflicts": [], + "skipped_rooms": ["!dm:example.org"], +} +``` + +### Pattern 4: Reset Modes Are Explicit +**What:** Separate `local-only`, `server-leave-forget`, and `dry-run`. +**When to use:** For dev/QA only. Never mix destructive server cleanup into normal startup. +**Example:** +```bash +uv run python -m adapter.matrix.reset --mode local-only +uv run python -m adapter.matrix.reset --mode server-leave-forget --dry-run +``` + +### Anti-Patterns to Avoid +- **Provisioning during reconciliation:** Do not create a new Space or new rooms while trying to recover missing local state. +- **Treating `next_chat_index` as primary truth:** Derive it from recovered `chat_id` values after scan; do not trust a missing or stale counter. +- **Routing unknown rooms straight through:** `unregistered:{room_id}` is a signal to reconcile, not a stable runtime identity. +- **Destructive reset by default:** Startup must never leave/forget rooms automatically. +- **Blindly trusting local `surface_ref`:** If `chat:*` and `matrix_room:*` disagree, rebuild from Matrix room metadata and repair the chat row. + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Room discovery | Custom DB-only reconstruction heuristics | `AsyncClient.joined_rooms()` plus synced room state | Server already knows which rooms the bot joined. | +| Space membership detection | Naming-convention parsing of room names | Matrix state: `m.room.create.type`, `m.space.child`, `m.space.parent` | Names are mutable and non-authoritative. | +| Room cleanup semantics | Custom “delete room” assumptions | `room_leave()` + `room_forget()` semantics | Client API supports leave/forget, not guaranteed global deletion. | +| Chat ID recovery | Hardcoded `C1/C2/...` reset | Rebuild from existing `matrix_room:*`/server state and compute next index | Prevents collisions after partial DB loss. | +| Diagnostic output | Ad hoc `print()` strings | Structured reconciliation/reset report via `structlog` | Easier manual QA and failure triage. | + +**Key insight:** The homeserver already persists the bot’s room graph. This phase should rehydrate local cache from that graph, not attempt to replace it with a second custom truth model. + +## Common Pitfalls + +### Pitfall 1: Joining the sync loop before reconciliation +**What goes wrong:** Commands arrive while local metadata is still missing, producing `unregistered:{room_id}` routing or `ChatManager` misses. +**Why it happens:** Current `main()` enters `sync_forever()` immediately after login. +**How to avoid:** Perform initial sync and reconciliation first. +**Warning signs:** `unregistered_room` logs immediately after restart; `ValueError("Chat ... not found")` on `!rename` or `!archive`. + +### Pitfall 2: Recovering room metadata but not chat rows +**What goes wrong:** Room routing works, but `ChatManager.rename/archive/list_active` still fails because `chat:{user}:{chat_id}` rows were not recreated. +**Why it happens:** Matrix adapter metadata and core chat metadata live in different keyspaces. +**How to avoid:** Reconciliation must repair both stores in one pass. +**Warning signs:** `matrix_room:*` exists but `chat:*` keys do not. + +### Pitfall 3: Trusting stale `next_chat_index` +**What goes wrong:** New chats reuse existing `C` IDs after local recovery. +**Why it happens:** `next_chat_id()` increments a persisted counter that may be absent or behind. +**How to avoid:** After scan, set `next_chat_index = max(recovered_chat_numbers) + 1`. +**Warning signs:** New room gets `C1` even though Space already contains prior rooms. + +### Pitfall 4: Assuming room names identify chat rooms safely +**What goes wrong:** Reconciliation binds the wrong room because a user renamed a room or Space. +**Why it happens:** Names are user-facing labels, not stable identifiers. +**How to avoid:** Prefer room state and existing `chat_id` metadata; use display names only as fallback. +**Warning signs:** Duplicate “Чат 1” names or renamed rooms break matching. + +### Pitfall 5: Over-promising full cleanup +**What goes wrong:** Reset script claims a “clean slate” but rooms still exist in Element or for other members. +**Why it happens:** Leaving/forgetting affects the bot account’s membership/history, not necessarily global room deletion. +**How to avoid:** Name the mode accurately and print the manual client steps when needed. +**Warning signs:** QA reruns still show old rooms in the user’s client. + +## Code Examples + +Verified patterns from official sources and the installed library surface: + +### Initial Sync Before Reconcile +```python +# Source: matrix-nio AsyncClient.sync/sync_forever +await client.sync(timeout=0, full_state=True) +report = await reconcile_matrix_state(client, store, chat_mgr) +await client.sync_forever(timeout=30000) +``` + +### Space Child Link Creation +```python +# Source: Matrix client-server API state event + current auth/new-chat flow +await client.room_put_state( + room_id=space_id, + event_type="m.space.child", + content={"via": [homeserver]}, + state_key=chat_room_id, +) +``` + +### Bot-Side Leave/Forget Cleanup +```python +# Source: matrix-nio AsyncClient.room_leave / room_forget +for room_id in room_ids: + await client.room_leave(room_id) + await client.room_forget(room_id) +``` + +### Router Recovery Trigger +```python +# Source: repo room_router contract +chat_id = await resolve_chat_id(store, room_id, matrix_user_id) +if chat_id.startswith("unregistered:"): + await reconcile_single_room(client, store, chat_mgr, room_id, matrix_user_id) +``` + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| Local adapter DB treated as the operational truth | Rebuildable local cache from server room graph | Mature Matrix client practice; supported by current Matrix CS API and `matrix-nio` | Restart no longer requires destructive local reset. | +| Manual room cleanup in client after experiments | Scripted leave/forget plus explicit manual instructions | Current `matrix-nio` 0.25.x API surface | QA becomes repeatable and auditable. | +| Immediate steady-state sync after login | Initial sync/full-state bootstrap before long polling | Supported by current `AsyncClient.sync()` / `sync_forever()` behavior | Reconciliation can run before any user traffic is handled. | + +**Deprecated/outdated:** +- `README.md` Matrix manual QA instruction `rm -f lambda_matrix.db` as the primary restart flow: outdated for this phase. +- DM-first Matrix recovery assumptions in `docs/matrix-prototype.md`: outdated relative to Phase 1 Space+rooms decisions. + +## Open Questions + +1. **How exactly should reconciliation identify the owning Matrix user for a recovered room when local `matrix_room:*` is gone?** + - What we know: the bot can enumerate joined rooms and fetch room state; current healthy metadata stores `matrix_user_id` and `space_id`. + - What's unclear: whether Phase 1-created rooms also expose enough server-side structure to recover owner deterministically without existing local metadata in every case. + - Recommendation: Plan a proof test against a real homeserver/client. If room-state-only ownership is ambiguous, persist a tiny bot-authored marker state event going forward, but keep that addition narrowly scoped. + +2. **Should runtime recovery happen only on startup, or also lazily on first unknown room access?** + - What we know: startup repair satisfies D-02/D-07 for common restart loss; `room_router` already surfaces unknown rooms cleanly. + - What's unclear: whether partial DB corruption during runtime is common enough to justify lazy single-room repair in Phase 01.1. + - Recommendation: Make startup reconciliation required, lazy room repair optional if it stays small. + +3. **How much of server cleanup should Phase 01.1 automate?** + - What we know: `room_leave()` and `room_forget()` are available; global room deletion is not what the client API guarantees. + - What's unclear: whether automating bot-side leave/forget is worth the extra risk for this urgent phase. + - Recommendation: Keep `local-only` mandatory. Make server cleanup optional and clearly labeled experimental/dev-only if included. + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| Python | Runtime, scripts, tests | ✓ | 3.14.3 | — | +| `uv` | Standard install/run workflow | ✓ | 0.9.30 | `python -m` + existing venv | +| `pytest` | Automated verification | ✓ | 9.0.2 | `uv run pytest` | +| Matrix homeserver credentials | Real restart/reset manual QA | ✗ in current shell | — | Manual-only after `.env` is configured | +| Matrix bot local DB/store paths | Reset workflow | ✓ | defaults in code | Can override with `MATRIX_DB_PATH` / `MATRIX_STORE_PATH` | + +**Missing dependencies with no fallback:** +- Live Matrix credentials for real manual reconciliation/reset QA. + +**Missing dependencies with fallback:** +- None for repository-only implementation and tests. + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | `pytest 9.0.2` + `pytest-asyncio 1.3.0` | +| Config file | `pyproject.toml` | +| Quick run command | `pytest tests/adapter/matrix -v` | +| Full suite command | `pytest tests/ -v` | + +### Phase Requirements → Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| PH01.1-BOOT | Startup rebuilds missing `matrix_user:*`, `matrix_room:*`, and `chat:*` from existing rooms without creating new rooms | unit/integration | `pytest tests/adapter/matrix/test_reconcile.py -v` | ❌ Wave 0 | +| PH01.1-ROUTER | Unknown room fallback can trigger repair or yields diagnosable warning without crashing commands | unit | `pytest tests/adapter/matrix/test_room_router_reconcile.py -v` | ❌ Wave 0 | +| PH01.1-COUNTER | Reconciliation resets `next_chat_index` to recovered max + 1 | unit | `pytest tests/adapter/matrix/test_reconcile.py -k next_chat_index -v` | ❌ Wave 0 | +| PH01.1-RESET | Dev reset `local-only` removes local DB/store paths and prints next steps | unit/smoke | `pytest tests/adapter/matrix/test_reset.py -v` | ❌ Wave 0 | +| PH01.1-NONDESTRUCTIVE | Reconciliation never calls room creation APIs | unit | `pytest tests/adapter/matrix/test_reconcile.py -k no_create -v` | ❌ Wave 0 | + +### Sampling Rate +- **Per task commit:** `pytest tests/adapter/matrix -v` +- **Per wave merge:** `pytest tests/ -v` +- **Phase gate:** Full suite green before `/gsd:verify-work` + +### Wave 0 Gaps +- [ ] `tests/adapter/matrix/test_reconcile.py` - startup reconciliation scenarios +- [ ] `tests/adapter/matrix/test_reset.py` - CLI/script reset modes and output +- [ ] `tests/adapter/matrix/test_room_router_reconcile.py` - lazy recovery or warning behavior +- [ ] Integration fixture for a fake `AsyncClient` response surface matching `joined_rooms()` and `room_get_state()` + +## Sources + +### Primary (HIGH confidence) +- Matrix Client-Server API - room state, leave, forget, joined rooms, Spaces semantics: https://spec.matrix.org/latest/client-server-api/index.html +- `matrix-nio` installed 0.25.2 API surface verified locally on 2026-04-03 via `AsyncClient.sync`, `sync_forever`, `joined_rooms`, `room_get_state`, `room_leave`, `room_forget` +- Repo code: [adapter/matrix/bot.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/bot.py), [adapter/matrix/store.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/store.py), [adapter/matrix/room_router.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/room_router.py), [adapter/matrix/handlers/auth.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/handlers/auth.py), [core/chat.py](/Users/a/MAI/sem2/lambda/surfaces-bot/core/chat.py) +- PyPI release metadata: https://pypi.org/project/matrix-nio/ , https://pypi.org/project/pytest/ , https://pypi.org/project/pytest-asyncio/ , https://pypi.org/project/structlog/ , https://pypi.org/project/python-dotenv/ + +### Secondary (MEDIUM confidence) +- [README.md](/Users/a/MAI/sem2/lambda/surfaces-bot/README.md) - current manual reset habit and run commands +- [docs/matrix-prototype.md](/Users/a/MAI/sem2/lambda/surfaces-bot/docs/matrix-prototype.md) - original Matrix UX intent, noting outdated DM/reaction sections +- [01-CONTEXT.md](/Users/a/MAI/sem2/lambda/surfaces-bot/.planning/phases/01-matrix-qa-polish/01-CONTEXT.md) - locked Phase 1 Matrix decisions +- [01-VERIFICATION.md](/Users/a/MAI/sem2/lambda/surfaces-bot/.planning/phases/01-matrix-qa-polish/01-VERIFICATION.md) - what has already been verified and what still needs human Matrix QA + +### Tertiary (LOW confidence) +- None + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH - verified against installed environment, PyPI metadata, and official Matrix spec +- Architecture: HIGH - directly grounded in current repo flow plus current `matrix-nio`/Matrix capabilities +- Pitfalls: HIGH - derived from concrete gaps in current startup/store/router code + +**Research date:** 2026-04-03 +**Valid until:** 2026-05-03 diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-VALIDATION.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-VALIDATION.md new file mode 100644 index 0000000..336cbd6 --- /dev/null +++ b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-VALIDATION.md @@ -0,0 +1,80 @@ +--- +phase: 01.1 +slug: matrix-restart-reconciliation-and-dev-reset-workflow +status: draft +nyquist_compliant: false +wave_0_complete: false +created: 2026-04-03 +--- + +# Phase 01.1 — Validation Strategy + +> Per-phase validation contract for feedback sampling during execution. + +--- + +## Test Infrastructure + +| Property | Value | +|----------|-------| +| **Framework** | `pytest 9.0.2` + `pytest-asyncio 1.3.0` | +| **Config file** | `pyproject.toml` | +| **Quick run command** | `pytest tests/adapter/matrix -v` | +| **Full suite command** | `pytest tests/ -v` | +| **Estimated runtime** | ~20 seconds | + +--- + +## Sampling Rate + +- **After every task commit:** Run `pytest tests/adapter/matrix -v` +- **After every plan wave:** Run `pytest tests/ -v` +- **Before `$gsd-verify-work`:** Full suite must be green +- **Max feedback latency:** 20 seconds + +--- + +## Per-Task Verification Map + +| Task ID | Plan | Wave | Requirement | Test Type | Automated Command | File Exists | Status | +|---------|------|------|-------------|-----------|-------------------|-------------|--------| +| 01.1-01-01 | 01 | 1 | PH01.1-BOOT | unit/integration | `pytest tests/adapter/matrix/test_reconcile.py -v` | ❌ W0 | ⬜ pending | +| 01.1-01-01 | 01 | 1 | PH01.1-COUNTER | unit | `pytest tests/adapter/matrix/test_reconcile.py -k next_chat_index -v` | ❌ W0 | ⬜ pending | +| 01.1-01-01 | 01 | 1 | PH01.1-NONDESTRUCTIVE | unit | `pytest tests/adapter/matrix/test_reconcile.py -k no_create -v` | ❌ W0 | ⬜ pending | +| 01.1-02-01 | 02 | 2 | PH01.1-BOOT | unit | `pytest tests/adapter/matrix/test_dispatcher.py -k startup -v` | ✅ | ⬜ pending | +| 01.1-02-02 | 02 | 2 | PH01.1-ROUTER | unit | `pytest tests/adapter/matrix/test_dispatcher.py -k reconcile -v` | ✅ | ⬜ pending | +| 01.1-03-01 | 03 | 1 | PH01.1-RESET | unit/smoke | `pytest tests/adapter/matrix/test_reset.py -v` | ❌ W0 | ⬜ pending | +| 01.1-03-02 | 03 | 1 | PH01.1-RESET | smoke | `python -m adapter.matrix.reset --help` | ❌ W0 | ⬜ pending | + +*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky* + +--- + +## Wave 0 Requirements + +- [ ] `tests/adapter/matrix/test_reconcile.py` — startup reconciliation scenarios, `next_chat_index`, and no-provisioning assertions +- [ ] `tests/adapter/matrix/test_reset.py` — CLI reset modes, dry-run behavior, and operator guidance output +- [ ] `tests/adapter/matrix/test_dispatcher.py` — startup bootstrap order and targeted unknown-room recovery coverage +- [ ] Fake `AsyncClient` fixture surface for joined rooms, room state, leave, and forget behavior + +--- + +## Manual-Only Verifications + +| Behavior | Requirement | Why Manual | Test Instructions | +|----------|-------------|------------|-------------------| +| Reconciled Space/chat rooms render correctly in a real Matrix client after restart | PH01.1-BOOT | Client UX and homeserver state cannot be fully trusted from fake nio fixtures | 1. Start the bot with existing Space/chat rooms. 2. Verify the bot does not create duplicate Space or chat rooms. 3. Send a command in a recovered room and confirm it routes normally. | +| Server-side cleanup leaves the account in a usable Element state after `server-leave-forget` | PH01.1-RESET | Element/archive behavior and homeserver retention are client/server integration concerns | 1. Run `python -m adapter.matrix.reset --mode server-leave-forget --dry-run`. 2. Run without `--dry-run` on a test account. 3. Confirm joined rooms disappear for the bot and fresh invites can be accepted cleanly. | + +--- + +## Validation Sign-Off + +- [ ] All tasks have `` verify or Wave 0 dependencies +- [ ] Sampling continuity: no 3 consecutive tasks without automated verify +- [ ] Wave 0 covers all MISSING references +- [ ] No watch-mode flags +- [ ] Feedback latency < 20s +- [ ] `nyquist_compliant: true` set in frontmatter + +**Approval:** pending diff --git a/.planning/phases/02-prototype/.continue-here.md b/.planning/phases/02-prototype/.continue-here.md new file mode 100644 index 0000000..a2d4619 --- /dev/null +++ b/.planning/phases/02-prototype/.continue-here.md @@ -0,0 +1,72 @@ +--- +phase: 02-prototype +task: 4 +total_tasks: 4 +status: paused +last_updated: 2026-04-07T23:54:30.473Z +--- + + +The Matrix direct-agent prototype is implemented and manually proven working on branch `feat/matrix-direct-agent-prototype`. The current code path can log into Matrix, accept invites, provision the first Space/chat tree for a fresh user, and send live text messages to a patched local `platform-agent` over WebSocket. The immediate remaining engineering gap is not feature delivery but resilience: backend/provider failures can still bubble up as `PlatformError` and crash the Matrix bot process. + + + + +- Task 1: Added `sdk/agent_session.py` and transport tests for direct WebSocket messaging with collision-safe `thread_key` generation. +- Task 2: Added `sdk/prototype_state.py` and tests for stable local user mapping, settings defaults, and mutation-safe settings copies. +- Task 3: Added `sdk/real.py` as the `PlatformClient` implementation, fixed import-time dependency leakage, and aligned thread-key tests to the actual dispatcher contract. +- Task 4: Wired Matrix runtime selection through `MATRIX_PLATFORM_BACKEND=real`, documented usage in `README.md`, and added dispatcher coverage for real backend selection. +- Fixed repeat Matrix invites so the bot now `join()`s before the existing-user early return path. +- Added Russian runbook doc `docs/matrix-direct-agent-prototype-ru.md` and pushed the branch. +- Manually validated live bring-up using a local patched `external/platform-agent` on port 8000 plus the Matrix homeserver `https://matrix.lambda.coredump.ru`. + + + + +- Add graceful degradation for backend/provider failures so `PlatformError` does not crash the Matrix process. +- Decide whether to upstream or separately push the required `external/platform-agent` patch (`1dca2c1`) that enables WebSocket `thread_id`. +- Optionally clean up repeat-invite UX if Space/chat reprovisioning should ever happen for already-known users. +- Optionally prepare a PR from `feat/matrix-direct-agent-prototype`. + + + + +- Keep the prototype in this repo, not a separate Matrix-only repo. +- Keep Matrix adapter logic intact and absorb backend differences inside `sdk/`. +- Split the real backend into `AgentSessionClient` and `PrototypeStateStore` behind `RealPlatformClient`. +- Patch only `platform-agent` for per-thread memory instead of changing both `agent` and `agent_api`. +- Use a serialized collision-safe thread key because Matrix user IDs contain colons. +- For repeat invites, join the room but do not recreate Space/chat state if the user is already provisioned locally. + + + +- Technical: provider/backend errors still crash the Matrix bot instead of returning a user-facing failure reply. +- External: the required `platform-agent` patch exists only in the local clone under `external/` and is not yet upstream. +- Operational: credentials used during manual bring-up were exposed in-session and should be rotated. + + + +The important mental model is stable. `platform/master` is still not the backend for surfaces, so the working prototype goes directly to `platform-agent` over `/agent_ws/`. The live setup that worked was: +- `surfaces-bot` branch: `feat/matrix-direct-agent-prototype` +- Matrix bot env: `MATRIX_PLATFORM_BACKEND=real`, `AGENT_WS_URL=ws://127.0.0.1:8000/agent_ws/` +- patched local `external/platform-agent` with `thread_id` support +- provider configured through OpenRouter using model `qwen/qwen3.5-122b-a10b` + +Important files: +- `sdk/agent_session.py` +- `sdk/prototype_state.py` +- `sdk/real.py` +- `adapter/matrix/bot.py` +- `adapter/matrix/handlers/auth.py` +- `docs/matrix-direct-agent-prototype-ru.md` + +Important local-only dependency: +- `external/platform-agent` commit `1dca2c1` (`feat: support websocket thread ids`) + +Likely running background process at pause time: +- local `platform-agent` server on port 8000, PID 13499 + + + +Start with the failure path: catch `PlatformError` around Matrix message handling so a bad provider response becomes a normal reply like “backend unavailable, try again later” instead of killing the process. After that, either upstream `external/platform-agent` commit `1dca2c1` or document it as an explicit prerequisite in the platform repo. + diff --git a/.planning/reports/20260422-session-report.md b/.planning/reports/20260422-session-report.md new file mode 100644 index 0000000..9044d2b --- /dev/null +++ b/.planning/reports/20260422-session-report.md @@ -0,0 +1,92 @@ +# GSD Session Report + +**Generated:** 2026-04-21T22:33:11.666Z +**Project:** surfaces-bot +**Milestone:** v1.0 — Production-ready surfaces + +--- + +## Session Summary + +**Duration:** Single session +**Phase Progress:** Phase 04 implemented; current follow-up work is audit, stabilization, and platform bug localization +**Plans Executed:** 0 formal GSD plans executed in this session; work was focused on post-implementation audit and cleanup +**Commits Made:** 6 + +## Work Performed + +### Phases Touched + +- **Phase 04** — Matrix MVP follow-up after implementation: + - completed audit of platform patches vs surface-owned responsibilities + - removed dependence on local platform modifications for `chat_id` + - switched Matrix integration to numeric `platform_chat_id` mapping on our side + - cleaned transport layer to a thin adapter over upstream `AgentApi` + - updated README and run instructions + - produced final Russian bug report with raw-trace-based diagnosis + +### Key Outcomes + +- Platform repos are clean and synced to pinned upstream commits. +- Matrix real backend works with numeric surrogate `platform_chat_id`. +- `surfaces` transport layer no longer owns custom stream semantics. +- Final diagnosis was narrowed: missing-first-chunk bug is now considered platform-side with direct raw evidence. +- Working state was committed and pushed on `feat/matrix-direct-agent-prototype`. + +### Decisions Made + +- Do not patch vendored platform repos for the working implementation. +- Keep `surfaces` transport layer thin and upstream-aligned. +- Treat the current streaming bug as platform-side unless new evidence disproves it. +- Do not add new local stream workarounds that would blur responsibility. + +## Files Changed + +- `README.md` +- `adapter/matrix/bot.py` +- `sdk/agent_api_wrapper.py` +- `sdk/real.py` +- `tests/platform/test_real.py` +- `tests/adapter/matrix/test_dispatcher.py` +- `tests/core/test_integration.py` +- `docs/reports/2026-04-22-platform-streaming-final-bug-report-ru.md` + +Planning / handoff artifacts updated: + +- `.planning/HANDOFF.json` +- `.planning/phases/04-matrix-mvp-shared-agent-context-and-context-management-comma/.continue-here.md` +- `.planning/reports/20260422-session-report.md` + +## Blockers & Open Items + +- Platform-side streaming bug after tool/file flow. +- Duplicate `END` from platform. +- Image path failure on oversized `data:` URI. +- `tokens_used` remains unavailable from pinned upstream client. + +## Estimated Resource Usage + +| Metric | Estimate | +|--------|----------| +| Commits | 6 | +| Files changed | 8 code/docs files in the main deliverable, plus planning artifacts | +| Plans executed | 0 formal plans in this session | +| Subagents spawned | 0 | + +> **Note:** Token and cost estimates require API-level instrumentation. +> These metrics reflect observable session activity only. + +--- + +### Recent Commits + +- `0c2884c` — `refactor: use thin upstream transport adapter` +- `569824e` — `refactor: shrink agent api wrapper to thin adapter` +- `4d917ac` — `docs: add thin transport adapter plan` +- `3a3fcdc` — `docs: add thin transport adapter design` +- `7a2ad86` — `docs: clarify matrix file sending flow` +- `4524a6a` — `feat: finalize matrix platform audit and docs` + +--- + +*Generated by `$gsd-session-report`* diff --git a/.planning/threads/matrix-dev-prototype-agent-platform-state.md b/.planning/threads/matrix-dev-prototype-agent-platform-state.md new file mode 100644 index 0000000..facd575 --- /dev/null +++ b/.planning/threads/matrix-dev-prototype-agent-platform-state.md @@ -0,0 +1,133 @@ +# Thread: Matrix dev prototype — состояние агента и платформы + +## Status: IN PROGRESS + +## Goal + +Зафиксировать текущее состояние платформы для последующей разработки Matrix dev прототипа, +в котором команды разработки скиллов смогут быстро добавлять и обкатывать скиллы. + +## Context + +*Исследование проведено 2026-04-14. Репозитории: `external/platform-agent`, `external/platform-agent_api`, `external/platform-master`.* + +### Решение по деплою: локальный контейнер у каждого разработчика + +`platform-master` не готов для общего деплоя: +- lifecycle management контейнеров (TTL, cleanup, переиспользование сессий) — в ветке `feat/storage`, не смержено в main +- без него при общем деплое контейнеры висят вечно, ресурсы не освобождаются + +Локальный вариант: `make up-dev` — полностью рабочий, volume mount `./workspace:/workspace/`, hot reload src. + +### Архитектура изоляции контекстов + +`AgentService` — singleton с `thread_id = "default"` — это **намеренно**. Архитектура Master предполагает один контейнер `platform-agent` на один чат. Изоляция на уровне контейнеров, не thread_id. Фиксить не нужно. + +### Система скиллов (deepagents) + +`SkillsMiddleware` в `deepagents` полностью готов: +- скилл = директория с `SKILL.md` (YAML frontmatter + markdown инструкции) +- progressive disclosure: агент видит имя+описание в system prompt, читает полный файл по требованию +- загружается один раз при старте сессии, кэшируется в LangGraph state + +**НЕ подключено** в `platform-agent/src/agent/base.py` — отсутствует одна строка: +```python +skills=["/workspace/skills/"] +``` +Это задача для команды платформы. + +### Workflow разработчика скилла + +``` +workspace/ + skills/ + my-skill/ + SKILL.md ← редактируешь здесь (live через volume mount) + helper.py ← вспомогательные файлы + config/ + my-skill.json ← токены и настройки (пишет агент при первом запуске) +``` + +1. Редактируешь `SKILL.md` +2. `!new` в Matrix (новая сессия = скиллы перечитываются) +3. Проверяешь поведение +4. Повторяешь + +Агент может **сам установить скилл** из GitHub: +- `execute` → git clone +- `write_file` → положить в `/workspace/skills/` +- после `!new` скилл активен + +### Конфигурация скиллов (токены, API ключи) + +Агент управляет конфигом сам: +- первый запуск: спрашивает пользователя → пишет в `/workspace/config/skill-name.json` +- последующие запуски: читает из файла +- файл персистентен между сессиями (volume mount) + +### Входящий протокол (что принимает агент) + +`ClientMessage` — только `text: str`. Файлы и изображения не поддерживаются. +Задача для платформы — расширить протокол. + +### Исходящий протокол (что шлёт агент) + +Новые события с `origin/main` (апрель 2026): +- `AGENT_EVENT_TOOL_CALL_CHUNK` — агент вызывает инструмент +- `AGENT_EVENT_TOOL_RESULT` — результат инструмента +- `AGENT_EVENT_CUSTOM_UPDATE` — произвольный прогресс + +**Наш `sdk/agent_session.py` падает на этих событиях** (`raise PlatformError("Unexpected agent message")`). +Нужно починить — это наша задача, ~10 строк. + +### AgentApi из lambda_agent_api + +Готовый production-клиент с правильным lifecycle (`connect()`, `close()`, `send_message()` как `AsyncIterator`). +Наш `sdk/agent_session.py` дублирует его функциональность. Стоит заменить. + +### Инструменты агента из коробки + +- `ls`, `read_file`, `write_file`, `edit_file`, `glob`, `grep` — файловые операции в workspace +- `execute` — shell под изолированным OS-пользователем `agent` +- `write_todos` — список задач +- `task` — вызов субагентов + +### Запуск локально + +```bash +# .env минимально необходимый: +PROVIDER_URL=https://openrouter.ai/api/v1 +PROVIDER_API_KEY=<ключ> +PROVIDER_MODEL=anthropic/claude-sonnet-4-6 + +# Dev контейнер: +make up-dev # требует AGENT_API_PATH=../platform-agent_api в env +``` + +Dev Dockerfile монтирует `./workspace:/workspace/` и `./src:/app/src` (hot reload). + +## Что нужно от платформы + +1. Добавить `skills=["/workspace/skills/"]` в `platform-agent/src/agent/base.py` +2. Поддержка файлов/изображений в `ClientMessage` (не срочно для MVP) +3. Lifecycle management контейнеров в Master (для общего деплоя, не срочно) + +## Что делаем мы + +1. Починить `sdk/agent_session.py` — обработка tool-событий вместо исключения +2. (опционально) Заменить `AgentSessionClient` на `AgentApi` из `lambda_agent_api` + +## References + +- `external/platform-agent` — локальный клон, наш патч `1dca2c1` (thread_id) поверх `1e9fa1f` +- `external/platform-agent_api` — локальный клон, актуальный (origin/master = `bb20a84`) +- `external/platform-master` — локальный клон, активная разработка в `feat/storage-s02` +- `docs/superpowers/specs/2026-04-08-matrix-direct-agent-prototype-design.md` +- `docs/superpowers/plans/2026-04-08-matrix-direct-agent-prototype.md` + +## Next Steps + +1. Запросить у команды платформы: подключение `SkillsMiddleware` в `base.py` +2. Починить `sdk/agent_session.py` — обработать tool-события +3. Написать первый тестовый скилл (`workspace/skills/hello/SKILL.md`) и проверить end-to-end +4. Документировать workflow для разработчиков скиллов diff --git a/.planning/threads/matrix-file-ingestion-context.md b/.planning/threads/matrix-file-ingestion-context.md new file mode 100644 index 0000000..0ccb079 --- /dev/null +++ b/.planning/threads/matrix-file-ingestion-context.md @@ -0,0 +1,81 @@ +# Thread: Matrix file ingestion and agent-visible storage contract + +## Status: IN PROGRESS + +## Goal + +Сохранить текущий контекст сессии для следующего агента и зафиксировать следующую архитектурную развилку: как принимать вложения из Matrix и делать их доступными агенту. + +## Current State + +Phase 4 Matrix MVP уже собран и проверен на уровне per-room routing: +- обычные сообщения теперь идут в `platform_chat_id`, а не в общий локальный `C1/C2` +- `!context` показывает состояние текущего Matrix-чата +- `!save` и `!load` привязаны к текущему room-context +- `PrototypeStateStore` хранит live state per context +- последние изменения закоммичены в `feat/matrix-direct-agent-prototype` + +Коммиты, которые важно знать: +- `c11c8ec` `feat(task-5): scope matrix context state per room` +- `07c5078` `feat(task-7): verify matrix per-room context routing` + +## What We Learned About Platform Runtime + +Текущий `external/platform-agent` не является отдельным контейнером на чат. +Фактическая модель сейчас такая: +- один FastAPI-процесс +- singleton `AgentService` +- `thread_id` используется как ключ памяти в LangGraph, а не как контейнерная изоляция +- файловой изоляции на чат сейчас нет +- `/workspace` как общий mount для Matrix bot и platform-agent сейчас не настроен +- отдельного upload API для вложений в текущем коде не видно + +Ключевые файлы: +- `external/platform-agent/src/api/external.py` +- `external/platform-agent/src/agent/service.py` +- `external/platform-agent/src/agent/base.py` + +## File Handling Requirement + +Пользовательский запрос на текущем этапе: +- принимать файл или сообщение с файлом из Matrix +- сохранять файл локально +- передавать агенту явный сигнал, что к сообщению есть вложения +- сообщать, где лежит файл + +Но есть техническое ограничение: +- если Matrix bot пишет файл только в своём контейнере, platform-agent его не увидит +- значит нужен либо общий storage, либо upload в платформу, либо контейнеризация platform-agent с общим volume + +## Recommended Design Direction + +Самый прагматичный MVP-вариант: +- хранить вложения в общем каталоге, который виден и Matrix bot, и platform-agent +- формировать для агента структурированный payload с: + - локальным путём + - original filename + - mime type + - attachment type +- если есть текст пользователя, дополнять сообщение краткой summary-подсказкой про вложения +- если прислан только файл, отправлять synthetic message вроде “пользователь прислал файл” + +Если общий каталог невозможен в текущем runtime: +- следующий вариант это upload endpoint в platform-agent +- Matrix surface скачивает файл и загружает его в платформу, а платформа уже кладёт его в своё доступное хранилище + +## Open Questions + +1. Где должен жить shared storage: host path, docker volume или platform-side volume? +2. Нужен ли немедленный upload API в platform-agent, или сначала достаточно shared path? +3. Должны ли файлы быть scoped per room/platform_chat_id, а не per user? + +## Next Step For Another Agent + +1. Подтвердить runtime-модель хранения файлов. +2. Проверить, как сейчас запускаются Matrix bot и platform-agent в реальной dev-схеме. +3. После выбора storage contract начать с изменений в Matrix attachment ingestion. + +## Notes + +- Контекст этой сессии сохранён как отдельный thread, потому что текущий следующий рискованный шаг уже не про context routing, а про файловый transport. +- Не смешивать этот трек с незавершённой историей про `!branch`: upstream branch/snapshot API всё ещё не подтверждён.