diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md index 9c859f8..d90b47e 100644 --- a/.planning/PROJECT.md +++ b/.planning/PROJECT.md @@ -2,56 +2,44 @@ ## What This Is -Telegram и Matrix боты для взаимодействия пользователя с AI-агентом Lambda. Каждый бот — тонкий адаптер поверх общего ядра (`core/`), изолирующего бизнес-логику от транспорта. Платформа подключается через `sdk/interface.py` Protocol; сейчас используется `MockPlatformClient`. +Surfaces (поверхности) — это тонкие адаптеры-клиенты, соединяющие мессенджеры с агентами платформы Lambda. +Текущая и главная реализация — **Matrix MVP**. Бот работает как stateless-прослойка: преобразует события Matrix во внутренний протокол `core/` и маршрутизирует их на внешние контейнеры агентов (через `AgentApi` по WebSocket). ## Core Value -Пользователь может вести диалог с Lambda-агентом через любой из поддерживаемых мессенджеров без изменения ядра системы. +Пользователь может бесшовно взаимодействовать с изолированными AI-агентами через нативные интерфейсы мессенджеров (с поддержкой пересылки файлов и работы в комнатах), в то время как сама платформа агентов не зависит от транспорта. ## Requirements ### Validated -- ✓ core/ — унифицированный протокол событий, EventDispatcher, StateStore, ChatManager, AuthManager, SettingsManager — existing -- ✓ adapter/telegram/ — forum-first адаптер (Threaded Mode), `/start`, `/new`, `/archive`, `/rename`, `/settings`, стриминг ответов — existing, QA passed -- ✓ adapter/matrix/ — Space+rooms адаптер, invite flow, `!new`, `!archive`, `!rename`, `!settings`, room-per-chat — existing -- ✓ sdk/mock.py — MockPlatformClient: `stream_message`, `get_or_create_user`, `get_settings`, `update_settings` — existing +- ✓ `core/` — унифицированный протокол событий, EventDispatcher, StateStore, ChatManager. +- ✓ `adapter/matrix/` — Space+rooms адаптер. Прием инвайтов, автосоздание иерархии комнат, команды `!new`, `!archive`, `!clear`, `!yes`/`!no`. +- ✓ `sdk/real.py` — интеграция с AgentApi. Поддержка WebSocket для обмена сообщениями и передачи вложений в обе стороны. +- ✓ Shared Volume — прямая передача файлов в локальные рабочие папки агентов (`/agents/`). +- ✓ Dynamic Routing — маршрутизация чатов к агентам на основе `config/matrix-agents.yaml`. +- ✓ Deployment — Разделение окружений на `docker-compose.prod.yml` (только бот) и `docker-compose.fullstack.yml` (бот + локальный агент для E2E). -### Active +### Out of Scope / Deferred -- [ ] Matrix QA — ручное тестирование Matrix адаптера, фиксация багов -- [ ] SDK integration — заменить MockPlatformClient реальным Lambda SDK (когда платформа готова) -- [ ] Production hardening — конфиг для деплоя, логирование, мониторинг - -### Out of Scope - -- E2EE для Matrix (python-olm не собирается на macOS/ARM) — инфраструктурная задача, отдельный трек -- Supergroup forum mode для Telegram — заменён Threaded Mode как основным режимом -- Telegram DM-first режим — заменён forum-first (Threaded Mode) +- E2EE для Matrix (отложено из-за сложностей сборки `python-olm` на кросс-платформенных средах). +- Интеграция с Master-сервисом платформы (временно используется прямое соединение с `platform-agent` через AgentApi). +- Telegram-адаптер (вынесен в легаси ветку `feat/telegram-adapter`, MVP фокусируется на Matrix). ## Context -- Python 3.11+, aiogram 3.4+, matrix-nio 0.21+, SQLite, pytest-asyncio -- Threaded Mode — Bot API 9.3, Mac клиент имеет известные баги (новые топики не сразу видны в сайдбаре) -- Lambda platform SDK ещё не готов, всё работает через MockPlatformClient -- Архитектура: Hexagonal / Ports-and-Adapters; `core/` не зависит от транспорта - -## Constraints - -- **Tech stack**: aiogram 3.x для Telegram, matrix-nio для Matrix — не менять без обсуждения -- **Platform**: SDK подключается только через `sdk/interface.py` Protocol — core/ и adapters не трогаются при смене реализации -- **Telegram**: Threaded Mode — единственный поддерживаемый режим; `closeForumTopic`/`deleteForumTopic` не работают в personal chat forums -- **E2EE**: python-olm не собирается на текущей среде — Matrix работает только без шифрования +- Стек: Python 3.11+, `matrix-nio`, `uv`, `pydantic`. +- Бот хранит только локальную привязку (`room_id` <-> `platform_chat_id`) в SQLite. Вся долговременная память и история диалогов хранятся на стороне агента. +- Жизненный цикл контейнеров агентов управляется платформой, а не ботом. ## Key Decisions | Decision | Rationale | Outcome | |----------|-----------|---------| -| Forum-first (Threaded Mode) для Telegram | Bot API 9.3 позволяет личный чат как форум — чище, без суперпруппы | ✓ Good | -| (user_id, thread_id) как PK в chats | Изоляция контекстов по топику | ✓ Good | -| MockPlatformClient через sdk/interface.py | Не ждать SDK, разрабатывать независимо | ✓ Good | -| Space+rooms для Matrix | Room-based UX и явные чаты важнее DM-first упрощений | ✓ Good | -| Отказ от E2EE в Matrix | python-olm не собирается на macOS/ARM | — Pending | +| Space+rooms для Matrix | Room-based UX и явные чаты (по одному на тред) удобнее, чем DM-каша | ✓ Good | +| Прямая интеграция AgentApi | Master API не был готов, прямое WebSocket соединение позволяет передавать стейт и файлы | ✓ Good | +| Shared Volume для файлов | Избавляет от необходимости гонять base64 по сети, быстрый прямой доступ к файлам | ✓ Good | +| Stateless бот | Бот легко перезапускать и масштабировать, память изолирована в агентах | ✓ Good | ## Evolution @@ -61,10 +49,5 @@ Telegram и Matrix боты для взаимодействия пользова 3. New requirements emerged? → Add to Active 4. Decisions to log? → Add to Key Decisions -**After each milestone:** -1. Full review of all sections -2. Core Value check — still the right priority? -3. Update Context with current state - --- -*Last updated: 2026-04-02 after initialization* +*Last updated: 2026-05-03 after codebase consolidation* diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 4e8799b..ffd6801 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -1,101 +1,32 @@ # Roadmap — v1.0 -## Milestone: v1.0 — Production-ready surfaces - -### Phase 1: Matrix QA & Polish - -**Goal:** Переработать Matrix адаптер с DM-first на Space+rooms, убрать реакции в пользу !yes/!no, довести до уровня "приемлемо работает" как Telegram. - -**Depends on:** Telegram QA complete - -**Plans:** 6 plans - -Plans: -- [x] 01-01-PLAN.md — Space+rooms infrastructure (store helpers, handle_invite rewrite, room_router) -- [x] 01-02-PLAN.md — Chat command handlers (!new, !archive, !rename) Space-aware -- [x] 01-03-PLAN.md — Reaction removal + !yes/!no confirmation + settings dashboard -- [x] 01-04-PLAN.md — Test suite (fix 4 broken + 12 new MAT-01..MAT-12) -- [x] 01-05-PLAN.md — Gap closure for Matrix `!yes` / `!no` pending-confirm scope -- [x] 01-06-PLAN.md — Remaining Phase 01 gap closure work (completed 2026-04-03) +## Milestone: v1.0 — Production-ready Matrix MVP +### Phase 01: Matrix QA & Polish +**Goal:** Переработать Matrix адаптер с DM-first на Space+rooms, убрать реакции в пользу `!yes`/`!no`. +**Status:** Completed **Deliverables:** - Space+rooms architecture for Matrix adapter -- !yes/!no text-based confirmation (no reactions) -- Read-only !settings dashboard -- 96+ tests green - ---- - -### Phase 01.1: Matrix restart reconciliation and dev reset workflow (INSERTED) - -**Goal:** Сделать Matrix-адаптер пригодным для повторяемого локального рестарта и ручного QA: бот восстанавливает минимальный local state из существующих Space/rooms и даёт явный dev reset workflow вместо ручного ritual reset. -**Requirements**: none explicitly mapped -**Depends on:** Phase 1 -**Plans:** 3 plans - -Plans: -- [ ] 01.1-01-PLAN.md — Non-destructive Matrix reconciliation module and tests -- [ ] 01.1-02-PLAN.md — Wire startup/bootstrap recovery into the Matrix runtime -- [ ] 01.1-03-PLAN.md — Dev reset CLI and updated Matrix restart runbook - -### Phase 2: SDK Integration - -**Goal:** Заменить MockPlatformClient реальным Lambda SDK — бот начинает работать с настоящим AI-агентом. - -**Depends on:** Phase 1, Lambda platform SDK готов +- !yes/!no text-based confirmation +- Test suite green +### Phase 04: Matrix MVP: Agent Integration +**Goal:** Подключить реального агента через `AgentApi`, добавить команды управления контекстом (`!clear`). +**Status:** Completed **Deliverables:** -- `sdk/real.py` — реализация PlatformClient через реальный SDK -- `bot.py` для обоих адаптеров переключается на реальный клиент через env var -- `stream_message` работает с реальным стримингом -- Интеграционные тесты с реальным SDK (или staging) - -### Phase 4: Matrix MVP: shared agent context and context management commands - -**Goal:** Привести Matrix-бот к рабочему состоянию для MVP-деплоя: заменить AgentSessionClient на AgentApi, добавить !save/!load/!reset/!context команды управления контекстом агента, упаковать в Docker. -**Requirements**: Replace AgentSessionClient with AgentApi; Wire AgentApi lifecycle; Implement !save, !load, !reset, !context commands; Dockerfile + docker-compose -**Depends on:** Phase 1 (Matrix adapter complete) -**Plans:** 3 plans - -Plans: -- [x] 04-01-PLAN.md — Replace AgentSessionClient with AgentApi; update sdk/real.py, bot.py, broken tests -- [x] 04-02-PLAN.md — !save, !load, !reset, !context handlers; PrototypeStateStore extensions; numeric interception -- [x] 04-03-PLAN.md — Dockerfile + docker-compose.yml + .env.example update - ---- +- `sdk/real.py` — реализация `PlatformClient` через реальный SDK (`AgentApi`). +- Поддержка WebSocket стриминга. +- Команды управления контекстом. +- Обертка в Docker. ### Phase 05: MVP Deployment - -**Goal:** Подготовить Matrix-бот к реальному деплою на lambda.coredump.ru без потери Space+rooms UX: закрепить per-room `platform_chat_id`, реальный `!clear`, reconciliation, file transfer через shared volume и разделение prod/fullstack compose. - -**Depends on:** Phase 4 - -**Plans:** 4/4 plans complete - -Plans: -- [x] 05-01-PLAN.md — Startup reconciliation from authoritative Matrix Space topology before live sync -- [x] 05-02-PLAN.md — Room-local `platform_chat_id` routing and real `!clear` semantics -- [x] 05-03-PLAN.md — Shared-volume attachment path hardening for `/agents` deployment -- [x] 05-04-PLAN.md — Split bot-only prod compose from internal fullstack compose and update docs - +**Goal:** Подготовить Matrix-бот к реальному деплою на lambda.coredump.ru с маршрутизацией по агентам и передачей файлов. +**Status:** Completed **Deliverables:** -- Space+rooms onboarding remains the primary Matrix UX -- Per-room `platform_chat_id` provides true context isolation and `!clear` -- Reconciliation restores room metadata and routing after restart -- File transfer uses shared `/agents/` volume with room-safe behavior -- `docker-compose.prod.yml` is bot-only handoff; `docker-compose.fullstack.yml` is for internal E2E testing +- Загрузка `matrix-agents.yaml` для маппинга пользователей к агентам. +- Per-room `platform_chat_id` routing. +- File transfer через shared `/agents/` volume. +- Разделение `docker-compose.prod.yml` и `docker-compose.fullstack.yml`. --- - -### Phase 3: Production Hardening - -**Goal:** Подготовить боты к реальному деплою — конфиг, логирование, мониторинг, обработка ошибок. - -**Depends on:** Phase 2 - -**Deliverables:** -- Docker / systemd конфиг для деплоя -- Структурированное логирование в production формате -- Health-check endpoint (если нужен) -- Rate limiting и защита от спама -- Graceful shutdown +*Note: Легаси-фазы, связанные с Telegram, прототипами и Mock-платформой, были удалены из Roadmap после закрепления архитектуры MVP в ветке main.* diff --git a/.planning/STATE.md b/.planning/STATE.md index eb05f42..47a860b 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,12 +2,12 @@ gsd_state_version: 1.0 milestone: v1.0 milestone_name: — Production-ready surfaces -status: Phase 05 Paused -last_updated: "2026-04-29T08:49:04Z" +status: MVP Deployed +last_updated: "2026-05-03T23:00:00Z" progress: - total_phases: 6 + total_phases: 3 completed_phases: 3 - total_plans: 16 + total_plans: 13 completed_plans: 13 --- @@ -15,115 +15,35 @@ progress: ## Project Reference -See: .planning/PROJECT.md (updated 2026-04-02) +See: `.planning/PROJECT.md` (updated 2026-05-03) -**Core value:** Пользователь ведёт диалог с Lambda через любой мессенджер без изменения ядра -**Current focus:** Phase 05 paused — latest file-contract change needs a new image build before platform redeploy +**Core value:** Пользователь бесшовно взаимодействует с изолированными агентами через нативные интерфейсы (Matrix), в то время как платформа агентов не зависит от транспорта. +**Current focus:** Итерационное развитие текущей архитектуры, добавление новых фич и поверхностей (по мере необходимости). ## Current Phase -**Phase 05** paused: MVP deployment hardening is in place, but the latest attachment workspace-root change is not yet published +Текущий MVP успешно завершен. Все базовые механизмы внедрены и работают: +- Маршрутизация к `AgentApi` +- Shared Volume файловый обмен (`/agents/`) +- Dynamic config через `matrix-agents.yaml` +- Изоляция контекстов через `platform_chat_id` -Deployment handoff follow-up is external. The last published image predates the latest file-handling change; the next step is to rebuild and publish a fresh image, then ask the platform to redeploy Matrix with the shared `/agents` volumes and `config/matrix-agents.yaml`. - -Plan `05-01` is complete. Matrix startup now reconciles managed Space rooms from synced topology before live traffic, restoring local metadata and deterministic legacy `platform_chat_id` bindings on restart. - -- `a75b26a` — failing restart reconciliation regressions for recovery, idempotence, startup ordering, and legacy backfill -- `8a80d00` — startup reconciliation module and pre-sync wiring in the Matrix runtime - -Verified with `MATRIX_AGENT_REGISTRY_PATH='' MATRIX_PLATFORM_BACKEND='' UV_CACHE_DIR=/tmp/uv-cache-surfaces uv run pytest tests/adapter/matrix/test_invite_space.py tests/adapter/matrix/test_chat_space.py tests/adapter/matrix/test_reconciliation.py tests/adapter/matrix/test_restart_persistence.py tests/adapter/matrix/test_dispatcher.py -v`. - -Plan `05-02` is complete. Matrix room-local context commands now rely on repaired per-room `platform_chat_id` bindings, and `!clear` rotates only the active room's upstream context when prototype room state is available. - -- `ae37476` — failing regressions for clear registration, room-local rotation, and strict routed-platform metadata requirements -- `85e2fda` — room-local clear semantics, compatibility alias wiring, and strict context resolution without shared chat fallbacks - -Verified with `MATRIX_AGENT_REGISTRY_PATH='' MATRIX_PLATFORM_BACKEND='' UV_CACHE_DIR=/tmp/uv-cache-surfaces uv run pytest tests/adapter/matrix/test_context_commands.py tests/adapter/matrix/test_routed_platform.py tests/adapter/matrix/test_dispatcher.py -v`. - -Plan `05-03` is complete. Shared-volume attachment handling now preserves relative agent paths while tolerating both `/workspace` and `/agents` absolute prefixes during normalization and Matrix file rendering. - -- `7a12a71` — failing regressions for shared-volume path normalization and room-safe attachment handling -- `5eddf16` — `/agents` deployment path hardening for Matrix files and routed platform attachments - -Verified with `uv run pytest tests/adapter/matrix/test_files.py tests/platform/test_real.py tests/adapter/matrix/test_send_outgoing.py -v`. - -Plan `05-04` is complete. Production handoff now uses `docker-compose.prod.yml` for a bot-only runtime, while internal end-to-end verification uses `docker-compose.fullstack.yml` with shared `/agents` volume guidance and health-gated startup. - -- `df6d8bf` — split prod and full-stack compose artifacts with the shared `/agents` contract -- `22a3a2b` — operator and deployment docs aligned to the split compose artifacts - -Verified with `docker compose -f docker-compose.prod.yml config`, `docker compose -f docker-compose.fullstack.yml config`, and docs grep checks for `docker-compose.prod.yml`, `docker-compose.fullstack.yml`, and `/agents`. +Проект находится в чистом состоянии для начала нового планирования. Неактуальные легаси фазы и прототипы (Telegram, MockPlatformClient) удалены из Roadmap и трекинга. ## Decisions -- Продолжаем с Threaded Mode несмотря на баги Mac клиента (2026-04-02) -- Invite flow Matrix переведён на idempotent-проверку через `user_meta.space_id`, а не через invite-room metadata (2026-04-02) -- Неизвестные Matrix rooms больше не auto-register в роутере; используется явный fallback `unregistered:{room_id}` с warning-логом (2026-04-02) -- [Phase 01]: Use ChatContext.surface_ref as the Matrix room identifier for !rename updates. -- [Phase 01]: Keep !archive limited to core archive state in Phase 1; Space child removal remains deferred. -- [Phase 01]: Matrix OutgoingUI no longer emits reactions; confirmation state is persisted and resumed via `!yes` / `!no`. -- [Phase 01]: `!settings` now renders a dashboard snapshot instead of advertising mutable subcommands. -- [Phase 01]: Split Matrix regression coverage into dedicated invite/chat/send_outgoing/confirm test modules. -- [Phase 01]: Kept 01-04 scoped to test coverage without widening into production-code changes. -- [Phase 01]: Matrix command callbacks now include room_id in payload for !yes and !no so confirm handlers can resolve runtime state without changing core protocol types. -- [Phase 01]: Pending confirmations are stored under the D-08 composite key of matrix user id plus room id, with a narrow legacy fallback only for callers that omit room context. -- [Phase 01]: Removed Matrix reaction conversion entirely and kept command callbacks limited to !yes/!no. -- [Phase 01]: Kept !settings as a pure snapshot surface while preserving mutable subcommands outside the dashboard. -- [Phase 01]: Seeded invite and dispatcher tests with explicit next_chat_index and room ids instead of treating C1 as Matrix transport identity. -- [Phase 04]: Replaced AgentSessionClient with AgentApiWrapper and persistent agent connection lifecycle in Matrix runtime. -- [Phase 04]: Added !save, !load, !reset, and !context commands with pending-state interception and local prototype session metadata. -- [Phase 04]: Added Matrix-only Docker packaging for MVP deployment; platform services remain external to this compose setup. -- [Phase 04]: Replaced the Matrix prod path again with direct upstream `AgentApi` per request; removed the local runtime wrapper from the prod flow. -- [Phase 04]: Adopted `AGENT_BASE_URL` as the primary runtime contract and kept `AGENT_WS_URL` only as backward-compatible env fallback. -- [Phase 04 follow-up]: Kept shared PlatformClient unchanged; introduced Matrix-specific RoutedPlatformClient to avoid breaking Telegram adapter. -- [Phase 04 follow-up]: agent_routing_enabled flag on MatrixRuntime activates stale-room check only in real multi-agent mode (RoutedPlatformClient). -- [Phase 04 follow-up]: !new binds agent_id at room creation time using selected_agent_id from user metadata. -- [Phase 04 follow-up]: platform_chat_seq (PLATFORM_CHAT_SEQ_KEY) is stored in SQLiteStore and survives restart — confirmed by test. -- [Phase 05 reset]: Discard the single-chat / DM-first deployment direction. Replan around Space+rooms, per-room `platform_chat_id`, real `!clear`, reconciliation, and split prod/fullstack compose artifacts. -- [Phase 05]: Keep adapter/matrix/files.py as the sole path builder; sdk/real.py only normalizes shared-volume attachment references. -- [Phase 05]: Normalize /workspace and /agents absolute file paths back to relative workspace_path values before agent transport and Matrix file rendering. -- [Phase 05]: Treat synced Matrix topology as authoritative for startup recovery; keep SQLite rebuildable. -- [Phase 05]: Backfill missing platform_chat_id values during startup reconciliation before routed handling begins. -- [Phase 05]: Expose `clear` only when prototype room-context support is available, while keeping `reset` as a compatibility alias. -- [Phase 05]: Require recovered `platform_chat_id` for save/context/clear flows instead of falling back to shared local chat ids. -- [Phase 05]: Split Compose artifacts by runtime intent: bot-only prod handoff vs internal full-stack verification. -- [Phase 05]: Document /agents as the bot-side shared volume root while internal platform-agent keeps /workspace on the same named volume. +- **Space+rooms для Matrix**: Разделение тредов на отдельные Matrix-комнаты, собранные в едином Space пользователя. +- **AgentApi**: Прямая интеграция с локальным агентом без Master-прослойки по WebSocket. +- **Shared Volume**: Файлы кладутся напрямую в рабочую папку агента, избавляя от необходимости гонять их по сети в Base64. +- **Статическая маршрутизация**: На данном этапе пользователи маппятся на агентов жестко через YAML. ## Blockers -- Lambda platform SDK не готов — Phase 2 заблокирована до готовности платформы -- Full production verification depends on the platform team's real multi-agent orchestration, production Matrix credentials, `config/matrix-agents.yaml`, and shared `/agents/N` volume mounts. +- Отсутствуют. Проект готов к деплою (см. `docker-compose.prod.yml`). ## Accumulated Context ### Roadmap Evolution -- Phase 01.1 inserted after Phase 01: Matrix restart reconciliation and dev reset workflow (URGENT) -- Phase 4 added: Matrix MVP: shared agent context and context management command -- Phase 04 follow-up added inline: multi-agent routing (RoutedPlatformClient, !agent, stale room blocking, restart persistence) -- Phase 05 reset on 2026-04-28: erroneous single-chat deployment artifacts were removed before fresh planning. - -## Performance Metrics - -| Phase | Plan | Duration | Tasks | Files | Recorded | -| --- | --- | --- | --- | --- | --- | -| 01 | 01 | 1 min | 3 | 3 | 2026-04-02T19:50:50Z | -| 01 | 02 | 1 min | 2 | 2 | 2026-04-02 | -| 01 | 03 | 3 min | 2 | 5 | 2026-04-02T19:57:34Z | -| 01 | 04 | 3 min | 2 | 7 | 2026-04-02T20:03:38Z | -| 01 | 05 | 2 min | 2 | 7 | 2026-04-03T09:28:47Z | -| 01 | 06 | 4 min | 2 | 7 | 2026-04-03T09:35:47Z | -| 04 | 01 | 1 session | 1 wave | 8 | 2026-04-17 | -| 04 | 02 | 1 session | 2 commits + summary | 8 | 2026-04-17 | -| 04 | 03 | 1 session | 1 commit + summary | 4 | 2026-04-17 | -| 04 | follow-up | 1 session | 5 tasks | 10+ | 2026-04-24 | -| 05 | 03 | 3 min | 2 | 3 | 2026-04-27T22:06:43Z | -| 05 | 01 | 8 min | 2 | 4 | 2026-04-27T22:09:28Z | -| 05 | 02 | 16 min | 2 | 4 | 2026-04-27T22:15:58Z | -| 05 | 04 | 3 min | 2 | 5 | 2026-04-27T22:17:10Z | - -## Session - -- Last session: 2026-04-29T08:49:04Z -- Stopped at: Handoff updated after attachment workspace-root change; waiting for image rebuild and platform redeploy -- Resume file: .planning/phases/05-mvp-deployment/.continue-here.md +- Изначальный Roadmap включал множество ответвлений (прототипы Telegram, локальный mock-клиент). После закрепления MVP в `main` Roadmap был очищен, чтобы отражать только актуальный путь продукта. +- Следующие фазы будут добавляться по мере возникновения новых задач (например, переход от YAML-конфига к БД для реестра агентов). diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.continue-here.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.continue-here.md deleted file mode 100644 index 6de8f62..0000000 --- a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.continue-here.md +++ /dev/null @@ -1,63 +0,0 @@ ---- -phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow -task: 1 -total_tasks: 2 -status: paused -last_updated: 2026-04-07T21:29:48.982Z ---- - - -Formally, the most recently active execution artifact inside the roadmap is still `01.1-03-PLAN.md`, which has not been implemented yet. In parallel, the platform-integration track has moved forward: the direct-agent Matrix prototype design is now approved, the implementation plan is written, and the next useful session should evaluate that spec/plan pair against the live platform repos before starting execution. - - - - -- Re-analysed live platform repos on 2026-04-07 by cloning `platform/agent`, `platform/agent_api`, `platform/master`, and `platform/docs`. -- Confirmed `master` is still only a thin HTTP skeleton with `/health` and `/users/{user_id}`, not a chat/session/settings backend for surfaces. -- Confirmed `agent` exposes a working `/agent_ws/` WebSocket and `agent_api` provides enough protocol/client code to stream model output. -- Identified the real technical gap for a prototype: `agent` currently uses a singleton service with a fixed `thread_id="default"`, so all conversations would share memory unless that is parameterized. -- Derived and got approval for the prototype path: keep Matrix adapter logic largely intact, add `sdk/agent_session.py`, `sdk/prototype_state.py`, and `sdk/real.py`, keep settings local, and use the direct `agent` WebSocket for real messaging. -- Resolved the repo-placement question: the prototype stays in this repo on its own branch, not in a separate prototype repo. -- Resolved the platform-change minimization question: prefer patching only `platform/agent`, not `platform/agent_api`, and use a tiny local WebSocket client in this repo. -- Wrote and committed the approved design spec: `docs/superpowers/specs/2026-04-08-matrix-direct-agent-prototype-design.md`. -- Wrote the implementation plan: `docs/superpowers/plans/2026-04-08-matrix-direct-agent-prototype.md`. - - - - -- Task 1: Implement `adapter.matrix.reset` with `local-only`, `server-leave-forget`, and `--dry-run`, plus tests. -- Task 2: Update `README.md` so restart vs explicit reset workflow is documented and the old manual reset ritual is removed. -- Prototype evaluation follow-up: review the approved spec and plan against the platform repos before starting execution. -- Future prototype work: introduce `sdk/real.py` plus a narrow compatibility boundary that keeps Matrix adapter logic stable while allowing later expansion toward a fuller platform split. - - - - -- Do not integrate with `master` yet; it is still not the backend surfaces needs. -- Use the direct `agent` WebSocket as the only realistic path for a working prototype right now. -- Keep consumer-facing Matrix logic as intact as possible and absorb backend differences inside a shim under `sdk/`. -- Treat future platform evolution as likely to split into at least two concerns: control-plane access and direct agent session streaming. -- Keep the prototype in this repo on its own branch. -- Minimize platform-side changes by patching only `platform/agent` if possible. - - - -- Phase 01.1 itself is not blocked; it is simply paused. -- Prototype blocker: the `agent` repo currently hardcodes a shared `thread_id`, so per-user/per-chat conversation isolation requires either a small upstream change or a careful workaround. -- Platform contract blocker remains for the longer-term Phase 02 direction: `master` still lacks stable user/chat/session/settings APIs for surfaces. - - - -The important mental model is now stable enough to execute. Full SDK integration through `master` is still premature, but a working Matrix prototype can be built now by talking directly to the `agent` WebSocket and hiding the split backend reality behind `sdk/real.py`. The approved design keeps the prototype in this repo, keeps settings local, and minimizes platform changes by preferring a tiny `platform/agent` patch over broader protocol churn. For evaluation and implementation context, inspect: -- local spec: `docs/superpowers/specs/2026-04-08-matrix-direct-agent-prototype-design.md` -- local plan: `docs/superpowers/plans/2026-04-08-matrix-direct-agent-prototype.md` -- remote repos: `https://git.lambda.coredump.ru/platform/agent`, `https://git.lambda.coredump.ru/platform/master`, `https://git.lambda.coredump.ru/platform/agent_api` -- local clones: `/tmp/platform-agent`, `/tmp/platform-master`, `/tmp/platform-agent_api` - - - -Resume with one of these depending on priority: -1. Evaluate the approved prototype spec and implementation plan against the live platform repos and decide whether to start in this repo or patch `platform/agent` first. -2. If staying on roadmap execution, implement `01.1-03-PLAN.md` Task 1 (`adapter.matrix.reset`) first. -3. If starting prototype execution immediately, begin with Task 1 of `docs/superpowers/plans/2026-04-08-matrix-direct-agent-prototype.md`. - diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.gitkeep b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md deleted file mode 100644 index 187baa9..0000000 --- a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow -plan: 01 -type: execute -wave: 1 -depends_on: [] -files_modified: - - adapter/matrix/reconcile.py - - tests/adapter/matrix/test_reconcile.py -autonomous: true -requirements: [] - -must_haves: - truths: - - "A normal Matrix restart can rebuild missing local metadata from already joined Space/chat rooms instead of requiring a destructive reset." - - "Reconciliation restores the minimal local state needed for routing and chat operations: `matrix_user:*`, `matrix_room:*`, and missing `chat:{user}:{chat_id}` rows." - - "Reconciliation never provisions new Matrix rooms or Spaces while repairing local state." - - "Recovered users get `next_chat_index` advanced past the highest recovered `C*` chat id." - artifacts: - - path: "adapter/matrix/reconcile.py" - provides: "Matrix bootstrap reconciliation helpers and structured report objects." - - path: "tests/adapter/matrix/test_reconcile.py" - provides: "Regression coverage for startup and single-room reconciliation behavior." - key_links: - - from: "adapter/matrix/reconcile.py" - to: "adapter/matrix/store.py" - via: "set_user_meta and set_room_meta restore Matrix metadata" - pattern: "set_(user|room)_meta" - - from: "adapter/matrix/reconcile.py" - to: "core/chat.py" - via: "chat_mgr.get_or_create repairs missing `chat:*` rows" - pattern: "chat_mgr\\.get_or_create" ---- - - -Create the non-destructive Matrix reconciliation layer that Phase 01.1 depends on. - -Purpose: Per D-01 through D-07, the adapter must stop treating local SQLite state as the only truth in dev. Startup and recovery code need a single helper module that can rebuild local metadata from the homeserver room graph without creating duplicate Spaces or chats. -Output: `adapter/matrix/reconcile.py` with full-run and single-room recovery helpers, plus targeted pytest coverage. - - - -@/Users/a/.codex/get-shit-done/workflows/execute-plan.md -@/Users/a/.codex/get-shit-done/templates/summary.md - - - -@.planning/PROJECT.md -@.planning/ROADMAP.md -@.planning/STATE.md -@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md -@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md -@.planning/phases/01-matrix-qa-polish/01-01-SUMMARY.md -@adapter/matrix/store.py -@adapter/matrix/handlers/auth.py -@core/chat.py -@tests/adapter/matrix/test_invite_space.py - - -From `adapter/matrix/store.py`: - -```python -async def get_room_meta(store: StateStore, room_id: str) -> dict | None -async def set_room_meta(store: StateStore, room_id: str, meta: dict) -> None -async def get_user_meta(store: StateStore, matrix_user_id: str) -> dict | None -async def set_user_meta(store: StateStore, matrix_user_id: str, meta: dict) -> None -``` - -From `core/chat.py`: - -```python -async def get_or_create( - self, - user_id: str, - chat_id: str, - platform: str, - surface_ref: str, - name: str | None = None, -) -> ChatContext -``` - -From Phase 01 room metadata shape: - -```python -{ - "room_type": "chat", - "chat_id": "C4", - "display_name": "Чат 4", - "matrix_user_id": "@alice:example.org", - "space_id": "!space:example.org", -} -``` - - - - - - - Task 1: Add reconciliation module for startup and single-room recovery - adapter/matrix/reconcile.py, tests/adapter/matrix/test_reconcile.py - adapter/matrix/store.py, adapter/matrix/handlers/auth.py, core/chat.py, tests/adapter/matrix/test_invite_space.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md - - - Test 1: `reconcile_matrix_state(...)` recreates missing `matrix_user:*`, `matrix_room:*`, and `chat:*` entries from joined Matrix rooms without calling `room_create`. - - Test 2: `reconcile_matrix_state(...)` leaves already-correct local metadata intact and reports restored vs skipped/conflicting rooms. - - Test 3: `reconcile_single_room(...)` can repair one `unregistered:{room_id}` chat room on demand and recompute `next_chat_index` for that user. - - Test 4: Space rooms or unrelated joined rooms are skipped, not converted into chat rows. - - -Create `adapter/matrix/reconcile.py` as the authoritative recovery module for this phase. Implement a small, explicit API that Plan 02 can wire directly: - -```python -async def reconcile_matrix_state(client: Any, store: StateStore, chat_mgr: ChatManager) -> dict: ... -async def reconcile_single_room( - client: Any, store: StateStore, chat_mgr: ChatManager, room_id: str, matrix_user_id: str -) -> dict: ... -``` - -Inside this module, add focused private helpers as needed for room classification, extracting room names, parsing `C` ids, and recomputing `next_chat_index`. Keep the logic non-destructive per D-04: -- never call `room_create`, `room_invite`, or provisioning code from `handlers/auth.py` -- prefer already-hydrated room data from the post-sync client object, and only fall back to explicit room-state fetches if required for room classification -- rebuild only the minimal metadata required by D-03: `matrix_user:*`, `matrix_room:*`, and missing `chat:{user}:{chat_id}` records -- if `chat:*` exists but points at the wrong `surface_ref`, repair it from Matrix room metadata and include the fix in the returned report -- derive `next_chat_index` from the highest recovered `C` for that user instead of trusting stale local counters - -Return a structured reconciliation report with stable keys such as: -`joined_rooms`, `restored_user_meta`, `restored_room_meta`, `restored_chat_rows`, `repaired_chat_rows`, `skipped_rooms`, and `conflicts`. - -Write `tests/adapter/matrix/test_reconcile.py` with lightweight `SimpleNamespace`/fake-client fixtures following the existing Matrix test style. Cover both full startup reconciliation and `reconcile_single_room(...)`. Assert that no provisioning calls are made during reconciliation, because D-04 forbids creating new Space/room topology while recovering local state. - - - cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_reconcile.py -q - - - - `adapter/matrix/reconcile.py` exports `reconcile_matrix_state` and `reconcile_single_room`. - - Reconciliation restores missing `matrix_user:*`, `matrix_room:*`, and `chat:*` entries for already-joined Matrix chat rooms per D-02 and D-03. - - Reconciliation does not call `room_create` or otherwise provision new server-side rooms per D-04. - - The report returned by reconciliation clearly distinguishes restored items, skipped rooms, and conflicts. - - `tests/adapter/matrix/test_reconcile.py` proves `next_chat_index` is recomputed from recovered chat ids rather than stale local state. - - The repository has an executable, tested reconciliation layer that can rebuild local Matrix metadata after dev-state loss without duplicating server-side rooms. - - - - - -Run `pytest tests/adapter/matrix/test_reconcile.py -q` and confirm startup and single-room reconciliation paths are covered. - - - -- Matrix recovery logic exists as a dedicated module instead of being scattered through handlers. -- Reconciliation is idempotent, non-destructive, and sufficient to restore routing/chat metadata from existing Matrix rooms. -- Plan 02 can wire startup and first-access recovery by calling exported functions rather than inventing new recovery logic. - - - -After completion, create `.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-SUMMARY.md` - diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-PLAN.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-PLAN.md deleted file mode 100644 index bdfdaf8..0000000 --- a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-PLAN.md +++ /dev/null @@ -1,167 +0,0 @@ ---- -phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow -plan: 02 -type: execute -wave: 2 -depends_on: ["01.1-01"] -files_modified: - - adapter/matrix/bot.py - - tests/adapter/matrix/test_dispatcher.py -autonomous: true -requirements: [] - -must_haves: - truths: - - "The Matrix bot performs an initial sync and reconciliation before entering steady-state `sync_forever()`." - - "If a room still arrives as `unregistered:{room_id}` after startup, the bot makes one targeted recovery attempt before dispatching or failing." - - "When reconciliation cannot repair a room, the bot logs a clear diagnostic reason instead of crashing on downstream commands like `!rename`." - artifacts: - - path: "adapter/matrix/bot.py" - provides: "Startup bootstrap flow with initial sync, reconciliation, and targeted runtime retry." - - path: "tests/adapter/matrix/test_dispatcher.py" - provides: "Matrix runtime coverage for pre-sync reconcile and on-message recovery behavior." - key_links: - - from: "adapter/matrix/bot.py" - to: "adapter/matrix/reconcile.py" - via: "startup bootstrap and single-room recovery calls" - pattern: "reconcile_(matrix_state|single_room)" - - from: "adapter/matrix/bot.py" - to: "adapter/matrix/room_router.py" - via: "unregistered room detection before dispatch" - pattern: "unregistered:" ---- - - -Wire the new reconciliation layer into the actual Matrix runtime. - -Purpose: D-05 through D-07 require restart recovery to be the default developer path. The bot must bootstrap itself from existing Matrix rooms on startup and make one on-demand repair attempt before routing an unknown room through the dispatcher. -Output: `adapter/matrix/bot.py` performs initial sync + reconciliation before `sync_forever()`, and runtime tests prove the bot recovers or logs clearly instead of blindly dispatching broken state. - - - -@/Users/a/.codex/get-shit-done/workflows/execute-plan.md -@/Users/a/.codex/get-shit-done/templates/summary.md - - - -@.planning/PROJECT.md -@.planning/ROADMAP.md -@.planning/STATE.md -@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md -@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md -@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-01-PLAN.md -@adapter/matrix/bot.py -@adapter/matrix/room_router.py -@adapter/matrix/reconcile.py -@tests/adapter/matrix/test_dispatcher.py - - -From `adapter/matrix/bot.py`: - -```python -class MatrixBot: - async def on_room_message(self, room: MatrixRoom, event: RoomMessageText) -> None - -async def main() -> None -``` - -From `adapter/matrix/reconcile.py`: - -```python -async def reconcile_matrix_state(client: Any, store: StateStore, chat_mgr: ChatManager) -> dict -async def reconcile_single_room( - client: Any, store: StateStore, chat_mgr: ChatManager, room_id: str, matrix_user_id: str -) -> dict -``` - -From `adapter/matrix/room_router.py`: - -```python -async def resolve_chat_id(store: StateStore, room_id: str, matrix_user_id: str) -> str -``` - - - - - - - Task 1: Run initial sync and reconciliation before the long-poll loop - adapter/matrix/bot.py, tests/adapter/matrix/test_dispatcher.py - adapter/matrix/bot.py, adapter/matrix/reconcile.py, tests/adapter/matrix/test_dispatcher.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md - - - Test 1: `main()` performs `client.sync(timeout=0, full_state=True)` before `sync_forever()`. - - Test 2: `main()` calls `reconcile_matrix_state(...)` after the initial sync and logs the returned report. - - Test 3: startup still reaches `sync_forever()` when reconciliation reports recoverable skips/conflicts instead of fatal failure. - - -Modify `adapter/matrix/bot.py` so normal startup follows the two-phase bootstrap recommended in research: -1. build client and runtime -2. authenticate -3. register callbacks -4. run `await client.sync(timeout=0, full_state=True)` -5. run `await reconcile_matrix_state(client, runtime.store, runtime.chat_mgr)` -6. log a structured `matrix_reconcile_complete` event with the report fields -7. enter `await client.sync_forever(timeout=30000)` - -Do not move provisioning logic into startup. The startup step only rehydrates local state from server-side rooms per D-02 through D-04. - -Update or add focused tests in `tests/adapter/matrix/test_dispatcher.py` using `monkeypatch`/fake-client patterns already used in the repo so the verify command proves the call order and logging-safe behavior. The test should fail if `sync_forever()` starts before reconciliation. - - - cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_dispatcher.py -q - - - - `adapter/matrix/bot.py` runs an initial full-state sync before steady-state polling. - - `adapter/matrix/bot.py` invokes `reconcile_matrix_state(...)` exactly once during startup. - - Startup logs a structured reconciliation summary instead of silently skipping the recovery step. - - `tests/adapter/matrix/test_dispatcher.py` asserts the bootstrap order explicitly. - - Normal Matrix bot startup now includes a recovery pass before the event loop begins handling user traffic. - - - - Task 2: Retry unknown-room routing once before dispatching broken state - adapter/matrix/bot.py, tests/adapter/matrix/test_dispatcher.py - adapter/matrix/bot.py, adapter/matrix/room_router.py, adapter/matrix/reconcile.py, tests/adapter/matrix/test_dispatcher.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md - - - Test 1: `MatrixBot.on_room_message(...)` detects `unregistered:{room_id}`, runs `reconcile_single_room(...)`, then retries `resolve_chat_id(...)`. - - Test 2: if retry succeeds, the event is dispatched against the recovered logical chat id. - - Test 3: if retry still fails, the bot does not crash; it logs a clear warning and sends a user-facing diagnostic message to that room. - - -Extend `MatrixBot.on_room_message(...)` so D-07 is satisfied even when startup could not repair a room yet. Keep `resolve_chat_id(...)` as the room-router source of truth, but treat `unregistered:{room_id}` as a recovery trigger rather than a stable runtime identity: -- first call `resolve_chat_id(...)` -- if the result starts with `unregistered:`, call `reconcile_single_room(client, runtime.store, runtime.chat_mgr, room.room_id, event.sender)` -- immediately retry `resolve_chat_id(...)` -- only dispatch once a concrete logical chat id exists -- if the retry still returns `unregistered:{room_id}`, log a structured warning with room id, matrix user id, and reconciliation report, then send a short `OutgoingMessage`-equivalent Matrix text explaining that local state could not be restored automatically and a dev reset/restart may be required - -Do not invent a new fallback chat id and do not auto-create rooms here; that would violate D-04. Keep this change inside `adapter/matrix/bot.py` so file ownership stays isolated for this plan. - - - cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_dispatcher.py -q - - - - Unknown Matrix rooms trigger one targeted reconciliation attempt before dispatch. - - Successful targeted recovery leads to normal dispatch with a real logical `chat_id`. - - Failed targeted recovery logs a clear diagnostic and avoids a handler crash on missing chat state per D-06. - - No code path in this task provisions new Matrix rooms or Spaces. - - The runtime treats unknown rooms as recoverable state drift first, not as a silent routing failure or crash path. - - - - - -Run `pytest tests/adapter/matrix/test_dispatcher.py -q` and confirm both startup-bootstrap and first-access recovery behaviors are covered. - - - -- A standard Matrix restart now attempts recovery before the bot starts processing live events. -- Unknown-room events are diagnosable and recoverable instead of falling straight into broken command handling. -- The runtime never provisions new server-side rooms during restart reconciliation. - - - -After completion, create `.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-02-SUMMARY.md` - diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-PLAN.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-PLAN.md deleted file mode 100644 index bd78891..0000000 --- a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-PLAN.md +++ /dev/null @@ -1,149 +0,0 @@ ---- -phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow -plan: 03 -type: execute -wave: 1 -depends_on: [] -files_modified: - - adapter/matrix/reset.py - - tests/adapter/matrix/test_reset.py - - README.md -autonomous: true -requirements: [] - -must_haves: - truths: - - "Developers have an explicit dev-only reset command instead of relying on memory or ad hoc shell history." - - "The default reset mode clears only local Matrix state and explains the manual Matrix-client cleanup that may still be needed." - - "Optional server cleanup is clearly named around leave/forget semantics and supports dry-run output." - artifacts: - - path: "adapter/matrix/reset.py" - provides: "Dev reset CLI for local-only, server-leave-forget, and dry-run workflows." - - path: "tests/adapter/matrix/test_reset.py" - provides: "CLI coverage for local reset behavior and printed operator guidance." - - path: "README.md" - provides: "Updated developer instructions for normal restart vs explicit reset." - key_links: - - from: "adapter/matrix/reset.py" - to: "README.md" - via: "documented invocation and manual Matrix cleanup guidance" - pattern: "adapter\\.matrix\\.reset" ---- - - -Ship the dev reset workflow that complements normal restart reconciliation. - -Purpose: D-08 through D-10 require a repeatable, explicit reset path for clean-room QA without making destructive cleanup the default restart flow. This plan creates the tool and updates the runbook developers actually use. -Output: `adapter/matrix/reset.py`, pytest coverage, and README instructions that replace the old `rm -f lambda_matrix.db` ritual. - - - -@/Users/a/.codex/get-shit-done/workflows/execute-plan.md -@/Users/a/.codex/get-shit-done/templates/summary.md - - - -@.planning/PROJECT.md -@.planning/ROADMAP.md -@.planning/STATE.md -@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md -@.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md -@README.md -@adapter/matrix/bot.py -@core/store.py - - -From `adapter/matrix/bot.py` env usage: - -```python -db_path = os.environ.get("MATRIX_DB_PATH", "lambda_matrix.db") -store_path = os.environ.get("MATRIX_STORE_PATH", "matrix_store") -homeserver = os.environ.get("MATRIX_HOMESERVER") -user_id = os.environ.get("MATRIX_USER_ID") -``` - -From `core/store.py`: - -```python -class SQLiteStore: - def __init__(self, db_path: str) -> None: ... -``` - - - - - - - Task 1: Add a dev-only Matrix reset CLI with explicit modes - adapter/matrix/reset.py, tests/adapter/matrix/test_reset.py - adapter/matrix/bot.py, core/store.py, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md - - - Test 1: `--mode local-only` deletes the configured local DB/store paths or reports what would be deleted in dry-run mode. - - Test 2: `--mode server-leave-forget --dry-run` prints the exact rooms it would leave/forget and does not mutate local files. - - Test 3: when server cleanup is not executed, the command prints the manual Matrix-client steps required by D-10. - - -Create `adapter/matrix/reset.py` as a CLI entrypoint runnable via `uv run python -m adapter.matrix.reset`. Use `argparse` and keep the tool explicitly dev-only in its help text and logs. - -Implement the following modes from research and locked decisions: -- `local-only` (default destructive mode for local QA): remove `MATRIX_DB_PATH` and `MATRIX_STORE_PATH` if they exist; if not, report that they were already absent -- `server-leave-forget`: for the bot account only, log in using the same Matrix env vars as `adapter/matrix/bot.py`, inspect joined rooms, and call `room_leave()` + `room_forget()` for each joined room; support `--dry-run` so the operator can inspect the target set before mutation -- `--dry-run` must work with both modes and print a structured summary instead of mutating files or Matrix membership - -Always print a post-run summary that distinguishes: -- what local files/directories were deleted or would be deleted -- what server-side leave/forget actions were executed or would be executed -- the manual Matrix client steps still required for a true clean-room QA rerun (leave/archive old rooms or Space in Element, accept fresh invites, etc.) when those actions are outside this phase - -Write `tests/adapter/matrix/test_reset.py` to cover local-only deletion, dry-run output, and server-leave-forget dry-run behavior with fake clients/temporary directories. Follow the repo’s existing lightweight async test style. - - - cd /Users/a/MAI/sem2/lambda/surfaces-bot && pytest tests/adapter/matrix/test_reset.py -q - - - - `adapter/matrix/reset.py` supports `local-only`, `server-leave-forget`, and `--dry-run`. - - `local-only` reset targets both `lambda_matrix.db` and `matrix_store` via env-aware paths per D-09. - - The tool never claims to globally delete Matrix rooms; it uses leave/forget semantics or prints manual cleanup instructions per D-10. - - `tests/adapter/matrix/test_reset.py` proves dry-run mode is non-destructive. - - The repository contains a repeatable dev reset tool that replaces the undocumented shell ritual and names server-side cleanup honestly. - - - - Task 2: Replace the README reset ritual with the new restart and reset workflow - README.md - README.md, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md, .planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md - -Update `README.md` so Matrix development instructions reflect Phase 01.1 instead of the old destructive reset ritual. Replace the current manual QA block that tells developers to `rm -f lambda_matrix.db` with a short, explicit split: -- normal restart: `PYTHONPATH=. uv run python -m adapter.matrix.bot` now performs reconciliation automatically -- explicit clean-room reset: `PYTHONPATH=. uv run python -m adapter.matrix.reset --mode local-only` -- optional server cleanup preview: `PYTHONPATH=. uv run python -m adapter.matrix.reset --mode server-leave-forget --dry-run` - -State clearly that normal restart is the default path per D-05, and that full server-side cleanup may still require manual steps in the Matrix client. Keep the README concise; do not add production guidance or Phase 2 SDK content. - - - cd /Users/a/MAI/sem2/lambda/surfaces-bot && python -m adapter.matrix.reset --help >/tmp/matrix-reset-help.txt && rg -n "adapter.matrix.reset|local-only|server-leave-forget|reconciliation" README.md /tmp/matrix-reset-help.txt - - - - `README.md` no longer recommends raw `rm -f lambda_matrix.db` as the default Matrix restart workflow. - - `README.md` documents the normal restart path and the explicit reset path separately. - - The documented reset commands match the CLI implemented in `adapter/matrix/reset.py`. - - Developers can follow a repeatable README workflow for ordinary restart and clean-room QA reset without relying on tribal knowledge. - - - - - -Run `pytest tests/adapter/matrix/test_reset.py -q` and `python -m adapter.matrix.reset --help`, then confirm the README commands and help text stay aligned. - - - -- Dev reset is an explicit tool, not a remembered shell sequence. -- Local-only reset is automated and documented. -- Server cleanup semantics are honest, dry-runnable, and accompanied by manual Matrix-client guidance where needed. - - - -After completion, create `.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-03-SUMMARY.md` - diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md deleted file mode 100644 index 665061e..0000000 --- a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-CONTEXT.md +++ /dev/null @@ -1,121 +0,0 @@ -# Phase 01.1: Matrix restart reconciliation and dev reset workflow - Context - -**Gathered:** 2026-04-03 -**Status:** Ready for planning - - -## Phase Boundary - -Сделать Matrix-адаптер пригодным для повторяемой локальной разработки и ручного QA без ручного “ритуала” удаления БД, выхода из всех комнат и пересоздания пользователя. - -В scope этой фазы: -- безопасный restart flow для Matrix-бота после потери локального state -- reconciliation локального store с уже существующими Matrix rooms / Space -- отдельный dev reset workflow для controlled clean-room QA -- диагностируемое поведение при несогласованности local state и server-side Matrix state - -Вне scope: -- реальный Lambda SDK -- новые пользовательские Matrix features -- E2EE -- production-grade multi-user migration framework - - - - -## Implementation Decisions - -### Matrix state lifecycle - -- **D-01:** Локальный SQLite store больше не должен считаться единственной точкой истины для Matrix runtime в dev workflow. -- **D-02:** При старте бот должен пытаться восстановить минимально необходимое локальное состояние из уже существующих Matrix rooms / Space, а не требовать full reset. -- **D-03:** Reconciliation должен восстанавливать как минимум `matrix_user:*`, `matrix_room:*` и missing `chat:{user}:{chat_id}` записи, если серверные комнаты уже существуют. -- **D-04:** Reconciliation не должен создавать новые Space/rooms, если задача — именно восстановление локального state после рестарта. - -### Dev restart behavior - -- **D-05:** Обычный restart бота должен быть основным путём для разработки; удаление `lambda_matrix.db` и `matrix_store` не должно быть обязательным для проверки workflow. -- **D-06:** Если local state неполон, бот должен либо восстановить его, либо логировать понятную причину, а не падать на командах вроде `!rename`. -- **D-07:** Несогласованность между `room_meta` и `ChatManager` должна обнаруживаться и устраняться автоматически на startup или при первом обращении. - -### Dev reset workflow - -- **D-08:** Нужен отдельный dev-only reset tool/script для controlled QA, вместо ручного набора shell-команд. -- **D-09:** Reset workflow должен как минимум поддерживать `local-only` reset: удаление `lambda_matrix.db` и `matrix_store` с понятной инструкцией, что делать с server-side Matrix rooms. -- **D-10:** Если full server-side cleanup не автоматизируется в этой фазе, tool должен явно печатать, какие ручные шаги обязательны в Matrix client. - -### The agent's Discretion - -- Точное место вызова reconciliation в startup flow -- Внутренняя структура helper-модуля (`bootstrap.py`, `reconcile.py` или аналог) -- Формат dev reset script и уровень автоматизации server-side cleanup -- Детали debug-logging и dry-run режима, если они помогают без раздувания scope - - - - -## Specific Ideas - -- Главный критерий: после обычного restart бот не должен ломаться только потому, что local DB была сброшена или частично потеряна. -- Reset workflow должен быть явным и repeatable, а не завязанным на память разработчика. -- Нужно различать две ситуации: - - broken because code is wrong - - broken because local dev state was deliberately reset and requires reconciliation - - - - -## Canonical References - -**Downstream agents MUST read these before planning or implementing.** - -### Matrix phase artifacts -- `.planning/phases/01-matrix-qa-polish/01-CONTEXT.md` — locked Matrix decisions from Phase 1 -- `.planning/phases/01-matrix-qa-polish/01-VERIFICATION.md` — what Phase 1 already validated and what manual QA still expects -- `.planning/phases/01-matrix-qa-polish/01-HUMAN-UAT.md` — remaining real-client Matrix checks - -### Current Matrix runtime -- `adapter/matrix/bot.py` — startup flow, sync loop, runtime wiring, DB/store env vars -- `adapter/matrix/store.py` — persisted Matrix metadata and pending confirmation keys -- `adapter/matrix/room_router.py` — room_id to chat_id resolution and current unregistered fallback -- `adapter/matrix/handlers/auth.py` — invite bootstrap that creates Space and first chat room -- `core/chat.py` — `ChatManager` persistence contract that currently breaks when local state is missing - -### Supporting docs -- `docs/matrix-prototype.md` — intended Matrix UX and architecture direction -- `README.md` — current run instructions and existing manual QA/reset habits - - - - -## Existing Code Insights - -### Reusable Assets -- `adapter/matrix/store.py` already persists room/user metadata and is the obvious place to anchor reconciliation inputs. -- `adapter/matrix/room_router.py` already detects unknown rooms via `unregistered:{room_id}` fallback; this is a useful reconciliation trigger point. -- `core/chat.py` already has `get_or_create`, `rename`, `archive`, `list_active`; missing chat records can be rebuilt through this API instead of inventing a parallel format. - -### Established Patterns -- Matrix runtime uses `SQLiteStore` for adapter-local metadata and `matrix-nio` room callbacks for transport events. -- Phase 1 already moved Matrix to Space+rooms and command-only confirmations, so this phase must preserve that model rather than reverting to DM-first simplifications. - -### Integration Points -- Startup path in `adapter/matrix/bot.py:main()` is the natural place to run reconciliation before `sync_forever`. -- Invite/bootstrap path in `adapter/matrix/handlers/auth.py` is the existing source of truth for what metadata a healthy first room should have. -- `ChatManager` records and `matrix_room:*` metadata must stay consistent enough that commands like `!rename`, `!archive`, and `!chats` work after restart. - - - - -## Deferred Ideas - -- Full production-grade migration of historical Matrix state across schema versions -- Automatic server-side deletion/leave for all Matrix rooms and Space during reset, if it requires broader admin semantics -- Any Phase 2 SDK integration work - - - ---- - -*Phase: 01.1-matrix-restart-reconciliation-and-dev-reset-workflow* -*Context gathered: 2026-04-03* diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md deleted file mode 100644 index 792031d..0000000 --- a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-RESEARCH.md +++ /dev/null @@ -1,350 +0,0 @@ -# Phase 01.1: Matrix restart reconciliation and dev reset workflow - Research - -**Researched:** 2026-04-03 -**Domain:** Matrix adapter restart reconciliation, local state recovery, dev reset workflow -**Confidence:** HIGH - - -## User Constraints (from CONTEXT.md) - -### Locked Decisions -- **D-01:** Локальный SQLite store больше не должен считаться единственной точкой истины для Matrix runtime в dev workflow. -- **D-02:** При старте бот должен пытаться восстановить минимально необходимое локальное состояние из уже существующих Matrix rooms / Space, а не требовать full reset. -- **D-03:** Reconciliation должен восстанавливать как минимум `matrix_user:*`, `matrix_room:*` и missing `chat:{user}:{chat_id}` записи, если серверные комнаты уже существуют. -- **D-04:** Reconciliation не должен создавать новые Space/rooms, если задача — именно восстановление локального state после рестарта. -- **D-05:** Обычный restart бота должен быть основным путём для разработки; удаление `lambda_matrix.db` и `matrix_store` не должно быть обязательным для проверки workflow. -- **D-06:** Если local state неполон, бот должен либо восстановить его, либо логировать понятную причину, а не падать на командах вроде `!rename`. -- **D-07:** Несогласованность между `room_meta` и `ChatManager` должна обнаруживаться и устраняться автоматически на startup или при первом обращении. -- **D-08:** Нужен отдельный dev-only reset tool/script для controlled QA, вместо ручного набора shell-команд. -- **D-09:** Reset workflow должен как минимум поддерживать `local-only` reset: удаление `lambda_matrix.db` и `matrix_store` с понятной инструкцией, что делать с server-side Matrix rooms. -- **D-10:** Если full server-side cleanup не автоматизируется в этой фазе, tool должен явно печатать, какие ручные шаги обязательны в Matrix client. - -### Claude's Discretion -- Точное место вызова reconciliation в startup flow -- Внутренняя структура helper-модуля (`bootstrap.py`, `reconcile.py` или аналог) -- Формат dev reset script и уровень автоматизации server-side cleanup -- Детали debug-logging и dry-run режима, если они помогают без раздувания scope - -### Deferred Ideas (OUT OF SCOPE) -- Full production-grade migration of historical Matrix state across schema versions -- Automatic server-side deletion/leave for all Matrix rooms and Space during reset, if it requires broader admin semantics -- Any Phase 2 SDK integration work - - -## Summary - -Phase 01.1 should be planned as a bootstrap/recovery phase, not as another chat-feature phase. The current Matrix adapter has no startup reconciliation path: `adapter/matrix/bot.py` logs in and goes directly to `sync_forever()`, while routing and command handlers assume `matrix_room:*`, `matrix_user:*`, and `chat:*` keys already exist. That means local DB loss currently produces logical corruption, not just missing cache. - -The safe standard approach is: perform a first sync that hydrates joined-room state, inspect the bot's current joined rooms and room state from the homeserver, rebuild the minimal local metadata needed for command routing, and only then enter the long-running sync loop. Reconciliation should be non-destructive and idempotent: if local keys already exist and match server state, leave them alone; if they are missing, recreate them; if they conflict, prefer the server room topology for Matrix-specific metadata and recreate missing `ChatManager` rows from that. - -For reset, separate two workflows explicitly. `local-only` reset is the default and should be automated. Optional server-side cleanup may leave/forget rooms for the bot account, but it cannot promise global deletion of Matrix rooms for all members; if that is not automated, the tool must print the exact manual steps for the Matrix client. - -**Primary recommendation:** Add a startup `reconcile_matrix_state()` step before `sync_forever()`, and ship a dev-only reset CLI with `local-only`, `server-leave-forget`, and `dry-run` modes. - -## Project Constraints (from CLAUDE.md) - -- Do not treat missing Lambda SDK as a blocker. -- Keep all platform calls behind `platform/interface.py`. -- Current runtime implementation is `platform/mock.py`; recommendations must work with that. -- Prefer architecture changes in adapters and core without coupling to future SDK internals. -- Use pytest-based verification. -- Do not recommend committing `.env`. -- Respect dependency order: `core/` first, then `platform/`, then adapters. - -## Standard Stack - -### Core -| Library | Version | Purpose | Why Standard | -|---------|---------|---------|--------------| -| Python | 3.14.3 installed | Runtime for bot and scripts | Already available locally; codebase targets `>=3.11`. | -| `matrix-nio` | 0.25.2, published 2024-10-04 | Matrix client, sync, room membership/state APIs | Already installed; exposes the exact bootstrap/reset APIs this phase needs. | -| `SQLiteStore` (repo) | local | Adapter/core KV persistence | Existing persistence contract for `matrix_user:*`, `matrix_room:*`, and `chat:*`. | -| Matrix Client-Server API | spec latest | Authoritative room membership/state semantics | Needed to reason about restart recovery and leave/forget behavior correctly. | - -### Supporting -| Library | Version | Purpose | When to Use | -|---------|---------|---------|-------------| -| `pytest` | 9.0.2, published 2025-12-06 | Test runner | For targeted adapter/bootstrap regression tests. | -| `pytest-asyncio` | 1.3.0, published 2025-11-10 | Async test execution | For async reconciliation/reset flows. | -| `structlog` | 25.5.0, published 2025-10-27 | Diagnostics | For reconciliation summaries and conflict logging. | -| `python-dotenv` | 1.2.2, published 2026-03-01 | Env loading | Already used by `adapter/matrix/bot.py` for Matrix config. | - -### Alternatives Considered -| Instead of | Could Use | Tradeoff | -|------------|-----------|----------| -| Startup reconciliation from joined rooms + state | Force developers to wipe local DB and recreate rooms | Simpler code, but directly violates D-01, D-02, D-05. | -| Non-destructive local rebuild | Full auto-recreate of Space/rooms on missing local state | Easier to implement, but causes duplicate Matrix rooms and breaks D-04. | -| Dev reset script | README-only manual ritual | Lower code cost, but not repeatable and fails D-08..D-10. | - -**Installation:** -```bash -uv sync -``` - -**Version verification:** Verified via installed environment and PyPI metadata on 2026-04-03: -- `matrix-nio` `0.25.2` - 2024-10-04 -- `pytest` `9.0.2` - 2025-12-06 -- `pytest-asyncio` `1.3.0` - 2025-11-10 -- `structlog` `25.5.0` - 2025-10-27 -- `python-dotenv` `1.2.2` - 2026-03-01 - -## Architecture Patterns - -### Recommended Project Structure -```text -adapter/matrix/ -├── bot.py # startup flow calls reconciliation before sync loop -├── reconcile.py # bootstrap/rebuild logic from Matrix server state -├── reset.py # dev-only reset CLI / entrypoint -├── room_router.py # room_id -> chat_id with recovery hook -├── store.py # metadata helpers, prefix scans, derived counters -└── handlers/ - ├── auth.py # first-time provisioning only - └── chat.py # uses recovered state, no provisioning fallback -``` - -### Pattern 1: Two-Phase Startup Bootstrap -**What:** Split startup into `login -> initial sync/full_state -> reconcile -> steady-state sync_forever`. -**When to use:** Always for Matrix bot startup when local DB may be missing or stale. -**Example:** -```python -# Source: matrix-nio AsyncClient docs/source + repo startup flow -client = AsyncClient(...) -runtime = build_runtime(store=SQLiteStore(db_path), client=client) - -await login_or_restore_session(client) -await client.sync(timeout=0, full_state=True) -report = await reconcile_matrix_state(client, runtime.store, runtime.chat_mgr) -logger.info("matrix_reconcile_complete", **report) -await client.sync_forever(timeout=30000) -``` - -### Pattern 2: Rebuild Local Metadata From Joined Rooms -**What:** Enumerate joined rooms, inspect local hydrated room objects or room state, and recreate missing `matrix_room:*`, `matrix_user:*`, and `chat:*` records. -**When to use:** On startup and optionally on `unregistered:{room_id}` fallback at runtime. -**Example:** -```python -# Source: matrix-nio AsyncClient.joined_rooms/room_get_state + repo store contracts -joined = await client.joined_rooms() -for room_id in joined.rooms: - state = await client.room_get_state(room_id) - # detect: space room vs chat room, owner user, child relationship, display name - # rebuild matrix_room:{room_id} - # rebuild chat:{matrix_user_id}:{chat_id} if absent -``` - -### Pattern 3: Non-Destructive Reconciliation Report -**What:** Return a structured report: scanned rooms, restored rooms, restored chats, conflicts, skipped rooms. -**When to use:** Every reconciliation run, including dry-run. -**Example:** -```python -{ - "joined_rooms": 4, - "restored_user_meta": 1, - "restored_room_meta": 3, - "restored_chat_rows": 3, - "conflicts": [], - "skipped_rooms": ["!dm:example.org"], -} -``` - -### Pattern 4: Reset Modes Are Explicit -**What:** Separate `local-only`, `server-leave-forget`, and `dry-run`. -**When to use:** For dev/QA only. Never mix destructive server cleanup into normal startup. -**Example:** -```bash -uv run python -m adapter.matrix.reset --mode local-only -uv run python -m adapter.matrix.reset --mode server-leave-forget --dry-run -``` - -### Anti-Patterns to Avoid -- **Provisioning during reconciliation:** Do not create a new Space or new rooms while trying to recover missing local state. -- **Treating `next_chat_index` as primary truth:** Derive it from recovered `chat_id` values after scan; do not trust a missing or stale counter. -- **Routing unknown rooms straight through:** `unregistered:{room_id}` is a signal to reconcile, not a stable runtime identity. -- **Destructive reset by default:** Startup must never leave/forget rooms automatically. -- **Blindly trusting local `surface_ref`:** If `chat:*` and `matrix_room:*` disagree, rebuild from Matrix room metadata and repair the chat row. - -## Don't Hand-Roll - -| Problem | Don't Build | Use Instead | Why | -|---------|-------------|-------------|-----| -| Room discovery | Custom DB-only reconstruction heuristics | `AsyncClient.joined_rooms()` plus synced room state | Server already knows which rooms the bot joined. | -| Space membership detection | Naming-convention parsing of room names | Matrix state: `m.room.create.type`, `m.space.child`, `m.space.parent` | Names are mutable and non-authoritative. | -| Room cleanup semantics | Custom “delete room” assumptions | `room_leave()` + `room_forget()` semantics | Client API supports leave/forget, not guaranteed global deletion. | -| Chat ID recovery | Hardcoded `C1/C2/...` reset | Rebuild from existing `matrix_room:*`/server state and compute next index | Prevents collisions after partial DB loss. | -| Diagnostic output | Ad hoc `print()` strings | Structured reconciliation/reset report via `structlog` | Easier manual QA and failure triage. | - -**Key insight:** The homeserver already persists the bot’s room graph. This phase should rehydrate local cache from that graph, not attempt to replace it with a second custom truth model. - -## Common Pitfalls - -### Pitfall 1: Joining the sync loop before reconciliation -**What goes wrong:** Commands arrive while local metadata is still missing, producing `unregistered:{room_id}` routing or `ChatManager` misses. -**Why it happens:** Current `main()` enters `sync_forever()` immediately after login. -**How to avoid:** Perform initial sync and reconciliation first. -**Warning signs:** `unregistered_room` logs immediately after restart; `ValueError("Chat ... not found")` on `!rename` or `!archive`. - -### Pitfall 2: Recovering room metadata but not chat rows -**What goes wrong:** Room routing works, but `ChatManager.rename/archive/list_active` still fails because `chat:{user}:{chat_id}` rows were not recreated. -**Why it happens:** Matrix adapter metadata and core chat metadata live in different keyspaces. -**How to avoid:** Reconciliation must repair both stores in one pass. -**Warning signs:** `matrix_room:*` exists but `chat:*` keys do not. - -### Pitfall 3: Trusting stale `next_chat_index` -**What goes wrong:** New chats reuse existing `C` IDs after local recovery. -**Why it happens:** `next_chat_id()` increments a persisted counter that may be absent or behind. -**How to avoid:** After scan, set `next_chat_index = max(recovered_chat_numbers) + 1`. -**Warning signs:** New room gets `C1` even though Space already contains prior rooms. - -### Pitfall 4: Assuming room names identify chat rooms safely -**What goes wrong:** Reconciliation binds the wrong room because a user renamed a room or Space. -**Why it happens:** Names are user-facing labels, not stable identifiers. -**How to avoid:** Prefer room state and existing `chat_id` metadata; use display names only as fallback. -**Warning signs:** Duplicate “Чат 1” names or renamed rooms break matching. - -### Pitfall 5: Over-promising full cleanup -**What goes wrong:** Reset script claims a “clean slate” but rooms still exist in Element or for other members. -**Why it happens:** Leaving/forgetting affects the bot account’s membership/history, not necessarily global room deletion. -**How to avoid:** Name the mode accurately and print the manual client steps when needed. -**Warning signs:** QA reruns still show old rooms in the user’s client. - -## Code Examples - -Verified patterns from official sources and the installed library surface: - -### Initial Sync Before Reconcile -```python -# Source: matrix-nio AsyncClient.sync/sync_forever -await client.sync(timeout=0, full_state=True) -report = await reconcile_matrix_state(client, store, chat_mgr) -await client.sync_forever(timeout=30000) -``` - -### Space Child Link Creation -```python -# Source: Matrix client-server API state event + current auth/new-chat flow -await client.room_put_state( - room_id=space_id, - event_type="m.space.child", - content={"via": [homeserver]}, - state_key=chat_room_id, -) -``` - -### Bot-Side Leave/Forget Cleanup -```python -# Source: matrix-nio AsyncClient.room_leave / room_forget -for room_id in room_ids: - await client.room_leave(room_id) - await client.room_forget(room_id) -``` - -### Router Recovery Trigger -```python -# Source: repo room_router contract -chat_id = await resolve_chat_id(store, room_id, matrix_user_id) -if chat_id.startswith("unregistered:"): - await reconcile_single_room(client, store, chat_mgr, room_id, matrix_user_id) -``` - -## State of the Art - -| Old Approach | Current Approach | When Changed | Impact | -|--------------|------------------|--------------|--------| -| Local adapter DB treated as the operational truth | Rebuildable local cache from server room graph | Mature Matrix client practice; supported by current Matrix CS API and `matrix-nio` | Restart no longer requires destructive local reset. | -| Manual room cleanup in client after experiments | Scripted leave/forget plus explicit manual instructions | Current `matrix-nio` 0.25.x API surface | QA becomes repeatable and auditable. | -| Immediate steady-state sync after login | Initial sync/full-state bootstrap before long polling | Supported by current `AsyncClient.sync()` / `sync_forever()` behavior | Reconciliation can run before any user traffic is handled. | - -**Deprecated/outdated:** -- `README.md` Matrix manual QA instruction `rm -f lambda_matrix.db` as the primary restart flow: outdated for this phase. -- DM-first Matrix recovery assumptions in `docs/matrix-prototype.md`: outdated relative to Phase 1 Space+rooms decisions. - -## Open Questions - -1. **How exactly should reconciliation identify the owning Matrix user for a recovered room when local `matrix_room:*` is gone?** - - What we know: the bot can enumerate joined rooms and fetch room state; current healthy metadata stores `matrix_user_id` and `space_id`. - - What's unclear: whether Phase 1-created rooms also expose enough server-side structure to recover owner deterministically without existing local metadata in every case. - - Recommendation: Plan a proof test against a real homeserver/client. If room-state-only ownership is ambiguous, persist a tiny bot-authored marker state event going forward, but keep that addition narrowly scoped. - -2. **Should runtime recovery happen only on startup, or also lazily on first unknown room access?** - - What we know: startup repair satisfies D-02/D-07 for common restart loss; `room_router` already surfaces unknown rooms cleanly. - - What's unclear: whether partial DB corruption during runtime is common enough to justify lazy single-room repair in Phase 01.1. - - Recommendation: Make startup reconciliation required, lazy room repair optional if it stays small. - -3. **How much of server cleanup should Phase 01.1 automate?** - - What we know: `room_leave()` and `room_forget()` are available; global room deletion is not what the client API guarantees. - - What's unclear: whether automating bot-side leave/forget is worth the extra risk for this urgent phase. - - Recommendation: Keep `local-only` mandatory. Make server cleanup optional and clearly labeled experimental/dev-only if included. - -## Environment Availability - -| Dependency | Required By | Available | Version | Fallback | -|------------|------------|-----------|---------|----------| -| Python | Runtime, scripts, tests | ✓ | 3.14.3 | — | -| `uv` | Standard install/run workflow | ✓ | 0.9.30 | `python -m` + existing venv | -| `pytest` | Automated verification | ✓ | 9.0.2 | `uv run pytest` | -| Matrix homeserver credentials | Real restart/reset manual QA | ✗ in current shell | — | Manual-only after `.env` is configured | -| Matrix bot local DB/store paths | Reset workflow | ✓ | defaults in code | Can override with `MATRIX_DB_PATH` / `MATRIX_STORE_PATH` | - -**Missing dependencies with no fallback:** -- Live Matrix credentials for real manual reconciliation/reset QA. - -**Missing dependencies with fallback:** -- None for repository-only implementation and tests. - -## Validation Architecture - -### Test Framework -| Property | Value | -|----------|-------| -| Framework | `pytest 9.0.2` + `pytest-asyncio 1.3.0` | -| Config file | `pyproject.toml` | -| Quick run command | `pytest tests/adapter/matrix -v` | -| Full suite command | `pytest tests/ -v` | - -### Phase Requirements → Test Map -| Req ID | Behavior | Test Type | Automated Command | File Exists? | -|--------|----------|-----------|-------------------|-------------| -| PH01.1-BOOT | Startup rebuilds missing `matrix_user:*`, `matrix_room:*`, and `chat:*` from existing rooms without creating new rooms | unit/integration | `pytest tests/adapter/matrix/test_reconcile.py -v` | ❌ Wave 0 | -| PH01.1-ROUTER | Unknown room fallback can trigger repair or yields diagnosable warning without crashing commands | unit | `pytest tests/adapter/matrix/test_room_router_reconcile.py -v` | ❌ Wave 0 | -| PH01.1-COUNTER | Reconciliation resets `next_chat_index` to recovered max + 1 | unit | `pytest tests/adapter/matrix/test_reconcile.py -k next_chat_index -v` | ❌ Wave 0 | -| PH01.1-RESET | Dev reset `local-only` removes local DB/store paths and prints next steps | unit/smoke | `pytest tests/adapter/matrix/test_reset.py -v` | ❌ Wave 0 | -| PH01.1-NONDESTRUCTIVE | Reconciliation never calls room creation APIs | unit | `pytest tests/adapter/matrix/test_reconcile.py -k no_create -v` | ❌ Wave 0 | - -### Sampling Rate -- **Per task commit:** `pytest tests/adapter/matrix -v` -- **Per wave merge:** `pytest tests/ -v` -- **Phase gate:** Full suite green before `/gsd:verify-work` - -### Wave 0 Gaps -- [ ] `tests/adapter/matrix/test_reconcile.py` - startup reconciliation scenarios -- [ ] `tests/adapter/matrix/test_reset.py` - CLI/script reset modes and output -- [ ] `tests/adapter/matrix/test_room_router_reconcile.py` - lazy recovery or warning behavior -- [ ] Integration fixture for a fake `AsyncClient` response surface matching `joined_rooms()` and `room_get_state()` - -## Sources - -### Primary (HIGH confidence) -- Matrix Client-Server API - room state, leave, forget, joined rooms, Spaces semantics: https://spec.matrix.org/latest/client-server-api/index.html -- `matrix-nio` installed 0.25.2 API surface verified locally on 2026-04-03 via `AsyncClient.sync`, `sync_forever`, `joined_rooms`, `room_get_state`, `room_leave`, `room_forget` -- Repo code: [adapter/matrix/bot.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/bot.py), [adapter/matrix/store.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/store.py), [adapter/matrix/room_router.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/room_router.py), [adapter/matrix/handlers/auth.py](/Users/a/MAI/sem2/lambda/surfaces-bot/adapter/matrix/handlers/auth.py), [core/chat.py](/Users/a/MAI/sem2/lambda/surfaces-bot/core/chat.py) -- PyPI release metadata: https://pypi.org/project/matrix-nio/ , https://pypi.org/project/pytest/ , https://pypi.org/project/pytest-asyncio/ , https://pypi.org/project/structlog/ , https://pypi.org/project/python-dotenv/ - -### Secondary (MEDIUM confidence) -- [README.md](/Users/a/MAI/sem2/lambda/surfaces-bot/README.md) - current manual reset habit and run commands -- [docs/matrix-prototype.md](/Users/a/MAI/sem2/lambda/surfaces-bot/docs/matrix-prototype.md) - original Matrix UX intent, noting outdated DM/reaction sections -- [01-CONTEXT.md](/Users/a/MAI/sem2/lambda/surfaces-bot/.planning/phases/01-matrix-qa-polish/01-CONTEXT.md) - locked Phase 1 Matrix decisions -- [01-VERIFICATION.md](/Users/a/MAI/sem2/lambda/surfaces-bot/.planning/phases/01-matrix-qa-polish/01-VERIFICATION.md) - what has already been verified and what still needs human Matrix QA - -### Tertiary (LOW confidence) -- None - -## Metadata - -**Confidence breakdown:** -- Standard stack: HIGH - verified against installed environment, PyPI metadata, and official Matrix spec -- Architecture: HIGH - directly grounded in current repo flow plus current `matrix-nio`/Matrix capabilities -- Pitfalls: HIGH - derived from concrete gaps in current startup/store/router code - -**Research date:** 2026-04-03 -**Valid until:** 2026-05-03 diff --git a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-VALIDATION.md b/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-VALIDATION.md deleted file mode 100644 index 336cbd6..0000000 --- a/.planning/phases/01.1-matrix-restart-reconciliation-and-dev-reset-workflow/01.1-VALIDATION.md +++ /dev/null @@ -1,80 +0,0 @@ ---- -phase: 01.1 -slug: matrix-restart-reconciliation-and-dev-reset-workflow -status: draft -nyquist_compliant: false -wave_0_complete: false -created: 2026-04-03 ---- - -# Phase 01.1 — Validation Strategy - -> Per-phase validation contract for feedback sampling during execution. - ---- - -## Test Infrastructure - -| Property | Value | -|----------|-------| -| **Framework** | `pytest 9.0.2` + `pytest-asyncio 1.3.0` | -| **Config file** | `pyproject.toml` | -| **Quick run command** | `pytest tests/adapter/matrix -v` | -| **Full suite command** | `pytest tests/ -v` | -| **Estimated runtime** | ~20 seconds | - ---- - -## Sampling Rate - -- **After every task commit:** Run `pytest tests/adapter/matrix -v` -- **After every plan wave:** Run `pytest tests/ -v` -- **Before `$gsd-verify-work`:** Full suite must be green -- **Max feedback latency:** 20 seconds - ---- - -## Per-Task Verification Map - -| Task ID | Plan | Wave | Requirement | Test Type | Automated Command | File Exists | Status | -|---------|------|------|-------------|-----------|-------------------|-------------|--------| -| 01.1-01-01 | 01 | 1 | PH01.1-BOOT | unit/integration | `pytest tests/adapter/matrix/test_reconcile.py -v` | ❌ W0 | ⬜ pending | -| 01.1-01-01 | 01 | 1 | PH01.1-COUNTER | unit | `pytest tests/adapter/matrix/test_reconcile.py -k next_chat_index -v` | ❌ W0 | ⬜ pending | -| 01.1-01-01 | 01 | 1 | PH01.1-NONDESTRUCTIVE | unit | `pytest tests/adapter/matrix/test_reconcile.py -k no_create -v` | ❌ W0 | ⬜ pending | -| 01.1-02-01 | 02 | 2 | PH01.1-BOOT | unit | `pytest tests/adapter/matrix/test_dispatcher.py -k startup -v` | ✅ | ⬜ pending | -| 01.1-02-02 | 02 | 2 | PH01.1-ROUTER | unit | `pytest tests/adapter/matrix/test_dispatcher.py -k reconcile -v` | ✅ | ⬜ pending | -| 01.1-03-01 | 03 | 1 | PH01.1-RESET | unit/smoke | `pytest tests/adapter/matrix/test_reset.py -v` | ❌ W0 | ⬜ pending | -| 01.1-03-02 | 03 | 1 | PH01.1-RESET | smoke | `python -m adapter.matrix.reset --help` | ❌ W0 | ⬜ pending | - -*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky* - ---- - -## Wave 0 Requirements - -- [ ] `tests/adapter/matrix/test_reconcile.py` — startup reconciliation scenarios, `next_chat_index`, and no-provisioning assertions -- [ ] `tests/adapter/matrix/test_reset.py` — CLI reset modes, dry-run behavior, and operator guidance output -- [ ] `tests/adapter/matrix/test_dispatcher.py` — startup bootstrap order and targeted unknown-room recovery coverage -- [ ] Fake `AsyncClient` fixture surface for joined rooms, room state, leave, and forget behavior - ---- - -## Manual-Only Verifications - -| Behavior | Requirement | Why Manual | Test Instructions | -|----------|-------------|------------|-------------------| -| Reconciled Space/chat rooms render correctly in a real Matrix client after restart | PH01.1-BOOT | Client UX and homeserver state cannot be fully trusted from fake nio fixtures | 1. Start the bot with existing Space/chat rooms. 2. Verify the bot does not create duplicate Space or chat rooms. 3. Send a command in a recovered room and confirm it routes normally. | -| Server-side cleanup leaves the account in a usable Element state after `server-leave-forget` | PH01.1-RESET | Element/archive behavior and homeserver retention are client/server integration concerns | 1. Run `python -m adapter.matrix.reset --mode server-leave-forget --dry-run`. 2. Run without `--dry-run` on a test account. 3. Confirm joined rooms disappear for the bot and fresh invites can be accepted cleanly. | - ---- - -## Validation Sign-Off - -- [ ] All tasks have `` verify or Wave 0 dependencies -- [ ] Sampling continuity: no 3 consecutive tasks without automated verify -- [ ] Wave 0 covers all MISSING references -- [ ] No watch-mode flags -- [ ] Feedback latency < 20s -- [ ] `nyquist_compliant: true` set in frontmatter - -**Approval:** pending diff --git a/.planning/phases/02-prototype/.continue-here.md b/.planning/phases/02-prototype/.continue-here.md deleted file mode 100644 index a2d4619..0000000 --- a/.planning/phases/02-prototype/.continue-here.md +++ /dev/null @@ -1,72 +0,0 @@ ---- -phase: 02-prototype -task: 4 -total_tasks: 4 -status: paused -last_updated: 2026-04-07T23:54:30.473Z ---- - - -The Matrix direct-agent prototype is implemented and manually proven working on branch `feat/matrix-direct-agent-prototype`. The current code path can log into Matrix, accept invites, provision the first Space/chat tree for a fresh user, and send live text messages to a patched local `platform-agent` over WebSocket. The immediate remaining engineering gap is not feature delivery but resilience: backend/provider failures can still bubble up as `PlatformError` and crash the Matrix bot process. - - - - -- Task 1: Added `sdk/agent_session.py` and transport tests for direct WebSocket messaging with collision-safe `thread_key` generation. -- Task 2: Added `sdk/prototype_state.py` and tests for stable local user mapping, settings defaults, and mutation-safe settings copies. -- Task 3: Added `sdk/real.py` as the `PlatformClient` implementation, fixed import-time dependency leakage, and aligned thread-key tests to the actual dispatcher contract. -- Task 4: Wired Matrix runtime selection through `MATRIX_PLATFORM_BACKEND=real`, documented usage in `README.md`, and added dispatcher coverage for real backend selection. -- Fixed repeat Matrix invites so the bot now `join()`s before the existing-user early return path. -- Added Russian runbook doc `docs/matrix-direct-agent-prototype-ru.md` and pushed the branch. -- Manually validated live bring-up using a local patched `external/platform-agent` on port 8000 plus the Matrix homeserver `https://matrix.lambda.coredump.ru`. - - - - -- Add graceful degradation for backend/provider failures so `PlatformError` does not crash the Matrix process. -- Decide whether to upstream or separately push the required `external/platform-agent` patch (`1dca2c1`) that enables WebSocket `thread_id`. -- Optionally clean up repeat-invite UX if Space/chat reprovisioning should ever happen for already-known users. -- Optionally prepare a PR from `feat/matrix-direct-agent-prototype`. - - - - -- Keep the prototype in this repo, not a separate Matrix-only repo. -- Keep Matrix adapter logic intact and absorb backend differences inside `sdk/`. -- Split the real backend into `AgentSessionClient` and `PrototypeStateStore` behind `RealPlatformClient`. -- Patch only `platform-agent` for per-thread memory instead of changing both `agent` and `agent_api`. -- Use a serialized collision-safe thread key because Matrix user IDs contain colons. -- For repeat invites, join the room but do not recreate Space/chat state if the user is already provisioned locally. - - - -- Technical: provider/backend errors still crash the Matrix bot instead of returning a user-facing failure reply. -- External: the required `platform-agent` patch exists only in the local clone under `external/` and is not yet upstream. -- Operational: credentials used during manual bring-up were exposed in-session and should be rotated. - - - -The important mental model is stable. `platform/master` is still not the backend for surfaces, so the working prototype goes directly to `platform-agent` over `/agent_ws/`. The live setup that worked was: -- `surfaces-bot` branch: `feat/matrix-direct-agent-prototype` -- Matrix bot env: `MATRIX_PLATFORM_BACKEND=real`, `AGENT_WS_URL=ws://127.0.0.1:8000/agent_ws/` -- patched local `external/platform-agent` with `thread_id` support -- provider configured through OpenRouter using model `qwen/qwen3.5-122b-a10b` - -Important files: -- `sdk/agent_session.py` -- `sdk/prototype_state.py` -- `sdk/real.py` -- `adapter/matrix/bot.py` -- `adapter/matrix/handlers/auth.py` -- `docs/matrix-direct-agent-prototype-ru.md` - -Important local-only dependency: -- `external/platform-agent` commit `1dca2c1` (`feat: support websocket thread ids`) - -Likely running background process at pause time: -- local `platform-agent` server on port 8000, PID 13499 - - - -Start with the failure path: catch `PlatformError` around Matrix message handling so a bad provider response becomes a normal reply like “backend unavailable, try again later” instead of killing the process. After that, either upstream `external/platform-agent` commit `1dca2c1` or document it as an explicit prerequisite in the platform repo. -