docs: add matrix staged attachments design
This commit is contained in:
parent
8b04fcaf77
commit
105ecc68ed
1 changed files with 262 additions and 0 deletions
|
|
@ -0,0 +1,262 @@
|
||||||
|
# Matrix Staged Attachments Design
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Make file sending in the Matrix surface usable for an AI agent despite current Matrix client behavior, especially in Element where media is often sent immediately as separate events without a shared text composer.
|
||||||
|
|
||||||
|
The result should be:
|
||||||
|
|
||||||
|
- files can arrive before the user writes the actual instruction
|
||||||
|
- the surface stages those files instead of immediately sending them to the agent
|
||||||
|
- the next normal user message in the same chat commits all staged files as one agent turn
|
||||||
|
- the user can inspect and remove staged files with short chat commands
|
||||||
|
|
||||||
|
## Core Decision
|
||||||
|
|
||||||
|
The selected UX model is:
|
||||||
|
|
||||||
|
`incoming Matrix media -> staged attachments for (chat_id, user_id) -> next normal message commits them`
|
||||||
|
|
||||||
|
This means:
|
||||||
|
|
||||||
|
- attachment-only events do not immediately invoke the agent
|
||||||
|
- the bot acknowledges staged files with a service message
|
||||||
|
- the next normal user message sends text plus all currently staged files to the agent
|
||||||
|
- staged files are then cleared
|
||||||
|
|
||||||
|
## Why This Decision
|
||||||
|
|
||||||
|
Matrix natively models messages as separate events, and common clients do not provide a reliable "one text message with many attachments" composer flow.
|
||||||
|
|
||||||
|
In practice this causes two UX failures for an AI bot:
|
||||||
|
|
||||||
|
- users may send files first and only then write the task
|
||||||
|
- users may send multiple files as multiple independent Matrix events
|
||||||
|
|
||||||
|
If the surface treats each incoming file as a full agent turn, the bot becomes noisy and context-fragmented. If it ignores file-only messages, file handling feels broken.
|
||||||
|
|
||||||
|
Staging is the smallest surface-side abstraction that fixes both problems without fighting the Matrix event model.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This design covers:
|
||||||
|
|
||||||
|
- staging inbound Matrix attachments before agent submission
|
||||||
|
- per-chat attachment state for a specific user
|
||||||
|
- user-facing service messages for staged attachments
|
||||||
|
- short commands for listing and removing staged files
|
||||||
|
- commit behavior on the next normal message
|
||||||
|
|
||||||
|
This design does not cover:
|
||||||
|
|
||||||
|
- edits or redactions of original Matrix media events as attachment controls
|
||||||
|
- cross-surface shared staging
|
||||||
|
- thread-aware staging beyond the existing `chat_id` boundary
|
||||||
|
- changes to the platform attachment contract
|
||||||
|
|
||||||
|
## State Model
|
||||||
|
|
||||||
|
### Staging key
|
||||||
|
|
||||||
|
Staged attachments are isolated by:
|
||||||
|
|
||||||
|
- `chat_id`
|
||||||
|
- `user_id`
|
||||||
|
|
||||||
|
This means:
|
||||||
|
|
||||||
|
- files staged by a user in one chat never appear in another chat
|
||||||
|
- files staged by one user do not mix with another user's files in the same room
|
||||||
|
|
||||||
|
### Staged attachment record
|
||||||
|
|
||||||
|
Each staged attachment must track at least:
|
||||||
|
|
||||||
|
- stable internal id
|
||||||
|
- display filename
|
||||||
|
- workspace-relative path
|
||||||
|
- MIME type if known
|
||||||
|
- created timestamp
|
||||||
|
|
||||||
|
User-visible commands operate on the current ordered list, not on internal ids.
|
||||||
|
|
||||||
|
### Lifecycle
|
||||||
|
|
||||||
|
A staged attachment is in exactly one of these states:
|
||||||
|
|
||||||
|
1. `staged`
|
||||||
|
2. `committed`
|
||||||
|
3. `removed`
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
|
||||||
|
- only `staged` attachments appear in `!list`
|
||||||
|
- `committed` attachments are no longer user-removable
|
||||||
|
- `removed` attachments are excluded from future commits
|
||||||
|
|
||||||
|
## Inbound Behavior
|
||||||
|
|
||||||
|
### Attachment-only event
|
||||||
|
|
||||||
|
If the Matrix surface receives one or more file/media events from a user without a normal text message to commit them:
|
||||||
|
|
||||||
|
1. download each file into shared `/workspace`
|
||||||
|
2. add each file to the staged set for `(chat_id, user_id)`
|
||||||
|
3. do not call the agent yet
|
||||||
|
4. send a service acknowledgment message
|
||||||
|
|
||||||
|
### Service acknowledgment
|
||||||
|
|
||||||
|
The service message must communicate:
|
||||||
|
|
||||||
|
- the current staged attachment list with indices
|
||||||
|
- that the next normal message will be sent to the agent together with those files
|
||||||
|
- available commands: `!list`, `!remove <n>`, `!remove all`
|
||||||
|
|
||||||
|
Example shape:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Staged attachments:
|
||||||
|
1. screenshot.png
|
||||||
|
2. invoice.pdf
|
||||||
|
|
||||||
|
Your next message will be sent to the agent with these files.
|
||||||
|
Commands: !list, !remove <n>, !remove all
|
||||||
|
```
|
||||||
|
|
||||||
|
### Burst handling
|
||||||
|
|
||||||
|
Matrix clients may send multiple files as separate consecutive events.
|
||||||
|
|
||||||
|
To avoid bot spam, service acknowledgments should be debounced over a short window and aggregated into one reply where feasible.
|
||||||
|
|
||||||
|
The acknowledgment must reflect the full current staged set, not only the most recently received file.
|
||||||
|
|
||||||
|
## Commit Behavior
|
||||||
|
|
||||||
|
### Commit trigger
|
||||||
|
|
||||||
|
The commit trigger is:
|
||||||
|
|
||||||
|
- the next normal user message in the same `(chat_id, user_id)` scope
|
||||||
|
|
||||||
|
Normal user message means:
|
||||||
|
|
||||||
|
- not a staging control command
|
||||||
|
- not a pure attachment event being staged
|
||||||
|
|
||||||
|
### Commit action
|
||||||
|
|
||||||
|
When a commit-triggering message arrives:
|
||||||
|
|
||||||
|
1. collect all currently staged attachments for `(chat_id, user_id)`
|
||||||
|
2. send the user text plus those attachments to the agent as one turn
|
||||||
|
3. mark all included staged attachments as `committed`
|
||||||
|
4. clear the staged set
|
||||||
|
|
||||||
|
After commit:
|
||||||
|
|
||||||
|
- the just-sent attachments must no longer appear in `!list`
|
||||||
|
- a later file upload starts a new staged set
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
### `!list`
|
||||||
|
|
||||||
|
Shows the current staged attachment list for the user in the current chat.
|
||||||
|
|
||||||
|
If the list is empty, the response should be short and explicit.
|
||||||
|
|
||||||
|
### `!remove <n>`
|
||||||
|
|
||||||
|
Removes the staged attachment at the current 1-based index.
|
||||||
|
|
||||||
|
Behavior:
|
||||||
|
|
||||||
|
- if the index is valid, remove that staged attachment and return the updated staged list
|
||||||
|
- if the index is invalid, return a short error without repeating the list
|
||||||
|
|
||||||
|
### `!remove all`
|
||||||
|
|
||||||
|
Clears the entire staged set for the user in the current chat.
|
||||||
|
|
||||||
|
The response should be short and explicit.
|
||||||
|
|
||||||
|
## Ordering Rules
|
||||||
|
|
||||||
|
The staged list is ordered by staging time.
|
||||||
|
|
||||||
|
User-facing indices:
|
||||||
|
|
||||||
|
- are 1-based
|
||||||
|
- are recalculated from the current staged set
|
||||||
|
- may change after removals
|
||||||
|
|
||||||
|
Therefore:
|
||||||
|
|
||||||
|
- `!list` always shows the current authoritative numbering
|
||||||
|
- after a successful `!remove <n>`, the bot should reply with the refreshed list
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
### Download failure
|
||||||
|
|
||||||
|
If a file cannot be downloaded or stored:
|
||||||
|
|
||||||
|
- do not add it to the staged set
|
||||||
|
- do not pretend it will be sent later
|
||||||
|
- send a short user-visible failure message
|
||||||
|
|
||||||
|
### Invalid command
|
||||||
|
|
||||||
|
If the command is malformed or uses an invalid index:
|
||||||
|
|
||||||
|
- return a short error
|
||||||
|
- do not commit staged attachments
|
||||||
|
- do not clear the staged set
|
||||||
|
|
||||||
|
### Agent submission failure
|
||||||
|
|
||||||
|
If commit fails when sending the text plus staged files to the agent:
|
||||||
|
|
||||||
|
- staged attachments must remain available for retry unless the failure is known to be irreversible
|
||||||
|
- the user-visible error should make it clear that the files were not consumed
|
||||||
|
|
||||||
|
This prevents silent loss of staged context.
|
||||||
|
|
||||||
|
## Interaction with Shared Workspace Design
|
||||||
|
|
||||||
|
This design assumes the shared-workspace contract defined in
|
||||||
|
[2026-04-20-matrix-shared-workspace-file-flow-design.md](/Users/a/MAI/sem2/lambda/surfaces-bot/docs/superpowers/specs/2026-04-20-matrix-shared-workspace-file-flow-design.md).
|
||||||
|
|
||||||
|
Specifically:
|
||||||
|
|
||||||
|
- staged files are stored in shared `/workspace`
|
||||||
|
- the final commit still passes workspace-relative paths to `platform-agent`
|
||||||
|
- staging changes only when the surface chooses to invoke the agent, not how attachments are represented
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
The implementation must cover:
|
||||||
|
|
||||||
|
- file-only Matrix events are staged and do not immediately invoke the agent
|
||||||
|
- service acknowledgment includes staged filenames and command hints
|
||||||
|
- `!list` returns the current staged set for the correct `(chat_id, user_id)`
|
||||||
|
- `!remove <n>` removes the correct staged attachment and refreshes numbering
|
||||||
|
- `!remove all` clears the staged set
|
||||||
|
- invalid `!remove <n>` returns a short error and keeps state unchanged
|
||||||
|
- the next normal message commits all staged attachments with the text as one agent turn
|
||||||
|
- committed attachments disappear from staging after success
|
||||||
|
- failed commits preserve staged attachments
|
||||||
|
- staging in one chat does not leak into another chat
|
||||||
|
- staging for one user does not leak to another user in the same room
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
This design intentionally does not attempt to:
|
||||||
|
|
||||||
|
- emulate Telegram-style albums in Matrix
|
||||||
|
- rely on special support from Element or other Matrix clients
|
||||||
|
- introduce a rich interactive attachment management UI
|
||||||
|
|
||||||
|
The goal is a reliable chat-native workflow that works within Matrix's actual event model.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue