OpenBot Remote

This directory holds our own control plane for WebRTC connections. The Remote API relays SDP and ICE. Team files, chats, commands and video never pass through the Remote API or Cloudflare. The exceptions are the OpenBot Slack app, the OpenBot Discord bot and the OpenBot Telegram bot: see Slack requests, Discord and Telegram updates.

Flow

1. The app signs the user in through the Cloudflare API.
2. Cloudflare D1 checks membership and creates a logical session.
3. Cloudflare signs a short ES256 ticket. The ticket names the host, the user, the role, the protocol version and authEpoch.
4. The client and the host open WSS /v1/signal. Signal validates the ticket locally through JWKS.
5. Signal relays SDP and ICE and issues a resume token along with time-limited coturn credentials.
6. Chromium sets up WebRTC. The Team API uses the rpc, events and files channels.
7. ICE picks either a direct p2p path or relay through this coturn. Cloudflare is never on the data path.

Signal has no database. It stores no tokens, SDP, ICE, file names or message contents. Room and presence data exist only in process memory. The one exception is the queue of a hosted server that sleeps (see Sleeping hosted servers): Signal keeps a Slack, Discord or Telegram event for it in memory, sealed to the host's key so that Signal cannot read it, for at most 10 minutes.

A resume token is valid for 10 minutes. Signal keeps the issued token in a bounded in-memory cache. A normal reconnect validates that token locally. After a Signal restart the cache is empty, so the first reconnect for a given token makes one signed request to the control plane. The same request is needed once the token has expired. The control plane checks the session, the membership and authEpoch. Signal then issues a new token. Subsequent reconnects are local again. There are no heartbeats and no periodic refresh through Cloudflare.

Ending a session and changing access both write a revocation event to a durable D1 outbox. The Worker tries to deliver it to Signal immediately. If Signal is unreachable, the Worker retries the delivery from cron. An ordinary reconnect still does not ask Cloudflare. Ended and expired sessions are removed after the 10-minute validation window.

Slack requests

The OpenBot Slack app sends the events and button presses of every workspace to one request URL, https://signal.openbot.run/v1/slack/events. Signal checks Slack's signature with the app's signing secret, answers Slack's url_verification challenge, and reads only the app ID (api_app_id), the workspace ID, and the event's type, channel type, thread and bot ID (to know whether it addresses OpenBot, not its text). SLACK_SIGNING_SECRET is a comma-separated list of <app ID>:<signing secret>, because the production and development apps share Signal. A request must name the app whose secret signed it, so one app's secret cannot reach the other app's hosts; a malformed value turns off only the Slack route. Then Signal passes the exact request body to the ingress socket of the host that the app and workspace are linked to, and returns the host's answer, or 503 when no host holds the workspace or the host does not answer in 2.5 seconds. Slack then sends the request again. For a hosted server that sleeps, Signal answers 200 instead (see Sleeping hosted servers).

An ingress socket names its workspaces with a Slack route ticket: an ES256 JWT with the audience openbot-slack-route, signed by the Worker with SLACK_ROUTE_PRIVATE_JWK (key id SLACK_ROUTE_KEY_ID) for the workspaces, each with its app, that the account service links to that host. Its public key must be in the ticket JWKS that Signal loads (REMOTE_TICKET_PUBLIC_JWKS on the Worker, or REMOTE_TICKET_PUBLIC_KEYS here). Signal does not store or log the body. Without SLACK_SIGNING_SECRET, the Slack route answers 503.

GitHub events

The OpenBot GitHub App sends the events of every installation to one webhook URL, https://signal.openbot.run/v1/github/events. Signal checks GitHub's X-Hub-Signature-256 with the App's webhook secret (GITHUB_WEBHOOK_SECRET, at least 32 bytes), and reads only the event name, the delivery ID, the installation ID and the repository ID. Then it passes the exact body to the ingress socket of every host whose GitHub route names the repository, at the same time. It answers 202 when a host saved the event, 204 when no host watches the repository or every host refused it, and 503 when a host did not answer. A body larger than 512 KiB gets 413. GitHub does not send an event again by itself: an event for a host that is offline is lost, and the host shows the gap in its event history.

An ingress socket names its repositories with a GitHub route ticket: an ES256 JWT with the audience openbot-github-route, signed by the Worker with the remote ticket key, for 5 minutes. The Worker checks, with the host's GitHub sign-in, which of the repositories that the host's GitHub triggers watch the sign-in can see through an installation of the App. D1 keeps none of it. Signal does not store or log the body. Without GITHUB_WEBHOOK_SECRET, the GitHub route answers 503.

Discord

The OpenBot Discord bot is one bot for every guild that installs it. Signal holds its bot token (DISCORD_BOT_TOKEN) and its application ID (DISCORD_APPLICATION_ID). Without both, or with a malformed value, Discord is off: Signal logs one line without the value, keeps no Gateway connection, and POST /v1/discord/api answers 503. The remote sessions and the Slack route keep running.

Signal asks for the privileged Message Content intent, so that OpenBot gets the text of the message a person replies to and of the conversation history. Turn on Message Content Intent in the Developer Portal. When Discord refuses the intent (close code 4014), Signal logs one line and connects without it. DISCORD_MESSAGE_CONTENT=false stops Signal asking for it.

Signal keeps the bot's Gateway connection with the intents Guilds, GuildMessages and DirectMessages, plus MessageContent unless DISCORD_MESSAGE_CONTENT is false. Without the Message Content intent, Discord gives the bot the text of a message only when the message mentions the bot. With it, Discord sends the text of every message in the channels the bot can see; Signal still passes on only the messages that mention the bot. From the Gateway, Signal reads:

• the names of the guilds, channels and threads, and which guild each channel is in;
• each guild message that mentions the bot and that a person wrote. Signal passes the message ID, channel, author, the author's role IDs, text without the bot's mention, the replied message (with its author, text and time when a person wrote it and Discord gave its text), the Discord CDN addresses of the attachments and the time to the ingress socket of the host that the guild is linked to;
• each press of an OpenBot button. Signal acknowledges it to Discord at once and passes it on;
• the removal of the bot from a guild, which ends that guild's connection on the host.

A direct message to the bot gets one fixed answer, at most once an hour for each user. Signal does not pass it on.

An ingress socket names its guilds with a Discord route ticket: an ES256 JWT with the audience openbot-discord-route, signed by the Worker for the guilds that the account service links to that host. Its public key must be in the ticket JWKS that Signal loads, as for the Slack route ticket. The account service revokes a guild link through /internal/auth-events. After a start, Signal asks the account service which links are current until every older ticket has expired.

A socket with a Discord route gets a discord-session token and the list of the guilds routed to it. The host sends it as the bearer of its calls to POST /v1/discord/api: post, edit or delete a message, add or remove a reaction, read the messages after one message, read a member or channel name, list the guild's roles, answer a button press, and upload one file of at most 10 MB. Signal makes each call with the bot token, only in a guild that is routed to that socket, and only in a channel of that guild. Only a new text post can ping, and only a user that it mentions, never a role or @everyone. The token is valid while the socket is open.

So the answers and files of Discord conversations pass through Signal. Signal does not store them or log them, and it does not log a token or Discord's error text.

When the bot leaves a guild, Signal drops its route and asks the account service to unlink it (/v2/remote/discord-route/removed). After the Gateway lists the bot's guilds, and every 30 minutes, Signal sends the account service the guild IDs that the bot is in (/v2/remote/discord-route/reconcile), and the account service unlinks each older link of another guild. Signal sends nothing while a Gateway shard is closed, because the list can then miss a new guild. So a link goes also when its host is off and the first unlink failed, or Signal restarted.

Telegram updates

One OpenBot bot serves every user, and only Signal has its token. TELEGRAM_BOT_TOKENS is a comma-separated list of bot tokens (<bot ID>:<secret>), because the production and development bots share Signal. TELEGRAM_WEBHOOK_SECRET (at least 32 bytes) is required with them: the webhook secret of each bot is the base64url HMAC-SHA256 of telegram-webhook:<bot ID> with it. A malformed value turns off only Telegram. With TELEGRAM_WEBHOOK_ORIGIN, Signal sets the webhook of each bot to <origin>/v1/telegram/updates/<bot ID> when it starts; a failure is logged and Signal runs on.

Telegram posts each update to /v1/telegram/updates/<bot ID>. Signal checks the secret header, reads only the chat ID, the callback query ID and a /start <code> link code, and passes the exact body to the ingress socket of the chat's host. A link code makes Signal ask the account service to link the chat (/v2/remote/telegram-route/link), and routes the chat to that host at once. Signal answers 200 to each signed update, also when no host holds the chat: Telegram holds back the bot's other updates while one fails.

An ingress socket names its chats with a Telegram route ticket: an ES256 JWT with the audience openbot-telegram-route, signed with the key of the Slack route ticket. When ready names the telegram capability, the host calls the Bot API through the socket (telegram-call). Signal accepts only the methods and parameters of TelegramCallParams, only for the chats routed to that socket, and returns only the reduced result. A host downloads a file from /v1/telegram/files/<token> and posts a document to /v1/telegram/uploads/<token>. These tokens are signed, expire after two minutes, and an upload token works once. Signal does not store or log an update or a file, except for the sealed queue of a hosted server that sleeps.

Sleeping hosted servers

All messaging platforms share one path (src/ingress-queue.ts). For an event whose route has no ingress socket, Signal asks the account service (/v2/remote/route-wake, signed), at most once a minute for each route. An event that addresses OpenBot starts a hosted server that sleeps. Signal then keeps the event's delivery frame, sealed with the host's queue key (queueKey in the ingress hello) and the host ID, for at most 10 minutes, 64 events and 192 KB for each host. When the host's socket holds the route again, Signal sends each kept event in order as queued-delivery, and keeps it until the host sends queued-delivery-ack. While a host has kept events, Signal asks for its start again once a minute, because a start can fail after the account service answered. A host that sent no key gets nothing kept. A restart of Signal loses the queue and the keys: until a sleeping host connects again, its events start it but are not kept. Signal remembers at most 100,000 keys and 100,000 route answers, and drops the oldest first. A generic webhook for a route with no socket starts a sleeping hosted server too, at most once in 30 minutes for each host, but Signal keeps nothing and answers 503: only the host can check the signature, and the sender sends again. Signal names the ingress-queue capability in ready only when the account service answered /v2/remote/route-wake when Signal started: only then does a hosted server with a live connection sleep. Deploy the account service before Signal.

Production requirements

• Linux with Docker Engine and Docker Compose.
• A static public IPv4 address.
• signal.openbot.run and turn.openbot.run records in DNS only mode.
• Open ports TCP 443, TCP/UDP 3478, TCP 5349 and UDP 16384-65535 (the coturn relay range). With ufw: ufw allow 16384:65535/udp comment 'OpenBot TURN relay UDP'. coturn uses host networking, so ufw applies to it. Without this rule, the firewall drops the first packets that a peer sends to a relay address, and relay connections become slow or fail.
• A Cloudflare token scoped to DNS edit for the openbot.run zone.

The signal.openbot.run and turn.openbot.run records must be in DNS only mode. Do not enable the Cloudflare proxy for them. api.openbot.run stays a Cloudflare Worker.

If you already run a reverse proxy on port 443, set REMOTE_SIGNAL_BIND_ADDRESS=127.0.0.1, REMOTE_SIGNAL_PUBLIC_PORT=8081, REMOTE_SIGNAL_PORT=8081 and REMOTE_TLS_DISABLED=true. The reverse proxy must terminate TLS and forward the WebSocket to http://127.0.0.1:8081. Also set REMOTE_TRUST_PROXY=true so the IP limits use the client address from X-Forwarded-For.

Each Signal socket uses two Nginx connections: one from the client and one to Signal. The Nginx default of 768 connections for each worker holds only about 12,000 sockets on a 32-core machine, less than one machine must hold after a failover. Set these in /etc/nginx/nginx.conf, and add reuseport to the listen 443 lines of the Signal server, so that new sockets go to all workers:

worker_rlimit_nofile 65535;
events {
  worker_connections 8192;
}

The configuration in nginx/signal.openbot.run.conf uses the certificate from the ACME volume. Install openbot-remote-nginx-reload.path and openbot-remote-nginx-reload.service into /etc/systemd/system/ as well. The unit reloads Nginx after the certificate is renewed:

sudo cp remote/nginx/openbot-remote-nginx-reload.* /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now openbot-remote-nginx-reload.path

Production secrets are stored in the tracked .env.production file as encrypted Dotenvx values. They use the same key pair as apps/auth-api/.env.production: DOTENV_PRIVATE_KEY_PRODUCTION in the root .env.keys. Do not make a new key pair. On the server, the ignored remote/.env.keys holds only a copy of that key. On a development computer, decrypt with -fk .env.keys. Do not pass the encrypted file straight to docker compose --env-file, because Compose will not decrypt the values. The remote:* scripts run Compose through Dotenvx and decrypt the values only in process memory. The --overload option keeps empty host environment variables from overriding the decrypted values. The scripts use remote/bin/dotenvx. The wrapper picks the pinned Node from the application runtime if the system Node is too old.

Validate the environment and start the services:

bun run remote:env:validate
bun run remote:up
bun run remote:check

Updating Signal may close the WebSocket. An active WebRTC connection stays up. remote:update puts coturn into drain and waits for the allocations to finish. Forcing a coturn restart ends active relay sessions. The client performs an ICE restart and resumes a file transfer from the last acknowledged offset once the service is back.

Two Signal machines

Two Signal machines can run at the same time. Each machine has its own name, and one more name, the entry (signal.openbot.run), goes to one of them. Set these on each machine:

• SIGNAL_ID: the name of this machine in the signal ticket claim, ^[a-z0-9-]{1,32}$.
• SIGNAL_ENTRY_HOST: the entry name, for example signal.openbot.run.
• SIGNAL_PUBLIC_IPV4: the public IPv4 address of this machine.

Set SIGNAL_ENTRY_HOST and SIGNAL_PUBLIC_IPV4 together, or Signal does not start. Without them, the machine is always the entry.

The claim rule: the account service can put a signal claim in a ticket. A Signal with SIGNAL_ID refuses a ticket with another signal claim (authentication_required, close 1008). A resume token keeps the claim. A ticket with the claim of this machine works whether this machine is the entry or not. Without SIGNAL_ID, Signal does not check the claim.

The entry rule (src/entry.ts): Signal resolves SIGNAL_ENTRY_HOST (IPv4, all addresses) once before it listens, then every 30 seconds. An answer with SIGNAL_PUBLIC_IPV4 makes this machine the entry. Two answers in a row without it end the entry. A failed lookup keeps the state. So the entry name must be DNS-only (not proxied): a proxied name never shows the address of a machine. While a machine is not the entry:

• A ticket or resume token without a signal claim gets close 1013 and no error frame. The peer connects again through the entry name. When the machine stops being the entry, it closes these sockets (host, client and ingress) with 1013.
• The Slack, Telegram, GitHub and generic webhook routes answer 503.
• The Discord Gateway is stopped, and /v1/discord/api answers 503 (unavailable). The Gateway starts again when the machine becomes the entry again.

Set TELEGRAM_WEBHOOK_ORIGIN and the Slack, GitHub and webhook URLs to the entry name.

On failover, the entry name moves. With it move the bots (Slack, Telegram, Discord, GitHub, webhooks), the ingress sockets, and the hosts and clients whose tickets have no signal claim. A peer whose ticket names a machine stays on that machine, and needs a new ticket to move.

/metrics shows openbot_remote_signal_entry{signal="<SIGNAL_ID>"}: 1 while this machine is the entry, 0 when it is not. Without the entry rule the metric is absent. The health routes answer 200 on both machines.

TURN relays

TURN_EXTRA_HOSTS names other coturn servers, comma-separated, that use the same TURN_SHARED_SECRET and the same TURN ports. Signal then gives each host one relay: TURN_HOST or one of these. The host and all its clients get the same relay, and the hosts spread over the relays. A peer gets only one relay, because a browser allocates on each TURN server that it gets.

Signal makes one real allocation on each relay every minute (src/turn-relays.ts). A relay that does not allocate, because it is full or does not answer, gets no new peers until it allocates again. When no relay allocates, Signal gives all of them. When a relay stops under a connected client, the client loses its path, opens a new Signal socket (recoverPath in team-webrtc.ts), gets the relay that Signal picks then, and restarts ICE. /metrics shows the last probe of each relay as openbot_remote_turn_relay_available{host="<relay>"}. /health/turn still reports only the relay of TURN_HOST; the Signal on the machine of each other relay reports that relay.

Each allocation uses one UDP port of the relay range (min-port to max-port in coturn/turnserver.conf), also for a client that connects over TCP or TLS. So the range limits the allocations of one relay: 16384-65535 gives 49,152 ports, and total-quota (40,000) stays below it. A browser allocates for each network interface and for each TURN URL (UDP, TCP and TLS), on both peers, also when the connection does not use the relay. Production had about 4 allocations for each peer connection. When the relay is full, coturn refuses each new allocation with 486, and every connection that needs a relay fails. The signal_capacity analytics event shows the allocations of each relay every 5 minutes.

Production uses turn2.openbot.run on the second Signal machine as the second relay; see the deployment procedure.

Current hosts advertise multiplex: true and support independent device sessions at the same time. Reconnecting one session replaces only that session's Signal socket. Disconnecting or revoking one device must not end another device's session. Legacy hosts that omit multiplex keep the one-client limit: a second session receives host_busy without interrupting the first.

REMOTE_MAX_CONNECTIONS_PER_USER (default 32) limits the authenticated client Signal sockets of one account. Each open desktop, phone, or browser keeps a socket for each saved host. A reconnect of the same session does not count its old socket. Host sockets do not count, and the limit never refuses a host. A host hello moves the clients of the older sockets of the same host to the new socket. The older sockets stay open; when the new socket closes, the last remaining one takes the clients. A host keeps at most 4 sockets: a fifth hello closes the oldest with code 1013, and that host can connect again. Change the value on the Signal server: a desktop environment does not change it. This also applies when the old client socket waits for its host to reconnect. Token signing finishes before Signal admits the new socket, so concurrent reconnects do not hold extra account slots.

Production deployments keep /opt/openbot/remote/.env.production. A value in that file overrides the default, even when a release changes the repository's environment file. Check the running container's effective limit after a configuration change and Signal restart.

See the issue #325 deployment procedure for the production evidence, a Signal-only update, rollback commands, and the required desktop/mobile checks.

To run Signal with your own account service, see Self-hosted remote access.