Troubleshoot a deployment

Start with the last boundary that worked. If sign-in succeeds but a universe will not open, investigate the Platform’s runtime access. If a run is accepted but does not progress, inspect its state and workers. This narrows the problem without repeatedly restarting services that are already doing their job.

Record the deployed release, time, exact error, universe UUID, and the affected session/run, bot, environment, or channel account IDs. Capture the relevant logs before changing state. Keep credential values and private conversation content out of reports unless they are necessary and the report’s recipients are authorized to see them.

Locate the failing boundary

Symptom	Start here
Runtime or Platform does not start	Startup and configuration
Sign-in or universe access fails	Authentication and universe access
A run is queued, waiting, or apparently stuck	Session progress
The model rejects a call or tools are absent	Models and MCP
A machine is offline or a command cannot see a file	Execution environments
A larger upload or stored payload fails	Blob writes
Chat messages do not reach the agent or return	Chat delivery
Closing work does not reclaim storage or compute	Retention and cleanup

The operations guide lists health endpoints and their limits. A runtime /health response proves the listener is serving; it does not check Temporal pollers, model access, or useful session progress.

Startup and configuration

For the self-hosting containers, inspect the process output:

docker logs --tail 100 lightspeed-runtime
docker logs --tail 100 lightspeed-platform

If the runtime exits before serving, check the error against these dependencies:

Error area	What to inspect
PostgreSQL connection	The runtime database URL, credentials, network path, and required connection options. Container-local localhost is not the database host.
Schema verification	Use the deployed release’s schema-version diagnostic. The runtime never migrates implicitly; follow the explicit migration procedure.
Temporal connection	The frontend address and an existing namespace, reachable from the process.
Object-store configuration	A nonempty bucket when any LIGHTSPEED_OBJECT_STORE_* variable is set, plus the intended endpoint and credentials.
Role or environment routing	Valid roles and the internal environment gateway URL/token on every process without that role.
Secret configuration	A base64 master key decoding to 32 bytes, matching the stored encrypted state.

The Platform needs its own database and authentication settings. Its migrations run before HTTP serving. Do not point it at the runtime database or reset either database to work around a startup error. Use Configuration for settings and Upgrades and recovery for ledger failures.

For local development, ./dev.sh status reports the supervised stack. Check the relevant service’s logs and prerequisites before restarting it.

Authentication and universe access

If sign-in fails, compare the browser’s actual origin with LIGHTSPEED_PLATFORM_BASE_URL and any configured trusted origins. Check HTTPS termination and forwarded host/scheme handling. Retain the Platform’s authentication secret across restarts.

The administrator bootstrap variables apply only while the users table is empty. They cannot reset an existing password. Use Admin → Users from an authorized administrator account for account management.

When sign-in works but a page fails, distinguish a Platform permission denial from an upstream runtime error. Sessions, profiles, workspaces, integrations, and setup controls require owner/admin or platform administrator access. The ordinary member role is narrower.

For runtime errors, verify that the Platform calls a reachable private trusted-header gateway and that its mapped runtime UUID exists. The browser slug is not the runtime UUID. Admin → Universes can identify a missing runtime or Platform record; creating an empty missing universe does not restore its former contents.

For a direct API client, confirm the listener’s authentication mode. An lsk_ key authenticates at an API-key gateway, and the client must not also send tenant/principal headers. A trusted-header listener does not switch modes because a bearer key is present. The access guide covers keys, roles, and the separate offboarding steps for membership and runtime access.

Session progress

Open the session and inspect the current run and pending work before assuming a worker failure. A later submission can queue behind an active run. A run waiting for tool approvals, a timer, an environment job, or a sub-agent can also be behaving as intended. Complete all required approval decisions and inspect the source it is waiting for.

If there is no expected wait, inspect Temporal using the correct namespace and workflow ID, <universe-uuid>/<session-id>. Check workflow status, pending activities, failures, and pollers on the configured session queue. Workflow and activity polling must both be available. A process serving the bots queue does not substitute for a missing sessions worker.

Use the Temporal execution chain when a session has continued as new. The Temporal run ID differs from the Lightspeed run ID shown for one agent task. Operations explains the log identifiers.

If Temporal says the workflow is terminal or missing while the session still shows an active run, look for session_workflow_failed, stale_active_projection, or session_bootstrap_failed. The maintenance reaper does not automatically resume an arbitrary failed workflow. Preserve the history and error, then follow recovery guidance. Terminating, resetting, or force-closing work changes its lifecycle; it is not an ordinary retry of a read request.

See Sessions and runs for queueing, steering, cancellation, and closure behavior.

Models and MCP

For a model failure, inspect the selected provider ID, API kind, and model name together. Verify the integration in the same universe as the session. Then check the provider’s returned error: authentication, quota, unsupported parameters, and network failures require different corrections.

A disabled or unusable universe provider record deliberately blocks a deployment fallback key. Removing the record permits fallback again, so do not confuse removal with disabling access. A coding-agent subscription login also does not authenticate Lightspeed’s own session inference. Follow Models and credentials.

If discovery succeeds but an agent has no expected tool, inspect its profile’s capability grants and the MCP server/tool selection. Registering a server does not grant every session permission to use it. Also distinguish a tool waiting for approval from one that was never exposed.

For private MCP endpoints, native discovery and execution require both the server’s private-network opt-in and the runtime’s host/CIDR allowlist. OAuth network permission is separate. Provider-hosted execution requires the model provider to reach the MCP endpoint, which is a different network path from the runtime reaching it. Tools and MCP explains execution modes and discovery behavior.

Execution environments

Begin with the environment’s source, lifecycle, and power state. A paused or stopped VM needs a wake path; an offline registered machine needs its daemon; a provisioning failure needs the provider and template that created it.

For a registered daemon, inspect its own output and both public WebSocket paths. The control connection can be established while data routing is still broken. Verify the deployment has one environment gateway, that the proxy preserves upgrades, and that runtime processes use the correct internal URL and token. Retain the daemon state directory so a restart proves the same identity. A closed identity cannot reconnect as a new environment merely by reusing its key.

For Incus, inspect provider /health, the universe provider binding, selected template, instance boot, and guest daemon. Follow the specific procedures in Bring your own compute, Incus VMs, and Networking and ingress.

If a process cannot see a workspace file, check which filesystem contains it. VFS workspace files do not automatically appear on an execution environment’s disk. Transfer required inputs explicitly and verify the process working directory. Processes and jobs also explains why a daemon restart interrupts persisted active jobs and loses interactive process handles.

Chat delivery

Follow delivery in three stages: connector account, runtime routing, and outbound response.

1. Inspect the connector host’s /healthz and expected account inventory. Verify its provider/account selection, runtime API access, Temporal namespace, and current provider login. Platform’s Waiting for connector display can mean health aggregation is missing; check LIGHTSPEED_PLATFORM_CHANNELS_HEALTH_URLS before concluding the host is down.
2. If the account is ready but no agent runs, inspect the chat trigger, pairing, sender allowlist, group activation policy, and paused bot/trigger state. Connector readiness does not establish that a message has a route to a bot.
3. If the agent produced a response but chat did not receive it, inspect the delivery activity and the connector’s per-account activity worker. Check provider authentication and WhatsApp linked-device state where applicable.

Also check discovery freshness: /readyz can remain successful after a later discovery failure, and a host with no selected accounts can be ready. Duplicate hosts serving the same account can conflict, particularly when polling one Telegram token. Use nonoverlapping account ownership as described in operations.

Chat channels provides the complete account, pairing, and routing setup.

Blob writes

If the runtime starts successfully but a larger file, attachment, model payload, or diagnostic dump fails with blob exceeds inline threshold (65536 bytes) but no object store is configured, configure S3-compatible storage on the runtime processes using CAS. The current PostgreSQL-only setup accepts blobs up to 64 KiB; larger blobs require that backend. Follow Choose the blob backend and retry a controlled write after restarting the affected processes.

When object storage is already configured, distinguish a failed write from a missing existing object. Check its endpoint and permissions for writes; for reads, also check the exact physical location retained by the blob catalog. A bucket or endpoint change does not copy the old bytes.

Retention and cleanup

Closing a session retains history unless a deletion policy later removes it. Automatic deletion waits for the root’s configured deadline and for its retained descendant tree to be closed. Blob collection then follows durable references and its own grace period; repeated reads do not renew that period.

A machine’s lifecycle is separate. Inspect environment ownership, retention, and provider cleanup before expecting a session close to destroy a VM. A bring-your-own machine remains under its operator’s control. The retention guide and environment cleanup guide explain the individual decisions.

Universe archive is also separate from shutdown. It does not stop automation or revoke access, and permanent purge is not a complete infrastructure cleanup operation. Follow Multitenancy when retiring an entire tenant.