# Environment protocol This page states the environment protocol's rules: what every peer sends, accepts and answers. It is written for anyone who implements a provider controller, a relay in front of the daemon, or a client. The [provider guide](https://ls.bot/docs/integrating-and-extending/environment-providers.md) explains how to build a provider; this page is the contract that guide refers to. The wire types live in the [environment protocol crate](https://github.com/smartcomputer-ai/lightspeed/blob/main/crates/environment-protocol/src/lib.rs). Its doc comments use wire names and state the per-field rules. Where this page and the crate differ, the crate is authoritative. ## Planes and versions The protocol has three planes. Each runs over its own WebSocket connection. | Plane | Client (dials) | Server | Version number | Purpose | | ------------ | ---------------------- | ------------------------------------ | ----------------------------- | --------------------------------------------------------------------- | | Data | The Lightspeed runtime | `lightspeed-envd` in the environment | `ENVD_PROTOCOL_VERSION` | Filesystem, transfer, process, and job operations in one environment. | | Controller | The Lightspeed runtime | A provider controller | `CONTROLLER_PROTOCOL_VERSION` | Create, observe, power, and close the provider's targets. | | Registration | `lightspeed-envd` | The Lightspeed environment gateway | `ENVD_PROTOCOL_VERSION` | Admit an outbound daemon and ask it for reverse-dialed data streams. | Data and registration currently use `5`; the controller uses `4`. They are matched exactly. A peer that states any other number is refused with `versionMismatch`. There are no version ranges and no negotiation. A change to a plane's wire shape, or to the meaning of an existing field or method, needs a new number for that plane. The data plane and the registration channel share one number, so a data-plane change also changes the number in the gateway's challenge, which is what prompts a registered daemon to upgrade. `release/metadata.env` records the same numbers as `LIGHTSPEED_ENVD_PROTOCOL_VERSION` and `LIGHTSPEED_CONTROLLER_PROTOCOL_VERSION`. `scripts/release/verify-metadata.sh` checks them against the crate. The controller plane does not depend on data-plane types. A provider that changes only its controller does not force a new daemon, and a daemon change does not force a provider redeploy unless the provider bakes the daemon into its images. ## Contract documents The [method manifest](https://github.com/smartcomputer-ai/lightspeed/blob/main/crates/environment-protocol/src/manifest.rs) lists every method with its plane, direction, family, params and response types, and the facts servers act on: the capability a method needs, whether it counts as use of the environment, and whether `$/cancelRequest` aborts it. A method exists exactly when it has a manifest entry. Generated from the manifest and committed under [`crates/environment-protocol/contract/`](https://github.com/smartcomputer-ai/lightspeed/tree/main/crates/environment-protocol/contract): | File | Contents | | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `data.openrpc.json`, `controller.openrpc.json`, `registration.openrpc.json` | One OpenRPC document per plane: every method, its params, result, and summary. | | `data.schema.json`, `controller.schema.json`, `registration.schema.json` | JSON Schema bundles of every type the plane uses. | The controller documents also name two types that no method carries as a whole: `x-lightspeed-data-route`, the data route refusal body with its hold and `Retry-After` bounds, and `x-lightspeed-idle-policy`, the idle policy providers run. Each method also has a fixture under [`fixtures//`](https://github.com/smartcomputer-ai/lightspeed/tree/main/crates/environment-protocol/fixtures), named after the method with `.` for `/`, such as `fixtures/data/process.read.json`. It holds a valid params object, a valid response, and a typed error. The crate's tests decode each fixture strictly and check that the committed contract matches the current types. ## Envelope Every plane uses one JSON-RPC 2.0 envelope: - Each WebSocket **text** frame holds exactly one JSON-RPC message. Binary frames are not part of any plane: a server answers one with `invalidRequest` and a null id, and a client closes the connection. - `jsonrpc` must be `"2.0"`. An unknown member of the envelope is `invalidRequest`. - Params are passed by name, as an object. Absent params mean `{}`. Params that are not an object are `invalidParams`. - A request has an integer or string `id`. A notification has no `id` and is never answered, not even with an error. A malformed frame that names a string `method` and has no `id` is treated as a malformed notification: it is dropped without an answer. - A batch (a JSON array) is answered with one `invalidRequest` error and a null id. - A frame that is not JSON is answered with `parseError` and a null id. - The connection stays open after every framing error. On the data and controller planes, the server never sends a request or a notification. A client that receives one treats it as a protocol violation and closes the connection. The registration channel carries requests and notifications in both directions; see [Registration channel](#registration-channel). ## Handshake The data and controller planes each begin with `initialize`. The server enforces these rules per connection: 1. Before a successful `initialize`, any other request is answered with `notInitialized`, and notifications are dropped. 2. An `initialize` whose `protocolVersion` differs from the server's number is answered with `versionMismatch`. Its `expected` field is the server's number and `received` is the number the client stated. The connection stays open and uninitialized; the client closes it. 3. A second `initialize` after a successful one is `invalidRequest`. 4. The response names its `plane`. A client checks `plane` and `protocolVersion` before it uses the connection, so a client that reached the wrong socket fails at the handshake. `clientInfo` and `serverInfo` describe the peer's build: a `name` and optional `version`, `gitSha`, and `target`, each 1 to 256 bytes without control characters. They are descriptive only. Nothing selects, gates, or authenticates on them. Data `initialize` also carries `platform`: the remote `os`, `pathConvention`, verified shell catalog and optional default shell. Shell entries identify a kind, absolute executable path and observed version. An absent catalog/default does not authorize a client to guess a shell. Windows defaults to verified PowerShell 7, otherwise Windows PowerShell; Git Bash requires explicit selection. Clients validate these facts against cwd/home and execution capabilities before using the connection. The daemon probes installed shells once at startup, recording only successful, bounded version probes. Windows probes PowerShell 7 on PATH and in its standard installation directory, and Windows PowerShell in the system directory. Git Bash requires an adjacent verified Git for Windows installation in a standard system or per-user installation directory; a PATH `bash.exe` is not accepted. Restart the daemon after installing a shell to refresh the catalog. Model-facing `exec_command` and `Bash` accept an optional `shell` enum: `bash`, `powerShell`, `windowsPowerShell`, or `gitBash`. Omission uses the discovered default. `exec_command` preserves Unix Bash's login default; PowerShell and Git Bash default to no profiles. Explicit `login: true` enables profiles. PowerShell pipe requests use `-NonInteractive`; terminal requests omit it. Direct process and job argv bypass these shell adapters. The active environment's model context includes its OS, native cwd/home, shell paths and versions, and relevant execution capabilities. Model file paths and cwd overrides are resolved using the remote path convention, independently of the controller OS. Windows accepts native drive paths or relative paths; the wire still uses canonical `C:/...` spelling. VFS paths retain their separate logical POSIX grammar. ## Decoding Every params, response, and message type on the three planes decodes strictly: an unknown member is refused, and defaults exist only for documented optional knobs. An omitted `protocolVersion` is never assumed. The frozen bootstrap surface is the exception. It is the gateway's first frame (the `registration/challenge` notification), the deployment's daemon discovery document, and the JSON that `lightspeed-envd --print-build` prints. These only gain fields, and they are decoded tolerantly, so a daemon of any version can read them. ## Encoding conventions | Topic | Rule | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Methods | `domain/verb[Object]` in lowerCamel, singular domains, no plane prefix. | | Fields | camelCase. Lists are plural nouns; counts end in `Count`, sizes in `Bytes`, durations in `Ms`, instants in `AtMs`, ids in `Id`. Payload bytes are always `data`. | | Optionality | No `null` except JSON-RPC's own `id`. Absent options and empty collections are omitted. Flags are booleans that default to false and are omitted when false. | | Numbers | Byte counts, offsets, and durations are unsigned 64-bit; counts, indices, and depths unsigned 32-bit; instants are signed 64-bit milliseconds since the Unix epoch. Every value stays below 2^53. | | Unions | Data unions carry a `kind` tag. | | Ids | Lightspeed-assigned ids follow the general grammar `^[A-Za-z0-9][A-Za-z0-9_.:-]{0,127}$`. Provider-assigned ids (`targetId`, `templateId`) are opaque: 1 to 255 bytes without control characters. | | Bytes | Payloads are padded standard base64. Fixed-size cryptographic values are lowercase hex. Content digests are `sha256:<64 hex>`. Opaque fingerprints carry their own prefix, such as `scan:v1:`. | | Paths | `EnvironmentPath`: absolute, normalized, at most 4096 bytes; `.` and `..` segments are refused, not rewritten. Tree entries use relative `TreePath` values; the selected root itself is `.`. | | Text | Descriptive text such as a target `message` is 1 to 1024 bytes, without control characters other than line feed and tab. URLs are at most 2048 bytes. | ## Errors An error is a JSON-RPC error object. Its `code` is fixed by its kind, and its `data` carries the kind and its typed fields: ```json {"jsonrpc": "2.0", "id": 7, "error": {"code": -32004, "message": "unknown process id: p1", "data": {"kind": "notFound", "reason": "expired"}}} ``` `data.kind` is authoritative. Receivers branch on it, never on the code or the message. A message is at most 4096 bytes and is never parsed. Retryability is a property of the kind and is never sent. The same error object appears inside results that report per-entry failures, such as `job/read` entries, `fs/scan` roots, and a failed transfer's status. | Kind | Code | Retryable | Typed fields | Use | | ----------------------- | ------ | --------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `parseError` | -32700 | No | | The frame is not JSON. | | `invalidRequest` | -32600 | No | | A malformed envelope, a batch, a binary frame, a repeated `initialize`. | | `methodNotFound` | -32601 | No | | No such method on this plane. | | `invalidParams` | -32602 | No | `field`, `ceiling` | Params do not decode, a value is out of range, or a value exceeds an advertised ceiling. `field` is the wire path of the offending member, such as `jobs[0].dependsOn`. | | `internal` | -32603 | No | | A defect inside the server. | | `notInitialized` | -32002 | No | | A request before `initialize`. | | `permissionDenied` | -32003 | No | `reason` | Confinement, operating-system permissions, or registration credentials. | | `notFound` | -32004 | No | `reason` | An unknown handle, path, target, or job. | | `versionMismatch` | -32005 | No | `expected`, `received` | The stated protocol version differs. | | `capabilityUnavailable` | -32006 | No | `capability` | The endpoint lacks a capability family or fact the request needs. | | `failedPrecondition` | -32007 | No | `reason` (required) | The current state forbids the request. | | `busy` | -32008 | Yes | `reason`, `retryAfterMs` | A concurrency, capacity, or rate limit. | | `alreadyExists` | -32009 | No | | The thing to create already exists. | | `payloadTooLarge` | -32013 | No | `limitBytes`, `limitEntries` | A message, payload, or result exceeds its limit. | | `deadlineExceeded` | -32015 | No | | A server-side duration bound ran out. | | `expired` | -32016 | No | | A credential or token is past its validity. | | `aborted` | -32017 | Yes | `reason` | Cancelled, or the input changed underneath the operation. | | `processFailed` | -32018 | No | | A process could not be started or supervised. | | `unavailable` | -32019 | Yes | `retryAfterMs` | A dependency of the server is temporarily unreachable. | Reasons by kind: | Kind | Reasons | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `permissionDenied` | `outsideRoot`, `daemonState`, `filePermission`, `invalidSignature`, `invalidRegistrationKey`, `registrationKeyRevoked` | | `notFound` | `expired` (the handle was evicted), `daemonRestarted` (the handle belongs to an earlier daemon instance), `unknownDaemon` | | `capabilityUnavailable` | `filesystem`, `filesystemWrite`, `process`, `processPty`, `job`, `power`, `ingress` | | `failedPrecondition` | `inputClosed`, `ifMatchFailed`, `createCondition`, `notEmpty`, `isDirectory`, `notDirectory`, `notRegularFile`, `symlinkLoop`, `symlinkRejected`, `filesystemRoot`, `crossDevice`, `noSpace`, `jobIdInUse`, `operationIdInUse`, `transferPhase`, `transferOffset`, `digestMismatch`, `environmentClosed`, `identityInUse`, `ownerMismatch`, `templateDeprecated`, `ingressUnsupported`, `targetNotActive` | | `busy` | `requestLimit`, `transferCapacity`, `rateLimited`, `capacityExhausted` | | `aborted` | `cancelled`, `sourceChanged` | An endpoint that cannot honour a parameter refuses the request with `capabilityUnavailable`. It never ignores the parameter. ## Liveness A peer that is waiting for a response sends a WebSocket ping at least every 20 seconds while any request it sent is unanswered. It may also ping while idle. The daemon, providers, and the gateway answer pings and never originate them on data or controller connections. A server answers pings even while it is busy writing responses. Intermediaries, such as a gateway route, a provider relay, or an edge proxy, forward ping and pong frames, count every frame as activity, and never close a connection for idleness sooner than 60 seconds. Request deadlines, not pings, detect a server that has stopped answering. The registration channel has its own heartbeat; see [Registration channel](#registration-channel). ## Limits The protocol has three kinds of limit: 1. **Protocol constants.** Every message, and therefore every frame, is at most 16 MiB (16,777,216 bytes) on every hop. Each reader and writer configures that limit explicitly, with the frame limit equal to the message limit. A sender whose request would exceed it fails locally instead of sending. A server whose response would exceed it answers `payloadTooLarge` for the same id instead. 2. **Endpoint ceilings.** The daemon advertises its operational ceilings in the data-plane `initialize` response as `limits`. A request above a ceiling is `invalidParams` naming the field and the ceiling. Nothing is clamped. 3. **Caller quotas.** Each request states its own bounds as required fields, such as `maxBytes`, `waitMs`, or a `limits` object. Response sizes are bounded by construction. A `process/read` page or one `job/read` response holds at most 4096 chunks plus gaps. Search and glob matches, and the entries, skips, and content of all roots of one scan, stop at a 12 MiB result budget. A scan root or transfer lists at most 100 skipped entries and counts the rest. A leftover-process report lists at most 64 processes, each command cut to 1024 bytes. ## Pipelining and cancellation Clients may pipeline requests. Responses arrive in any order and are matched by id. Servers accept at least 16 requests in flight per connection; the daemon advertises its own number as `limits.maxConcurrentRequests`. Beyond it, a server answers `busy` with reason `requestLimit` at once. On the data plane, the client notification `$/cancelRequest {id}` aborts a pending request. The manifest marks which methods it stops: `process/read`, `job/read`, `fs/searchText`, `fs/globFiles`, and `fs/scan`. They stop waiting or walking and answer `aborted` with reason `cancelled`; an aborted `process/read` never advances the output cursor. Every other method ignores the notification and runs to a consistent state. An unknown or finished id is ignored. The controller plane has no cancellation. When a data connection closes, the daemon cancels that connection's pending requests as if each had been cancelled, and drops their replies. ## Controller plane A provider controller manages targets: machines, containers, or other units of compute. The runtime is the client. Calls go one way only: the runtime calls the provider, and the provider never calls Lightspeed. The provider owns everything that happens to a running target: its power, its idle policy, its ingress, its placement, and any pools or snapshots behind it. Lightspeed decides only two things, that an environment is created and that it is closed, and it retries both until the provider accepts them. Lightspeed records what the provider reports for display; it stores no power or ingress intent and never reverts a change the provider made. ### Transport and credential The controller is a WebSocket endpoint at the URL the operator registers in Lightspeed: `ws://`, `wss://`, or `unix:///absolute/path`. Any path is allowed. The credential is optional. When the provider is registered with one, every connection to the controller and to its data routes carries it as `Authorization: Bearer `; the provider compares it in constant time and answers HTTP 401 before the upgrade when it is missing or wrong. When the provider is registered without one, the runtime sends no `Authorization` header. Run a provider without a credential only where nothing but Lightspeed can reach its listener. ### `initialize` The params carry `protocolVersion` and `clientInfo`. The response carries: | Field | Meaning | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `protocolVersion` | The controller's number. | | `plane` | Always `controller`. | | `serverInfo` | The controller's build. | | `capabilities` | Optional methods: `power` serves `target/setPower`, and `ingress` serves `ingress/ensure` and `ingress/remove`. `template/list`, `target/create`, `target/get`, and `target/close` are mandatory. | ```json { "protocolVersion": 4, "plane": "controller", "serverInfo": {"name": "lightspeed-provider-incus", "version": "0.5.0"}, "capabilities": {"power": true, "ingress": true} } ``` The runtime reads `capabilities` from the live handshake of each connection and never stores it. It refuses an API request for a power or ingress change up front when the provider does not advertise the method. A controller that receives an optional method it does not serve answers `capabilityUnavailable`. ### Methods | Method | Params | Semantics | Errors | | ----------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `template/list` | `universeId` | The templates the universe may create targets from. A provider may offer each universe a different set. | `unavailable` | | `target/create` | `owner`, `templateId`, `displayName?`, `idle?` | Idempotent per owner. If the provider already holds a target for the owner, answer its observation and apply no creation policy (deprecation or image checks). Otherwise apply policy, start creation, and answer promptly with `phase: provisioning`. Handing out a target from a warm pool counts as a create. `idle` replaces the template's default idle policy for this target. | `failedPrecondition` (`templateDeprecated`, `ownerMismatch`), `notFound` (template), `invalidParams` (including an `idle` that does not validate), `unavailable` | | `target/get` | `target` | A pure read: it never creates, starts, or changes anything. Resolves by `targetId` when given, otherwise by owner. | `notFound`, `failedPrecondition` (`ownerMismatch`), `unavailable` | | `target/setPower` | `target`, `power` | Needs `power`. Move the target to one of its template's `powerStates`. A target already there, or moving there, is observed without action. Answer promptly with the transitional observation. The provider stays the owner of power afterwards. | `capabilityUnavailable`, `notFound`, `invalidParams` (a state outside `powerStates`), `failedPrecondition` (`targetNotActive`), `unavailable` | | `target/close` | `target` | Destroy everything the owner's target holds, public ingress included: gracefully within the provider's own bound, then forced. Answer `closing` while it runs, `closed` once nothing remains, and `closed` without `targetId` when the provider holds nothing for the owner. Idempotent. | `unavailable` | | `ingress/ensure` | `target` | Needs `ingress`. Route the template's public ingress to the target. Idempotent. | `capabilityUnavailable`, `notFound`, `failedPrecondition` (`ingressUnsupported`) | | `ingress/remove` | `target` | Needs `ingress`. Stop routing public ingress to a target that stays. Idempotent. | `capabilityUnavailable`, `notFound` | Every target and ingress method answers `{"target": }`. ### Ownership A target belongs to exactly one owner, the Lightspeed environment it serves: ```json {"universeId": "u1", "environmentId": "env_1"} ``` A provider holds at most one target per owner. Every target method names the owner in a target reference, `{owner, targetId?}`. Without `targetId`, the provider resolves the owner's target itself; this is how the runtime finds and closes a target whose id it never learned. A target that belongs to another owner is refused with `failedPrecondition` (`ownerMismatch`). Knowing a provider's target id is never enough to reach a target. An environment id is never reused, so a closed environment's owner cannot reach a later environment's target. Record ownership durably enough to survive the provider's own restarts and an ambiguous create, where the target was created but the response was lost. The Incus provider uses deterministic names and ownership metadata on its native resources; another provider may use its own store. ### Bounded calls Every method answers within a bounded time, a few seconds. Slow work, such as an image pull, a boot, or a graceful shutdown, continues in the background, and the runtime follows it with `target/get`. An asynchronous provider may answer `target/create` before it has assigned an id; the observation then omits `targetId`, and the runtime records the id from the first observation that carries one. ### Observation ```json { "targetId": "ls-3f2a", "owner": {"universeId": "u1", "environmentId": "env_1"}, "phase": "active", "power": "paused", "ingress": {"endpoint": "https://acorn.env.example.com"} } ``` | Field | Meaning | | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `targetId` | The provider's id. Absent only while provisioning before an id exists, and in a `closed` observation of an owner with nothing. | | `owner` | Always echoed from the request. | | `phase` | `provisioning`, `active`, `closing`, `closed`, `failed` (broken beyond the provider's own recovery), or `unknown` (the provider cannot tell). | | `power` | Present exactly when `phase` is `active`: `running`, `paused` (execution halted, memory retained), `suspended` (execution state persisted, compute released), or `stopped` (powered off, disks retained). | | `message` | Why the target is `failed` or `unknown`, or what a transition waits for. | | `ingress` | Realized public ingress, an `http` or `https` URL. Absent when disabled. | Report what the infrastructure shows, not what a caller asked for. An observation says nothing about whether the daemon answers: the runtime proves that itself by dialing the data route. ### Templates A template has a `templateId`, a non-empty `displayName`, and these optional fields: | Field | Meaning | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `description` | A short description for people choosing a template. | | `powerStates` | The power states `target/setPower` can move this template's targets to. Absent or empty when the provider offers no power control beyond create and close. | | `publicIngress` | Targets of this template can serve public ingress. | | `idle` | The idle policy a target gets when `target/create` names none. Absent when the provider applies none by default. | | `deprecated` | The template creates no new targets; existing targets keep working. | | `resources` | `vcpus`, `memoryBytes`, and `diskBytes` in provider-neutral units, each omitted when the provider does not fix it. | | `metadata` | Provider display facts, such as an image fingerprint. | ### Idle policy An idle policy names when a provider powers down or closes an idle target: ```json {"pauseAfterMs": 600000, "stopAfterMs": 3600000, "closeAfterMs": 86400000} ``` `pauseAfterMs`, `suspendAfterMs`, `stopAfterMs`, and `closeAfterMs` are each optional, but a policy sets at least one. Every set threshold is positive, and the set thresholds are non-decreasing in the order pause, suspend, stop, close. A policy that breaks these rules is `invalidParams`. The policy belongs to the provider. A template names a default, and `target/create` may replace it for one target; the runtime validates a policy and passes it through without interpreting it. The provider runs the clock. For a running target it reads the daemon's `env/getIdle` over its own channel to the guest and applies the most escalated stage whose threshold `idleForMs` has reached and whose power state the template supports; `close` always qualifies. The protocol crate's [`IdlePolicy::due_action`](https://github.com/smartcomputer-ai/lightspeed/blob/main/crates/environment-protocol/src/idle.rs) implements that rule. Running processes, running jobs, and leftover process groups hold `idleForMs` at zero, so they keep the target awake. ### Data routes A provider serves the daemon's data plane for each target on a data route: the provider's data route base followed by `/{universeId}/{environmentId}/{targetId}`, each segment percent-encoded (everything except RFC 3986 unreserved characters). The runtime dials it with the bearer credential when one is configured. The provider serves the data plane unchanged on that route, under the liveness rule and the message limit above, and refuses a route whose owner does not own the target. The data route base is a `ws://` or `wss://` URL with a lowercase scheme, a host and optional port, an optional path that does not end with `/`, and no user information, query, or fragment. The operator may set it when registering the provider. Without one, the runtime derives it from the controller URL: the controller's scheme, host, and port followed by `/routes`, so `wss://provider.example.com/control` gives `wss://provider.example.com/routes`. A `unix://` controller has no derivable base, so its registration must name one. A configured base for a controller reached over `wss` (or `https`) must itself use `wss`, so the credential never travels in cleartext; registration refuses a downgrade. A dial of a powered-down target's data route wakes the target. This is the only wake signal: Lightspeed has no separate wake call. The provider may hold the dial for at most 5 seconds (`MAX_DATA_ROUTE_HOLD_MS`) while it resumes the target. If the daemon answers in time, the route upgrades and relays as usual. Otherwise the provider refuses the upgrade with a JSON body, `Content-Type: application/json`, tagged by `state`: | Status | Body | Meaning | | ------ | --------------------------------------- | ---------------------------------------------------- | | 503 | `{"state": "waking"}` | The target is resuming from a powered-down state. | | 503 | `{"state": "booting"}` | The target runs, but the daemon does not answer yet. | | 410 | `{"state": "gone"}` | The target no longer exists. | | 409 | `{"state": "failed", "message": "..."}` | The target is broken beyond the provider's recovery. | Every 503 carries `Retry-After` in whole seconds, from 1 to 60 (`RETRY_AFTER_MAX_SECS`); receivers clamp a larger value. A 503 may succeed on a later dial; 410 and 409 do not. The body is an HTTP refusal, not a JSON-RPC message. The controller schema bundle names it as `x-lightspeed-data-route`, together with both bounds. The environment gateway's worker route uses the same body. It adds three states of its own and forwards a provider's refusal unchanged, status and `Retry-After` included: | Status | Body | Meaning | | ------ | --------------------------- | ----------------------------------------------------------------- | | 503 | `{"state": "provisioning"}` | The environment is still provisioning. | | 503 | `{"state": "disconnected"}` | The registered daemon is not connected. | | 410 | `{"state": "away"}` | The registered daemon has been away longer than it is waited for. | ### Provider obligations The provider owns its targets' power, idle policy, and ingress. Every provider: 1. makes `target/create` and `target/close` idempotent per owner; 2. wakes a powered-down target when its data route is dialed, holding the upgrade at most 5 seconds before it answers a typed refusal; 3. decides idleness from the daemon's `env/getIdle`, so running processes, jobs, and leftover process groups keep the target awake; 4. never powers a target down while it relays a data connection to it; 5. reports `closed`, or `notFound` from `target/get`, for a target it removed on its own, such as by a lifetime limit or an idle `closeAfterMs`; 6. never snapshots or clones a daemon state directory that holds a daemon identity. ### How the runtime drives a controller Providers can rely on this behaviour: - Until it has recorded a target id, the runtime sends `target/create`, with the same `displayName` and `idle` on every attempt. Afterwards it observes with `target/get` and never sends create again. - `phase: provisioning` is checked again after 2 seconds. - `phase: active` leads the runtime to run the data-plane handshake on the target's route. This handshake is the only readiness proof: success opens the environment and records the daemon's build. A daemon that speaks another protocol version fails the environment, naming both versions. A 503 refusal is retried after its `Retry-After`; `gone` and `failed` refusals fail the environment. - `phase: failed` fails the environment with its `message`. `closing` or `closed` before the environment opened fails it too. `phase: unknown` is retried with backoff. - Retryable kinds, `internal`, transport failures, timeouts, and failed handshakes are retried with backoff: 2 seconds, doubling per attempt, at most 5 minutes. Other refusals, `notFound` included, fail a provisioning environment at once, so reserve them for requests that can never succeed. An environment that has not opened within 30 minutes of entering provisioning fails with the last recorded problem. - Every 5 minutes, the runtime reads each open environment's target with `target/get` and records the observation for display. `phase: failed` fails the environment. `closed`, or `notFound`, closes it with the reason `providerEnded`. This sweep never changes power. - Close sends `target/close` by owner, with the target id when one is recorded, until the provider answers `closed` or `notFound`. `closing` is checked again after 2 seconds, and errors are retried with backoff. Close has no deadline, and the runtime never removes ingress before it. - `target/setPower` and the ingress methods are sent only when a user asks for a change, synchronously, with a 30-second deadline, and only when the controller advertises them. No loop calls them, and nothing retries them. - Connect and `initialize` each have a 10-second deadline, and the runtime's other controller calls a 120-second deadline. The runtime keeps at most eight calls in flight per provider, over one connection. ## Data plane The daemon, `lightspeed-envd`, is the data server on every provider. The data plane is specified for that daemon; a provider routes to it rather than reimplementing it. ### `initialize` The response carries `protocolVersion`, `plane: data`, `serverInfo`, and: | Field | Meaning | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `daemonInstanceId` | Random per daemon start. A different value than on an earlier connection means the daemon restarted and the earlier instance's process handles are gone. | | `capabilities` | Capability families (below). | | `limits` | The daemon's ceilings: `maxConcurrentRequests`, `maxWaitMs`, and per-family `process`, `job`, and `filesystem` limits. | | `defaultCwd` | The daemon's working directory, the default `cwd` of commands. | | `homeDirectory` | The home directory of the user commands run as, when it lies inside the filesystem root. | A present `filesystem` capability covers `fs/*`, including search, glob and scan. `filesystem.transfer` separately enables `transfer/*`; an omitted flag means transfers are unavailable. `process` covers `process/*`; `job` covers `job/*`. `filesystem.write` is false on a daemon started read-only, and `process.pty` says whether `process/start` accepts `pty`. A request that needs an absent family or fact is `capabilityUnavailable` before any side effect. ### Methods | Method | Counts as use | Abortable | Needs | Purpose | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | --------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- | | `initialize` | No | | | Handshake. | | `env/getIdle` | No | | | `idleForMs` since the last use, 0 while any process root or job runs or any leftover process group has live members, and diagnostic counts. | | `$/cancelRequest` | No | | | Notification: abort a pending abortable request. | | `fs/readFile` | Yes | | `filesystem` | Read at most `maxBytes` of a regular file from `offset`. | | `fs/writeFile` | Yes | | `filesystem.write` | Write a file atomically, with `create`, `ifMatch`, and `executable`; answers the new `version`. | | `fs/createDirectory` | Yes | | `filesystem.write` | Create a directory, optionally with its parents. | | `fs/getMetadata` | Yes | | `filesystem` | Describe one entry, optionally following a symlink. | | `fs/readDirectory` | Yes | | `filesystem` | List one page of a directory, sorted by name, with an opaque cursor. | | `fs/remove` | Yes | | `filesystem.write` | Remove a file, symlink, or directory without following links. | | `fs/rename` | Yes | | `filesystem.write` | Rename atomically, without replacing an existing destination unless `overwrite` is set. | | `fs/searchText` | Yes | Yes | `filesystem` | Search file contents below a root with a regular expression. | | `fs/globFiles` | Yes | Yes | `filesystem` | List non-directory entries below a root that match patterns. | | `fs/scan` | No | Yes | `filesystem` | Observe small trees below several roots, each reported on its own. | | `transfer/begin`, `transfer/advance`, `transfer/getStatus`, `transfer/readInventory`, `transfer/append`, `transfer/listMissing`, `transfer/read`, `transfer/write`, `transfer/commit`, `transfer/abort` | Yes | | `filesystem.transfer` (writes also `filesystem.write`) | One capture, materialize, or file write as many bounded calls. | | `process/start` | Yes | | `process` | Start a process under a caller-chosen `processId`. | | `process/read` | Yes | Yes | `process` | Wait for and read a page of output and the process state. | | `process/write` | Yes | | `process` | Queue bytes for a pipe or PTY input. | | `process/terminate` | Yes | | `process` | Interrupt, terminate, or kill the owned group or job using the advertised mechanisms. | | `job/start` | Yes | | `job` | Start jobs, or replay an identical earlier start. | | `job/read` | Yes | Yes | `job` | Read jobs' summaries and output pages. | | `job/cancel` | Yes | | `job` | Cancel jobs and optionally their dependents. | Methods that count as use reset the environment's idle clock. Handshakes, idle reports, cancellations, and scans do not, so discovery and a provider's idle policy can look at an environment without keeping it awake. The idle clock also restarts when the last running process, job, or leftover process group ends: a service that a command started and left running keeps `idleForMs` at zero like running work. ### Paths and confinement A wire path is absolute, in the environment's own filesystem namespace: the one its processes see. The daemon's filesystem root is not a chroot. With the root narrowed to `/workspace`, `/workspace/src` names that directory and `/etc` lies outside the root. One rule applies to every method and to every command's `cwd`: - A path that is neither the root nor below it is `permissionDenied` (`outsideRoot`). - The daemon resolves the remaining segments one at a time beneath the root. A symlink is followed only while its target stays inside the root; a link that leaves it is refused the same way. - No method addresses the daemon's state directory. A path that names it or lies inside it, and a remove, rename, write, or capture whose subject contains it, is `permissionDenied` (`daemonState`). Walks prune it. Paths are UTF-8. POSIX spellings remain unchanged; Windows drive paths use an uppercase drive and `/` separators (`C:/work`). Native conversion takes the remote convention explicitly and renders Windows paths with backslashes. UNC, drive-relative and device namespaces, alternate streams, reserved device names and trailing dots/spaces are unsupported. Windows names also exclude control characters and Win32-reserved punctuation. Path equality and prefixes are lexical, not filesystem identity or authorization; confinement needs native handles and identity checks. Entries that cannot be represented are skipped and counted. Relative transfer paths retain their grammar but must be validated against the destination convention before mutation. VFS paths are unchanged. The native Windows backend opens path components relative to directory handles, retains ancestors against rename and junction replacement, and excludes daemon state by volume/file identity. Writes stage privately and preserve the target's DACL when replacing an existing file. New files retain the destination's inherited DACL. Same-directory rename is supported; cross-directory rename and `executable: true` are `capabilityUnavailable`. Executable metadata is false on Windows. Publication is atomic, but directory crash-durability is not promised. Windows advertises `filesystem.transfer: true`. Captures, publication to a new tree path, and atomic file writes are supported. Replacing an existing materialized tree is `capabilityUnavailable`; the daemon never emulates an atomic tree replacement by deleting the destination first. ### Pattern grammar and walks `fs/searchText`, `fs/globFiles`, `fs/scan`, and transfer captures share one filter: `include` and `exclude` patterns, `ignoreCase`, `includeHidden`, `includeIgnored`, and `maxDepth`. Patterns match paths relative to the walk's root, gitignore-style: `*` and `?` match within one segment, `**` spans segments, `[...]` is a character class, and `{a,b}` an alternation. A pattern without `/` matches the entry's name at any depth; a leading `/` anchors it to the root. An empty `include` selects everything, and an `exclude` that matches a directory prunes it. The root's direct children are depth 1. By default, walks respect `.gitignore`, `.ignore`, and git excludes, and skip names that start with `.`. `includeIgnored` and `includeHidden` opt out. `.git`, `.hg`, `.svn`, `.jj`, and the daemon state directory are always pruned. Entries a walk cannot read or represent are skipped and counted; they never end the walk. A file larger than the remaining byte budget is skipped and counted too. ### Process output Process and job capabilities advertise `tree` (`posixProcessGroup` or `windowsJobObject`) and `termination` (`gracefulThenKill` or `immediateKill`). Immediate termination has zero grace. Processes additionally advertise separate `interrupt.pipe` and `interrupt.pty` flags; unsupported interruption is `capabilityUnavailable`, never a substitute forced kill. A Windows ownership unit is a non-breakaway Job Object attached before the child runs. Normal process root exit keeps descendants; job root exit sweeps them. Timeout, cancellation, environment close and daemon death terminate the owned tree. The daemon keeps non-inheritable kill-on-close handles while any owned descendants survive, independently of retained output. Restart never adopts raw Windows PIDs. POSIX group and restart behavior is unchanged. Windows returns native exit codes through the signed i32 `code` field, preserving all 32 bits; it never represents Windows termination as a Unix signal. Native pipe execution preserves bytes and supports closing stdin. ConPTY uses UTF-8 terminal streams and supports Ctrl-C; pipe interruption is unavailable. Closing ConPTY input with `close: true` is `capabilityUnavailable`, because closing the pseudoconsole would terminate attached processes rather than supply EOF. Terminal dimensions are set at start; the backend supports resize, but the protocol currently has no resize method. Direct Windows execution requires an `.exe`; scripts and batch files require an explicit interpreter. A process has one combined output stream: stdout, stderr, or PTY output interleaved in arrival order. Positions are byte offsets into it. `process/read` returns a page of `chunks`, the `next` offset after the page, the `end` offset of everything produced so far, and `gaps` for ranges it cannot return (`overflow` when the retention cap dropped them, `expired` when they were released after delivery). A page cut at `maxBytes` loses nothing. A read starts at `after`, at `end - tailBytes`, or at the daemon-owned cursor when it names neither. A cursor read advances the cursor only after the reply was written to the socket, so a lost connection never consumes output. Reads at an explicit `after` or `tailBytes` never move the cursor, so a second reader can follow without consuming. An optional `ack` advances the cursor to an offset the caller has already processed, which gives a caller that tracks positions exactly-once delivery. `returnOn` chooses when a wait ends: `anyOutput`, `window` (enough bytes for `maxBytes`), or `exit`. Every mode returns when `waitMs` elapses. `state` is `running` or `exited`. `exited` means the root process exited and its output has settled. An exit is `{kind: code, code}` or `{kind: signal, signal: "SIGSEGV", coreDumped}`, with signals named because their numbers differ by platform. `stoppedBy` (`timeout`, `terminate`, `kill`) is set when the daemon stopped a running root. A process that leaves descendants running reports them as `leftovers` with disposition `kept`; processes never sweep their group on a natural exit. An unknown handle is `notFound`. An evicted handle is `notFound` with reason `expired`, and a handle from before a daemon restart is `notFound` with reason `daemonRestarted`. ### Jobs A job's identity is its `jobId` together with the `requestId` of the `job/start` that created it. A start that repeats an existing job's `requestId` and spec replays it; any other reuse of the id is `failedPrecondition` (`jobIdInUse`). Statuses are `queued`, `running`, `succeeded`, `failed`, `timedOut`, `cancelled`, `dependencyFailed`, and `interrupted`. Dependencies name a job id or a name within the same request. Jobs that share a `queueKey` run one at a time, in start order. At job completion, the daemon sweeps the job's owned group or Job Object and reports what it stopped as `leftovers` with disposition `swept`. `job/read` takes a list of entries, each reading one job from `after` or the last `tailBytes`; a job may appear twice to read its head and tail in one call. The request's `maxBytes` is a total budget across entries. An unknown job is answered with a per-entry `notFound` error. ### Files and transfers `fs/writeFile` stages the new content next to the final target and renames it into place, so readers see the old or the new file and never a partial one. An in-root symlink at the path is followed to its final target. An existing target keeps its mode. `ifMatch` makes the write conditional on the version a read returned. A transfer moves a whole file or tree in bounded calls under a caller-chosen `operationId`. Retrying a call with the same input is safe. A capture reads a tree out of the environment; a materialize replaces a destination tree atomically at commit; a file write replaces one file with `fs/writeFile` semantics. Content moves by `sha256` digest, so content already present is not sent again. The first failure moves the operation to `failed`; later calls answer that error and nothing is committed. Finished operations keep a receipt for the advertised retention period. The [transfer module](https://github.com/smartcomputer-ai/lightspeed/blob/main/crates/environment-protocol/src/data/transfer.rs) documents the phases. Windows validates incoming inventory names against its native path rules and rejects case-colliding paths, including collisions across inventory pages. Parents must precede children with the same spelling. Rejected pages leave the inventory unchanged. Unix executable bits cannot be represented on Windows: `executable: true` in a write selection or inventory is `capabilityUnavailable`. VFS tree paths keep their portable grammar; these checks occur when publishing into Windows. Windows stages the selected root beside its destination and retains ancestor handles through publication. Staging is private to the execution account and SYSTEM. Materialized trees retain that protected ACL; file writes restore the existing target's DACL or, for a new file, its inherited DACL. File writes recheck `create` and `ifMatch` at commit. Existing-tree materialization is refused at begin, or at commit if a destination appeared during the operation. Transfer journals tag native path encoding: Unix bytes or Windows UTF-16LE. Legacy untagged records are Unix records. Foreign or undecodable records are quarantined, never interpreted as native cleanup paths. A daemon restart fails interrupted operations and retries staging/spool cleanup; a sharing violation retains the cleanup record even after its receipt expires. A restart during commit reports an uncertain outcome because publication may already have occurred. Completed captures retain their inventories with their receipts. Windows flushes journal files before write-through replacement; this recovery contract covers daemon restart and does not promise directory durability after power loss. ## Registration channel An outbound daemon dials `wss://[/prefix]/environment-gateway/connect` and keeps that control connection open. The daemon is the JSON-RPC client and the gateway the server; both directions carry requests and notifications. ```text gateway -> daemon notification registration/challenge {protocolVersion, nonce} always first daemon -> gateway request registration/register {protocolVersion, daemonPublicKey, signature, registrationKey?, displayName?, metadata?, implementation} gateway -> daemon response {receipt, heartbeatIntervalMs} or an error gateway -> daemon request registration/openStream {token, stream} any number of times daemon -> gateway response {} once the stream socket is upgraded, or unavailable daemon -> gateway notification registration/goodbye {reason: shutdown} before a deliberate stop gateway -> daemon notification registration/closed {reason, message?} superseded or environmentClosed ``` Rules: - The challenge is frozen. A daemon reads its `protocolVersion` before decoding anything else, and a mismatch leads to an upgrade or a refusal, never to a decode loop. - The daemon signs `"lightspeed-envd-registration/v2" ‖ nonce ‖ sha256(origin)` with its Ed25519 key, where `origin` is the canonical WebSocket origin of the gateway URL it dialed. The gateway accepts a signature over any of its configured registration origins: the canonical origin of its public base URL, plus any further registration URLs the operator lists, such as a cluster-internal service address. Every other origin is refused, so a gateway cannot relay another gateway's challenge. - The registration key is sent only until the daemon's first admission. `displayName` and `metadata` are read only then. - Before admission the gateway accepts only `registration/register`. Out of bounds build facts, display names, or metadata refuse admission with `invalidRequest`; other undecodable params are `invalidParams`. - The receipt names `environmentId`, `daemonId`, `controlConnectionId`, `identityMode` (`persistent` or `ephemeral`), and `admission` (`created` or `reconnected`). - `registration/goodbye` from a persistent identity marks its environment disconnected; the environment stays open and waits for the daemon to return. From an ephemeral identity, it closes the environment. - For each `registration/openStream`, the daemon dials the stream URL derived from its configured connect URL, replacing the `/environment-gateway/connect` suffix with `/environment-gateway/data`, with the one-time token as a bearer header, and serves the data plane on that socket. The gateway never names a URL. - The gateway pings every `heartbeatIntervalMs`. Either side treats three intervals without any frame as a dead connection. - The newest connection for a daemon identity wins. The gateway sends `registration/closed` with reason `superseded` to the older one before closing it. - When the environment closes while its daemon is connected, the gateway sends `registration/closed` with reason `environmentClosed`. A closed identity that registers again is refused with `failedPrecondition` (`environmentClosed`). Either way the identity is spent: the daemon exits or parks, as configured, and never reconnects. The daemon's exit status says why it stopped: 0 after a deliberate shutdown, 1 for an unexpected failure, 2 for a local configuration or receipt-file error, 3 for a terminal registration refusal or an unsupported gateway protocol, 4 when its environment was closed, and 5 when another process with the same identity superseded it. A daemon configured to park on close instead stops registering and listening, waits for `SIGTERM` or `SIGINT`, and then exits with 0. Retryable refusals, disconnects, and heartbeat timeouts never end the process. ## Conformance Two suites check an implementation against this contract without provisioning anything else: - The controller suite, [`environment_client::conformance::controller`](https://github.com/smartcomputer-ai/lightspeed/blob/main/crates/environment-client/src/conformance/controller.rs) behind the client crate's `conformance` feature, drives any controller URL with one template, and with the credential when the provider uses one. It checks the handshake rules, the credential refusal, per-owner create idempotency, pure `target/get`, owner fencing, close by owner and its repetition, strict decoding of every observation, and `capabilityUnavailable` for methods the controller does not advertise. It creates and closes one target. - The data-route helpers, [`environment_client::conformance::data_route`](https://github.com/smartcomputer-ai/lightspeed/blob/main/crates/environment-client/src/conformance/data_route.rs), check that a refused data route answers a typed body with the status it names, a 503 with `Retry-After`, within the hold bound. - The data suite, [`tools::environment_protocol::conformance`](https://github.com/smartcomputer-ai/lightspeed/blob/main/crates/tools/src/environment_protocol/conformance.rs), drives a data endpoint and checks file writes and reads, confinement, walks, scans, process output and exits, and job identity. When transfers are advertised it also checks capture, chunked file publication, stale-version rejection at commit, and receipt replay. It works in a fresh directory and removes it afterwards.