Zero-Downtime Deployments in Financial Systems: Shipping Code Without Breaking the Market

Most industries deploy code. Financial systems deploy risk.

Financial systems rarely offer a clean window in which nothing is happening. During a rollout, markets may still be trading, payments may still be settling, and downstream systems may still be acting on previously published data. A release therefore enters a system already in motion, carrying open orders, changing balances, unsettled positions, and active risk decisions.

The consequences of failure are different too. In many software systems, a bad deployment creates downtime, broken workflows, or delayed user actions. In financial systems, it may produce duplicate orders, stale risk checks, broken hedges, inconsistent balances, or unintended exposure. The dangerous case is a system that remains available while producing incorrect results.

Deployment safety in finance is therefore not limited to service reachability. The system must preserve correctness while money moves.

What Zero-Downtime Means

In financial systems, zero downtime means preserving the system's external contracts throughout a deployment. Clients, counterparties, downstream services, and operators must not observe behavior that violates defined guarantees.

For some part of the rollout, old and new service versions will coexist. They may serve requests, consume events, and interact with shared state concurrently, so compatibility must extend beyond API shapes to behavior, event formats, and database schemas. Neither version should make assumptions that render the other unsafe. (See: Schema Evolution Under Live Order Flow)

The transition must also preserve work already in progress. Orders, settlements, risk updates, and ledger operations cannot be dropped during a restart or repeated because processing authority moved between instances or versions.

Every deployment also needs a safe recovery path. That may mean rolling back, rolling forward, disabling new behavior, or cutting traffic back to the previous path. The available options depend on how far the deployment has progressed: whether code has merely been rolled out, new behavior has been released, or processing authority has already been transferred.

Three transitions matter during a deployment, and they may occur at different times:

• Rollout: propagating new code through the fleet.
• Release: exposing new behavior to users or downstream systems.
• Cutover: transferring authority from one version, path, or system to another.

Keeping these transitions separate limits how much changes at once and preserves more recovery options. Code can be rolled out with new behavior disabled, released gradually, and given authority only after the new path has been validated.

Safety must hold through every intermediate state: while versions are mixed, traffic is partially shifted, or processing authority is moving between paths.

A deployment is safe only if every externally visible state during the transition satisfies the system's contracts and business invariants.

How Deployments Fail

An outage is only the most visible form of deployment failure. A rollout can look healthy on the surface: instances pass readiness checks, latency remains stable, and error rates stay low. Meanwhile, a gateway may acknowledge an order that was never durably recorded, a replayed event may trigger the same action twice, or two processors may both believe they have authority to act.

These failures can be partial and delayed: they may affect a particular instrument, venue, or partition, and may not become visible until reconciliation or settlement processing. Deployment safety must therefore account for lost work, duplicate actions, stale decisions, and conflicting ownership.

Accepted work is lost

A gateway can acknowledge an order before durably recording it or handing it to the execution service. If the gateway then restarts, the client believes the order was accepted, but the platform has no record of it.

Consumer restarts do not inherently lose work. Durable queues and pending-entry recovery allow another consumer to resume processing, but message acknowledgement and side effects must be coordinated with recovery. Loss occurs when work is acknowledged too early, retained only in memory, skipped during recovery, or an external side effect occurs without enough durable state to recover it safely.

One instruction executes twice

A restarted consumer may replay messages from its last durable checkpoint. If it submitted an order but crashed before recording the exchange’s response, replaying the message may submit the order again.

Timeouts create the same ambiguity. An order gateway may send an order to an exchange and time out before receiving the response. The exchange may already have accepted the order. Retrying with a new identifier can place a second live order.

Risk decisions use stale state

Restarts can increase consumer lag and leave caches incomplete while they rebuild. If recent fills have not reached the position state, a risk service may understate current exposure and approve an order that should have been rejected. The defect lies in the completeness and freshness of the input state.

Execution authority overlaps

Multiple instances may run concurrently, but each non-idempotent action needs one effective owner. During cutover, the old and new processors may both believe they are responsible for the same order or partition. Both can then submit the order or commit conflicting state transitions.

The handoff needs a durable boundary, such as an ownership epoch or fencing token, that prevents a former owner from committing further work.

Dependent operations are reordered

Restarts and parallel replay can change the order in which related events reach internal services. A fill may be processed before the corresponding order state has been restored, or risk capacity reserved for an open order may be released before the venue confirms its cancellation.

Causally dependent operations must either retain their required order or be handled safely when they arrive late, out of order, or more than once.

Deployment patterns do not eliminate these failure modes. They only control how the new version is introduced and how much production traffic or authority it receives.

Deployment Patterns

Deployment patterns differ in how traffic, state, and processing authority move between versions. What matters is not only how the code reaches production, but how those three things move while old and new versions coexist.

Blue/Green

Blue/green deployment works best when the green environment can be initialized and validated before receiving production traffic, with the required state already synchronized or accessible. Traffic may move all at once or progressively, but routing and processing ownership must remain unambiguous throughout the transfer.

Keeping blue warm makes traffic reversal fast, but does not guarantee a safe rollback. Green may have written data that blue cannot read or initiated financial actions that switching traffic back cannot undo.

Canary Releases

A canary release routes a small, controlled share of production traffic to the new version. Exposure increases only while predefined technical and business metrics remain within acceptable limits. Traffic assignment must be stable, canary outcomes must be attributable to the new version, and further exposure must be stoppable quickly.

In financial systems, traffic share is a poor measure of blast radius. Orders processed by the canary may update positions, consume shared risk capacity, trigger hedges, or change an order book used by other flows. A more useful blast-radius boundary may be the accounts, instruments, order books, or risk capacity affected rather than the percentage of requests routed to the canary.

Work that shares the same mutable state or lifecycle should remain on the same version. For example, routing all orders for an account or all events in a partition is usually safer than sampling individual requests at random. Returning traffic to the previous version prevents the canary from processing new work, but it cannot undo orders, payments, or other effects already committed.

Shadow Deployments

A shadow deployment mirrors production inputs to a new version without giving it control over production outcomes. The shadow processes live orders, market data, or risk requests and records what it would have done. Its results can then be compared with those produced by the authoritative system.

Shadowing exposes the new version to real traffic distributions and load patterns. Comparing its decisions and resulting state with production can reveal behavioral drift before the new version receives authority.

A shadow may write to state that is separate from the authoritative production state.

Feature Flags and Kill Switches

Feature flags separate rollout from release. New code can be deployed with its behavior disabled, then enabled for a controlled set of accounts, instruments, or partitions without another deployment.

When a flag affects an operation across multiple processing stages, its value must remain consistent for that operation's lifetime. Once an order enters the workflow, every subsequent stage should apply the same behavior version, even if the flag changes while the order is being processed.

Kill switches stop new work from entering a disabled path. Their behavior must be defined in advance: whether new orders are rejected or queued, whether accepted orders continue processing, and whether cancellations remain available.

State Handoff and Authority Transfer

Moving traffic to a new system is insufficient for stateful workloads. Before taking ownership, the new system must have a consistent view of orders, positions, balances, and risk state at a defined handoff point.

Once synchronization reaches that point, authority can move to the new owner. The new owner may then commit further state transitions, while the previous owner must be prevented from doing so.

State determines which decisions a system can make correctly; authority determines which system may commit them.

Handoff Scope

Authority can move for the whole system or progressively for smaller scopes, such as an account, event partition, or an instrument's order book. The chosen scope defines both the state that must be synchronized and the actions the new owner is permitted to commit.

For a matching engine, an instrument's order book cannot be handed one price level at a time. An incoming order may match against any resting order, so the complete book and its sequence position must remain under the same authority.

Narrower scopes reduce the impact of a failed handoff and make gradual cutover possible. They are safe only when decisions within one scope do not require mutable state controlled by another owner.

Defining the Handoff Point

Each scope needs an exact boundary between work assigned to the old owner and work assigned to the new one. Depending on the scope, this may be a sequence number or a snapshot tied to a specific log position.

If an order-book snapshot contains every event through sequence 8,412, the new owner can load that snapshot and process events from 8,413 onward. The snapshot and sequence number must refer to the same state.

Handoff points may differ across scopes. One order book may already belong to the new system while another remains under the old system. For each scope, the handoff point states where the old owner's responsibility ends and the new owner's begins.

Transferring Authority

Once the new system has reached the handoff point for a scope, that scope's ownership record can be updated. The record identifies the system currently permitted to commit new state transitions for that scope.

The record also carries a version number that increases whenever ownership changes. An order book assigned to blue under version 41 may be reassigned to green under version 42.

Authority can therefore move progressively: both systems may remain active during the migration, but for different scopes.

Fencing the Previous Owner

Each operation carries the authority version under which it was produced. A commit point accepts only the current version, so after an order book moves from version 41 to 42, writes carrying version 41 are rejected.

Fencing must protect the boundary where an action becomes irreversible. If a stale owner can submit an order to a venue before its authority is checked, rejecting the subsequent internal commit is too late. For external actions, authority must be checked before submission, or retries must carry a stable command identity that lets duplicate or ambiguous submissions be detected and reconciled.

Persistent Connections, Draining, and Session Continuity

A backend cutover can be correct and still break clients. FIX and WebSocket sessions carry state of their own, so that state must either remain on the draining instance or be recoverable elsewhere.

Stateless requests can be shifted incrementally through a load balancer, whereas persistent connections may remain bound to a serving instance for hours while carrying active session state. A WebSocket connection may carry market-data subscriptions, while a FIX session maintains sequence numbers and unacknowledged messages across a long-lived connection.

Draining is therefore more than removing an instance from the load balancer: active sessions must either complete on that instance or recover safely elsewhere.

Draining a Serving Instance

Before shutdown, an instance marks itself as draining and is removed from service discovery or load-balancer rotation. Routing changes take time to propagate, so it must continue handling requests that arrive during that interval.

Removal from rotation prevents new connections but does not close existing ones. Persistent sessions may continue submitting orders or receiving updates. The drain policy must define whether those sessions remain active, stop accepting new commands, or receive a signal to reconnect elsewhere.

The instance can terminate once in-flight requests have completed and the state required for session recovery (such as sequence numbers, unacknowledged messages, and subscription state) has been persisted. If the shutdown deadline expires first, enough recovery state must already be durable for reconnecting clients to determine which commands were accepted and which may be retried.

Reconnect and Recover

A deployment may close persistent connections even while the service remains available. Whether this is acceptable depends on the protocol contract: some sessions resume on another instance, while others require the client to reconnect and rebuild its state.

For streaming data, recovery usually starts from a known sequence number or a fresh snapshot followed by incremental updates. This allows the client to detect gaps and restore a consistent view before returning to the live stream. Other protocols, such as FIX, define their own sequence and replay mechanisms.

Recovery must also resolve commands whose outcome was unknown when the connection closed. Before repeating a request, the client needs a way to determine whether the original command was accepted; otherwise, reconnection can produce duplicate financial actions.

Zero downtime may still involve connection interruptions, provided clients can recover without data gaps or duplicate actions.

Operating the Deployment

Deployment Gates

Deployment gates control whether production exposure can increase. Each gate applies to a defined scope and holds the rollout at that scope until its behavior can be evaluated.

Infrastructure metrics establish whether the new version is operational. Business metrics provide evidence that observed behavior remains within predefined business thresholds. The gate must remain closed long enough for the failures it is intended to detect to surface. Some correctness defects cannot be ruled out by rollout metrics alone.

A failed gate stops the rollout from expanding but does not automatically imply a rollback.

Continue, Pause, or Recover

Exposure increases only while technical and business signals remain within the deployment gate. A threshold breach stops further expansion and triggers the predefined response: for example, disabling the new behavior, routing new work away, or freezing further cutover.

Recovery options shrink as the deployment progresses. Before release, the new behavior can remain disabled. After release, new traffic can be removed from the affected path. Once state has changed or authority has moved, returning to the previous version is safe only if it can process the current state correctly. Otherwise, recovery requires a corrected version.

Containment prevents further impact but does not erase effects already committed. Orders, payments, and state changes produced before the rollout stopped must still be reconciled and, where necessary, compensated.

Completing the Deployment

Fleet convergence does not mark the end of a deployment. Every operation accepted during the transition must have a known outcome, and internal records must agree with relevant venue, counterparty, or ledger records. Missing operations, unexplained duplicates, and unresolved submissions keep the deployment open.

The previous path can be retired once reconciliation shows no unexplained differences and the new path holds authority for its intended scope. Temporary routing rules, compatibility code, and feature flags can then be removed.

Zero-downtime deployment is therefore less about keeping every process or connection alive than about preserving correctness while the system changes underneath active work. Accepted work still has to survive, old and new behavior must coexist safely, authority has to move without overlap, and every committed outcome has to be accounted for.