Platform

Solutions

Industries

About

Blog

Book a demo

Published Jul 20, 2026

Snapshots Are Not Rollback: What Rollback by Design Actually Requires

In May 2024, a provisioning error caused Google Cloud to delete UniSuper’s entire Google Cloud VMware Engine private cloud.

According to Google Cloud’s incident report, an internal provisioning tool left a parameter blank. The system assigned an unintended one-year term, and when that period expired, the private cloud was automatically deleted. Infrastructure, configuration, and data were affected.

Recovery required rebuilding the environment, restoring its network and security configuration, recovering applications, and restoring data. The work continued around the clock for several days. UniSuper ultimately recovered with help from an independent backup held outside Google Cloud.

That backup made the incident survivable. It did not make recovery immediate.

This is the distinction many teams miss:

A snapshot is an artifact. Rollback is a tested path back to a known-good operating state.

As AI agents begin operating production infrastructure, this distinction becomes critical. An operation should not be considered safely reversible merely because a snapshot exists somewhere. The real question is whether the organization can use that snapshot to restore service within an acceptable time, with an understood amount of data loss and without creating a second incident.

A snapshot is an artifact. Rollback is a path.

A snapshot captures some state at a particular moment. It does not contain the complete operational path required to return a running system to that state.

Consider an ordinary Amazon RDS snapshot. Restoring it does not overwrite the existing database instance. Amazon RDS creates a new instance with a new endpoint.

Returning the application to service may therefore require additional work:

Provisioning the restored instance
Selecting and verifying its configuration
Reconnecting networking and permissions
Redirecting applications to the new endpoint
Verifying data integrity and application behavior
Coordinating the final cutover

Every one of these steps can delay or prevent recovery. None is solved merely by possessing the snapshot.

The snapshot proves that a previous state was captured. A demonstrated rollback path proves that the organization can operate from that state again.

Three gaps a snapshot does not close

Even when a snapshot can be restored successfully, three important gaps remain.

1. Recovery takes time

Restoring from a snapshot can take minutes, hours, or longer, depending on the platform, data volume, dependencies, and surrounding environment.

The honest recovery time is measured during an exercise—not estimated during an incident.

If an AI agent can make a damaging change in thirty seconds but restoring service takes two days, calling the operation “reversible” hides an important part of the risk. The organization may eventually recover, but the operational consequences can still be severe.

Reversibility is therefore not simply yes or no. Time matters.

A recovery mechanism that restores service in ten minutes provides a very different level of safety from one that requires two days, even if both eventually recover the same data.

2. Recovery may discard legitimate changes

A snapshot represents a point in time. The system usually continues changing after that point.

Restoring the snapshot may therefore remove both the unwanted change and every valid change made afterward. This is the purpose of the Recovery Point Objective, or RPO: to define how much data loss the organization is prepared to accept during recovery.

But rollback often asks for something more precise. Ideally, it should undo the damaging change while preserving unrelated work that followed it.

A full restoration cannot always make that distinction. It may return the entire system to an earlier state—correct data and incorrect data alike.

This is the difference between recovering from a disaster and surgically reversing one operation.

3. Recovery can create its own blast radius

A restore operation may affect far more than the resource that originally failed.

Consider an etcd snapshot in Kubernetes. The snapshot contains the cluster’s Kubernetes state. Restoring it returns control-plane state to an earlier point, potentially reversing deployments, configuration changes, and other work performed after the snapshot.

At the same time, external resources and application data may remain in their newer states. The restored Kubernetes state and the surrounding infrastructure can therefore stop agreeing with each other.

Recovery is itself a consequential production operation. It requires planning, coordination, verification, and clear responsibility.

A recovery mechanism that unexpectedly damages unrelated workloads is not a clean undo. It is another high-risk change performed during an already difficult incident.

What about point-in-time recovery?

Point-in-time recovery, or PITR, significantly improves the situation.

Instead of selecting only from periodic snapshots, PITR allows a database to be restored to a selected point within its available retention window. This can narrow the potential data-loss window considerably.

But PITR does not remove every recovery problem.

In Amazon RDS, point-in-time recovery still creates a new database instance. Applications may still need to be redirected, configuration must still be verified, and the restored database must still be tested before cutover.

PITR also restores the database as a whole to an earlier moment. It does not automatically remove one unwanted transaction while preserving every valid transaction that followed it.

PITR is therefore a powerful recovery capability, but it is not automatically a surgical undo mechanism. Its operational value still depends on the surrounding recovery path.

A recovery artifact must survive the incident

A recovery artifact is useful only if it survives the event it is supposed to reverse.

In the PocketOS incident, public accounts reported that an AI coding agent deleted a production volume hosted on Railway. The volume-level backups were lost with it, while the most recent independently recoverable backup was approximately three months old.

UniSuper demonstrates the opposite outcome. The Google Cloud deletion affected the private cloud and resources held within it, but an independent backup outside Google Cloud remained available and contributed to the recovery.

The lesson is straightforward:

A recovery artifact must not share the same practical failure boundary as the system it protects.

That boundary is not only physical location. It can also include:

The same provider or account
The same administrative credentials
The same deletion operation
The same encryption keys
The same lifecycle policy
The same region or control plane

A copy can exist and still be vulnerable to the same event as the original. Independence must be evaluated against the failures that matter.

An untested restore is a hypothesis

A documented restore procedure is valuable. It is not proof that recovery will work.

Recovery paths decay.

An encrypted snapshot may become unusable if its encryption key is deleted or inaccessible. A procedure tested before a major platform upgrade may no longer work afterward. Permissions change. Dependencies disappear. Network architecture evolves. People who understood the original process move to other roles.

These failures often remain invisible until recovery is urgently needed.

Regular restore exercises answer questions that documentation cannot:

Can the artifact still be accessed?
Can it be restored in the current environment?
How long does recovery actually take?
What configuration must be reconstructed?
What data will be lost?
Can applications operate correctly after restoration?
Does the recovery process affect unrelated systems?

Until those questions have been answered through evidence, the organization has a recovery artifact—not a demonstrated recovery capability.

A practical recovery review

For each important recovery source, teams should be able to complete a simple review using records rather than memory.

Recovery source	Last tested	Measured recovery time	Independent failure boundary	Expected data loss
RDS snapshot or PITR
EBS snapshot
etcd snapshot
S3 versioning
Kubernetes backup

A blank row should not be interpreted as proof that recovery is impossible. It means the organization has not yet demonstrated the capability.

Each column answers a different operational question:

Last tested: Has the recovery path been exercised against the current environment?
Measured recovery time: How long does returning to service actually take?
Independent failure boundary: Is the recovery source likely to survive the event affecting the original?
Expected data loss: What valid state may be lost during restoration?

This is not merely backup administration. It determines how much operational confidence the organization can place in the word “reversible.”

Rollback by Design begins before the incident

Rollback by Design does not mean that every operation can be perfectly undone. Many infrastructure and data operations cannot be reversed without cost, delay, or loss.

It means that the organization understands those limitations before relying on recovery as a safety control.

Before treating an operation as safely reversible, teams should know:

What recovery capability exists
Whether it has been tested
How long it takes
What state may be lost
What else the recovery operation could affect
Whether the recovery source can survive the original failure

Snapshots remain essential. They can turn catastrophic loss into a survivable incident. But a snapshot alone is not evidence of an operational rollback capability.

The artifact is only the beginning. The real design work is the path back.

A snapshot proves that a previous state once existed. A tested rollback path proves that you can operate from it again.

Aokumo provides governance software for AI agents that operate cloud infrastructure, helping enterprises introduce production automation without losing control.