Series II: From infrastructure to agentic projects in production. Episode 15 of 20.
Observability reveals a problem in the portal, but cannot bring data back by itself. For an SME, the essential decision is to fund demonstrable recovery, not just storage for copies. A “backup successful” message confirms that an operation ran; it does not prove that people can resume work within the time the business needs.
We continue the series' fictional B2B requests and documents portal through a hypothetical scenario: some files no longer open, and document processing stops. The cause could be an error or a compromise. We attribute no incident to a client and do not assume that restoration alone fixes the cause.
Availability and backup solve different problems
High availability aims to maintain service through redundant components. Replication copies changes to another instance or location. It helps when a component fails, but can also propagate accidental deletion or corrupted data. It does not replace recoverable history.
A snapshot preserves the state of data at a particular time. If it depends on the same storage and administrative permissions, their loss can also affect the snapshot. It can form part of a backup strategy, but is not independent protection by itself.
A backup is a copy retained for restoration, with versions and retention rules defining how long it is kept. The useful distinction is not “local versus cloud”, but which incident could affect production and copies simultaneously. Two directories on one system do not cover losing that system.
Decide acceptable loss before choosing the solution
RPO, recovery point objective, expresses the point to which data must be recovered after disruption. In practical discussions, we express it as the interval of data the business accepts losing. RTO, recovery time objective, sets the target duration for restoring a resource before its unavailability causes unacceptable impact.
For the portal, we propose a strictly illustrative RPO of 30 minutes and RTO of four hours. If an incident occurs at 14:00 and the last recoverable clean state is from 13:40, the potentially lost interval is 20 minutes. If only yesterday's copy proves usable, the target is missed regardless of how many operations were reported as successful.
These values are neither i8 guarantees nor universal recommendations. The process owner establishes the impact, while the technical team tests feasibility through exercises. We specify when outage measurement starts and include dependencies and validation, not just file transfer.
Inventory what must be recovered together
The portal needs its requests database, uploaded documents, configuration, application version, and access to required services. The backup register must show what each mechanism covers and excludes. Separately retained code helps reconstruction, but does not automatically contain customer data.
If we choose PostgreSQL 18 for this scenario, PITR, point-in-time recovery, can combine a base backup with archived WAL, the database's write-ahead change log. Its documentation requires continuity of the necessary log files. This mechanism does not automatically recover document files or configuration changes.
We therefore check consistency between records and files: a document referenced by the restored database must exist and be readable. For an agent later integrated into the portal, we also inventory job state. We do not blindly restart tasks that could resend notifications or perform other external actions.
Protect copies and the access path
We propose historical copies in storage separate from production, including at least one offline or modification-protected copy. Immutability prevents changes or deletion under the chosen mechanism's conditions. It does not mean the copied information was correct or every account is invulnerable.
For example, Amazon S3 Object Lock protects object versions; Governance mode allows retention bypass with special permissions, unlike Compliance mode's protection during the specified period. We check the actual configuration, not merely the commercial label “immutable”.
We separate backup administration permissions from application permissions and protect encryption keys. Encryption makes data unreadable without its key, but losing that key can prevent restoration. We test authorized emergency access and keep instructions available even when the portal is down.
We also check recovery resources: free space, connectivity, access to application packages, and availability of the person authorized to approve intervention. A plan assuming a nonexistent spare server does not meet the target. In the cloud, we explicitly check which services and data the contract covers, without assuming the provider protects against every application mistake.
Recovery follows an order, not an impulse
For the hypothetical incident, we propose this workflow, adapted to the diagnosis:
Step | Proposed owner | Required evidence |
|---|---|---|
Limit impact | Incident coordinator and administrator | Affected operations stopped under control; suspected areas isolated |
Investigate | Technical owner, with a specialist if needed | Timeline, logs, and changes retained with controlled access |
Choose a recoverable state | Data administrator | Verified copies and a justified recovery point |
Rebuild in isolation | Technical team | Clean environment, compatible application, and consistent data |
Reopen | Coordinator and process owner | Accepted tests and active monitoring |
Isolation and evidence preservation need coordination; we do not erase the suspected system before assessment. If compromise is indicated, we seek specialist help, check attacker persistence, and change affected credentials from a trusted environment. We choose a defensible clean copy, not necessarily the newest.
NIST SP 800-61 Revision 3, published in April 2025, recommends checking backup integrity before use and addressing root causes before returning to production. We use it as technical guidance, not a claim of legal compliance for a Romanian SME.
Restoration cannot retrieve documents already disclosed to an attacker. If investigation indicates data exposure, the coordinator separately addresses consequences and communication with the appropriate specialists. Restoring the online interface does not automatically close the incident.
Test the process, not just the download
We propose an initial exercise, repeated after significant changes, followed by a risk-based schedule. We restore into an isolated environment without real emails or external actions. We check authentication, organizational separation, documents, and request states. We measure time until essential functions are accepted and the gap against targets.
Amvrosiadis and Bhadkamkar's study, presented at USENIX ATC in 2016, identified misconfiguration as the cause of most backup job errors in its dataset. The finding concerns installations of the studied product, not current systems' failure rates. It supports checking procedures without turning research into a commercial promise.
The portal's final recovery record contains the inventory, approved RPO and RTO, backup locations, emergency access, recovery order, owners, and the latest exercise results. It also covers informing users and reconciling requests received during the outage.
Agents developing the portal can prepare scripts and tests, but receive no implicit permission to delete copies or reopen production. Agents integrated into the application remain stopped until their state is verified. The next episode covers APIs and safe retries to reduce duplicate actions during resumption.
Sources
Sources checked on 16 October 2026:
- NIST SP 800-34 Revision 1, Contingency Planning Guide for Federal Information Systems, 2010, updated 11 November 2010, for recovery objectives and priorities.
- NIST SP 800-61 Revision 3, April 2025, particularly RC.RP-03 and RC.RP-05, for backup integrity and safe return to service.
- PostgreSQL 18, Continuous Archiving and Point-in-Time Recovery, for database recovery conditions and limitations.
- Amazon S3, Locking objects with Object Lock, for object-version protection and retention modes.
- Amvrosiadis and Bhadkamkar, Getting Back Up: Understanding How Enterprise Data Backups Fail, USENIX ATC, June 2016, for the empirical study of backup errors.
Want to check whether your systems can recover within the time your business needs? Talk to i8 about backups, restoration tests, and continuity.
