A team preparing a release for its Node order API wants to remove old test rows from Postgres. The team runs a migration against the live database while the API continues taking orders. The migration finishes without an error, but an hour later a customer reports missing orders. The team finds that the cleanup deleted real orders while customers continued placing new ones.
Restore the database to a time before the mistake and inspect that copy separately. Before pointing the live app at it, decide how to keep the valid orders customers placed afterward. Otherwise the switch could recover the deleted orders while leaving the newer ones behind.
Set the recovery goals before the next migration
Before running a migration like the one at Parcel, the example order app, decide how much recent work the app can afford to lose and how long it can be down. These limits are called the recovery point objective (RPO) and recovery time objective (RTO).
Write the plain-language limits beside those names. An internal tool might tolerate rebuilding an hour of work. A billing app may need to preserve every paid order. Those needs guide the choice of backup plan and the work needed to restore the app.
Assign the people who can pause writes, request a restore, check the data, and approve the switch. If the team has to find someone with the right access during an outage, that wait adds to the time customers spend without the app.
Find a time from before the mistake
Parcel's team uses release records, the change log, and safe data checks to pin down when the migration changed the orders. The restore point needs to fall before that change.
| Time | What happened |
|---|---|
| 09:00 | A scheduled snapshot completed |
| 10:00 | The migration deleted the wrong orders |
| 10:05 | Customers added more valid orders |
| 11:00 | The team found the data loss |
The 09:00 snapshot contains the earlier state. A point-in-time restore to just before 10:00 could preserve more valid work from before the migration, if the provider's restore window supports that time.
Before choosing the restore point, check the earliest and latest times the provider reports. The retention setting describes the intended window, and the reported times show which part of that window is available now.
On Amazon RDS, point-in-time restore creates a new DB instance and leaves the source unchanged. Read the restorable window and check the new instance's settings as part of the restore. RDS point-in-time restore
Keep the restored copy separate while you inspect it
Parcel should connect a test copy of the app to the restored database before changing production. Check known missing order IDs and the fields the migration touched. Also inspect related records, such as order items, so the restored order is usable as a whole.
Once the provider reports that the restore finished, use those records to confirm you picked the right time and that the test app can read the recovered orders.
Keep payment collection, email delivery, and scheduled work disabled in the test app. A job that reads the older data may see an order as unpaid or a receipt as unsent even if the live app already completed that work.
Save the checks and expected values in the incident note. Another reviewer should be able to confirm why this copy is a good recovery source.
Preserve the valid orders made later
Neither the 09:00 snapshot nor a restore just before 10:00 contains the good orders added at 10:05. Switching the whole app to that copy would leave those orders behind.
Decide how to keep those orders before making the switch. Depending on the error, the team can copy selected missing records into a repaired source or bring valid later changes into the restored copy. Include related records, updates, and deletes in the plan so a repair preserves more than just the new rows.
Check outside systems too. A later order may already have a completed payment or a sent confirmation. Rebuilding its database record should preserve that fact without repeating the external action.
If the team cannot tell which later writes are safe, keep the source and seek help with the repair. A rushed switch can trade a known loss for a wider, harder-to-understand one.
Pause writers for the final switch
When the chosen database is ready, pause browser requests that can write and stop background jobs and event handlers that write too. A maintenance page can keep customers from submitting orders while payment callbacks still reach the old database.
Preserve valid changes up to the point when you paused writes, then move the app's connections. Confirm the app is using the intended database without exposing its credentials, and test a core read and a harmless write before allowing normal traffic to resume.
Check pending events and jobs before restarting them. Decide which need to run and which have already completed elsewhere. Keep the old source available while you confirm the recovered app behaves as expected.
After the switch, check that Parcel's order history includes both the recovered orders and the later valid ones. A 200 response shows the request succeeded. Inspect the order items and payment state too, since the app can answer the request with incomplete data.
Rehearse the whole path with synthetic data
Before the next risky change, run a drill in a separate test setup. Create known orders, save their IDs and values, make a harmless test change, and restore to the chosen point.
Connect the test app and follow the same inspection and reconnection steps. Measure from the start of the drill through a successful app request, including the time to find the right backup, wait for support if required, and verify the data.
Use the result to improve the restore plan and the schema change. Change the schema in steps that let old code keep reading the database. That can make it easier to switch back if a release fails.
Retention should cover errors found late enough to matter. On RDS, later snapshots are incremental, and manual snapshots have their own lifecycle. Check changed data and any backup allowance before estimating the cost of longer retention. RDS backup behavior
Check the recovery path on Forte
Forte shared Postgres is open beta. It gets daily backups kept for seven days, with restores handled through support. Dedicated Postgres requires closed-alpha access and adds point-in-time restore with retention from one to 35 days. Check feature terms before you rely on the tier for a live app. Forte database tiers and recovery
For a dedicated database, the Backups tab shows the available restore window and workflow. The window starts 12 hours inside the retention period, so seven-day retention reaches back 6.5 days; a new database starts at its first backup. Choose a supported time and check the new database's name and size. Forte restore guide
The restore creates another database while the source keeps running. Services remain connected to the source. Follow the data checks and write pause before moving those connections. Dedicated databases are single node, and beta features have no GA commitments unless agreed in writing.
Forte handles backups and restores for the selected tier. Your team chooses the time, checks the recovered data, preserves later valid writes, and decides when to switch. Include any backup storage above the dedicated allowance when you budget for that work. Forte database pricing
Use the database guide to check the tier against Parcel's goals. Before the next migration, name the owner and date of a full restore drill.