Why Database p99 Latency Spikes During Backups and Checkpoints

The database looks fast during a quiet benchmark. Then the backup starts, a checkpoint lands, and a small share of customer requests becomes noticeably slower. The average still looks acceptable, so the dashboard hides the experience that users are reporting.

Short answer: backups and checkpoints can compete with foreground work for storage, CPU, memory and network capacity. Measure request latency while that maintenance is running, then correlate it with database and infrastructure evidence. A p99 spike is a symptom to investigate, not proof that the disk is the only bottleneck.

TL;DR

• Evaluate foreground queries and maintenance together.
• Record p50, p95 and p99 latency alongside throughput, errors and timeouts.
• Separate backup traffic, checkpoint writes, CPU compression work and lock or client waits.
• Keep durability settings and request rates comparable across tests.
• Choose storage using sustained application behavior and recovery results, not peak IOPS alone.

Why can maintenance change query latency?

Foreground queries, transaction-log writes and background maintenance may share parts of the same I/O path. A backup can add a large stream of reads. A checkpoint writes dirty database pages. Compression adds CPU work, and transferring backup data consumes network capacity. The limiting resource depends on the deployment and configuration.

PostgreSQL’s WAL configuration documentation explains checkpoint behavior and how checkpoint writes are spread. Its base-backup documentation describes backup controls, including compression and checkpoint modes. Those mechanisms explain possible pressure; measurements are needed to determine which one matters in your workload.

What does p99 actually tell you?

The 99th percentile describes the latency below which approximately 99% of observations fall, according to the measurement method. It focuses attention on the slower end of the distribution. It does not describe the single worst request, and a small sample can make it unstable.

Always report the observation count and time window. Do not average several p99 values and call the result an overall p99. Retain raw samples or mergeable histograms, and keep timed-out or failed requests visible. Our latency percentile guide explains the basics.

Run a controlled mixed-load test

Start with a representative database, query mix and write rate. Keep the instance, volume, database version, dataset, indexes, durability settings and connection pool constant. Record cache conditions. Run long enough to observe the relevant maintenance cycle, rather than stopping before it occurs.

Phase	Workload	Question
A: Baseline	Normal foreground reads and writes	What is the steady operating range?
B: Backup	The same foreground load plus the normal backup job	Does backup activity change latency, throughput or errors?
C: Checkpoint overlap	Observe checkpoint activity under the same foreground load	Do latency excursions align with checkpoint work?
D: One adjustment	Repeat with one documented change	Did that change improve the constrained resource and user result?
E: Recovery	Restore the resulting backup separately	Did the performance change preserve usable recovery?

Use a load generator that records offered request rate as well as completed rate. If clients simply wait longer before issuing the next request, the test can hide the queueing a real arrival stream would create. Keep the arrival pattern explicit and report overload instead of discarding it.

Which signals should you collect?

At the application boundary, collect latency, completed operations, failures, retries and timeout counts. At the host, collect CPU, memory, disk I/O, available capacity and network utilization. In the database, inspect wait events, checkpoint activity, WAL behavior and backup progress.

For PostgreSQL 18, these are useful read-only starting points; columns and interpretation must match your server version:

SELECT * FROM pg_stat_checkpointer;
SELECT * FROM pg_stat_io;
SELECT * FROM pg_stat_wal;
SELECT * FROM pg_stat_progress_basebackup;

The statistics reference and progress reporting reference describe these views. Many counters accumulate, so compare deltas over the same interval. Timing fields depend on relevant instrumentation settings; enabling additional timing can have overhead and should be evaluated deliberately.

How do you interpret the result?

Suppose an illustrative test records p99 of 20 ms during the baseline and 180 ms during a backup, while throughput stays similar. That is a reason to investigate, not a diagnosis. These numbers are an example, not measured Nirvana results.

• If storage latency and queueing rise with backup reads while CPU has headroom, investigate I/O contention.
• If CPU saturates with compression, evaluate the compression choice and where it runs.
• If database waits point to locks, inspect the conflicting transactions and workload design.
• If backup transfer saturates a shared network path, inspect bandwidth and topology.
• If only client queueing grows, verify connection-pool limits and the load-generator design.

Then change one relevant variable. Examples include a documented backup-rate limit, moving backup work to a suitable replica, revisiting checkpoint settings, or evaluating storage with more sustained headroom. Each choice has tradeoffs: a slower backup may reduce interference but lengthen the backup window, while replica-based backups still require a validated recovery procedure.

What should you ask a cloud storage provider?

Ask for the tested volume and instance configuration, sustained I/O behavior, relevant limits and the workload behind any performance claim. Require the same durability settings and offered load in comparisons. Keep compute, storage and network costs visible.

Nirvana’s high-performance storage cloud is designed for stateful workloads, but your application’s tail latency still needs measurement. Start from the published benchmark methods and results and adapt this mixed-load test to your database. For a narrower write-path investigation, use the PostgreSQL WAL and commit test guide.

Frequently asked questions

Does higher IOPS always lower p99?

No. Locks, CPU saturation, network delay and client queueing can dominate even when storage has spare capacity.

Should I disable durability to improve the number?

Changing durability changes the service being measured. Keep required durability behavior intact and document any configuration differences.

Is an overnight backup test enough?

Only if that load represents the operating scenario you need to support. Test the overlap customers actually experience, including expected growth and maintenance windows.

Primary documentation reviewed 20 October 2026. This article proposes a diagnostic method and reports no new benchmark measurements.

---

About Nirvana Labs

Nirvana Labs is a high-performance storage cloud built for blockchain, AI and databases: demanding, real-time, stateful workloads. Its high-performance block storage, compute, private networking, Nirvana Kubernetes Service and Agent Sandboxes help teams build and operate data-intensive applications.

Learn more at Nirvana Labs.

Nirvana Cloud | Pricing | Blog | Docs | Changelog | LinkedIn | Twitter | Telegram | YouTube