Reliability is kinda our whole thing at PlanetScale. We maintain a flawless uptime record and preach the gospel of high availability. It might seem counterintuitive, but this involves embracing failure.
A resilient system anticipates the server failures inherent to cloud-native environments.
When we'll block a cutover
Most stuff in your primary database makes its way into its replicas without issue. The write-ahead log (WAL) streams updates from the primary to replicas, so after a brief moment (replication lag), the databases are effectively the same.
In the event of a resize or configuration change, a replica that is an exact match of the primary is "promoted" to become the new primary. The only penalty is a few seconds of primary unavailability and dropped connections (except to PgBouncers).
Most commonly, you might think that a replica is promotion-ready because it has replayed WAL to completion. A less obvious condition is whether the replica has a synchronized, usable copy of logical replication slots.
In vanilla Postgres, you can proceed with a promotion even if replication slots aren't caught up, potentially breaking your connected applications.
On PlanetScale Postgres, we'll block you. It's for your own good.
Wait, what's a replication slot?
Postgres contains two types of replication slots, which act as "bookmarks" in the WAL.
- Physical slots belong to the replicas. Each one tracks how far a replica has gotten through the WAL.
- Logical slots decode WAL into per-row change events, which are typically piped to external subscribers such as search indexes, analytics tools, and queues.
This post is concentrated on the latter. Postgres is designed to be okay with replica promotion so long as the data is caught up, but with no concern for whether logical slots exist on the replica.
Without a copy of that bookmark on the promoted replica, the database is fine, but the change stream is not. Consumers of that slot miss every event since they last acknowledged one, or they stall until you take a new snapshot. Promoting an incomplete replica to primary is a data-loss event for your downstream applications.
Who's responsible for a cutover?
It's helpful to put a box around PlanetScale and Postgres and define their roles and responsibilities in creating a seamless cutover experience.
Click on any of the boxes in the diagram below for more details
Postgres' cutover features
Postgres provides a lot of functionality to create a high-availability architecture. It understands the concept of a primary and replicas, and the WAL allows the former to stream updates to the latter, keeping their data synchronized.
(Note: What PlanetScale calls replicas, Postgres documentation calls standbys. The additional layer of confusion this adds to writing about replication slots is not lost on the author of this post.)
Postgres can report a replica's current state, including whether it's connected, how far through the WAL it is, replication lag, and more.
Postgres won't provision servers, decide which replica to promote, or decide whether a replica is ready for promotion. The operator makes these decisions.
The PlanetScale operator
The joy of PlanetScale Postgres is its custom Kubernetes operator, which, among other things, makes critical operations like resizing, reconfiguring, or reviving a database from failure much safer.
In relation to logical replication slots, it will:
- Detect misconfigured slots. The operator watches
pg_replication_slots
and records any misconfigurations such as missingfailover = true
or whetherhot_standby_feedback
orsync_replication_slots
areoff
. The next section covers the correct configuration. - Alert you. In the event of a misconfiguration being detected you will receive email alerts and a promenant banner is displayed in the PlanetScale dashboard with details on how to correct.
- Block problematic planned cutovers. A resize, parameter change, or maintenance that would silently drop a slot is blocked by the operator, protecting downstream applications from data loss events.
- Wait for the slot to be usable. Before a promotion event takes place, the operator waits for all named slots to report a ready state. It also manages
synchronized_standby_slots
so one dead replica doesn't block all logical replication.
If you take no action, we will allow blocked cutovers to proceed after the grace period. However, you risk downstream consumers missing updates with no trustworthy position from which to resume.
Getting replication slots promotion-ready
This blog post isn't a full guide to setting up replication slots; it just highlights how to do it correctly on PlanetScale.
Say you're creating a slot called analytics_cdc
. The last parameter matters most: it ensures the slot stays in sync with replicas. You must set failover = true
. Double-check any implementation code from your CDC tooling.
SELECT pg_create_logical_replication_slot(
'analytics_cdc',
'pgoutput',
false, -- temporary
false, -- two_phase
true -- 👈 failover
);
If you already have a replication slot with failover = false
, you can modify it, just be aware this will hang if the slot is already being consumed. In a separate session you'll have to terminate th consumer to apply this change.
ALTER_REPLICATION_SLOT analytics_cdc (FAILOVER true);
If you haven't already updated your database configuration for replication slots, you should soon receive an email from PlanetScale notifying you that changes are required, and you'll see a new banner in the dashboard.
Setting failover = true
in the replication slot makes it sync to replicas, but doesn't ensure it's promotion-ready. PlanetScale needs to know the name of any replication slots your applications depend on to ensure they won't be deleted from the replica before promotion.
You can add the names of all required slots in the dashboard under Clusters > Parameters > Logical slot name.
Repeat this step for each new slot you add to your database. Thankfully, you'll also receive an email reminder for each one.
Additionally, you'll need to set two Postgres settings to on
in your parameters configuration.
hot_standby_feedback = 'on'
keeps that bookmark copy valid so it can be used after promotionsync_replication_slots = 'on'
instructs Postgres to copy slot state to replicas
This is a one-time operation that will cover all replication slots.
With this done, your replicas and their replication slots are cutover-ready.
Why are these parameters off by default?
Setting these two parameters to off
is the perfect default for a database with no CDC consumers. Some folks believe they should be on
by default. Since you're using logical replication slots, you need both on, but it's worth knowing the consequences.
hot_standby_feedback
determines if a replica tells a primary which old rows it is still reading. Theoff
default means a replica cannot pin the primary's vacuum horizon, whileon
deliberately pins that vacuum for correctness, but at the cost of adding bloat to the primary. Be aware, long-running transactions against replicas can cause problems with this enabled.sync_replication_slots
is the worker that copies slot state onto replicas. Turn iton
withouthot_standby_feedback
and the copy can be invalidated the first time vacuum runs past the slot's horizon.
Most databases never create a logical slot, so Postgres' defaults assume you're better off without the extra work that these introduce.
Conclusion
While Postgres understands high-availability architecture, its default behavior doesn't have downstream applications' best interests in mind. The combination of what Postgres can do and what PlanetScale lets you do saves you from finding that out the hard way.
Facts Only
* Reliability is a core focus at PlanetScale, maintaining high uptime.
* Replicas receive updates via the write-ahead log (WAL) from the primary.
* A replica can be promoted to primary by an exact match of the primary.
* Promotion results in a few seconds of primary unavailability and dropped connections.
* The state of logical replication slots is a key concern for promotion readiness.
* Postgres allows promotion even if logical replication slots are not caught up.
* Replication slots are two types: physical slots for replicas and logical slots for external subscribers.
* Logical slots track per-row change events piped to external consumers.
* The PlanetScale operator monitors `pgreplicationslots` for misconfigurations.
* The operator blocks planned cutovers that could silently drop a slot.
* To be promotion-ready, logical slots must have `failover = true`, and PostgreSQL settings `hotstandbyfeedback` and `syncreplicationslots` must be set to 'on'.
Executive Summary
Reliability is central to PlanetScale, achieved through high availability and the acceptance of failure in cloud-native environments. The process for handling database promotion involves understanding replication lag and the state of logical replication slots. Updates flow from the primary to replicas via a write-ahead log (WAL), meaning replicas are nearly identical shortly after an update. A replica can be promoted to primary if it is an exact match, though this introduces a brief period of primary unavailability. The critical factor for seamless cutover is ensuring that downstream applications, which often rely on logical replication slots, have synchronized state.
Postgres natively allows promotion without concern for the synchronization of logical replication slots, but this poses a risk to consumers relying on those slots if they are not properly managed. PlanetScale's custom Kubernetes operator intervenes to manage this risk by detecting misconfigured slots and blocking planned cutovers that could cause data loss. To ensure a replica is truly promotion-ready regarding logical slots, specific configurations must be set: the slot must have `failover = true`, and Postgres settings `hotstandbyfeedback` and `syncreplicationslots` must be enabled.
Full Take
The narrative pivots on the tension between the functional capability of core database systems like Postgres and the operational reality required for high-availability in cloud environments. The fundamental pattern observed is that default configurations prioritize internal consistency over external application continuity, creating a gap that necessitates an external control layer to enforce resilience. PostgreSQL’s design permits replication promotion without regard for downstream consumers' needs regarding logical change streams, which implies that data integrity at the database level does not equate to system-wide operational integrity.
The mechanism described, where the PlanetScale operator actively monitors and blocks operations based on the state of logical replication slots, introduces a crucial layer of enforced responsibility onto the infrastructure itself. This shifts the burden from trusting the default behavior of the core engine to relying on an external agent to manage complex distributed state. The parameters `hotstandbyfeedback` and `syncreplicationslots`, which are off by default for simple setups, illustrate how latent complexity can be safely suppressed, but that suppression requires explicit configuration when those complexities (like logical slots) are introduced. This suggests a pattern where perceived simplicity masks necessary operational complexity; the system defaults to the least intrusive state until specific consumer requirements necessitate stricter enforcement.
The implications point toward a necessary evolution in database management philosophy: robustness is not inherent in the data store alone but emerges from the layered control mechanisms applied around it. The necessity of explicitly setting slot synchronization parameters and relying on an operator to enforce these settings suggests that true resilience in distributed systems requires embedding application-level awareness into the infrastructure layer, rather than assuming a monolithic view of system health. What is missing is a deeper exploration of the historical trade-offs between transactional consistency, data stream integrity, and operational control in emergent cloud architectures.
Sentinel — Human
The text is highly specific, deeply technical, and weaves complex, nuanced information effectively, strongly suggesting authorship by an expert familiar with both database internals and cloud infrastructure architecture.