Upgrade a Cluster Running PCR

To safely perform a major version upgrade on a cluster with physical cluster replication (PCR), you must upgrade the standby cluster before the primary cluster to avoid version incompatibility. You cannot replicate data from a cluster on a newer version to a cluster on an older version.

This page describes the process for safely performing a major version upgrade on a pair of clusters that have PCR enabled.

Minor version upgrades

Minor versions are not relevant when determining PCR compatibility. There is no need to consider PCR compatibility when upgrading to a specific minor version within a major version, so you can follow the normal patch upgrade process.

Upgrade primary and standby clusters

To upgrade your primary and standby clusters:

1. Ensure that the virtual clusters on both your primary cluster and your standby cluster are finalized on the current version before beginning the upgrade process.

2. Upgrade the binaries on the standby cluster. On each node of the cluster, replace the binary and restart the node.

3. Connect to the standby cluster's system VC:

   cockroach sql --url \
   "postgresql://root@{standby node IP or hostname}:26257?options=-ccluster=system&sslmode=verify-full" \
   --certs-dir "certs"
   

4. Finalize the upgrade on the standby cluster's system VC:

   SET CLUSTER SETTING version '{VERSION}';
   

   Note:

   If you need to roll back an upgrade, you must do so before the upgrade has been finalized.

5. Confirm that finalization is complete on the standby cluster's system VC:

   > SHOW CLUSTER SETTING version;
   

6. Upgrade the binaries on the primary cluster. On each node of the cluster, replace the binary and restart the node.

7. Connect to the primary cluster's system VC:

   cockroach sql --url \
   "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster=system&sslmode=verify-full" \
   --certs-dir "certs"
   

8. Finalize the upgrade on the primary cluster's system VC:

   SET CLUSTER SETTING version '{VERSION}';
   

   Note:

   If you need to roll back an upgrade, you must do so before the upgrade has been finalized. Rolling back the upgrade on the primary cluster does not roll back the standby cluster.

9. Confirm that finalization is complete on the primary cluster's system VC:

   > SHOW CLUSTER SETTING version;
   

10. Connect to the primary cluster's app VC:

    cockroach sql --url \
    "postgresql://root@{primary node IP or hostname}:26257?options=-ccluster={app_virtual_cluster_name}&sslmode=verify-full" \
    --certs-dir "certs"
    

11. Finalize the upgrade on the primary cluster's app VC:

    SET CLUSTER SETTING version '{VERSION}';
    

    Upgrading the primary cluster's app VC also upgrades the standby cluster's app VC, since it replicates from the primary.

12. Confirm that finalization is complete on the primary cluster's app VC:

    > SHOW CLUSTER SETTING version;
    

Upgrade reader VC

If you have a reader virtual cluster (reader VC), you must drop and recreate it to upgrade after completing the main upgrade process on the primary and standby clusters. Follow these steps to upgrade your reader VC:

1. After upgrading the app VC on your primary cluster, wait for the replicated time to pass the time at which the upgrade completed.

2. On the standby cluster, stop the reader VC service:

   ALTER VIRTUAL CLUSTER <readervc-name> STOP SERVICE;
   

3. Drop the reader VC:

   DROP VIRTUAL CLUSTER <readervc-name>;
   

4. On the standby cluster, re-create the reader VC:

   ALTER VIRTUAL CLUSTER dest-system SET REPLICATION READ VIRTUAL CLUSTER;
   

At this point, the reader VC is on the same version as the standby cluster.

Failover and fast failback during upgrade

If needed, you can perform a failover while upgrading your clusters.

However, after performing a failover you cannot perform a fast failback to the original primary cluster while the standby cluster version is newer than the primary cluster version. This is because you cannot replicate data from a cluster on a newer version to a cluster on an older version.