Solace High Availability and Disaster Recovery: HA Groups and Replication

Solace high availability protects a broker against a failure inside one site, and disaster recovery protects it against losing the site. They are separate features. High availability uses a three-node HA group: a primary and a backup messaging node in active-standby, plus a monitoring node that breaks ties. Disaster recovery, which Solace calls replication, copies guaranteed messages from an active site to a standby site, synchronously or asynchronously. HA fails over automatically; site failover is, in Solace’s words, “a co-ordinated operation that must be performed by network administrators” (Solace replication overview).

This page explains both mechanisms, how they fail, and how to set recovery targets. It belongs to the Solace Event Broker overview in the Solace cluster.

How Solace high availability works

The Solace HA documentation describes an HA group of “three event broker instances: two acting as active-standby messaging nodes and a third acting as a monitoring node”.

• Active and standby. One messaging node, usually the primary, serves clients. The other waits.
• Config-Sync keeps the two messaging nodes’ configuration identical, so a queue created on the primary exists on the backup.
• The mate link synchronises guaranteed messages and their state between primary and backup, so a message spooled but not yet consumed survives a failover.
• The monitoring node carries no messages. It exists to “prevent split-brain scenarios”, where a network fault leaves both messaging nodes believing they should be active. It needs “a unique static IP address” in “the same subnet” as the others.
• Client failover uses host lists: each client knows both messaging nodes, and the APIs “reconnect to the newly active event broker without impacting the client applications”.

HA is available in both editions of the software broker (edition table). Solace Cloud services are HA by default. For the appliance, Solace states “99.999% uptime with built-in replication and fast failover” (appliance page). On Kubernetes the same three nodes become three pods, covered on the Kubernetes and OpenShift page.

What the docs do not state is a failover time. That number depends on detection timers, guaranteed state and client reconnect settings, so it has to be measured.

Disaster recovery: replication between sites

Replication works per Message VPN. Guaranteed messages published to a VPN in the active replication state at one site are “automatically propagated to matching Message VPNs with a standby replication state” at the other (replication overview). The documentation describes an active-standby model, with a typical distance between sites of “perhaps 50 or 100 miles”.

Each site keeps its own HA group. Replication protects against losing a whole site; HA still handles node failures at each one.

Replication is Enterprise only on the software broker. The free Standard Edition has HA but no DR (Standard vs Enterprise).

Site failover is deliberate. Solace explains why: “typically there are servers, critical applications, and other infrastructure that must be switched as part of the failover”, so it “does not happen automatically”.

Synchronous or asynchronous replication

The mode is set per replicated topic subscription for ordinary guaranteed messages, and per Message VPN for transactions (sync and async docs).

	Synchronous	Asynchronous
Publisher waits for the standby site	Yes	No
Latency cost	Every publish includes the round trip to the other site	Minimal
Data at risk on site loss	None that was acknowledged	Messages not yet replicated
If the standby site is unreachable	By default the broker switches to asynchronous and becomes “synchronous ineligible”	Continues; backlog grows
To forbid the downgrade	Enable reject-msg-when-sync-ineligible; publishers are then rejected	Not applicable
Use for	Events you cannot recreate: orders, confirmations, quality holds	Telemetry, status, anything you can republish

The default downgrade is a trade-off worth knowing. It keeps production running when the WAN between sites fails, but for that period synchronous topics are quietly asynchronous. If a lost confirmation is worse than a stopped publisher, configure the rejection instead, and make sure the publishing application handles it.

RPO and RTO for a Solace design

Recovery point objective (how much data can be lost) and recovery time objective (how long until service returns) should be written down per failure, before choosing modes.

Failure	Mechanism	Data at risk (RPO)	What drives recovery time (RTO)
Broker process or VM fails	HA failover	None for guaranteed messages synced over the mate link	Detection and client reconnect settings
Network split inside the site	Monitoring node arbitration	None, if quorum is reached	Same as above
Site lost, sync replication	Replication, admin failover	None acknowledged	Your runbook: decision, DNS or host lists, applications
Site lost, async replication	Replication, admin failover	Recent unreplicated messages	Same runbook
Bad configuration pushed	Config-Sync copies it to the backup	Depends on the change	Configuration backups or configuration as code

An illustrative plant case. A food plant routes three kinds of events through its broker: machine states every few seconds, production confirmations to the ERP, and quality holds from the lab system. Machine states can be lost for a minute without harm, because the next one replaces them. A lost confirmation means an inventory error someone finds at month end. A lost quality hold can mean a pallet ships that should not have. The sensible design replicates confirmations and holds synchronously, with rejection enabled for holds so the lab system knows when a hold did not land, and leaves machine states asynchronous or unreplicated. The RTO conversation is different again: the plant can run for hours without the ERP link, but the lab cannot release product without the hold system, so its consumers need the fastest path back.

Direct messages are a separate question. The replication documentation describes guaranteed messages. Do not assume direct (at-most-once) messages survive a failover; design consumers of direct data to tolerate gaps.

Testing failover

1. Write the targets from the table above for each event class.
2. Fail the primary under representative load: stop the VM or pod, and time how long until every client is publishing again.
3. Check the queues. Compare message counts before and after; no guaranteed message should be missing or duplicated beyond what consumers can handle.
4. Split the network between primary and backup and confirm the monitoring node arbitrates as expected.
5. Cut the replication link and confirm what synchronous topics do: downgrade or reject.
6. Run a site failover drill with the application owners, following the written runbook.
7. Repeat after upgrades and after any change to client libraries.

Where it gets hard

• The monitoring node needs the same subnet. That constrains stretching an HA group across buildings or availability zones. Plan the network with the OT network team early.
• Host lists must be right everywhere. One gateway configured with only the primary address turns a clean failover into an outage for that line.
• Site failover touches more than the broker. DNS, firewall rules, consuming applications and plant gateways all have to follow. Across many plants, see multi-site edge to cloud and the event mesh.
• No published failover figure. Measure your own, and repeat the measurement on each new version.

---

Related: Solace Event Broker · Kubernetes and OpenShift · Standard vs Enterprise · Solace hub