Rotating gossip encryption keys in your Consul cluster helps maintain strong security posture. In this tutorial, we’ll walk through:
- Reviewing the current configuration
- Generating a new encryption key
- Distributing the key across the cluster
- Promoting the new key to primary
- Removing the old key
1. Review Current Configuration
Inspect your existing gossip encryption key in consul.d/config.hcl:
{
"log_level": "INFO",
"node_name": "consul-node-b",
"server": true,
"ui": true,
"leave_on_terminate": true,
"data_dir": "/etc/consul.d/data",
"datacenter": "us-east-1",
"client_addr": "0.0.0.0",
"bind_addr": "10.0.101.248",
"advertise_addr": "10.0.101.248",
"retry_join": ["10.0.101.110"],
"bootstrap_expect": 2,
"enable_syslog": true,
"encrypt": "62qD/DH15Ax0lMRUpMKvttP53p4FAvu+FgARDU4MzA=",
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"connect": {
"enabled": true
},
"acl": {
"enabled": true,
"default_policy": "allow",
"down_policy": "extend-cache"
},
"performance": {}
}
Before you begin, back up your Consul configuration and data directory. This ensures you can recover quickly if something goes wrong.
2. Generate a New Gossip Encryption Key
Run the following command on any Consul server or client to create a fresh base64-encoded key:
consul keygen
# Example output:
wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4=
3. Distribute the New Key Across the Cluster
Install the newly generated key into the cluster keyring:
consul keyring -install wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4=
You should see:
Installing new key "wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4="
3.1. Verify Key Distribution
On another node, list installed keys:
Expected output:
==> Gathering installed encryption keys...
us-east-1 (LAN):
62qD/DH15Ax0lMRUpMKvttP53p4FAvu+FgARDU4MzA= [2/2]
wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4= [2/2]
Switch the cluster’s primary gossip encryption key:
consul keyring -use wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4=
You’ll see:
Changing primary gossip encryption key to "wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4="
Confirm the change:
Should display the new key first under both WAN and LAN segments.
5. Remove the Old Encryption Key
Once every node is using the new key, remove the old one:
consul keyring -remove 62qD/DH15Ax0lMRUpMKvttP53p4FAvu+FgARDU4MzA=
Output:
Removing encryption key "62qD/DH15Ax0lMRUpMKvttP53p4FAvu+FgARDU4MzA="
Verify only the new key remains:
==> Gathering installed encryption keys...
WAN:
wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4= [2/2]
us-east-1 (LAN):
wlVkhlSnyl7SEy63/XsXMJ/48gIQSghShhUqn/05C4= [2/2]
Do not remove the old key until all nodes report the new key as primary. Premature removal can lead to cluster partitions and service disruptions.
Command Reference
| Command | Description |
|---|
consul keygen | Generates a new base64-encoded key |
consul keyring -install <key> | Installs a key into the cluster keyring |
consul keyring -list | Lists installed keys and their usage counts |
consul keyring -use <key> | Promotes a key to be the primary encryption key |
consul keyring -remove <key> | Deletes an old key from the keyring |
Links and References