The installation will be performed on three virtual machines, preferably distributed across different physical hosts. I will be using Keepalived to support the cluster's virtual IP instead of an external hardware load balancer. I will install the cluster on SUSE 15 SP4 just only because I have a ready-made template for this operating system.
I created three virtual machines and added four IP addresses for hcv{1,2,3}.domain.com and hcv.domain.com to the DNS. The last name is for the cluster's VIP to be an entry point.
Then create a certificate for this hcv.domain.com name, including all hcv{1,2,3}.domain.coms as DNS alternate names. It is important to enable them because some calls between nodes will be made using hostnames. It is also useful to add the DNS name localhost and IP 127.0.0.1. Adding this will save you a lot of keystrokes when using the CLI.
If not already done, add your CA to be trusted by creating a file /etc/pki/trust/anchors/ca.crt and run update-ca-certificates.
Download RPM for RedHat. It is also suitable for SUSE. Install it with the rpm command.
Place the certificate, you had created in step above, and its key in the /etc/vault.d/ directory, then edit /etc/vault.d/vault.hcl. Here is an example of this file for hcv1 node:
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1
# Full configuration options can be found at https://developer.hashicorp.com/vault/docs/configuration
cluster_name = "HCV"
ui = true
disable_mlock = true
storage "raft" {
path = "/opt/vault/data"
node_id = "hcv1"
retry_join {
leader_api_addr = "https://hcv2.domain.com:8200"
}
retry_join {
leader_api_addr = "https://hcv3.domain.com:8200"
}
}
# HTTPS listener
listener "tcp" {
address = "127.0.0.1:8200"
tls_cert_file = "/etc/vault.d/hcv.domain.com.crt"
tls_key_file = "/etc/vault.d/hcv.domain.com.key"
}
listener "tcp" {
address = "10.0.0.11:8200"
tls_cert_file = "/etc/vault.d/hcv.domain.com.crt"
tls_key_file = "/etc/vault.d/hcv.domain.com.key"
}
api_addr = "https://hcv.domain.com:8200"
cluster_addr = "https://hcv1.domain.com:8201"
An api_addr points to our VIP DNS. Copy this file to the second node and replace the red node name with hcv2 and the blue node names with the names of neighboring nodes. Fix listener address to match hosts IP. Two listener lines required instead of common recommended "0.0.0.0:8200" because of keepalived load balancer. Copy the file to the third node and make similar changes.
Start vault service on every node by commands
# systemctl daemon-reload # systemctl enable --now vault.service
Initialization should only be done on one node (say hcv1). Open a shell on the node and run the following command:
# vault operator init | tee vault.credentials Unseal Key 1: l4/biMcRLPDG53ukLG/MSgD0BVnrVlJGtw95cycqR0Vb Unseal Key 2: CN089ZHSTIVEivxk03XGh48LD2GgL8dJ6ssiuLXZhda1 Unseal Key 3: f7oWJN7aWSxv8tSCI1+g5w8A40WofGFvf7c/cDrFvVXu Unseal Key 4: vGNyIRDVSUJ/o93N7FiXdOveAjStu+eV+fCI0VWQpMZw Unseal Key 5: tlWZh2Xmeqii4y6OKDTIxt8V5uIH5JyokQxv7Xfw8i7P Initial Root Token: hvs.3mr6C5BG4uEAg33Jw2Duti5j Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests. Vault does not store the generated root key. Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed! It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
This output is shown only once. Now. It shows the initial root token usefull for further configuration. Five Unseal Keys are meant to be stored with five different keepers, any three of which must be available at database startup. Because of the importance of this data, I have saved it now in the vault.credentials file. Don't forget to take care of it after installation.
Initialization does not mean that the database has become open. You need to "unseal" it using any three keys printed above. Here is an example of this process
# vault operator unseal l4/biMcRLPDG53ukLG/MSgD0BVnrVlJGtw95cycqR0Vb Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce 2cbde089-d2eb-907d-a41c-7a9cff956524 Version 1.15.2 Build Date 2023-11-06T11:33:28Z Storage Type raft HA Enabled true # vault operator unseal CN089ZHSTIVEivxk03XGh48LD2GgL8dJ6ssiuLXZhda1 Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 2/3 Unseal Nonce 2cbde089-d2eb-907d-a41c-7a9cff956524 Version 1.15.2 Build Date 2023-11-06T11:33:28Z Storage Type raft HA Enabled true # vault operator unseal f7oWJN7aWSxv8tSCI1+g5w8A40WofGFvf7c/cDrFvVXu Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.15.2 Build Date 2023-11-06T11:33:28Z Storage Type raft Cluster Name HCV Cluster ID 2f22afcc-d95e-b788-0526-41d97bd5653a HA Enabled true HA Cluster n/a HA Mode standby Active Node AddressRaft Committed Index 32 Raft Applied Index 32
Each command prints the output of the vault status command at the end of execution. You can see that the third output is very different and indicates that the storage becomes usable.
You are not required to unseal the database using the CLI. It is possible to open browser and point it to the address https://hcv1.domain.com:8200, and then paste the required key. This method is useful if you actually use three different key holders.
Once unsealed, you can log into the vault using the saved root token and make changes or view the cluster status.
# vault login Token (will be hidden): # vault operator raft list-peers Node Address State Voter ---- ------- ----- ----- hcv1 hcv1.domain.com:8201 leader true
Actually, our cluster is almost ready, because all three nodes are already trying to communicate with each other. However, they still have nothing in common.
To add another node, open a shell (or browser) on it and run the same unseal commands. If successful, repeat on the third node. The resulting status will be
# vault operator raft list-peers Node Address State Voter ---- ------- ----- ----- hcv1 hcv1.domain.com:8201 leader true hcv2 hcv2.domain.com:8201 follower true hcv3 hcv3.domain.com:8201 follower true
Install Keepalived. It usually comes with the distribution.
# zypper in -y iptables keepalived ipvsadm
Create a configuration file /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type AH
auth_pass fe75c3ac
}
virtual_ipaddress {
10.0.0.14/24
}
}
virtual_server 10.0.0.14 8200 {
delay_loop 6
lb_algo sh
lb_kind NAT
persistence_timeout 50
protocol TCP
real_server 10.0.0.11 8200 {
weight 10
SSL_GET {
connect_timeout 2
url {
path /
status_code 307
}
}
}
real_server 10.0.0.12 8200 {
weight 10
SSL_GET {
connect_timeout 2
url {
path /
status_code 307
}
}
}
real_server 10.0.0.13 8200 {
weight 10
SSL_GET {
connect_timeout 2
url {
path /
status_code 307
}
}
}
}
An important information about configuration options. The brown parameters must be unique for this cluster in the network, otherwise this cluster will interfere with other clusters. The preferences goes to localhost to save unnesesary traffic. Neigbour nodes used only if localhost is not serving. Because of lb_kind NAT, IP forwarding should be enabled on each node:
# echo 1 > /proc/sys/net/ipv4/ip_forward # echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/50-ip_forward.conf
Copy this file to other nodes, replacing MASTER to BACKUP.
You can start keepalived now.
The only thing you should be aware of when restarting a storage cluster node is that it starts up sealed (its database is still locked with the encryption key). The node will try to participate in the cluster, but will be unsuccessful. You can check its status using the vault status command. You should run the command vault operator unseal XXX until the status changes and the node becomes opened. Then you can move on to another node.