Load-Balancers
Now let's deploy the Load-Balancer cluster.
Load-Balancer Type
This step is only required when not using Load-Balancer-as-a-service (e.g. AWS ALB).
This is conditionned by the variable:
dalet_flex_platform_lbaas_enabled: bool
If enabled, Ansible will assume you're using a third-party managed application load-balancer instances. You're all set, you can now skip the rest of this section.
If disabled, Ansible will deploy a self-managed HAProxy cluster.
Self-Managed Load-Balancer
If instructed to do so, Ansible will be used to deploy a master-slave cluster with high-availability and failover support on the 2 server instances that are part of the haproxy group in your Ansible's inventory file.
Please ensure beforehand that the group exist with 2 defined LB instances (or more, but not really useful).
Redundancy and Failover
The deployed HAProxy cluster will run against 2 nodes:
- a master (or primary, active), which handles all incoming traffic requests.
- a slave (or secondary, inactive), ready to take over, should master instance fails.
This mechanism works through usage of a virtual IP (or VIP) within your private network. VIP is basically a floatting IP address, an IP moving from one server instance to another depending on various criteria.
For load-balancer cluster, LB VIP is assigned to the master instance (whoever that is from the pool). This ensures that HTTP(S) clients always connect to the very same IP address (no configuration change), whatever happens at the infrastructure level. Should HAProxy service fails on current master (crash, intended maintenance, disk issue, network issue, anything really ...), the VIP binding criteria will fail and the current slave will bind LB VIP and take over traffic. Once former master comes back to life, it'll boot as a slave, ready to take over again.
A key element to take into account is that VIP works through VRRP protocol at network L2 layer. It relies on unique virtual router identifier (VRID) to be known and shared only by the peers who are eligible for the given VIP.
The VIP address be unique on the given network L3 network, so one must be picked carefully. You must ensure that this is a free, unused IP address, that is not assigned to any existing (or future) machine (including the database instances themselves). The VIP must not be part of any DHCP server's pool, again, to ensure it won't be assigned to anyone.
The following parameters must be filled (example):
dalet_flex_lb_mysql_host: "LB VIP ADDRESS"
dalet_flex_lb_mysql_private_virtual_router_id: 1-255
dalet_flex_lb_mysql_private_interface: eth0
dalet_flex_lb_mysql_private_control_interface: eth0
This defines which VIP and VRID are to be used and which instance's network interface is to be used to share VRRP packets (control) and to bind the VIP (can be the same network interface).
TLS Security
Chances are, your platform is going to be exposed over Internet. Even if not, it is good practice to use (or even enforce) HTTPS usage, i.e. SSL/TLS traffic encryption. (actually, some browsers may even prevent you from accessing unsecure HTTP servers in the future).
The first thing to define is minimum supported TLS protocol. The following platform variable may be override if necessary:
dalet_flex_lb_tls_min_version: "1.3"
- TLS 1.3 is the latest, most secure version there is.
- TLS 1.2 is still strong and recommended.
- TLS versions prior to 1.2 are known to be weak and vulnerable and their usage is not recommended anymore.
Decreasing the minimum supported TLS version weakens global security but might be useful if using legacy/deprecated HTTP clients with limited encryption capabilities.
Next information you'd need is the SSL certificate itself. The certificate works with a typical private/public key echange, where server encrypt HTTP traffic with a private key and client decrypts it with the associated public key. It also ensures clients that the server they're conacting is to be trusted and who it pretends to be (no man-in-the-middle).
The SSL certificate is to be signed by a known Certificate Authority (CA) that is trusted by your clients. If you have your own PKI and CA, you must then ensure that CA certificate is well deployed on every client which will connect to Flex or connections will be refused (seen as untrusted).
Two options come next related to SSL certificate itself:
- Use your own previously generated custom certificate file. You will have to carefully manage the certificate, its security and lifetime.
- Instruct HAProxy to use Let's Encrypt CA to automatically issue (and renew, when required) an SSL certificate for you. This is the most seamless and recommended approach.
This is instructed through the following variable:
dalet_flex_lb_tls_certificate_issuer: 'auto' or 'custom'
Customer-provided SSL certificate
Dalet strongly recommends the use of wildcard SSL certificates (e.g. *.flex.acme.com, where * means an infinite number of sub-domains being supported) as the Dalet Flex software relies on the host part of the address to work.
If wildcard SSL certificate is not an option (due to customer’s internal security policy), a SAN (Subject Alternative Names) certificate must be provided, listing all possible subdomains to be ever used by Flex. Approximately 20 SAN entries must be provided when such a use case as to be implemented.
The mandatory DNS entries are:
- grafana.<domain_name>
- kibana.<domain_name>
.<domain_name> - master.<domain_name>
- metadata.<domain_name>
- workflow.<domain_name>
- consul.<domain_name>
Optional one might be added, depending on services existence or willigness to be exposed:
- upload.<domain_name>
- reviewer.<domain_name>
- publish.<domain_name>
- panels.<domain_name>
- prometheus.<domain_name>
- alertmanager.<domain_name>
Custom SSL certificate generation is out of the scope of this documentation and let to operator.
Dalet suggests that the provided SSL certificate is at least RSA-2048 or EC-256 encrypted, with a SHA-256 digital signature and a 1y minimal validity. These encryption requirements are considered to be safe enough to nowadays standards while keeping a wide compatibility with modern client OSes and browsers.
The operator is responsible for the proper management (including expiry) of the certificate provided. Certificate compromission is expiracy requires HAproxy cluster update (i.e. re-deployment).
When custom TLS certificate issuer is chosen, the certificate must be be provided in PEM format and copied into ansible/files directory, associated with the following variable:
dalet_flex_lb_tls_certificate_issuer_custom_cert_file: cert.pem
Load-Balancer expects certificates to be in fullchain format, i.e.
- The certificate for your domain
- The The intermediates in ascending order to the Root CA
- A Root CA, if any (usually none)
- Private Key
Example:
$ cat certificate.crt intermediates.pem private.key > cert.pem
It is highly recommended for certificates to be encrypted with Vault/SOPS (contains private key). This can be done through:
$ opsctl vault encrypt ansible/files/cert.pem
Auto-generated SSL certificate
The recommended approach is to use auto-generated certificates instead. This takes care of initial certificate generation and automatic renewal when close to expiry, nullifying the need for manual maintenance process.
Once configured to do so, the following variables are expected:
dalet_flex_lb_tls_certificate_issuer_auto_subdomains: ['*']
dalet_flex_lb_tls_certificate_issuer_auto_challenge: 'dns-route53' or 'http'
This instructs HAProxy to request Let's Encrypt for automatic wildcard certificate for the specified platform's domain. The challenge parameter defines whether Let's Encrypt should use DNS or HTTP for identity verification. It ensures that whoever ask for SSL certificate for a given domain actually owns the domain.
- If DNS-Route53 challenge is requested, the following variables are required to be filled in:
dalet_flex_lb_tls_certificate_issuer_auto_route53_access_key_id: ''
dalet_flex_lb_tls_certificate_issuer_auto_route53_secret_access_key: ''
so that Route53 API can be used to create the relevant records allowing Let's Encrypt to validate your identity.
- If HTTP challenge is requested the following variable is expected:
dalet_flex_lb_tls_certificate_issuer_auto_http_port: 8888
where HAProxy will bind the requested port to offer the relevant information Let's Encrypt requires to validate your identity.
If HTTP challenge is requested, don't forget to allow for incoming TCP packets on the configured port from firewall perspective.
Futher Tuning
Many other settings can be used to fine-tune your self-managed load-balancer and optimize its inherent behavior, as described in reference guide.
Deployment
Finally, let's deploy the HAProxy cluster:
$ opsctl deploy -p dalet.flex.haproxy