close
Skip to content

Basic Features

Proxying, TLS, monitoring, high availability, balancing, stickiness, logs, and statistics

This section will enumerate a number of features that HAProxy implements, some of which are generally expected from any modern load balancer, and some of which are a direct benefit of HAProxy’s architecture. More advanced features will be detailed in the next section.

3.3.1. Basic features : Proxying

Proxying is the action of transferring data between a client and a server over two independent connections. The following basic features are supported by HAProxy regarding proxying and connection management:

  • Provide the server with a clean connection to protect them against any client-side defect or attack;

  • Listen to multiple IP addresses and/or ports, even port ranges;

  • Transparent accept: intercept traffic targeting any arbitrary IP address that doesn’t even belong to the local system;

  • Server port doesn’t need to be related to listening port, and may even be translated by a fixed offset (useful with ranges);

  • Transparent connect: spoof the client’s (or any) IP address if needed when connecting to the server;

  • Provide a reliable return IP address to the servers in multi-site LBs;

  • Offload the server thanks to buffers and possibly short-lived connections to reduce their concurrent connection count and their memory footprint;

  • Optimize TCP stacks (e.g. SACK), congestion control, and reduce RTT impacts;

  • Support different protocol families on both sides (e.g. IPv4/IPv6/Unix);

  • Timeout enforcement: HAProxy supports multiple levels of timeouts depending on the stage the connection is, so that a dead client or server, or an attacker cannot be granted resources for too long;

  • Protocol validation: HTTP, SSL, or payload are inspected and invalid protocol elements are rejected, unless instructed to accept them anyway;

  • Policy enforcement: ensure that only what is allowed may be forwarded;

  • Both incoming and outgoing connections may be limited to certain network namespaces (Linux only), making it easy to build a cross-container, multi-tenant load balancer;

  • PROXY protocol presents the client’s IP address to the server even for non-HTTP traffic. This is an HAProxy extension that was adopted by a number of third-party products by now, at least these ones at the time of writing:

    • client: haproxy, stud, stunnel, exaproxy, ELB, squid
    • server: haproxy, stud, postfix, exim, nginx, squid, node.js, varnish

3.3.2. Basic features : SSL

HAProxy’s SSL stack is recognized as one of the most featureful according to Google’s engineers (http://istlsfastyet.com/). The most commonly used features making it quite complete are:

  • SNI-based multi-hosting with no limit on sites count and focus on performance. At least one deployment is known for running 50000 domains with their respective certificates;

  • support for wildcard certificates reduces the need for many certificates;

  • certificate-based client authentication with configurable policies on failure to present a valid certificate. This allows to present a different server farm to regenerate the client certificate for example;

  • authentication of the backend server ensures the backend server is the real one and not a man in the middle;

  • authentication with the backend server lets the backend server know it’s really the expected haproxy node that is connecting to it;

  • TLS NPN and ALPN extensions make it possible to reliably offload SPDY/HTTP2 connections and pass them in clear text to backend servers;

  • OCSP stapling further reduces first page load time by delivering inline an OCSP response when the client requests a Certificate Status Request;

  • Dynamic record sizing provides both high performance and low latency, and significantly reduces page load time by letting the browser start to fetch new objects while packets are still in flight;

  • permanent access to all relevant SSL/TLS layer information for logging, access control, reporting etc. These elements can be embedded into HTTP header or even as a PROXY protocol extension so that the offloaded server gets all the information it would have had if it performed the SSL termination itself.

  • Detect, log and block certain known attacks even on vulnerable SSL libs, such as the Heartbleed attack affecting certain versions of OpenSSL.

  • support for stateless session resumption (RFC 5077 TLS Ticket extension). TLS tickets can be updated from CLI which provides them means to implement Perfect Forward Secrecy by frequently rotating the tickets.

3.3.3. Basic features : Monitoring

HAProxy focuses a lot on availability. As such it cares about servers state, and about reporting its own state to other network components:

  • Servers’ state is continuously monitored using per-server parameters. This ensures the path to the server is operational for regular traffic;

  • Health checks support two hysteresis for up and down transitions in order to protect against state flapping;

  • Checks can be sent to a different address/port/protocol: this makes it easy to check a single service that is considered representative of multiple ones, for example the HTTPS port for an HTTP+HTTPS server.

  • Servers can track other servers and go down simultaneously: this ensures that servers hosting multiple services can fail atomically and that no one will be sent to a partially failed server;

  • Agents may be deployed on the server to monitor load and health: a server may be interested in reporting its load, operational status, administrative status independently from what health checks can see. By running a simple agent on the server, it’s possible to consider the server’s view of its own health in addition to the health checks validating the whole path;

  • Various check methods are available: TCP connect, HTTP request, SMTP hello, SSL hello, LDAP, SQL, Redis, send/expect scripts, all with/without SSL;

  • State change is notified in the logs and stats page with the failure reason (e.g. the HTTP response received at the moment the failure was detected). An e-mail can also be sent to a configurable address upon such a change;

  • Server state is also reported on the stats interface and can be used to take routing decisions so that traffic may be sent to different farms depending on their sizes and/or health (e.g. loss of an inter-DC link);

  • HAProxy can use health check requests to pass information to the servers, such as their names, weight, the number of other servers in the farm etc. so that servers can adjust their response and decisions based on this knowledge (e.g. postpone backups to keep more CPU available);

  • Servers can use health checks to report more detailed state than just on/off (e.g. I would like to stop, please stop sending new visitors);

  • HAProxy itself can report its state to external components such as routers or other load balancers, allowing to build very complete multi-path and multi-layer infrastructures.

3.3.4. Basic features : High availability

Just like any serious load balancer, HAProxy cares a lot about availability to ensure the best global service continuity:

  • Only valid servers are used; the other ones are automatically evicted from load balancing farms; under certain conditions it is still possible to force to use them though;

  • Support for a graceful shutdown so that it is possible to take servers out of a farm without affecting any connection;

  • Backup servers are automatically used when active servers are down and replace them so that sessions are not lost when possible. This also allows to build multiple paths to reach the same server (e.g. multiple interfaces);

  • Ability to return a global failed status for a farm when too many servers are down. This, combined with the monitoring capabilities makes it possible for an upstream component to choose a different LB node for a given service;

  • Stateless design makes it easy to build clusters: by design, HAProxy does its best to ensure the highest service continuity without having to store information that could be lost in the event of a failure. This ensures that a takeover is the most seamless possible;

  • Integrates well with standard VRRP daemon keepalived: HAProxy easily tells keepalived about its state and copes very well with floating virtual IP addresses. Note: only use IP redundancy protocols (VRRP/CARP) over cluster-based solutions (Heartbeat, …) as they’re the ones offering the fastest, most seamless, and most reliable switchover.

3.3.5. Basic features : Load balancing

HAProxy offers a fairly complete set of load balancing features, most of which are unfortunately not available in a number of other load balancing products:

  • no less than 10 load balancing algorithms are supported, some of which apply to input data to offer an infinite list of possibilities. The most common ones are round-robin (for short connections, pick each server in turn), leastconn (for long connections, pick the least recently used of the servers with the lowest connection count), source (for SSL farms or terminal server farms, the server directly depends on the client’s source address), URI (for HTTP caches, the server directly depends on the HTTP URI), hdr (the server directly depends on the contents of a specific HTTP header field), first (for short-lived virtual machines, all connections are packed on the smallest possible subset of servers so that unused ones can be powered down);

  • all algorithms above support per-server weights so that it is possible to accommodate from different server generations in a farm, or direct a small fraction of the traffic to specific servers (debug mode, running the next version of the software, etc);

  • dynamic weights are supported for round-robin, leastconn and consistent hashing; this allows server weights to be modified on the fly from the CLI or even by an agent running on the server;

  • slow-start is supported whenever a dynamic weight is supported; this allows a server to progressively take the traffic. This is an important feature for fragile application servers which require to compile classes at runtime as well as cold caches which need to fill up before being run at full throttle;

  • hashing can apply to various elements such as client’s source address, URL components, query string element, header field values, POST parameter, RDP cookie;

  • consistent hashing protects server farms against massive redistribution when adding or removing servers in a farm. That’s very important in large cache farms and it allows slow-start to be used to refill cold caches;

  • a number of internal metrics such as the number of connections per server, per backend, the amount of available connection slots in a backend etc makes it possible to build very advanced load balancing strategies.

3.3.6. Basic features : Stickiness

Application load balancing would be useless without stickiness. HAProxy provides a fairly comprehensive set of possibilities to maintain a visitor on the same server even across various events such as server addition/removal, down/up cycles, and some methods are designed to be resistant to the distance between multiple load balancing nodes in that they don’t require any replication:

  • stickiness information can be individually matched and learned from different places if desired. For example a JSESSIONID cookie may be matched both in a cookie and in the URL. Up to 8 parallel sources can be learned at the same time and each of them may point to a different stick-table;

  • stickiness information can come from anything that can be seen within a request or response, including source address, TCP payload offset and length, HTTP query string elements, header field values, cookies, and so on.

  • stick-tables are replicated between all nodes in a multi-master fashion;

  • commonly used elements such as SSL-ID or RDP cookies (for TSE farms) are directly accessible to ease manipulation;

  • all sticking rules may be dynamically conditioned by ACLs;

  • it is possible to decide not to stick to certain servers, such as backup servers, so that when the nominal server comes back, it automatically takes the load back. This is often used in multi-path environments;

  • in HTTP it is often preferred not to learn anything and instead manipulate a cookie dedicated to stickiness. For this, it’s possible to detect, rewrite, insert or prefix such a cookie to let the client remember what server was assigned;

  • the server may decide to change or clean the stickiness cookie on logout, so that leaving visitors are automatically unbound from the server;

  • using ACL-based rules it is also possible to selectively ignore or enforce stickiness regardless of the server’s state; combined with advanced health checks, that helps admins verify that the server they’re installing is up and running before presenting it to the whole world;

  • an innovative mechanism to set a maximum idle time and duration on cookies ensures that stickiness can be smoothly stopped on devices which are never closed (smartphones, TVs, home appliances) without having to store them on persistent storage;

  • multiple server entries may share the same stickiness keys so that stickiness is not lost in multi-path environments when one path goes down;

  • soft-stop ensures that only users with stickiness information will continue to reach the server they’ve been assigned to but no new users will go there.

3.3.7. Basic features : Logging

Logging is an extremely important feature for a load balancer, first because a load balancer is often wrongly accused of causing the problems it reveals, and second because it is placed at a critical point in an infrastructure where all normal and abnormal activity needs to be analyzed and correlated with other components.

HAProxy provides very detailed logs, with millisecond accuracy and the exact connection accept time that can be searched in firewalls logs (e.g. for NAT correlation). By default, TCP and HTTP logs are quite detailed and contain everything needed for troubleshooting, such as source IP address and port, frontend, backend, server, timers (request receipt duration, queue duration, connection setup time, response headers time, data transfer time), global process state, connection counts, queue status, retries count, detailed stickiness actions and disconnect reasons, header captures with a safe output encoding. It is then possible to extend or replace this format to include any sampled data, variables, captures, resulting in very detailed information. For example it is possible to log the number of cumulative requests or number of different URLs visited by a client.

The log level may be adjusted per request using standard ACLs, so it is possible to automatically silent some logs considered as pollution and instead raise warnings when some abnormal behavior happen for a small part of the traffic (e.g. too many URLs or HTTP errors for a source address). Administrative logs are also emitted with their own levels to inform about the loss or recovery of a server for example.

Each frontend and backend may use multiple independent log outputs, which eases multi-tenancy. Logs are preferably sent over UDP, maybe JSON-encoded, and are truncated after a configurable line length in order to guarantee delivery. But it is also possible to send them to stdout/stderr or any file descriptor, as well as to a ring buffer that a client can subscribe to in order to retrieve them.

3.3.8. Basic features : Statistics

HAProxy provides a web-based statistics reporting interface with authentication, security levels and scopes. It is thus possible to provide each hosted customer with his own page showing only his own instances. This page can be located in a hidden URL part of the regular web site so that no new port needs to be opened. This page may also report the availability of other HAProxy nodes so that it is easy to spot if everything works as expected at a glance. The view is synthetic with a lot of details accessible (such as error causes, last access and last change duration, etc), which are also accessible as a CSV table that other tools may import to draw graphs. The page may self-refresh to be used as a monitoring page on a large display. In administration mode, the page also allows to change server state to ease maintenance operations.

A Prometheus exporter is also provided so that the statistics can be consumed in a different format depending on the deployment.