HAProxy

The Reliable, High Performance TCP/HTTP Load Balancer

  Mirror Sites: Master, China
  Language: English, Chinese

Quick links

Quick News
Recent News
Description
Design Choices
Supported Platforms
Performance
Reliability
Security
Download
Documentation
Live demo
Commercial Support
Products using HAProxy
Contributions
Other Solutions
Contacts
External links
Mailing list archives
10GbE load-balancing (updated)


Willy TARREAU
You want to donate ?


 
Web 1wt.eu

Quick News

June 27th, 2009 : HAProxy to counter DoS attacks

    Since the announcement of the Slowloris tool, people seem to be discovering how fragile a default Apache setup can be ! Well, this is not news, as people who install Apache on high-traffic sites have been aware of this weakness for ages, and have been setting very low timeouts and disabling keep-alive in order to mitigate risks. Now that a tool is publicly advertised, I'm beginning to hear questions from worried site admins about what to do if their site is attacked. Also, we're seeing several sites and forums suggesting installing HAProxy in front of Apache servers to protect them (note that Nginx would probably do equally well).

    Indeed, HAProxy does not need a new thread nor process to accept a new connection, it only needs some RAM (16-32 kB per connection). Some people are already using it past 70000 concurrent connections, which cannot be achieved on Apache which needs an expensive thread or process per connection. More specifically, HAProxy will only forward complete and valid requests. This means that it will not bother Apache while the attacker is playing with its few thousands connections, and all valid requests will immediately pass through. And the icing on the cake is that HAProxy can kill requests which take too much time to complete, using timeout http-request (more than a few seconds is not to be considered normal).

    Once again, we observe a derivate use of a load-balancer, which is a bit expected : when a tool is designed to accept 10 times more load than the servers it feeds, there is nothing surprizing that it can be used to protect them ! Let's see if Apache evolves towards providing more tunables to mitigate such attacks... In the mean time, a drop-in anti-DoS configuration is available here.

May 10th, 2009 : 1.3.18

    Yan Qiao of Rocket Fuel Inc reported a crash on x86_64, which was pretty much unexpected ! He nicely offered to help troubleshooting by rebuilding with debugging on and leaving the process running in production to catch the error, then sent me an interesting core 1 week later, which revealed that a field in the struct session which was never touched had been changed due to the sharing of two pools of the same size. This field should have been initialized but was unfortunately not. The issue can only happen on x86_64 with HTTP logging enabled, due to the exact 1024 bytes of the struct session which allows its pool to be shared with the struct requri's. Thank you guys for your huge help and the risks you have taken leaving that process running!

    During a troubleshooting session with the T20 guys (Maxim Fedchishin, Jason Coward and Viktor Brilon from modX team, Hans from RightScale team), I came across an old leftover process doing nothing after a soft-reload. That issue is brought once in a while by various people, but it happens too rarely for anyone to get an opportunity to debug it. The guys accepted that I installed a debugger on their machine to see what the process was doing. It was deadlocked in free() during the reload. And that made sense : during a reload, the old process releases as much memory as possible to leave room for the new one. If the two signals sent by the second one are too close to each other, the second signal is sent while the first one has not completed releasing memory and we can have a recursion in the libc's free(), causing a deadlock. That has been fixed by implementing asynchronous signal delivery. Thank you guys for giving me the opportunity to catch that rare event!

    Problems aside, a few minor features were merged. The stats are now more readable, report max session rates and provide full 64-bit counters everywhere. It is now possible to forward invalid requests or responses without blocking them, but they will still be captured. The config parser now warns about possibly unwanted ordering of ACLs or reqxxx/rspxxx. Several wrong printf() format strings have been fixed. The build process now supports an alternative architecture, and the RPM spec file has been cleaned. A new balance hdr(header) algorithm has been added to balance depending on a header hash. A new option enables addition of the destination IP address in the X-Original-To header. And last but not least, the doc has been massively cleaned up and reorganised.

    With all these fixes, I released 1.3.18, as well as 1.3.15.9 and 1.3.14.13 which are probably among the last ones of their respective branches after 12 and 18 months of maintenance.

April 19th, 2009 : new performance record broken !

    It was a long time since my last 10 Gigabit tests, exactly one year. The Linux kernel has evolved a lot, so did HAProxy and even the Myri10GE driver. I knew we could get much throughput since we fixed the kernel splice() syscall. It was a good opportunity to start a new series of benchmarks again. In short, new records were broken. Full 10GbE line rate with 20% CPU, and the 100000 sessions/s barrier was crossed !

March 29th, 2009 : 1.3.17

    Bart Bobrowski of who's.amung.us reported abnormal CPU usage with the new version 1.3.16. After a full day of tests and code analysis, I failed to reproduce the issue here, and the bug appeared impossible to me. Bart then offered a lot of help with testing many patches, providing hundreds of megs of traces, so that I could finally fix the issue caused by a nasty race condition. I really appreciate it when users with extreme loads accept to take traces in production, with all the risks that this practise implies. Sometimes it's the only way to get a bug fixed.Thanks Bart!.
    Since other minor fixes and enhancements were pending, I released 1.3.17, which users of 1.3.16 are invited to upgrade to.

Recent news...

Latest versions

BranchDescriptionLast versionReleasedNotes
gitDevelopment GIT not releasedmay be broken
1.31.3-stable 1.3.18 2009/05/10 New stable version
1.3.151.3.15-maint 1.3.15.9 2009/05/10 Previous version
1.3.141.3.14-maint 1.3.14.13 2009/05/10 Critical fixes only
1.2.x1.2-stable 1.2.18 2008/05/25 Critical fixes only
1.1.x1.1-stable 1.1.34 2006/01/29 Unmaintained
1.0.x1.0-old1.0.22001/12/30Unmaintained

Description

HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net, such as below :

Currently, two major versions are supported (NB: comment is outdated):

  • version 1.3 - content switching and extreme loads
    This version has brought a lot of new features and improvements over 1.2, among which content switching to select a server pool based on any request criteria, ACL to write content switching rules, wider choice of load-balancing algorithms for better integration, content inspection allowing to block unexpected protocols, transparent proxy under Linux, which allows to directly connect to the server using the client's IP address, kernel TCP splicing to forward data between the two sides without copy in order to reach multi-gigabit data rates, layered design separating sockets, TCP and HTTP processing for more robust and faster processing and easier evolutions, fast and fair scheduler allowing better QoS by assigning priorities to some tasks, session rate limiting for colocated environments, etc...
  • version 1.2 - opening the way to very high traffic sites
    The same as 1.1 with some new features such as poll/epoll support for very large number of sessions, IPv6 on the client side, application cookies, hot-reconfiguration, advanced dynamic load regulation, TCP keepalive, source hash, weighted load balancing, rbtree-based scheduler, and a nice Web status page. This code is in deep feature freeze and may eventually receive critical fixes only.

Version 1.1, which has been maintaining critical sites online since 2002, is not maintained anymore. Users should upgrade to 1.2 or 1.3, keeping in mind that 1.2 will soon not be supported anymore either.

Unlike other free "cheap" load-balancing solutions, this product is only used by a few hundreds to a few thousands of people around the world, but those people run very big sites serving several millions hits and between several tens of gigabytes to several terabytes per day to hundreds of thousands of clients. They need 24x7 availability and have internal skills to risk to maintain a free software solution. Often, the solution is deployed for internal uses and I only know about it when they send me some positive feedback or when they ask for a missing feature ;-)

Design Choices and history

HAProxy implements an event-driven, single-process model which enables support for very high number of simultaneous connections at very high speeds. Multi-process or multi-threaded models can rarely cope with thousands of connections because of memory limits, system scheduler limits, and lock contention everywhere. Event-driven models do not have these problems because implementing all the tasks in user-space allows a finer resource and time management. The down side is that those programs generally don't scale well on multi-processor systems. That's the reason why they must be optimized to get the most work done from every CPU cycle.

It began in 1996 when I wrote Webroute, a very simple HTTP proxy able to set up a modem access. But its multi-process model cloberred its performance for other usages than home access. Two years later, in 1998, I wrote the event-driven ZProx, used to compress TCP traffic to accelerate modem lines. It was when I first understood the difficulty of event-driven models. In 2000, while benchmarking a buggy application, I heavily modified ZProx to introduce a very dirty support for HTTP header rewriting. HAProxy's ancestor was born. First versions did not perform the load-balancing themselves, but it quickly proved necessary.

Now in 2009, the core engine is reliable and very robust. Event-driven programs are robust and fragile at the same time : their code needs very careful changes, but the resulting executable handles high loads and supports attacks without ever failing. It is the reason why HAProxy only supports a fine set of features. HAProxy has never ever crashed in a production environment. This is something people are not used to nowadays, because the most common things new users tell me is that they're amazed it has never crashed ;-)

People often ask for SSL and Keep-Alive support. Both features will complicate the code and render it fragile for several releases. By the way, both features have a negative impact on performance :

  • Having SSL in the load balancer itself means that it becomes the bottleneck. When the load balancer's CPU is saturated, the overall response times will increase and the only solution will be to multiply the load balancer with another load balancer in front of them. the only scalable solution is to have an SSL/Cache layer between the clients and the load balancer. Anyway for small sites it still makes sense to embed SSL, and it's currently being studied. There has been some work on the CyaSSL library to ease integration with HAProxy, as it appears to be the only one out there to let you manage your memory yourself.
  • Keep-alive was invented to reduce CPU usage on servers when CPUs were 100 times slower. But what is not said is that persistent connections consume a lot of memory while not being usable by anybody except the client who openned them. Today in 2009, CPUs are very cheap and memory is still limited to a few gigabytes by the architecture or the price. If a site needs keep-alive, there is a real problem. Highly loaded sites often disable keep-alive to support the maximum number of simultaneous clients. The real downside of not having keep-alive is a slightly increased latency to fetch objects. Browsers double the number of concurrent connections on non-keepalive sites to compensate for this.

However, I'm planning on implementing both features in future versions, because it appears that there are users who mostly need availability above performance, and for them, it's understandable that having both features will not impact their performance, and will reduce the number of components.

Supported platforms

HAProxy is known to reliably run on the following OS/Platforms :

Highest performance should be achieved with haproxy versions newer than 1.2.5 running on Linux 2.6, or epoll-patched Linux kernel 2.4. It is only because of a very OS-specific optimization : the default polling system for version 1.1 is select(), which is common among most OSes, but can become slow when dealing with thousands of file-descriptors. Versions 1.2 and 1.3 uses poll() by default instead of select(), but on some systems it may even be slower. However, it is recommended on Solaris as its implementation is rather good. Haproxy 1.3 will automatically use epoll on Linux 2.6 and patched Linux 2.4, and kqueue on FreeBSD and OpenBSD. Both mechanisms achieve constant performance at any load thus are preferred over poll().

On very recent Linux 2.6 (>= 2.6.27.19), HAProxy can use the new splice() syscall to forward data between interfaces without any copy. Performance above 10 Gbps may only be achieved that way.

Based on those facts, people looking for a very fast load balancer should consider the following options on x86 or x86_64 hardware, in this order :

  1. HAProxy 1.3 on Linux 2.6.27+
  2. HAProxy 1.3 on Linux 2.4 + epoll
  3. HAProxy 1.3 on Linux 2.6.16 + scheduler starvation fixes
  4. HAProxy 1.3 on FreeBSD
  5. HAProxy 1.3 on Solaris 10

Current typical 1U servers equipped with a dual-core Opteron or Xeon generally achieve between 15000 and 30000 hits/s and have no trouble saturating 2 Gbps under Linux.

Performance

Well, since a user's testimony is better than a long demonstration, please take a look at Chris Knight's experience with haproxy saturating a gigabit fiber on a video download site. Another big data provider I know constantly pushes between 2 and 3 Gbps of traffic 24 hours a day. Also, my experiments with Myricom's 10-Gig NICs might be of interest.

HAProxy involves several techniques commonly found in Operating Systems architectures to achieve the absolute maximal performance :

  • a single-process, event-driven model considerably reduces the cost of context switch and the memory usage. Processing several hundreds of tasks in a millisecond is possible, and the memory usage is in the order of a few kilobytes per session while memory consumed in Apache-like models is more in the order of megabytes per process.
  • O(1) event checker on systems that allow it (Linux and FreeBSD) allowing instantaneous detection of any event on any connection among tens of thousands.
  • Single-buffering without any data copy between reads and writes whenever possible. This saves a lot of CPU cycles and useful memory bandwidth. Often, the bottleneck will be the I/O busses between the CPU and the network interfaces. At 10 Gbps, the memory bandwidth can become a bottleneck too.
  • MRU memory allocator using fixed size memory pools for immediate memory allocation favoring hot cache regions over cold cache ones. This dramatically reduces the time needed to create a new session.
  • work factoring, such as multiple accept() at once, and the ability to limit the number of accept() per iteration when running in multi-process mode, so that the load is evenly distributed among processes.
  • tree-based storage, making heavy use of the Elastic Binary tree I have been developping for several years. This is used to keep timers ordered, to keep the runqueue ordered, to manage round-robin and least-conn queues, with only an O(log(N)) cost.
  • optimized HTTP header analysis : headers are parsed an interpreted on the fly, and the parsing is optimized to avoid an re-reading of any previously read memory area. Checkpointing is used when an end of buffer is reached with an incomplete header, so that the parsing does not start again from the beginning when more data is read. Parsing an average HTTP request typically takes 2 microseconds on a Pentium-M 1.7 GHz.
  • careful reduction of the number of expensive system calls. Most of the work is done in user-space by default, such as time reading, buffer aggregation, file-descriptor enabling/disabling.

All these micro-optimizations result in very low CPU usage even on moderate loads. And even at very high loads, when the CPU is saturated, it is quite common to note figures like 5% user and 95% system, which means that the HAProxy process consumes about 20 times less than its system counterpart. This explains why the tuning of the Operating System is very important. I personnally build my own patched Linux 2.4 kernels, and finely tune a lot of network sysctls to get the most out of a reasonable machine.

This also explains why Layer 7 processing has little impact on performance : even if user-space work is doubled, the load distribution will look more like 10% user and 90% system, which means an effective loss of only about 5% of processing power. This is why on high-end systems, HAProxy's Layer 7 performance can easily surpass hardware load balancers' in which complex processing which cannot be performed by ASICs has to be performed by slow CPUs. Here is the result of a quick benchmark performed on haproxy 1.3.9 at EXOSEC on a single core Pentium 4 with PCI-Express interfaces:

    In short, a hit rate above 10000/s is sustained for objects smaller than 6 kB, and the Gigabit/s is sustained for objects larger than 40 kB.

In production, HAProxy has been installed several times as an emergency solution when very expensive, high-end hardware load balancers suddenly failed on Layer 7 processing. Hardware load balancers process requests at the packet level and have a great difficulty at supporting requests across multiple packets and high response times because they do no buffering at all. On the other side, software load balancers use TCP buffering and are insensible to long requests and high response times. A nice side effect of HTTP buffering is that it increases the server's connection acceptance by reducing the session duration, which leaves room for new requests. New benchmarks will be executed soon, and results will be published. Depending on the hardware, expected rates are in the order of a few tens of thousands of new connections/s with tens of thousands of simultaneous connections.

There are 3 important factors used to measure a load balancer's performance :

A load balancer's performance related to these factors is generally announced for the best case (eg: empty objects for session rate, large objects for data rate). This is not because of lack of honnesty from the vendors, but because it is not possible to tell exactly how it will behave in every combination. So when those 3 limits are known, the customer should be aware that he will generally be below all of them. A good rule of thumb on software load balancers is to consider an average practical performance of half of maximal session and data rates for average sized objects.

You might be interested in checking the 10-Gigabit/s page.

Reliability - keeping high-traffic sites online since 2002

Being obsessed with reliability, I tried to do my best to ensure a total continuity of service by design. It's more difficult to design something reliable from the ground up in the short term, but in the long term it reveals easier to maintain than broken code which tries to hide its own bugs behind respawning processes and tricks like this.

In single-process programs, you have no right to fail : the smallest bug will either crash your program, make it spin like mad or freeze. There has not been any such bug found in the code nor in production for the last 7 years.

HAProxy has been installed on Linux 2.4 systems serving millions of pages every day, and which have only known one reboot in 3 years for a complete OS upgrade. Obviously, they were not directly exposed to the Internet because they did not receive any patch at all. The kernel was a heavily patched 2.4 with Robert Love's jiffies64 patches to support time wrap-around at 497 days (which happened twice). On such systems, the software cannot fail without being immediately noticed !

Right now, it's being used in several Fortune 500 companies around the world to reliably serve millions of pages per day or relay huge amounts of money. Some people even trust it so much that they use it as the default solution to solve simple problems (and I often tell them that they do it the dirty way). Such people sometimes still use versions 1.1 or 1.2 which sees very limited evolutions and which targets mission-critical usages. HAProxy is really suited for such environments because the indicators it returns provide a lot of valuable information about the application's health, behaviour and defects, which are used to make it even more reliable. Version 1.3 has now received far more testing than 1.1 and 1.2 combined, so users are strongly encouraged to migrate to a stable 1.3 for mission-critical usages.

As previously explained, most of the work is executed by the Operating System. For this reason, a large part of the reliability involves the OS itself. Recent versions of Linux 2.4 offer the highest level of stability. However, it requires a bunch of patches to achieve a high level of performance. Linux 2.6 includes the features needed to achieve this level of performance, but is not yet as stable for such usages. The kernel needs at least one upgrade every month to fix a bug or vulnerability. Some people prefer to run it on Solaris (or do not have the choice). Solaris 8 and 9 are known to be really stable right now, offering a level of performance comparable to Linux 2.4. Solaris 10 might show performances closer to Linux 2.6, but with the same code stability problem. I have too few reports from FreeBSD users, but it should be close to Linux 2.4 in terms of performance and reliability. OpenBSD sometimes shows socket allocation failures due to sockets staying in FIN_WAIT2 state when client suddenly disappears. Also, I've noticed that hot reconfiguration does not work under OpenBSD.

The reliability can significantly decrease when the system is pushed to its limits. This is why finely tuning the sysctls is important. There is no general rule, every system and every application will be specific. However, it is important to ensure that the system will never run out of memory and that it will never swap. A correctly tuned system must be able to run for years at full load without slowing down nor crashing.

Security - Not even one vulnerability in 7 years

Security is an important concern when deploying a software load balancer. It is possible to harden the OS, to limit the number of open ports and accessible services, but the load balancer itself stays exposed. For this reason, I have been very careful about programming style. The only vulnerability found so far dates 7 years and only lasted for one week. It was introduced when logs were reworked. It could be used to cause BUS ERRORS to crash the process, but it did not seem possible to execute code : the overflow concerned only 3 bytes, too short to store a pointer (and there was a variable next).

Anyway, much care is taken when writing code to manipulate headers. Impossible state combinations are checked and returned, and errors are processed from the creation to the death of a session. A few people around the world have reviewed the code and suggested cleanups for better clarity to ease auditing. By the way, I'm used to refuse patches that introduce suspect processing or in which not enough care is taken for abnormal conditions.

I generally suggest starting HAProxy as root because it can then jail itself in a chroot and drop all of its privileges before starting the instances. This is not possible if it is not started as root because only root can execute chroot().

Logs provide a lot of information to help to maintain a satisfying security level. They can only be sent over UDP because once chrooted, the /dev/log UNIX socket is unreachable, and it must not be possible to write to a file. The following information are particularly useful :

  • source IP and port of requestor make it possible to find their origin in firewall logs ;
  • session set up date generally matches firewall logs, while tear down date often matches proxies dates ;
  • proper request encoding ensures the requestor cannot hide non-printable characters, nor fool a terminal.
  • arbitrary request and response header and cookie capture help to detect scan attacks, proxies and infected hosts.
  • timers help to differentiate hand-typed requests from browsers's.

HAProxy also provides regex-based header control. Parts of the request, as well as request and response headers can be denied, allowed, removed, rewritten, or added. This is commonly used to block dangerous requests or encodings (eg: the Apache Chunk exploit), and to prevent accidental information leak from the server to the client. Other features such as Cache-control checking ensure that no sensible information gets accidentely cached by an upstream proxy consecutively to a bug in the application server for example.

Download

The source code is covered by GPL v2. Source code and pre-compiled binaries for Linux/x86 and Solaris/Sparc can be downloaded right here :

Documentation

There are three types of documentation now : the Reference Manual which explains how to configure HAProxy but which is outdated, the Architecture Guide which will guide you through various typical setups, and the new Configuration Manual which replaces the Reference Manual with more a explicit configuration language explanation.

Commercial Support (France)

If you think you don't have the time and skills to setup and maintain a free load balancer, or if you're seeking for commercial support to satisfy your customers or your boss, you should contact EXOSEC. Another solution would be to use Exceliance's ALOHA appliances (see below).

Products using HAProxy

The following products or projects use HAProxy :

  • redWall Firewall
    From the site : "redWall is a bootable CD-ROM Firewall. Its goal is to provide a feature rich firewall solution, with the main goal, to provide a webinterface for all the logfiles generated!"
  • Exceliance's ALOHA Load Balancer appliance
    Exceliance is a french company who sells a complete haproxy-based solution embedding an optimized and hardened version of Formilux packaged for ease of use via a full-featured Web interface, reduced maintenance, and enhanced availability through the use of VRRP for box fail-over, bonding for link fail-over, configuration synchronization, SSL, transparent mode, etc... (check differences between HAProxy and Aloha). An evaluation version running in VMWare Player is available on the site. Since this is where I work, a lot of features are created there :-)
  • Loadbalancer.org
    This company based in the UK has recently added HAProxy to their load-balancing solution in order to provide the basic layer 7 support that some customers were asking for. They're also among the rare commercial product makers who admit to use HAProxy and who have donated to the project.

Contributions

Some happy users have contributed code which may or may not be included. Others spent a long time analysing the code, and there are some who maintain ports up to date.

Other Solutions

If you don't need all of HAProxy's features and are looking for a simpler solution, you may find what you need here :

  • Linux Virtual Servers (LVS)
    Very fast layer 3/4 load balancing merged in Linux 2.4 and 2.6 kernels. Should be coupled with Keepalived to monitor servers. This generally is the solution embedded by default in most IP-based load balancers.
  • Pure Load Balancer (PLB)
    The author adopted the same event-driven model as in HAProxy (but relying on libevent). Interestingly, he has the same conclusions about other models's limitations. However, his goal is just to achieve high performance and availability, without any particular HTTP processing nor persistence.
  • Pound
    Pound can be seen as a complement to HAProxy. It supports SSL, and can direct traffic according to the requested URL. Its code is very small and will stay small for easy auditing. Its configuration file is very small too. However, it does not support persistence, and the performance associated to its multi-threaded model limits its usage to medium sites only.
  • Pen
    Pen is a very simple load balancer for TCP protocols. It supports source IP-based persistence for up to 2048 clients. Supports IP-based ACLs. Uses select() and supports higher loads than Pound but will not scale very well to thousands of simultaneous connections.

Contacts

Feel free to contact me at for any questions or comments :

Some people regularly ask if it is possible to send donations, so I have set up a Paypal account for this. Click here if you want to donate.

An IRC channel for haproxy has been opened on FreeNode (but don't seek me there, I'm not) :

External links

Here are some links to possibly useful external contents I gathered on the net. I have found most of them due to their link to haproxy's site ;-)