HAProxy

Quick links

News

Three months ago I was approached by the Stack Overflow Team team who needed to get some improvements in HAProxy. Overall, their needs would have been addressed by the final release of version 1.5 scheduled around the end of the year, but having to wait that long was not practical due to some architectural constraints imposed by an intermediate solution. They proposed that we find an agreement on which we could work together. Since we were already having productive exchanges for some time, and I knew they were good guys (after all they already donated to the project last year), I accepted the deal.

Also, I must say that as a software engineer, it's always a lot better to have someone explain their needs with high expectations than having to guess how a feature will be used.

Geoff Dalgas and Jeff Atwood described to me in great details what they needed to do : perform request throttling per IP address, possibly based on various criteria, in order to limit risks of service abuse. That was very interesting, because that feature was being thought about for about 4 years without enough time to completely develop it, and also the new stickiness framework that was contributed by Exceliance and Loadbalancer.org was making that really possible, although an important design rework had to be operated first within the code.

During the tests with Geoff and Kyle Brandt, it appeared that some more changes had to be operated to be able to store any criteria in the tables (eg: bandwidth per IP address), and to be able to consult and change the table contents at runtime, leading to a more and more generic code. Kyle has been very patient and comprehensive, I think I have changed the mechanisms and configuration syntax at least 5 or 6 times during the tests, but he always took the time to understand the changes and adapt his configurations. If I had been at his place, I would have got bored earlier, so I owe him a big thanks for his patience !

Now the code has been running fine in production overthere, so it's time to release it and merge it into the master branch. I won't extend further on how it works, since Kyle has put an excellent explanation on his blog that is a lot more clear than the doc (that reminds me that the architecture guide really needs some lifting).

I'll add quick status on the current code. Some core changes that I wanted to do earlier will now start. But that means that 1.5-dev1 should be as stable as 1.4.8. I'm not saying that I would suggest to anyone to push it into production, but it can clearly be used to mitigate DDoS attacks as well as stop service abuses. I could get it to stop connection floods slightly above 200000 connections per second (yes, two hundreds thousands) and let the good traffic pass through. So for this reason, I think that people who are regularly exposed to such trouble may find it useful to keep it handy.

Now what's next ? Right now the data in the tables is local to one process, so it is not shared if you start multiple processes, nor it is across reloads. The second step of the stickiness extensions developped by Exceliance and Loadbalancer.org will include stickiness table synchronization between multiple hosts. Some work will still be needed to be able to share counters, but since this development is done in a flexible way, it should not be too hard to adapt it later. BTW, I also owe a big kudos to the Git versionning system, which has made it very easy to rework my patches after every change and bugfix until they were looking good, through massive abuse of branching and rebasing.

Too much talk. The code is available here, the ChangeLog is here and the doc is here. Since this is a development version, no binary is provided.

The last words naturally go to the really cool guys at Stack Overflow. It's very nice to see some sites and companies involve time and money and take risks to make Open Source products better. Of course they benefit from this work, but at no point during the whole development did they try to reduce the focus to their specific needs, quite the opposite. From the very first exchanges, their goal clearly was to make the product better, and that must be outlined. That's now achieved and I really appreciate their involvement. Thank you guys !

June 16th, 2010 : stable 1.4.8 and 1.3.25

Today, Ben Congleton, Morten Gade Sørensen and Hervé Commowick reported and diagnosed two bugs in 1.4.7. One was a regression introduced in 1.4.7, breaking the stick-table feature due to a side effect of the fix for a memory leak in it. The second one is quite old, it dates 1.3.16 (2 years old) ! It causes crashes soon after a connection has matched a monitor-net in a tcp-only instance. I wanted to quickly fix both issues so that users who have not yet upgraded to 1.4.7 don't waste their time, and I'm very grateful to the 3 guys above who were extremely reactive and allowed the bugs to be fixed in a few hours. Finally, Patrick Mézard's doc updates were merged too.

Version 1.4.8 was released, with sources, Linux and Solaris binaries at the usual places. Version 1.3.25 was released too with a few other pending fixes, and sources and Linux/Solaris binaries at the usual places too.

Users of 1.3 after 1.3.16, or 1.4 with TCP only frontends will likely want to upgrade, eventhough if they have not been hit yet, they will probably never be. Users of stick-tables will have to upgrade too. The fact that we're now spotting bugs that have been there for two years is an encouraging sign of stability.

June 10th, 2010 : stable 1.4.7

Jeff Persch debugged the trouble affecting consistent hashing about two weeks ago. I wanted to issue 1.4.7 right after that fix got merged but I was lucky enough to spot a few minor bugs in the following days, so that finally got delayed. Most of these issues are really minor and were found while reviewing the code in preparation of 1.5-dev. Some stats were not accounted properly (failed req instead of denied req), some termination flags in the logs could be wrong, which is a shame because we use them for debugging, a TCP to HTTP transition was not properly handled, and the dispatch and http_proxy modes were broken since 1.4-dev, but apparently nobody uses them anymore.

Some people will like the new halog, as it is able to report per-server stats on status codes, error ratios, average connection times and response times, which is very handy for quick checking of your prod's health.

The complete ChangeLog for version 1.4.7 is here and the sources are here.

On a side note, someone asked on the list how haproxy would perform on a Guruplug Server Plus. I ordered one a few months ago and finally received it. Well, it's a disaster, not only it is slower than the Geode, but it heats so much that metal parts hurt to the touch and power supplies fry after a few months. Definitely not something to buy. Check the full review on previous link if you're interested.

May 16th, 2010 : stable 1.4.6

Version 1.4.5 triggered a conflict in a macro name with some Linux distros shipping glibc >= 2.10. Since this affects most recent distros, it was better to upgrade now. A few points about RDP cookies were clarified in the doc. Last, a new ACL match srv_is_up() has been added, to consider a specific server's state in ACLs. Those who had no problem building 1.4.5 don't really need to upgrade. mixing them with fixes.

May 13th, 2010 : stable 1.4.5

No reliability issue was reported since 1.4.4 was released. This is a very good thing, because some people were asking for a few minor features, so it was a good opportunity to get them merged without mixing them with fixes. The ChangeLog for version 1.4.5 is here and the updates are here.

First, Cyril Bonté provided the new ignore-persist directive. it allows haproxy to ignore the persistence cookie on some requests which validate an ACL-based condition. It is particularly suited to optimise the load balancing of static or stateless objects in the middle of a stateful farm.

Second, it was planned 3 years ago to be able to feed ACLs with large data sets loaded from files, but it was still not implemented due to the lack of precise needs. Now, 3 years later, more and more people are reporting difficulties writing large configurations, and the last config I saw which was 104000 lines long convinced me that it was urgent to support this feature. But matching requests against very large datasets can be CPU intensive, so I have extended my Elastic Binary Trees to support new lookup methods and now it is possible to lookup a string or an IP address among tens of thousands in a few tens of nanoseconds. This means that it is now possible to use haproxy to perform geolocation. For instance, checking that a source address belongs to one of the 38400 european networks only consumes 2% CPU at 40000 requests per second.

The rest are just minor improvements. Tt's now possible to stick on an IP address extracted from an HTTP header, and I improved a bit more the halog analyser, which is now possible to report request counts by status codes. It also gained some nice performance boost as it can now parse about 1.3 Gigabytes of logs per second on a 3 GHz Core2.

I expect that this version will take some time to spread because it only contains new features and will likely not be backported to various distros. Still, some power users will probably interested in giving it a try.

April 7th, 2010 : stable 1.4.4

Some people were experiencing optimisation issues with Tomcat and Jetty, with which it was not possible to perform client side keep-alive when the server received a "Connection: close" header. This is due to a strange design choice by which they decide the client is not interested in the response length if it intends to close after the transfer! Well technically that works... most often... Sometimes users may get truncated objects without being aware of that. Anyway, Cyril Bonté had a very smart idea for a workaround : pretend to the server we'll maintain its session alive while it's false. This fixed the problem, and is now available by adding option http-pretend-keepalive to option http-server-close.

Jetty's HTTP implementation seems to be the flakiest though. It even manages to send "HTTP/1.1 100 Continue" intermediate responses when the client sends "Expect: 100-continue", but it closes the connection just after that message, resulting in a 502 error for the client.

Cyril also fixed an issue with appsession where a cookie whose name begins like the appsession cookie could be mistaken for it. Those issues were enough to justify a new release.

Very few other minor fixes were brought there, and a minor feature was added. It consists in being able to bind to a source address found in a header when connecting to a server. Normally this will be the X-Forwarded-For header. This requires use of the Linux kernel TPROXY patches, and makes it possible for backend servers to see the initial client's IP even when several layers of proxies have been passed through.

March 30th, 2010 : stable 1.4.3 and 1.3.24

A few remaining issues and one regression were fixed in 1.4. In 1.3, I have backported all pending fixes since 1.3.23, most of which were discovered in 1.4. The most interested people will probably be users of FreeBSD where they could randomly get a SIGPIPE if the program was compiled on a very recent version (8, maybe 7.2 too). Anyway, due to other minor to medium fixes, 1.3 users should upgrade.
It's worth noting that very few issues were reported on 1.4.2. The code has stabilized very quickly, and people in sensible environments will be able to start to think about evaluating it to plan an upgrade (from most reports, the new features such as client keep-alive and improved stats are very much demanded). If I could send them an advice, I'd say that we're going to release 1.4.4 soon with a few minor improvements and that if some people don't know what version they will start with, then let's start with 1.4.4 when it's available.

March 17th, 2010 : stable 1.4.2

A new batch of annoying issues got fixed. Among these issues, we can find a risk of crash (broken pipe) on very recent versions of FreeBSD, a segfault when using CLF logs and capturing a non existant header, a very rare case of stuck client session when using keep-alive, some garbage which appeared on the stats page after 1000 client resets, or when the host name was too long, a url_param hash bug which could result in a dead server to be used in very rare situations, a risk of getting an empty result on the stats socket if the input closes before haproxy responds, and endless loop at config parsing time on the error-limit keyword, status codes 501 and 505 which could cause a server to be marked down if on-error was used, and a risk of getting truncated HTTP responses when chunk-encoding was used with chunk sizes that are exact divisors of the buffer size, and an issue with anonymous ACLs which were merged together into a single one. A few improvements were made to the health checks which now support multi-packet responses, and the stats socket can now display more debug information about a specific connection. The list is quite huge, it was important to release 1.4.2 without waiting for any possible new issue to come, eventhough things seem to calm down.

March 5th, 2010 : stable 1.4.1

Some build issues on non-Linux platforms were preventing new 1.4 adopters from trying it. These issues are now fixed. Other issues concerned the appearance of more 502 errors in the logs than with 1.3. This was a bug that caused the status code to be changed to 502 even in case of connection abort during the data transfer. A few new error counters were added to the stats, and other minor issues were fixed. This new version now builds and works on FreeBSD, OpenBSD, OSX, Solaris, AIX and Linux, so let's not wait and release 1.4.1.

Also, Solaris users will now be happy, I unpacked and replugged my Ultra5 so the Sparc binary is available again.

On a side note, I have removed the link to the haproxy.org mirror because it has been outdated for the last 6 months and even remained 1 week on an expired DNS zone. I failed several times to contact Kevin Kuang there, so I don't even know who manages it now if any. If someone gets in touch with him, please ask him to contact me.

February 26th, 2010 : New stable branch 1.4 !

After 11 months of active development and a lot of external contributions, version 1.4 is now released. It has been tested for the last 3 weeks by many people, and only very minor bugs were reported (and fixed), so it's now time to officially stamp it as stable. Version 1.4 brings a lot of new features, among which the long-awaited support for client-side HTTP keep-alive, the RDP protocol, and stickiness on anything, as well as many other nice usability improvements on the stats interface, checks and the CLI. It is also much more powerful than version 1.3 and will support addition of new protocols faster than before due to a better structured internal architecture. 1.3 will still be supported for some time, and the old 1.3.15.X branch is now entering a deep freeze where only critical bugs will be fixed. Please consult the ChangeLog for more information about all the changes. I particularly want to thank all the persons and companies who contributed to this version by code, testing or development funding ; without their efforts and participation, we would still be far away from a release !

January 28th, 2010 : stable 1.3.23

Several minor bugs were fixed since 1.3.22, and the request-learn, force-persist and http-check send-state were backported because people need them for more transparent and reliable application updates. Since no new bug was in sight, and 3.5 months had elapsed since 1.3.22, it was the moment to release 1.3.23 with all that. As 1.4 approaches, 1.3.23 will probably be the last 1.3 which accepts new minor features. Future 1.3 versions will very likely be dedicated to bug fixes only.

January 25th, 2010 : 1.4-dev7 & 1.4-dev8

While trying to work on end-to-end keep-alive, I encountered issues that needed to be fixed, so this has delayed dev7 quite a lot, and it does still not have this end-to-end keepalive. Think of it as a much cleaner dev6 instead since many bugs were fixed. The stickiness code sponsored by Exceliance and Loadbalancer.org got merged. Currently, it can almost only learn IP addresses, but it has been designed with an amazing flexibility so that it will be very easy to add stickiness on any request or response criteria. MySQL checks have been introduced and this code will evolve for slightly deeper and more reliable checks. A new "force-persist" statement allows admins to test their servers without opening them to the world, which is very convenient to ensure they're correctly installed and that their customers will not face a lot of crap. And as always, a bunch of bugs in many areas were fixed. Update: 1.4-dev5 to 1.4-dev7 had a nasty bug with keep-alive enabled, so please update to 1.4-dev8.

January 16th, 2010 : 4-hour network outage

Some of you have noticed that the site was down from 11:45 to 15:45 local time, and it can be seen here. It was the longest outage I ever experienced in 8 years with my ISP (Nerim). The support told me the outage was at their ADSL provider, SFR. Well, 4 hours in 8 years is still 99.995% availability, I have nothing to complain about :-)

January 8th, 2010 : 1.4-dev6

As could be expected, 1.4-dev5 did not work very well. The rule is pretty clear : if you don't like your code, it will fail. Just reread the last post and you'll see that it was destined to fail. With the nice help of Cyril Bonté and Hank A. Paulson, we could spot a lot of bugs and I finally got rid of those parts I found ugly. Now curiously, it works a lot better :-) Also, Krzysztof Oledzki contributed a nice feature he talked about some time ago : the default-server setting. This makes it possible to specify some common settings globally and not have to repeat them for all servers. This is useful for check intervals, maxconn, etc.. So it was time to release 1.4-dev6 so that all those who had a bad experience with 1.4-dev5 can try again. This is the version currently running on the site, so it looks fine :-)

January 2nd, 2010 : New year, new features

After several weeks of work, I have committed the patch which introduces client-side HTTP keep-alive and pipelining support. The code is quite ugly and I'm not proud of it. This is because I got quite a bunch of last-minute surprises that I will have to workaround in a cleaner way. But since the code worked, I would have found it wasted to make you wait for it.
In order to enable pipelining on the client side, just comment out any "option httpclose" statement in the defaults, frontend and backend sections and set "option http-server-close" in any of them. As the name implies it, the connection is still closed on the server side. This way we can still have low ressource usage on servers and correctly enforce maxconn while retaining keep-alive with the client.
This code will be in 1.4-dev5 by the end of the week-end, but the impatient will be able to download a snapshot for their tests.
The new code has been put in test on the Formilux server and already shows decent load time savings on this page. Stay tuned...

October 18th, 2009

I have put online a matrix of all known bugs which affect stable versions 1.3 starting with 1.3.14. It took about 4 hours of work to put that in shape but I think it was worth it. Let's put it short : those of you running 1.3.15.2, 1.3.16 or 1.3.17 are doomed. Those running 1.3.15.X before 1.3.15.7, 1.3.19 or 1.3.21 are at risk. 1.3.14.14, 1.3.15.10 and 1.3.20 are pretty good, and 1.3.22 is the only one with no known bug yet.

October 14th, 2009 : 1.3.22

A few hours after 1.3.21 was issued, John Lauro reported an important regression causing a crash when connecting to the stats socket. This was caused by a minor backport which should have been modified for 1.3 and that I didn't detect during the tests because I did not use this socket. 1.3.22 was released to fix this issue. Please don't use 1.3.21 and update to 1.3.22!

October 12th, 2009 : 1.4-dev4, 1.3.21

A lot of changes have occured in only 3 weeks, so it was the right moment to release a new development version. It's worth noting that Krzysztof Oledzki has been very active, contributing no less than 1/3 of all the changes. This is nice because being two to work on the project, we progress faster. Concerning the changes, the stats interface (socket and page) certainly is the most affected area. It is now possible to reset counters and to change a server's weight without restarting... two features that many of you have been asking for years! The stats page now can also display a node name and description, as well as the exact status of a health check. The LB algorithms have now been moved to separate files, and a consistent hashing algorithm has been added. It allows hot addition or removal of servers without disturbing the load-balancing, which is desirable for caches. Also, the LB rework brought the opportunity to re-enable the old static round-robin algorithm, which can make sense for people who run more than 4000 servers in a single backend (practical limit of the dynamic RR algorithm). Last, some new ACLs have been added, to check for IP addresses in headers, and to check frontend's and backend's connections, queues and per-server average queue size. A few minor bugs were fixed, and those fixes as well as some minor riskless features have been backported into 1.3 to release 1.3.21.

September 26th, 2009

I found it was important to acknowledge some people and companies' efforts for the project. So I added a new page listing significant contributions, most often features but sometimes fixes, in the form of patches, code, time or even money. A minor bug on the stats page which remained in 1.4-dev3 has also been fixed and is available in the latest snapshot.

September 24th, 2009 : 1.4-dev3 + sponsors

Most of the internal changes planned for version 1.4 have been completed, so it was time to release a new clean snapshot. The architecture is now ready to permit keep-alive, SSL or FastCGI developments. Some more changes are planned but the remaining ones should be a lot easier to perform without breaking everything.

Compared to latest stable 1.3.20 version, 1.4-dev3 provides new features, among which support for the CLF log format, RDP protocol load-balancing and persistence, a new interactive CLI, an improved HTML stats page, support for inspecting HTTP contents in TCP frontends and switching to HTTP backends (allowing HTTP+SSL to coexist on the same port), support for forcing of the TCP MSS on frontends, smart network optimizations to reduce the number of TCP packets in a session, runtime-configurable buffer sizes, support for more than 64k concurrent connections, config parser support for "no option xxx" to disable options that were enabled by default, and correct 1xx status code processing. Developments to support keep-alive have already started, and if time permits, SSL integration will be attempted. The code looks amazingly stable for the amount of changes, and will probably not change much anymore, so any bug found in this version must be reported and fixed. Also, new feature submissions should be based on this version. It will be easier to implement for submitters and for me to merge.

Several large sites are already running on 1.4-dev2 with great success. This one should be even better, but given the number of changes, it should be monitored more closely at the beginning.

Last, I have a very good news that I hope will give ideas to others : Exceliance and Loadbalancer.org have both agreed to contribute some manpower and money to implement the complete persistence framework that everyone is dreaming about into haproxy. That's a tough work and I'm not certain it will be ready for 1.4 (though it might, depending if I'm as late on my releases as usual). I would personally like to thank them both for their contributions. When you have to put your money in commercial solutions, please never forget to consult first the guys who involve time and money in opensource projects, because they are the ones who help the projects evolve and live !

August 9th, 2009 : 1.3.20

Cristian Ditoiu from transfer.ro reported a major regression when testing 1.3.19. It would crash within a few minutes while 1.3.15.10 was OK. He offered to help so we could run gdb and debug the crash live. We finally found that the crash was the result of a regression introduced by a recent in 1.3.19. I really want to thank him because he spontaneously provided a lot of help and trust to debug this issue which at first glance looked impossible after reading the code and traces, but took less than an hour to spot and fix when caught live in gdb ! It's always pleasant when users show that level of involvement to chase bugs.

Another bug was reported by Romuald du Song, who found that option tcplog would log using global parameters if no logger was defined. It can be either helpful or annoying. This is now fixed and a warning is emitted when such a configuration is encountered, so that people running off erroneous configs can easily fix them.

This time I expect 1.3.20 to be the good one. It's always a good sign when we fix minor bugs or recent regressions introduced by bug fixes. 1.4-dev2 has also been released to help people track changes in the two versions in parallel.

July 27th, 2009 : 1.3.19

Since 1.3.18 was released two months ago, it has been widely deployed, in part thanks to the slowloris tool which has caused HAProxy downloads to jump by 20-30%. This results in more exposure and new kinds of bugs to be discovered. The most annoying ones concern too short sessions which may sometimes be reported as server errors, random delays under low traffic conditions due to a scheduler bug, and the last one reported today by an Exceliance customer who was kind enough to provide lots of traces, some occasional pauses on interactive TCP traffic which might also happen on the last chunk of an HTTP response, although extremely rarely. Each one alone would have been enough to issue a new maintenance release, so here it is, 1.3.19. It also brings a small bunch of nice new features backported from the dev tree, among which the support for multiple configuration files, the support for more than 64k concurrent connections (tested at 190k by the heavy user), and a highly better reporting of config warnings and errors. So, as usual with maintenance releases... everyone is highly encouraged to upgrade.

Since some of the bugs above were present in earlier versions, a new release was emitted for 1.3.15 and 1.3.14 too for the late users who have not upgraded yet. I really think it's the last one for 1.3.14. 1.3.15 might still see another one till the end of the year, and that will probably be all for this one after 20 months of free support :-)

The first development version should be released too, but I need to update my release scripts first, they are inadequate and take me too much time to use, so stay tuned !

June 27th, 2009 : HAProxy to counter DoS attacks

Since the announcement of the Slowloris tool, people seem to be discovering how fragile a default Apache setup can be ! Well, this is not news, as people who install Apache on high-traffic sites have been aware of this weakness for ages, and have been setting very low timeouts and disabling keep-alive in order to mitigate risks. Now that a tool is publicly advertised, I'm beginning to hear questions from worried site admins about what to do if their site is attacked. Also, we're seeing several sites and forums suggesting installing HAProxy in front of Apache servers to protect them (note that Nginx would probably do equally well).

Indeed, HAProxy does not need a new thread nor process to accept a new connection, it only needs some RAM (16-32 kB per connection). Some people are already using it past 70000 concurrent connections, which cannot be achieved on Apache which needs an expensive thread or process per connection. More specifically, HAProxy will only forward complete and valid requests. This means that it will not bother Apache while the attacker is playing with its few thousands connections, and all valid requests will immediately pass through. And the icing on the cake is that HAProxy can kill requests which take too much time to complete, using timeout http-request (more than a few seconds is not to be considered normal).

Once again, we observe a derivate use of a load-balancer, which is a bit expected : when a tool is designed to accept 10 times more load than the servers it feeds, there is nothing surprizing that it can be used to protect them ! Let's see if Apache evolves towards providing more tunables to mitigate such attacks... In the mean time, a drop-in anti-DoS configuration is available here.

May 10th, 2009 : 1.3.18

Yan Qiao of Rocket Fuel Inc reported a crash on x86_64, which was pretty much unexpected ! He nicely offered to help troubleshooting by rebuilding with debugging on and leaving the process running in production to catch the error, then sent me an interesting core 1 week later, which revealed that a field in the struct session which was never touched had been changed due to the sharing of two pools of the same size. This field should have been initialized but was unfortunately not. The issue can only happen on x86_64 with HTTP logging enabled, due to the exact 1024 bytes of the struct session which allows its pool to be shared with the struct requri's. Thank you guys for your huge help and the risks you have taken leaving that process running!

During a troubleshooting session with the T20 guys (Maxim Fedchishin, Jason Coward and Viktor Brilon from modX team, Hans from RightScale team), I came across an old leftover process doing nothing after a soft-reload. That issue is brought once in a while by various people, but it happens too rarely for anyone to get an opportunity to debug it. The guys accepted that I installed a debugger on their machine to see what the process was doing. It was deadlocked in free() during the reload. And that made sense : during a reload, the old process releases as much memory as possible to leave room for the new one. If the two signals sent by the second one are too close to each other, the second signal is sent while the first one has not completed releasing memory and we can have a recursion in the libc's free(), causing a deadlock. That has been fixed by implementing asynchronous signal delivery. Thank you guys for giving me the opportunity to catch that rare event!

Problems aside, a few minor features were merged. The stats are now more readable, report max session rates and provide full 64-bit counters everywhere. It is now possible to forward invalid requests or responses without blocking them, but they will still be captured. The config parser now warns about possibly unwanted ordering of ACLs or reqxxx/rspxxx. Several wrong printf() format strings have been fixed. The build process now supports an alternative architecture, and the RPM spec file has been cleaned. A new balance hdr(header) algorithm has been added to balance depending on a header hash. A new option enables addition of the destination IP address in the X-Original-To header. And last but not least, the doc has been massively cleaned up and reorganised.

With all these fixes, I released 1.3.18, as well as 1.3.15.9 and 1.3.14.13 which are probably among the last ones of their respective branches after 12 and 18 months of maintenance.

April 19th, 2009 : new performance record broken !

It was a long time since my last 10 Gigabit tests, exactly one year. The Linux kernel has evolved a lot, so did HAProxy and even the Myri10GE driver. I knew we could get much throughput since we fixed the kernel splice() syscall. It was a good opportunity to start a new series of benchmarks again. In short, new records were broken. Full 10Gig line rate with 20% CPU, and the 100000 sessions/s barrier was crossed !

March 29th, 2009 : 1.3.17

Bart Bobrowski of who's.amung.us reported abnormal CPU usage with the new version 1.3.16. After a full day of tests and code analysis, I failed to reproduce the issue here, and the bug appeared impossible to me. Bart then offered a lot of help with testing many patches, providing hundreds of megs of traces, so that I could finally fix the issue caused by a nasty race condition. I really appreciate it when users with extreme loads accept to take traces in production, with all the risks that this practise implies. Sometimes it's the only way to get a bug fixed.Thanks Bart!.
Since other minor fixes and enhancements were pending, I released 1.3.17, which users of 1.3.16 are invited to upgrade to.

March 22th, 2009 : 1.3.16 !

Minor fixes and enhancements have been added since the second release candidate. So, that's it, 1.3.16 is out and marks the new official stable release. As it has already received long testing from major users, I'm not worried about its stability, eventhough I expect that a few bugs will surface. Further development will continue in a new branch, and 1.3 will only receive fixes and minor enhancements.

March 10th, 2009 : 1.3.16 is getting closer !

Second release candidate of 1.3.16 has been published. It brings a lot of new long-awaited features, among which TCP splicing support, conditional redirection, TCP content filtering, session rate reporting and limiting, invalid request/response capture, binding to specific network interfaces, per-process affinity for frontends and backends, a monotonic internal clock, and many other features.
The internals have finally been reworked in layers so that forwarding can be processed without waking high levels up. HTTP is now on top of TCP and not a special case of it. A big advantage of these changes is that we can now touch the socket code without impacting HTTP and vice-versa, which had not yet been possible till there. This means that the risk of future regressions caused by feature additions will be significantly lowered. Thanks to these changes, a lot of complex tricks and specific cases are now handled more cleanly and in a more evolutive manner. New work on keep-alive, SSL integration and QoS will be easier.
Once 1.3.16 is out, branch 1.3 will become the new stable branch, and support for 1.2 as well as 1.3.14 and 1.3.15 will slowly phase out.

March 9th, 2009

Several minor bug fixes were pending since 1.3.15.7, so it was time to release 1.3.15.8 and 1.3.14.12. Those bugs are not stability bugs, rather load-time bugs (config parsing, etc...). Only one of them really justifies updating : if your configuration uses the "track" keyword in order to synchronize multiple servers states, the time taken to synchronize them grows with the number of servers. Among the changes, a backport of the doc updates was merged, covering the log format, so that the old docs should normally not be needed anymore.

December 4th, 2008

Kai Krueger reported a nasty problem he encountered and analyzed. When a server goes down, it requeues all of its connections waiting in the queue into the global queue. But when a session completes after that, haproxy checks if there are pending connections that this server can handle, without taking into account the fact that the server is dead. So the server can progressively suck all pending connections from the global queue just after it has been marked down. Yes, I know, this is stupid. A check has been added so that it does not dequeue global connections when it's marked down, and releases 1.3.15.7 and 1.3.14.11 have been issued. There are very few setups which will trigger this problem, however it's quite annoying for those experiencing it.

October 12th, 2008

Once in a while, a user reports that some old processes remain present after a soft-restart. I could never reproduce the issue until Manuel Soto sent me a truss output of a configuration with which the problem reproduces frequently. The cause is finally that haproxy still binds listening addresses to disabled instances, but does not try to stop them and refuses to exit as long as they remain present. I took the opportunity to fix a related problem causing warnings to be emitted when haproxy tried to stop backends, and a segfault in the configuration parser if ACLs were declared in a defaults section.

That was enough to release 1.3.15.5 and 1.3.14.9. I recommend that any user of 1.3.14 or 1.3.15 upgrades, as these fixes present very minor risk and fix really annoying problems.

September 14th, 2008

Several users reported on the mailing list that they were experiencing abnormal concurrent connection counts higher than the maxconn they configured. They were very prompt to send me configurations and screenshots of the stats report showing the problem. It was indeed a bug triggered every time a connection attempt to a server failed. I've fixed it along with another minor one, and released 1.3.15.4 and 1.3.14.8. Mongrel users are particularly exposed because they run with maxconn=1 and the server cannot accept more connections, so users may experience occasional errors when a server starts to reject connections. It would also be interesting to find why some connections fail to the servers.

September 3rd, 2008

A cool video demonstration of the connection regulation mechanism (maxconn) has been posted on 37signals. It's clearly explained and explicit enough for people not much aware of the mechanism to understand it. Check it there, it's not too long and really worth seeing.

September 2nd, 2008

While working on haproxy 1.3.16, I came across a few bugs in the code, so I issued 1.3.15.3 and 1.3.14.7. The only one annoying concerns 1.3.15 for people who use the "balance url_param ... check_post" construct to hash on parameters present in POST requests. There is a risk of crashing (but no server compromission though) with some invalid requests. Fortunately, this feature is very new and ver limited to niche users, but it needed a quick fix anyway. Other bugs are pretty minor and most of them concern small issues with how timeouts are handled.

July 20th, 2008 : two lines...

Two lines... That's all what is needed with the new TCP content inspection system to stop half of the spams I got home. One of my major customers who uses HAProxy a lot has sponsored the development of some preliminary content inspection which is used to decide whether to forward a connection or not. The very first usage of this feature consists in checking that only SSL is spoken on a connection. But most likely more protocols will come soon. As a nice side effect, I could now add a delay before the HELO message of my SMTP server, and reject all robots which talk first (forbidden). And since many spam bots have small timeout values, many of them abort before the timeout is reached, resulting in my incoming spam rate dropping from about 300/hour to "only" 150/hour. Those who keep up with the time out slow down due to limited resources. The small addition simply consists in adding those two lines in the frontend :

		tcp-request inspect-delay  35s
		tcp-request content reject if REQ_CONTENT
	  

June 21th, 2008

haproxy versions 1.3.15.2 and 1.3.14.6 have been released to fix a major bug in request queue handling. The problem was that due to a design problem, it was possible for new requests to be immediately served by a server before a request in queue would be served. That caused some requests to remain in queue until they reached the queue timeout, after which either they would eventually be served, or return a 503 error code to the client.

Since it was a design problem, it took a lot of time analyzing the root cause and finding a solution. However, as a positive side effect, the fix now makes the redispatch option work for requests which overflow a queue. That way, clients do not get a 503 error anymore but can be served by another server (which was the purpose of the redispatch option.

Note that it is possible that 1.2 is also affected by the issue since some parts of the faulty code have not changed since. But it is very hard to determine if it is faulty or not, and backporting the fix would take even more time. Maybe I will eventually take a look at it if people complain about the issue.

Update (2008/06/28): Alexander Staubo, who first noticed the problem, has run a new series of tests showing that the problem is definitely fixed. It also demonstrates the very nice positive effect of running with maxconn 1 with Rails servers.

May 25th, 2008

Released haproxy versions 1.3.15.1 and 1.3.14.5 with minor fixes : build fix for GCC 4.3, fix for early truncate of stats output in certain circumstances, and better handling of large amounts of highly active sockets. I indeed discovered during testing that the sepoll poller was so much efficient that when running at gigabit speed with 80000 active sockets fighting for their CPU share, almost all of them were running in speculative mode, causing starvation of the remaining ones, which in turn caused the accept() call to be very rarely called. Delays of about 40 seconds have been observed on a 3.4 GHz Pentium 4 to get the stats page under such a load. The other pollers were not better BTW. The fix consisted in ensuring that polled events are checked at much often as the speculative ones. With this fix, the stats page responds in less than one second on such a saturated machine. There is still room for improvement relying on events prioritization though. Version 1.3.15 has been promoted as the recommended one since there has been no regression report. Version 1.2.18 was also released for users of 1.2 which experienced trouble building on BSD.

April 19th, 2008

Released haproxy version 1.3.15 with many new features. The most important changes are stats updates (HTTP and UNIX), enhancements of server checks such as tracking and dynamic intervals, addition of the leastconn load-balancing algorithm, a fully transparent mode on Linux, better handling of connection failures (dead server avoidance and turn-around state), support for inter-site off-loading through redirects, updates to the build process, and large documentation updates. For more information, please check the ChangeLog. Due to the important number of changes, upgrade from earlier versions should be performed with a bit of care.

Once again, a lot of code comes from contributions. I'd like to specially thank Krzysztof Oledzki for a lot of useful contributions, including the SNMP agent, and the guys at Nokia for the good work they have done on POST parameter hashing.

March 30th, 2008

I finally assembled my new machines and installed the donated 10-Gig Myricom NICs. I ran a few benchmarks. Result: new bandwidth records set for HAPoxy: 9.897 Gbps and 35128 hits/s! It's possibly the highest bitrate achieved to date with an opensource load-balancer! BTW, even most commercial ones are commonly limited to 4 Gbps by hardware design. What's a bit frustrating for a precision-tweaker like me is that those NICs work out-of-the box on dirt cheap hardware, there's almost no joy passing beyond the first 4 Gbps :-)

March 8th, 2008

Released haproxy maintenance version 1.3.14.3 to address several minor bugs and clean up the configuration manual a little bit. One annoying issue with backup servers in round robin mode was fixed. Nothing really important was changed in this version, this makes it a good candidate for distro updates.

February 23rd, 2008

I finally decided to buy an expensive motherboard to upgrade my PC in order to begin testing with the 10Gig NICs. I selected an Asus P5E3-WS Pro because I needed PCI-X slots for my older cards. I've put a Core2 Duo E8200 (45nm) because I wanted a lot of silence. The mobo has 2 PCI-E 16x slots, which made it possible for me to run a back-to-back test between two 10Gig NICs. Since the board does not support PCI graphics cards, I had to boot on serial port (the only VGA card I've got running on this mobo was a cheap crappy GeForce 8400 GS which does not work under X). Well, my first test is quite encouraging : I can achieve 10 Gbps of HTTP traffic between the two NICs with the server and client on the same machine, which means that the hardware will be able to support haproxy under the same conditions. I tried with client + haproxy + server but the bit rate diminished to about 6-7 Gbps. I'm impatient to buy the 3 other mobos to build a full lab. I will mix 2 Athlons and one C2D so that I can experiment which one is better for which type of traffic. Stay tuned!

January 21th, 2008

Released haproxy maintenance version 1.3.14.2 to address several minor bugs as well as a major one affecting Linux 2.6 users with the sepoll poller, which can result in truncated responses if the client closes the connection before the server completes its response. Note that version 1.3.13.2 was released too with those bugs fixed. The GNU Makefile was crappy and caused trouble in some build environments. It has been rewritten in a more flexible manner, while still providing full variable compatibility with existing build systems. Distribution packagers are encouraged to migrate over to this one. The new configuration manual is almost finished. All keywords and all their options have been documented. Only the logs section remains to be completed. This version has been merged with 1.3.14.2. Some minor robustness and performance tuning parameters have been added, mostly timeouts and backlog.

January 13th, 2008

Worked all the day both in kernel and haproxy to get full transparent proxy to work on Linux. Now, with a small kernel patch, it's possible for haproxy to become completely transparent and just appear as a router, without touching either source nor destination addresses and ports. And all this without NAT, at the same performance level as in normal proxy mode. This will be great for people looking for SMTP/HTTPS/FTP relaying and load balancing. I'm even planning on installing it on my firewall ;-) Stay tuned for the updates, I will soon post the patches once cleaned up.

December 12th, 2007 : Santa Claus left a present for me at EXOSEC !

Some of you might have already got their hands on this. For those who don't know yet, this beautiful piece of art is a 10 Gbps Ethernet NIC from Myricom. For a long time, I had been tempted by their legendary high performance network cards, which were said everywhere to be able to saturate a 10 Gig wire under Linux without putting too much stress on the CPU, using a mainstream opensource driver, and without resorting to dirty tricks such as TOE. What would a performance addict like me need more ? I finally decided to mail these guys and described how I'm currently used to benchmark HAproxy with aggregated Gigabit NICs, with a minimum of 4 NICs in a setup (1 for the client, 2 for the proxy, 1 for the server). 4 hours later, when I woke up, I had a mail from Charles Seitz, Myricom's CEO. He explained to me that he was pleased to offer me 4 NICs with cables, plus one spare of each just in case, as their contribution to the project... yes, I'm talking about a donation of five 10Gig NICs! That's awsome! And if it would not be enough for some of you to find them really cool, he also provided me with french-speaking contacts, free access to their support and important advices for the choice of motherboards to get the best out of those wonderful NICs! I don't even know the polite words to say in such circumstances :-) Today I've been monitoring the shipping steps at UPS. This evening, I noticed that they arrived at EXOSEC. After leaving the customer's, I went back there to find this big parcel on my desk, with its contents very carefully packed. I must say that I was both very excited and extremely careful while opening the packaging. The first thing I noticed after extracting the first NIC from its packaging was that it had a very clean design, as can be seen on this photo. They are also very thin as shown on the picture on the right, so there will be no problem putting two of them side-by-side in the proxy. The CX4 connector looks a bit fragile, but careful manipulation is the minimal requirement to use the highest speed standard Ethernet. From what I understood, this is the same connector as used on Infiniband, except that 10GE has terminators on the board. Well, obviously, there are very nice companies out there who deserve to be talked about! Their very generous support to open source projects leaves many others far behind. People say that Santa Claus lives in the North Pole, but now I know he lives in Arcadia in California :-) Thank you very much Charles, thanks very much Myricom. Be sure to read about my first test results here.

December 6th, 2007

Released haproxy version 1.3.14. A good part of the changes comes from nice contributors of the mailing list. Most sensible changes include support for dynamic server weights offering support for slow start and graceful shutdown. The load balancer is now able to report its servers state to outer components, enabling the building of more complex multi-site architectures involving dynamic routing protocols such as BGP. People who were complaining about the rough configuration, rough statistics, or lack of logging to UNIX sockets, should really give this one a try. Rate of changes after this version should significantly drop in order to progressively switch the tree to a stable state.

October 18th, 2007

Released haproxy maintenance version 1.3.12.3 and new version 1.3.13. Thanks to quite a bunch of subscribers of the new mailing list, several nasty bugs have been ironed out and a handful of smart features have been added. The most visible add-ons are the UNIX socket to access the statistics page, and the support for Doug Lea's dlmalloc which dramatically reduces memory usage during soft-reconfigurations.

September 22th, 2007

The long awaited haproxy mailing list is finally there ! Special thanks to my friend Benoit who has spent some time setting it up on our Formilux server. For more info on how to subscribe, please check the Contacts entry at the bottom of the home page.

September 20th, 2007

Released haproxy version 1.3.12.2. It fixes several bugs affecting timeouts and retry counts when configs are split between frontends and backends. Some sanity checks on the configuration file were never executed, causing some erroneous configurations to be accepted without being fixed. Last, the license has been clarified in a few files from O'Reilly. All in all, it seems like keeping a supported version is already starting to pay off, as people are looking for something stable and report bugs very quickly. All version 1.3 users are encouraged to upgrade to 1.3.12.2.

September 5th, 2007

Released haproxy version 1.3.12.1. It fixes a few bugs discovered in 1.3.12, notably one which could lead to crashes under Linux with speculative I/O when clients disconnect before the connection has been established to the server. As a workaround, it is possible to specify "nosepoll" in the "global" section. A "stats refresh <interval>" option has also been added because some people like to have the stats page automatically refresh. It's also possible to hide all failed servers on the stats page now. This version also contains the new configuration manual which has just been started but which helps understand how to use ACL.

July 15th, 2007

Started writing the new Configuration Manual. It enumerates all configuration keywords and in what context they may be used. It also includes a few examples of ACLs. It is not finished yet, but I decided to publish it because people have really no other valuable sources of information to use content switching. It only covers version 1.3.12, and updates will only cover the latest version, making it far more readable. Please take a look at it and start from the examples in the examples/ directory from the sources. Any feedback is welcome :-)

June 17th, 2007

Released haproxy version 1.3.12. It completes integration of ACL with Content Switching, and allows you to customize your error responses. Except for the ACL and a few bugs, there have been few changes since 1.3.11.4, and I intend to support 1.3.12 during development and cleanups of the next versions which may not be as reliable. Several big content providers use 1.3 to regulate the traffic to/from their web servers, and there is a real demand for a stable version with the new features and performance of 1.3. And considering that some of them even pay for this, I understand they want something really reliable.

June 3rd, 2007

Released haproxy version 1.3.11.4. It fixes 2 long-standing bugs in timeout handling, which could sometimes cause 100% CPU usage during several seconds when a client had closed its write channel. Some small improvements to the I/O subsystem should save some CPU cycles on high bandwidth sites. It is now possible to finely tune the pollers for reduced latency.

May 14th, 2007

Released haproxy version 1.3.11.3. It fixes the (hopefully) last bug affecting Linux users with speculative I/O processing, introduced in 1.3.9. This bug was also causing random timeouts. Do not use versions 1.3.11 to 1.3.11.2 as they are all broken.

New in this release are a better timer management and a new memory manager which is able to self-manage its pools and free unused ones when memory is becoming scarce. It is also easier to code with this new one since it's not necessary anymore to declare pool sizes. Overall, yet another performance boost of 5% has been gained.

May 10th, 2007

Released haproxy version 1.3.10.2 to fix build on OpenBSD.

May 9th, 2007

Released haproxy version 1.3.10.1. It fixes a serious bug affecting Linux users with speculative I/O processing, introduced in 1.3.9. This bug was causing random timeouts on some traffic patterns, mostly noticeable in TCP mode but almost certainly in HTTP too. All Linux users of 1.3.9 and 1.3.10 should either upgrade or disable speculative I/O as a workaround, by starting haproxy with the -ds argument or by setting nosepoll in the global section.

May 9th, 2007

Released haproxy version 1.3.10. This one adds ACL, SMTP health checks (thanks to Peter van Dijk), and URI hashing (thanks to Guillaume Dallaire). Also, the rbtree was replaced with a much faster tree, leading to an overall performance boost around 5%.

The speculative I/O processing in 1.3.9 has introduced some bugs which have been fixed in this version. I feel confident that latest changes have brought their pile of bugs too. I will probably spend some time soon to do cleanup and stabilization work, eventhough both are not really compatible.

I also want to thank all the people who contribute code and testing. You are more and more at each release. I'm impatient to clean up the remains of the old code, so that even more people can contribute code. Interestingly, all contributions till now were of high quality. This is probably induced by some sort of selection caused by the technical aspect of the product, and the skills required to use the development version. Thanks again to you all !

Apr 22th, 2007

Done a quick benchmark at EXOSEC with haproxy 1.3.9 running on a nice single-core system equipped with many PCI-Express Gigabit NICs. The graph shows pretty decent results !

Apr 15th, 2007

Released haproxy version 1.3.9. This one adds modularization to the pollers, which made it possible for me to finally implement support for FreeBSD kqueue(). I'd like to thank Olivier Warin for providing me a FreeBSD account during this development.

A new concept was introduced too : speculative I/O. It is a new method consisting in reducing the number of calls to the expensive epoll_ctl() and epoll_wait() by attempting to access the file descriptors before being notified about their readiness. This provides an overall speed boost of 10%, which is quite much for just a poller.

Apr 3rd, 2007

Released haproxy version 1.3.8.2 to fix a minor and a major bug. The minor bug caused the response rewrite to fail on the status line. The major bug which was introduced in 1.3.6 could cause the process to crash in some circumstances when rewriting the request line (method and/or URI). All users of 1.3.6 and later must upgrade.

Apr 1st, 2007

Released haproxy version 1.3.8.1 to fix very minor bugs, and slightly improve performance. Request headers were not added if option httpclose was not set. Bruno Michel contributed a VIM script for syntax color highlighting.

Mar 25th, 2007

Released haproxy version 1.3.8. Several bugs which might have caused crashes on erroneous configurations have been fixed. The response processing is now completed, which means that real configurations can now be written ; HAProxy 1.3.8 now is at least equivalent to 1.2.17 in terms of features.

Just like with every release, several code optimization have led to small but noticeable performance increases, particularly on very high data bandwidth. The configuration errors are handled more gracefully now with indications about what failed and hints to resolve the issue. HAProxy now builds on MacOS 10.4 thanks to Dan Zinngrabe who provided a makefile. Also, it is now possible to send health checks to an alternate server address, thanks to a patch from Fabrice Dulaunoy.

Users of 1.3 are encouraged to upgrade to 1.3.8 as it both fixes known bugs and converges towards something less tricky than previous versions.

Mar 17th, 2007

Released haproxy version 1.2.17. I have backported Sin Yu's rbtree scheduler from version 1.3 since it proved to be stable. A few minor bugs were fixed, and two useful contributions were merged : support for user and group keywords as alternatives to numerical uid and gid from Marcus Rueckert, and the ability to prevent some source addresses from appearing in the X-Forwarded-For header, which is useful when combined with Stunnel for instance (patch from Bryan Germann). Thanks to both of them, contribs are always welcome !

The architecture manual was updated to reflect new features in branch 1.2, with examples for stunnel and for load mitigation.

Users of 1.2.16 with high loads are encouraged to upgrade to 1.2.17 as it offers them the high performance of branch 1.3 with the reliability of the stable branch 1.2.

Jan 27th, 2007

Released haproxy version 1.3.7. I found a critical bug in the new parser in development branch, causing crashes when an empty header is passed. This was caused by a missing pointer assignment in the empty header processing path. All 1.3.6 users MUST upgrade to 1.3.7.

Jan 22th, 2007

Released haproxy version 1.3.6. I spent a long time reworking the HTTP message parser. It now consists in a carefully hand-optimized 28-states FSM. The new code will look awful to goto haters, and will please FSM lovers. It's blazingly fast : parsing and indexing all of the 660 bytes of an HTTP request from Firefox on Freshmeat only takes 1.94 microsecond on my 1.7 GHz Pentium-M notebook, which means it can do it more than 500000 times a second!

The request code has been cleaned up a lot and adapted to this new FSM. Adding layer7 rules based on new criteria is now trivial. The response code will be ported next, but the code was so much cleaner and faster that it was worth releasing one version before breaking everything. Several bugs were fixed since 1.3.5. I really consider 1.3.6 as the most likely reliable 1.3 release to date.

Jan 7th, 2007

In order to support the new Linux Layer7 Switching project, I have implemented support for kernel TCP splicing using Alexandre Cassen's library. This is still experimental but already works remarkably well. On my notebook at 400 Mbps, haproxy's usage goes down from 65% to 5-10%. I have written some doc explaining how to setup up TCP splicing, with an example. Since the original code was provided for Linux kernel 2.6.19 only, I have backported the patches to kernel 2.6.16 and 2.4.33.

The second great news is that Sin Yu has provided me with a useful patch for the second time : the task scheduler is now based on an rbtree and not on the dirty old dual-linked list anymore. It means that people who had performance problems and who had to set all their timeouts to the same value as a workaround will not have to do this anymore. I have tested, and the code works like a charm ! Thanks again Sin !

Jan 2nd, 2007

After about 4500 new lines of code and some useful feedback from a bunch of brave beta-testers, I'm pleased to announce haproxy version 1.3.4 with the new Content-Switching features !!!

It is now possible to select a backend (server pool + load balancing algo) depending on any parameter in the request, such as any part of the URI, the host name, etc.... As of now, I've merged Sin Yu's patch to permit switching based on a request regex, but the framework is ready to accept many other criteria. The HTTP request parser has been completely rewritten to support unlimited header inspection, and the statistics page has been rewritten, as can be seen on the demo page. It is far from being finished right now, but it seems pretty usable. The server state machine should be adapted though.

There is still no doc, so please note that old configurations do still work, and that in order to switch from an instance to another backend, you need to use "reqisetbe <regex> <new_proxy>". Also, there's a config example here that will be worth any doc.

Dec 5th, 2006

The load balancing article has been linked to from LinuxFR. The small 128 kbps uplink is currently running at full speed but the site is still responding thanks to haproxy queuing the connections to smoothen the traffic. Next time, I should also write an article on setting up the QoS with tc, because typing remotely with SSH is still very responsive under full load :-)

Jul 4th, 2006

Opened development branch 1.3, which started with a major cleanup. Not sure yet about all features which will be merged, the first step is to clean up the code and make it modular. The API's licence has been switched to LGPL in order to later allow linking with binary external modules developped for applications covered by NDAs for example. Version 1.3.0 is exactly the same as 1.2.14+bugfixes so it is a stable starting point. It is available here.

⇐ Back to HAProxy

Contacts

• Main site : http://1wt.eu/
• e-mail :