Quick links
Quick News
Description
Design Choices
Supported Platforms
Performance
Reliability
Security
Download
Documentation
Live demo
Commercial Support
Products using HAProxy
Contributions
Other Solutions
Contacts
Mailing list archives
10-Gbps load-balancing
Contributions
Known bugs
Willy TARREAU
You want to donate ?
visitors online
|
|
March 5th, 2010 : stable 1.4.1
Some build issues on non-Linux platforms were preventing new 1.4 adopters
from trying it. These issues are now fixed.
Other issues
concerned the appearance of more 502 errors in the logs than with 1.3. This
was a bug that caused the status code to be changed to 502 even in case of
connection abort during the data transfer. A few new error counters were
added to the stats, and other minor issues were fixed. This new version now
builds and works on FreeBSD, OpenBSD, OSX, Solaris, AIX and Linux, so let's
not wait and release 1.4.1.
Also, Solaris users will now be happy, I unpacked and replugged my Ultra5 so
the Sparc binary is available again.
On a side note, I have removed the link to the haproxy.org mirror because it
has been outdated for the last 6 months and even remained 1 week on an expired
DNS zone. I failed several times to contact Kevin Kuang there, so I don't even
know who manages it now if any. If someone gets in touch with him, please ask
him to contact me.
February 26th, 2010 : New stable branch 1.4 !
After 11 months of active development and a lot of external contributions,
version 1.4 is now released.
It has been tested for the last 3 weeks by many people, and only very
minor bugs were reported (and fixed), so it's now time to officially stamp it as stable.
Version 1.4 brings a lot of new features, among which the long-awaited support for
client-side HTTP keep-alive, the RDP protocol, and stickiness on anything, as well as
many other nice usability improvements on the stats interface, checks and the CLI. It is
also much more powerful than version 1.3 and will support addition of new protocols faster
than before due to a better structured internal architecture. 1.3 will still be supported
for some time, and the old 1.3.15.X branch is now entering a deep freeze where only
critical bugs will be fixed. Please consult
the ChangeLog for more
information about all the changes. I particularly want to thank all the persons and
companies who contributed to this version by code, testing or development funding ;
without their efforts and participation, we would still be far away from a release !
January 28th, 2010 : stable 1.3.23
Several minor bugs were fixed since 1.3.22, and the request-learn,
force-persist and http-check send-state were backported because
people need them for more transparent and reliable application updates. Since no
new bug was in sight, and 3.5 months had elapsed since 1.3.22, it was the moment
to release 1.3.23 with all
that. As 1.4 approaches, 1.3.23 will probably be the last 1.3 which accepts new
minor features.
Future 1.3 versions will very likely be dedicated to bug fixes only.
January 25th, 2010 : 1.4-dev7 & 1.4-dev8
While trying to work on end-to-end keep-alive, I encountered issues that needed to
be fixed, so this has delayed dev7 quite a lot, and it does still not have this
end-to-end keepalive. Think of it as a much cleaner dev6 instead since many bugs
were fixed. The stickiness code sponsored by Exceliance
and Loadbalancer.org got merged. Currently, it can
almost only learn IP addresses, but it has been designed with an amazing flexibility so that
it will be very easy to add stickiness on any request or response criteria. MySQL checks have
been introduced and this code will evolve for slightly deeper and more reliable checks. A new
"force-persist" statement allows admins to test their servers without opening them to the world,
which is very convenient to ensure they're correctly installed and that their customers will not
face a lot of crap. And as always, a bunch of bugs in many areas were fixed.
Update: 1.4-dev5 to 1.4-dev7 had a nasty bug with keep-alive enabled, so please update to
1.4-dev8.
January 16th, 2010 : 4-hour network outage
Some of you have noticed that the site was down from 11:45 to 15:45 local time,
and it can be seen here.
It was the longest outage I ever experienced in 8 years with my ISP
(Nerim). The support told me the outage was
at their ADSL provider, SFR. Well, 4 hours in 8
years is still 99.995% availability, I have nothing to complain about :-)
January 8th, 2010 : 1.4-dev6
As could be expected, 1.4-dev5 did not work very well. The rule is pretty clear : if you don't like your code, it will
fail. Just reread the last post and you'll see that it was destined to fail. With the nice help of Cyril Bonté and Hank A. Paulson,
we could spot a lot of bugs and I finally got rid of those parts I found ugly. Now curiously, it works a lot better :-)
Also, Krzysztof Oledzki contributed a nice feature he talked about some time ago : the default-server setting.
This makes it possible to specify some common settings globally and not have to repeat them for all servers. This is
useful for check intervals, maxconn, etc..
So it was time to release 1.4-dev6 so that all those who
had a bad experience with 1.4-dev5 can try again. This is the version currently running on the site, so it looks
fine :-)
January 2nd, 2010 : New year, new features
After several weeks of work, I have
committed the patch which introduces client-side HTTP keep-alive and pipelining support. The code is quite ugly and I'm not proud
of it. This is because I got quite a bunch of last-minute surprises that I will have to workaround in a cleaner way. But since the code worked,
I would have found it wasted to make you wait for it.
In order to enable pipelining on the client side, just comment out any "option httpclose" statement
in the defaults, frontend and backend sections and set "option http-server-close"
in any of them. As the name implies it, the connection is still closed on the server side. This way we can still have
low ressource usage on servers and correctly enforce maxconn while retaining keep-alive with the client.
This code will be in 1.4-dev5 by the end of the week-end, but the impatient will be able to download a
snapshot for their tests.
The new code has been put in test on the Formilux server and already
shows decent load time savings
on this page. Stay tuned...
October 18th, 2009
I have put online a matrix of all known bugs
which affect stable versions 1.3 starting with 1.3.14. It took about 4 hours of
work to put that in shape but I think it was worth it. Let's put it short : those
of you running 1.3.15.2, 1.3.16 or 1.3.17 are doomed.
Those running 1.3.15.X before 1.3.15.7, 1.3.19 or 1.3.21 are
at risk. 1.3.14.14, 1.3.15.10 and 1.3.20 are pretty
good, and 1.3.22 is the only one with no known bug yet.
October 14th, 2009 : 1.3.22
A few hours after 1.3.21 was issued, John Lauro reported an important regression
causing a crash when connecting to the stats socket. This
was caused by a minor backport which should have been modified for 1.3 and that I
didn't detect during the tests because I did not use this socket. 1.3.22
was released to fix this issue. Please don't use 1.3.21 and
update to 1.3.22!
October 12th, 2009 : 1.4-dev4, 1.3.21
A lot of changes have occured in only 3 weeks, so it was the right moment to
release a new development version. It's worth noting that Krzysztof Oledzki has
been very active, contributing no less than 1/3 of all the changes. This is nice
because being two to work on the project, we progress faster. Concerning the changes,
the stats interface (socket and page) certainly is the most affected area. It is
now possible to reset counters and to change a server's weight without
restarting... two features that many of you have been asking for years! The stats page
now can also display a node name and description, as well as the exact status
of a health check. The LB algorithms have now been moved to separate files, and a
consistent hashing
algorithm has been added. It allows hot addition or removal of servers without
disturbing the load-balancing, which is desirable for caches. Also, the LB rework
brought the opportunity to re-enable the old static round-robin algorithm,
which can make sense for people who run more than 4000 servers in a single backend
(practical limit of the dynamic RR algorithm). Last, some new ACLs have been added,
to check for IP addresses in headers, and to check frontend's and backend's connections,
queues and per-server average queue size. A few minor bugs were fixed, and those fixes
as well as some minor riskless features have been backported into 1.3 to release 1.3.21.
September 26th, 2009
I found it was important to acknowledge some people and companies' efforts for
the project. So I added a new page listing significant
contributions, most often features but sometimes fixes, in the form of patches,
code, time or even money. A minor bug on the stats page which remained in 1.4-dev3
has also been fixed and is available in the latest snapshot.
September 24th, 2009 : 1.4-dev3 + sponsors
Most of the internal changes planned for version 1.4 have been completed, so it was time
to release a new clean snapshot. The architecture is now
ready to permit keep-alive, SSL or FastCGI developments. Some more changes are planned
but the remaining ones should be a lot easier to perform without breaking everything.
Compared to latest stable 1.3.20 version, 1.4-dev3 provides new features, among which
support for the CLF log format, RDP protocol load-balancing and persistence, a new
interactive CLI, an improved HTML stats page, support for inspecting HTTP contents in
TCP frontends and switching to HTTP backends (allowing HTTP+SSL to coexist on the same
port), support for forcing of the TCP MSS on frontends, smart network optimizations to
reduce the number of TCP packets in a session, runtime-configurable buffer sizes, support
for more than 64k concurrent connections, config parser support for "no option xxx"
to disable options that were enabled by default, and correct 1xx status code
processing. Developments to support keep-alive have already started, and if time
permits, SSL integration will be attempted. The code looks amazingly stable for the
amount of changes, and will probably not change much anymore, so any bug found in this
version must be reported and fixed. Also, new feature submissions should be based on
this version. It will be easier to implement for submitters and for me to merge.
Several large sites are already running on 1.4-dev2 with great success. This one should
be even better, but given the number of changes, it should be monitored more closely at
the beginning.
Last, I have a very good news that I hope will give ideas to others :
Exceliance and
Loadbalancer.org have both agreed to
contribute some manpower and money to implement the complete persistence framework
that everyone is dreaming about into haproxy. That's a tough work and I'm not certain
it will be ready for 1.4 (though it might, depending if I'm as late on my releases as
usual). I would personally like to thank them both for their contributions. When you
have to put your money in commercial solutions, please never forget to consult first
the guys who involve time and money in opensource projects, because they are the ones
who help the projects evolve and live !
August 9th, 2009 : 1.3.20
Cristian Ditoiu from transfer.ro reported a
major regression when testing 1.3.19. It would crash within a few minutes while
1.3.15.10 was OK. He offered to help so we could run gdb and debug the crash live.
We finally found that the crash was the result of a regression introduced by a recent
in 1.3.19. I really want to thank him because he spontaneously provided a lot of help
and trust to debug this issue which at first glance looked impossible after reading
the code and traces, but took less than an hour to spot and fix when caught live in
gdb ! It's always pleasant when users show that level of involvement to chase bugs.
Another bug was reported by Romuald du Song, who found that option tcplog
would log using global parameters if no logger was defined. It can be either helpful
or annoying. This is now fixed and a warning is emitted when such a configuration is
encountered, so that people running off erroneous configs can easily fix them.
This time I expect 1.3.20 to be the good one.
It's always a good sign when we fix minor bugs or recent regressions introduced by bug
fixes. 1.4-dev2 has also been released to help
people track changes in the two versions in parallel.
July 27th, 2009 : 1.3.19
Since 1.3.18 was released two months ago, it has been widely deployed, in part thanks
to the slowloris tool which has caused HAProxy downloads to jump by 20-30%. This
results in more exposure and new kinds of bugs
to be discovered. The most annoying ones concern too short sessions which may sometimes
be reported as server errors, random delays under low traffic conditions due to a
scheduler bug, and the last one reported today by an
Exceliance customer who was kind
enough to provide lots of traces, some occasional pauses on interactive TCP traffic
which might also happen on the last chunk of an HTTP response, although extremely
rarely. Each one alone would have been enough to issue a new maintenance release, so
here it is, 1.3.19. It also brings a small bunch
of nice new features backported from the dev tree, among which the support for multiple
configuration files, the support for more than 64k concurrent connections (tested at
190k by the heavy user), and a highly better reporting of config warnings and
errors. So, as usual with maintenance releases... everyone is highly encouraged to
upgrade.
Since some of the bugs above were present in earlier versions, a new release was
emitted for 1.3.15 and 1.3.14 too for the late users who have not upgraded yet.
I really think it's the last one for 1.3.14. 1.3.15 might still see another one
till the end of the year, and that will probably be all for this one after 20
months of free support :-)
The first development version should be released too, but I need to update my
release scripts first, they are inadequate and take me too much time to use,
so stay tuned !
June 27th, 2009 : HAProxy to counter DoS attacks
Since the announcement of the Slowloris
tool, people seem to be discovering how fragile a default Apache setup can be ! Well,
this is not news, as people who install Apache on high-traffic sites have been aware
of this weakness for ages, and have been setting very low timeouts and disabling
keep-alive in order to mitigate risks. Now that a tool is publicly advertised, I'm
beginning to hear questions from worried site admins about what to do if their site
is attacked. Also, we're seeing several sites and forums suggesting installing HAProxy
in front of Apache servers to protect them (note that Nginx would probably do equally
well).
Indeed, HAProxy does not need a new thread nor process to accept a new connection,
it only needs some RAM (16-32 kB per connection). Some people are already using it
past 70000 concurrent connections, which cannot be achieved on Apache which needs
an expensive thread or process per connection. More specifically, HAProxy will only
forward complete and valid requests. This means that it will
not bother Apache while the attacker is playing with its few thousands connections,
and all valid requests will immediately pass through. And the icing on the cake
is that HAProxy can kill requests which take too much time to complete, using
timeout http-request (more than a few seconds is not to be considered
normal).
Once again, we observe a derivate use of a load-balancer, which is a bit expected :
when a tool is designed to accept 10 times more load than the servers it feeds, there
is nothing surprizing that it can be used to protect them ! Let's see if Apache evolves
towards providing more tunables to mitigate such attacks... In the mean time, a drop-in
anti-DoS configuration is available here.
May 10th, 2009 : 1.3.18
Yan Qiao of Rocket Fuel Inc reported
a crash on x86_64, which was pretty much unexpected ! He nicely offered to help
troubleshooting by rebuilding with debugging on and leaving the process running
in production to catch the error, then sent me an interesting core 1 week later,
which revealed that a field in the struct session which was never touched
had been changed due to the sharing of two pools of the same size. This field
should have been initialized but was unfortunately not. The issue can only happen
on x86_64 with HTTP logging enabled, due to the exact 1024 bytes of the struct
session which allows its pool to be shared with the struct requri's.
Thank you guys for your huge help and the risks you have taken leaving that process
running!
During a troubleshooting session with the T20 guys
(Maxim Fedchishin, Jason Coward and Viktor Brilon from modX team, Hans from RightScale team),
I came across an old leftover process doing nothing after a soft-reload. That issue
is brought once in a while by various people, but it happens too rarely for anyone
to get an opportunity to debug it. The guys accepted that I installed a debugger on
their machine to see what the process was doing. It was deadlocked in free()
during the reload. And that made sense : during a reload, the old process releases as
much memory as possible to leave room for the new one. If the two signals sent by the
second one are too close to each other, the second signal is sent while the first
one has not completed releasing memory and we can have a recursion in the libc's
free(), causing a deadlock. That has been fixed by implementing asynchronous
signal delivery. Thank you guys for giving me the opportunity to catch that rare event!
Problems aside, a few minor features were merged. The stats are now more readable,
report max session rates and provide full 64-bit counters everywhere. It is now
possible to forward invalid requests or responses without blocking them, but they
will still be captured. The config parser now warns about possibly unwanted ordering
of ACLs or reqxxx/rspxxx. Several wrong printf() format strings have been fixed. The
build process now supports an alternative architecture, and the RPM spec file has
been cleaned. A new balance hdr(header) algorithm has been added to balance
depending on a header hash. A new option enables addition of the destination IP
address in the X-Original-To header. And last but not least, the doc has been
massively cleaned up and reorganised.
With all these fixes,
I released 1.3.18, as well as 1.3.15.9 and
1.3.14.13 which are probably among the last ones of their respective branches after
12 and 18 months of maintenance.
April 19th, 2009 : new performance record broken !
It was a long time since my last 10 Gigabit tests, exactly one year. The Linux kernel
has evolved a lot, so did HAProxy and even the Myri10GE driver. I knew we could get much
throughput since we fixed the kernel splice() syscall. It was a good opportunity to
start a new series of benchmarks again. In short, new records
were broken. Full 10Gig line rate with 20% CPU, and the 100000
sessions/s barrier was crossed !
March 29th, 2009 : 1.3.17
Bart Bobrowski of who's.amung.us reported
abnormal CPU usage with the new version 1.3.16. After a full day of tests and code
analysis, I failed to reproduce the issue here, and the bug appeared impossible
to me. Bart then offered a lot of help with testing many patches, providing
hundreds of megs of traces, so that I could finally fix the issue caused by a
nasty race condition. I really appreciate it when users with extreme loads
accept to take traces in production, with all the risks that this practise
implies. Sometimes it's the only way to get a bug fixed.Thanks Bart!.
Since other minor fixes and enhancements
were pending, I released 1.3.17, which
users of 1.3.16 are invited to upgrade to.
March 22th, 2009 : 1.3.16 !
Minor fixes and enhancements have been added since the second release candidate.
So, that's it, 1.3.16 is out and marks the
new official stable release. As it has already received long testing from major
users, I'm not worried about its stability, eventhough I expect that a few bugs
will surface. Further development will continue in a new branch, and 1.3 will
only receive fixes and minor enhancements.
March 10th, 2009 : 1.3.16 is getting closer !
Second release candidate of 1.3.16 has been
published. It brings a lot of new long-awaited features, among which TCP splicing
support, conditional redirection, TCP content filtering, session rate reporting and
limiting, invalid request/response capture, binding to specific network interfaces,
per-process affinity for frontends and backends, a monotonic internal clock, and many
other features.
The internals have finally been reworked in layers so that forwarding can be processed
without waking high levels up. HTTP is now on top of TCP and not a special case of it.
A big advantage of these changes is that we can now touch the socket code without
impacting HTTP and vice-versa, which had not yet been possible till there. This means
that the risk of future regressions caused by feature additions will be significantly
lowered. Thanks to these changes, a lot of complex tricks and specific cases are now
handled more cleanly and in a more evolutive manner. New work on keep-alive, SSL
integration and QoS will be easier.
Once 1.3.16 is out, branch 1.3 will become the new stable branch, and support for 1.2
as well as 1.3.14 and 1.3.15 will slowly phase out.
March 9th, 2009
Several minor bug fixes were pending since 1.3.15.7, so it was time to release
1.3.15.8 and 1.3.14.12. Those bugs are not
stability bugs, rather load-time bugs (config parsing, etc...). Only one of them really
justifies updating : if your configuration uses the "track" keyword in
order to synchronize multiple servers states, the time taken to synchronize them grows
with the number of servers. Among the changes, a backport of the doc updates was merged,
covering the log format, so that the old docs should normally not be needed anymore.
December 4th, 2008
Kai Krueger reported a nasty problem he encountered and analyzed. When a server goes down,
it requeues all of its connections waiting in the queue into the global queue. But when a
session completes after that, haproxy checks if there are pending connections that this
server can handle, without taking into account the fact that the server is dead. So the
server can progressively suck all pending connections from the global queue just after it
has been marked down. Yes, I know, this is stupid. A check has been added so that it does
not dequeue global connections when it's marked down, and releases
1.3.15.7 and 1.3.14.11 have been issued. There are
very few setups which will trigger this problem, however it's quite annoying for those experiencing it.
October 12th, 2008
Once in a while, a user reports that some old processes remain present after a soft-restart. I could never
reproduce the issue until Manuel Soto sent me a truss output of a configuration with which the problem
reproduces frequently. The cause is finally that haproxy still binds listening addresses to disabled instances,
but does not try to stop them and refuses to exit as long as they remain present. I took the opportunity to fix
a related problem causing warnings to be emitted when haproxy tried to stop backends, and a segfault in the
configuration parser if ACLs were declared in a defaults section.
That was enough to release 1.3.15.5 and 1.3.14.9. I recommend that any
user of 1.3.14 or 1.3.15 upgrades, as these fixes present very minor risk and fix really annoying problems.
September 14th, 2008
Several users reported on the mailing list
that they were experiencing abnormal concurrent connection counts higher than the maxconn they configured. They
were very prompt to send me configurations and screenshots of the stats report
showing the problem. It was indeed a bug triggered every time a connection attempt to a server failed. I've
fixed it along with another minor one, and released 1.3.15.4 and 1.3.14.8.
Mongrel users are particularly exposed because they run with maxconn=1 and the server cannot accept more
connections, so users may experience occasional errors when a server starts to reject connections. It would also
be interesting to find why some connections fail to the servers.
September 3rd, 2008
A cool video demonstration of the connection regulation mechanism (maxconn)
has been posted on 37signals.
It's clearly explained and explicit enough for people not much aware of the mechanism to understand it.
Check it there, it's not too long and really worth seeing.
September 2nd, 2008
While working on haproxy 1.3.16, I came across a few bugs in the code, so I issued
1.3.15.3 and 1.3.14.7. The only one annoying concerns
1.3.15 for people who use the "balance url_param ... check_post" construct to hash
on parameters present in POST requests. There is a risk of crashing (but no server compromission
though) with some invalid requests. Fortunately, this feature is very new and ver limited to
niche users, but it needed a quick fix anyway. Other bugs are pretty minor and most of them
concern small issues with how timeouts are handled.
July 20th, 2008 : two lines...
Two lines... That's all what is needed with the new TCP content inspection
system to stop half of the spams I got home. One of my major customers who
uses HAProxy a lot has sponsored the development of some preliminary content inspection which is
used to decide whether to forward a connection or not. The very first usage of this feature consists
in checking that only SSL is spoken on a connection. But most likely more protocols will come soon.
As a nice side effect, I could now add a delay before the HELO message of my SMTP server, and reject
all robots which talk first (forbidden). And since many spam bots have small timeout values, many of
them abort before the timeout is reached, resulting in my incoming spam rate dropping from about 300/hour
to "only" 150/hour. Those who keep up with the time out slow down due to limited resources. The small
addition simply consists in adding those two lines in the frontend :
tcp-request inspect-delay 35s
tcp-request content reject if REQ_CONTENT
June 21th, 2008
haproxy versions 1.3.15.2 and 1.3.14.6 have been released to
fix a major bug in request queue handling. The problem was that due to a design problem, it was
possible for new requests to be immediately served by a server before a request in queue would be
served. That caused some requests to remain in queue until they reached the queue timeout, after
which either they would eventually be served, or return a 503 error code to the client.
Since it was a design problem, it took a lot of time analyzing the root cause and finding a solution.
However, as a positive side effect, the fix now makes the redispatch option work for requests
which overflow a queue. That way, clients do not get a 503 error anymore but can be served by another
server (which was the purpose of the redispatch option.
Note that it is possible that 1.2 is also affected by the issue since some parts of the faulty code
have not changed since. But it is very hard to determine if it is faulty or not, and backporting the
fix would take even more time. Maybe I will eventually take a look at it if people complain about the
issue.
Update (2008/06/28): Alexander Staubo, who first
noticed the problem, has run
a
new series of tests showing that the problem is definitely fixed. It also demonstrates the very nice
positive effect of running with maxconn 1 with Rails servers.
May 25th, 2008
Released haproxy versions 1.3.15.1 and 1.3.14.5 with
minor fixes : build fix for GCC 4.3, fix for early truncate of stats output in certain circumstances,
and better handling of large amounts of highly active sockets. I indeed discovered during testing that
the sepoll poller was so much efficient that when running at gigabit speed with 80000
active sockets fighting for their CPU share, almost all of them were running in speculative mode, causing
starvation of the remaining ones, which in turn caused the accept() call to be very rarely called.
Delays of about 40 seconds have been observed on a 3.4 GHz Pentium 4 to get the stats page under such a
load. The other pollers were not better BTW. The fix consisted in ensuring that polled events are checked
at much often as the speculative ones. With this fix, the stats page responds in less than one second on
such a saturated machine. There is still room for improvement relying on events prioritization though.
Version 1.3.15 has been promoted as the recommended one since there has been no regression report.
Version 1.2.18 was also released for users of 1.2 which experienced trouble building on BSD.
April 19th, 2008
Released haproxy version 1.3.15 with many new features.
The most important changes are stats updates (HTTP and UNIX),
enhancements of server checks such as tracking
and dynamic intervals, addition of the leastconn
load-balancing algorithm, a fully transparent mode on Linux, better handling
of connection failures (dead server avoidance and turn-around state), support
for inter-site off-loading through redirects, updates to the build process,
and large documentation updates. For more information, please check the
ChangeLog. Due to the important number of changes, upgrade
from earlier versions should be performed with a bit of care.
Once again, a lot of code comes from contributions.
I'd like to specially thank Krzysztof Oledzki for a lot of useful contributions, including
the SNMP agent, and the guys at Nokia for the good work they have done on POST
parameter hashing.
March 30th, 2008
|
I finally assembled my new machines and installed the donated 10-Gig Myricom NICs.
I ran a few benchmarks. Result: new bandwidth records set for HAPoxy: 9.897 Gbps
and 35128 hits/s! It's possibly the highest bitrate achieved to date with an
opensource load-balancer! BTW, even most commercial ones are commonly limited to
4 Gbps by hardware design. What's a bit frustrating for a precision-tweaker like
me is that those NICs work out-of-the box on dirt cheap hardware, there's almost
no joy passing beyond the first 4 Gbps :-)
|

|
March 8th, 2008
Released haproxy maintenance version 1.3.14.3 to address several
minor bugs and clean up the configuration manual a little bit. One annoying issue with backup servers
in round robin mode was fixed. Nothing really important was changed in this version, this makes it a
good candidate for distro updates.
February 23rd, 2008
I finally decided to buy an expensive motherboard to upgrade my PC in order to begin testing with the
10Gig NICs. I selected an Asus P5E3-WS Pro because I needed PCI-X slots for my older cards. I've put
a Core2 Duo E8200 (45nm) because I wanted a lot of silence. The mobo has 2 PCI-E 16x slots, which made
it possible for me to run a back-to-back test between two 10Gig NICs. Since the board does not support
PCI graphics cards, I had to boot on serial port (the only VGA card I've got running on this mobo was
a cheap crappy GeForce 8400 GS which does not work under X). Well, my first test is quite encouraging :
I can achieve 10 Gbps of HTTP traffic between the two NICs with the server and client on the same machine,
which means that the hardware will be able to support haproxy under the same conditions. I tried with
client + haproxy + server but the bit rate diminished to about 6-7 Gbps. I'm impatient to buy the 3
other mobos to build a full lab. I will mix 2 Athlons and one C2D so that I can experiment which one
is better for which type of traffic. Stay tuned!
January 21th, 2008
Released haproxy maintenance version 1.3.14.2 to address several
minor bugs as well as a major one affecting Linux 2.6 users with the sepoll poller, which can
result in truncated responses if the client closes the connection before the server
completes its response. Note that version 1.3.13.2 was released too with those bugs fixed.
The GNU Makefile was crappy and caused trouble in some build environments. It has
been rewritten in a more flexible manner, while still providing
full variable compatibility with existing
build systems. Distribution packagers are encouraged to migrate over to this one.
The new configuration manual is almost finished.
All keywords and all their options have been documented. Only the logs section
remains to be completed. This version has been merged with 1.3.14.2. Some minor
robustness and performance tuning parameters have been added, mostly timeouts and backlog.
January 13th, 2008
Worked all the day both in kernel and haproxy to get full transparent proxy to
work on Linux. Now, with a small kernel patch, it's possible for haproxy to become
completely transparent and just appear as a router, without touching either source nor
destination addresses and ports. And all this without NAT, at the same performance level
as in normal proxy mode. This will be great for people looking for SMTP/HTTPS/FTP
relaying and load balancing. I'm even planning on installing it on my firewall ;-) Stay tuned for the
updates, I will soon post the patches once cleaned up.
December 12th, 2007 : Santa Claus left a present for me at EXOSEC !
|
Some of you might have already got their hands on this. For those who don't know yet, this
beautiful piece of art is a 10 Gbps Ethernet NIC from
Myricom. For a long time, I had been tempted
by their legendary high performance network cards,
which were said everywhere to be able to saturate a 10 Gig wire under Linux
without putting too much stress on the CPU, using a mainstream opensource driver, and
without resorting to dirty tricks such as TOE. What would a performance addict like me
need more ?
I finally decided to mail these guys and described how I'm currently used to benchmark
HAproxy with aggregated Gigabit NICs, with a minimum of 4 NICs in a setup (1 for the
client, 2 for the proxy, 1 for the server). 4 hours later, when I woke up, I had a
mail from Charles Seitz, Myricom's CEO. He explained to me that he was pleased to offer
me 4 NICs with cables, plus one spare of each just in case, as their contribution
to the project... yes, I'm talking about a donation of five 10Gig NICs! That's awsome! And if it
would not be enough for some of you to find them really cool, he also provided me with
french-speaking contacts, free access to their support and
important advices for the
choice of motherboards to get the best out of those wonderful NICs! I don't even know
the polite words to say in such circumstances :-)
Today I've been monitoring the shipping steps at UPS.
This evening, I noticed that they arrived at EXOSEC.
After leaving the customer's, I went back there to find this big parcel on my desk, with
its contents very carefully packed. I must say that I was both very excited and extremely
careful while opening the packaging.
The first thing I noticed after extracting the first NIC from its packaging was that it
had a very clean design, as can be seen on this photo.
They are also very thin as shown on the
picture on the right, so there will be no problem putting
two of them side-by-side in the proxy.
The CX4 connector looks a bit fragile, but careful
manipulation is the minimal requirement to use the highest speed standard Ethernet.
From what I understood, this is the same connector as used on Infiniband, except
that 10GE has terminators on the board.
Well, obviously, there are very nice companies out there who deserve to be talked about!
Their very generous support to open source projects leaves many others far behind. People
say that Santa Claus lives in the North Pole, but now I know he lives in Arcadia in California :-)
Thank you very much Charles, thanks very much Myricom.
Be sure to read about my first test results here.
|



|
December 6th, 2007
Released haproxy version 1.3.14. A good part of the
changes comes from nice contributors of the mailing list.
Most sensible changes include support for dynamic server weights offering support for
slow start and graceful shutdown. The load balancer is now able
to report its servers state to outer components, enabling the building of more complex
multi-site architectures involving dynamic routing protocols such as
BGP. People who were complaining about the rough configuration, rough statistics, or lack
of logging to UNIX sockets, should really give this one a try. Rate of changes after this version should significantly
drop in order to progressively switch the tree to a stable state.
October 18th, 2007
September 22th, 2007
September 20th, 2007
Released haproxy version 1.3.12.2. It fixes several bugs affecting timeouts and retry counts when configs are split between frontends and backends. Some sanity checks on the configuration file were never executed, causing some erroneous configurations to be accepted without being fixed. Last, the license has been clarified in a few files from O'Reilly. All in all, it seems like keeping a supported version is already starting to pay off, as people are looking for something stable and report bugs very quickly. All version 1.3 users are encouraged to upgrade to 1.3.12.2.
September 5th, 2007
Released haproxy version 1.3.12.1. It fixes a few bugs discovered in 1.3.12, notably one which could lead to crashes under Linux with speculative I/O when clients disconnect before the connection has been established to the server. As a workaround, it is possible to specify "nosepoll" in the "global" section. A "stats refresh <interval>" option has also been added because some people like to have the stats page automatically refresh. It's also possible to hide all failed servers on the stats page now. This version also contains the new configuration manual which has just been started but which helps understand how to use ACL.
July 15th, 2007
Started writing the new Configuration Manual. It enumerates all configuration keywords and in what context they may be used. It also includes a few examples of ACLs. It is not finished yet, but I decided to publish it because people have really no other valuable sources of information to use content switching. It only covers version 1.3.12, and updates will only cover the latest version, making it far more readable. Please take a look at it and start from the examples in the examples/ directory from the sources. Any feedback is welcome :-)
June 17th, 2007
Released haproxy version 1.3.12. It completes integration of ACL with Content Switching, and allows you to customize your error responses. Except for the ACL and a few bugs, there have been few changes since 1.3.11.4, and I intend to support 1.3.12 during development and cleanups of the next versions which may not be as reliable. Several big content providers use 1.3 to regulate the traffic to/from their web servers, and there is a real demand for a stable version with the new features and performance of 1.3. And considering that some of them even pay for this, I understand they want something really reliable.
June 3rd, 2007
Released haproxy version 1.3.11.4. It fixes
2 long-standing bugs in timeout handling, which could sometimes cause 100% CPU usage during
several seconds when a client had closed its write channel. Some small improvements to the I/O subsystem
should save some CPU cycles on high bandwidth sites. It is now possible to finely tune the pollers for
reduced latency.
May 14th, 2007
Released haproxy version 1.3.11.3. It fixes the (hopefully) last
bug affecting Linux users with speculative I/O processing, introduced in 1.3.9.
This bug was also causing random timeouts. Do not use versions 1.3.11 to
1.3.11.2 as they are all broken.
New in this release are a better timer management and a new
memory manager which is able to self-manage its pools and
free unused ones when memory is becoming scarce. It is also easier to code with this new one
since it's not necessary anymore to declare pool sizes. Overall, yet another
performance boost of 5% has been gained.
May 10th, 2007
May 9th, 2007
Released haproxy version 1.3.10.1. It fixes a serious bug
affecting Linux users with speculative I/O processing, introduced in 1.3.9. This bug was causing random
timeouts on some traffic patterns, mostly noticeable in TCP mode but almost certainly in HTTP too. All Linux users
of 1.3.9 and 1.3.10 should either upgrade or disable speculative I/O as a workaround, by starting
haproxy with the -ds argument or by setting nosepoll in the global section.
May 9th, 2007
Released haproxy version 1.3.10. This one adds
ACL, SMTP health checks (thanks to Peter van Dijk), and
URI hashing (thanks to Guillaume Dallaire). Also, the rbtree
was replaced with a much faster tree, leading to an overall performance boost around 5%.
The speculative I/O processing in 1.3.9 has introduced some bugs which have been fixed in this version. I
feel confident that latest changes have brought their pile of bugs too. I will probably spend some time soon
to do cleanup and stabilization work, eventhough both are not really compatible.
I also want to thank all the people who contribute code and testing. You are more and more at each release.
I'm impatient to clean up the remains of the old code, so that even more people can contribute code. Interestingly,
all contributions till now were of high quality. This is probably induced by some sort of selection caused by the
technical aspect of the product, and the skills required to use the development version. Thanks again to you all !
Apr 22th, 2007
Done a quick benchmark at EXOSEC with haproxy 1.3.9 running on a nice single-core system equipped with many PCI-Express Gigabit NICs. The graph shows pretty decent results !
Apr 15th, 2007
Released haproxy version 1.3.9. This one adds
modularization to the pollers, which made it possible for me to finally implement support for FreeBSD
kqueue(). I'd like to thank Olivier Warin for providing me a FreeBSD account during this development.
A new concept was introduced too : speculative I/O. It is a new method consisting in reducing
the number of calls to the expensive epoll_ctl() and epoll_wait() by attempting to access the file
descriptors before being notified about their readiness. This provides an overall speed boost
of 10%, which is quite much for just a poller.
Apr 3rd, 2007
Released haproxy version 1.3.8.2 to fix a minor and a major
bug. The minor bug caused the response rewrite to fail on the status line. The major bug which was
introduced in 1.3.6 could cause the process
to crash in some circumstances when rewriting the request line (method and/or URI).
All users of 1.3.6 and later must upgrade.
Apr 1st, 2007
Released haproxy version 1.3.8.1 to fix very minor bugs, and
slightly improve performance. Request headers were not added if option httpclose was not set.
Bruno Michel contributed a VIM script for syntax color highlighting.
Mar 25th, 2007
Released haproxy version 1.3.8. Several bugs which might have
caused crashes on erroneous configurations have been fixed. The response processing is now completed,
which means that real configurations can now be written ; HAProxy 1.3.8 now is at least equivalent to
1.2.17 in terms of features.
Just like with every release, several code optimization have led to small but noticeable performance
increases, particularly on very high data bandwidth. The configuration errors are handled more
gracefully now with indications about what failed and hints to resolve the issue. HAProxy now builds
on MacOS 10.4 thanks to Dan Zinngrabe who provided a makefile. Also, it is now possible to send
health checks to an alternate server address, thanks to a patch from Fabrice Dulaunoy.
Users of 1.3 are encouraged to upgrade to 1.3.8 as it both fixes known bugs and converges towards
something less tricky than previous versions.
Mar 17th, 2007
Released haproxy version 1.2.17.
I have backported Sin Yu's rbtree scheduler from version 1.3 since it proved to be stable.
A few minor bugs were fixed, and two useful contributions were merged : support for
user and group keywords as alternatives to numerical uid
and gid from Marcus Rueckert, and the ability to prevent some source addresses
from appearing in the X-Forwarded-For header, which is useful when combined
with Stunnel for instance (patch from Bryan Germann). Thanks to both of them, contribs
are always welcome !
The architecture manual was updated to reflect new
features in branch 1.2, with examples for stunnel and for load mitigation.
Users of 1.2.16 with high loads are encouraged to upgrade to 1.2.17 as it offers them
the high performance of branch 1.3 with the reliability of the stable branch 1.2.
Jan 27th, 2007
Released haproxy version 1.3.7. I found a critical bug
in the new parser in development branch, causing crashes when an empty header is passed. This was caused by a missing pointer
assignment in the empty header processing path. All 1.3.6 users MUST upgrade to 1.3.7.
Jan 22th, 2007
Released haproxy version 1.3.6. I spent a long time reworking the
HTTP message parser. It now consists in a carefully hand-optimized 28-states FSM. The new code
will look awful to goto haters, and will please FSM lovers.
It's blazingly fast : parsing and indexing all of the 660 bytes of an HTTP request from
Firefox on Freshmeat only takes 1.94 microsecond on my 1.7 GHz
Pentium-M notebook, which means it can do it more than 500000 times a second!
The request code has been cleaned up a lot and adapted to this new FSM. Adding layer7 rules based on new criteria is
now trivial. The response code will be ported next, but the code was so much cleaner and faster that it was worth
releasing one version before breaking everything. Several bugs were fixed since 1.3.5. I really consider 1.3.6
as the most likely reliable 1.3 release to date.
Jan 7th, 2007
In order to support the new Linux Layer7 Switching project,
I have implemented support for kernel TCP splicing using Alexandre Cassen's library. This is
still experimental but already works remarkably well. On my notebook at 400 Mbps, haproxy's
usage goes down from 65% to 5-10%. I have written some doc
explaining how to setup up TCP splicing, with an example.
Since the original code was provided for Linux kernel 2.6.19 only, I have backported the
patches to kernel 2.6.16 and
2.4.33.
The second great news is that Sin Yu has provided me with a useful patch for the second time :
the task scheduler is now based on an rbtree and not on the dirty old dual-linked list anymore.
It means that people who had performance problems and who had to set all their timeouts to the
same value as a workaround will not have to do this anymore. I have tested, and the code works
like a charm ! Thanks again Sin !
Jan 2nd, 2007
After about 4500 new lines of code and some useful feedback from a bunch of brave beta-testers,
I'm pleased to announce haproxy version 1.3.4 with the new
Content-Switching features !!!
It is now possible to select a backend (server pool + load balancing algo) depending on
any parameter in the request, such as any part of the URI, the host name, etc....
As of now, I've merged Sin Yu's patch to permit switching based on a request regex, but the framework is
ready to accept many other criteria. The HTTP request parser has been completely rewritten to support
unlimited header inspection, and the statistics page has been rewritten, as can be seen
on the demo page. It is far from being finished right
now, but it seems pretty usable. The server state machine should be adapted though.
There is still no doc, so please note that old configurations do still work, and that in order to switch
from an instance to another backend, you need to use "reqisetbe <regex> <new_proxy>".
Also, there's a config example here that will be worth any doc.
Dec 5th, 2006
The load balancing article has been linked to from LinuxFR. The small 128 kbps uplink is currently running at full speed but the site is still responding thanks to haproxy queuing the connections to smoothen the traffic. Next time, I should also write an article on setting up the QoS with tc, because typing remotely with SSH is still very responsive under full load :-)
Jul 4th, 2006
Opened development branch 1.3, which started with a major cleanup.
Not sure yet about all features which will be merged, the first step
is to clean up the code and make it modular. The API's licence has
been switched to LGPL in order to later allow linking with binary
external modules developped for applications covered by NDAs for
example. Version 1.3.0 is exactly the same as 1.2.14+bugfixes so it
is a stable starting point. It is available here.
⇐ Back to HAProxy
Feel free to contact me at for any questions or comments :
|
|