Quick summary: Stunnel 4.35 configured with four services. As clients connect, the main stunnel process grows a lot in vsz/rss memory. With a lot of clients connecting, it quickly grows to several gigabytes rss.
The use case is quite special: The multithreaded download accelerator 'axel' is used to download large files using 800 simultaneous threads. The downloads are from China, so the network performance is horrible, and connections are reset/broken all the time, so there is lots of new connections created constantly. Up to four axel instances run at the same time, so we're talking 3200 simultaneous connections. Over a short 2h timespan, nearly 30000 connections have been set up (ref attached stunnel.log).
The memory use increases over time, often in very large steps. As connections finish and the transfers stop (i.e. no active connections at all), the memory is not released.
A similar issue was reported in 2008 [1].
Attached are the following files: stunnel.conf: Four services, no special config. Debug level 5. massif.out.excerpt.txt: Valgrind Massif output.
The Massif log indicates that most of the memory is allocated through client.c:init_ssl(), by libssl and zlib. I haven't looked too much at the code yet, but could this be related to the high rate of connection resets/timeouts, combined with connection/session reuse?
sven
[1]: possible stunnel memory leak Message-ID: 47E8E0AD.8010709@eu.citrix.com http://www.stunnel.org/pipermail/stunnel-users/2008-March/001903.html
Sven Ulland wrote:
Quick summary: Stunnel 4.35 configured with four services. As clients connect, the main stunnel process grows a lot in vsz/rss memory. With a lot of clients connecting, it quickly grows to several gigabytes rss.
Thank you very much for the report. Stunnel does not call zlib directly, so OpenSSL should call the appropriate cleanup functions of zlib.
The Massif log indicates that most of the memory is allocated through client.c:init_ssl(), by libssl and zlib. I haven't looked too much at the code yet, but could this be related to the high rate of connection resets/timeouts, combined with connection/session reuse?
I guess you're right. A trivial workaround would be to build OpenSSL without zlib. 8-)
BTW: What is your version of OpenSSL?
Best regards, Mike
Is it possible that this is related to http://stunnel.mirt.net/pipermail/stunnel-users/2011-March/002982.html?
Basically, there is a memory leak in the code that creates pthreads.
Paul
-----Original Message----- From: stunnel-users-bounces@stunnel.org [mailto:stunnel-users-bounces@stunnel.org] On Behalf Of Michal Trojnara Sent: Monday, April 11, 2011 10:13 AM To: stunnel-users@stunnel.org Subject: Re: [stunnel-users] Possible leak in client.c:init_ssl()
Sven Ulland wrote:
Quick summary: Stunnel 4.35 configured with four services. As clients connect, the main stunnel process grows a lot in vsz/rss memory. With a lot of clients connecting, it quickly grows to several gigabytes rss.
Thank you very much for the report. Stunnel does not call zlib directly, so OpenSSL should call the appropriate cleanup functions of zlib.
The Massif log indicates that most of the memory is allocated through client.c:init_ssl(), by libssl and zlib. I haven't looked too much at the code yet, but could this be related to the high rate of connection resets/timeouts, combined with connection/session reuse?
I guess you're right. A trivial workaround would be to build OpenSSL without zlib. 8-)
BTW: What is your version of OpenSSL?
Best regards, Mike _______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org http://stunnel.mirt.net/mailman/listinfo/stunnel-users
On 04/11/2011 05:13 PM, Michal Trojnara wrote:
Sven Ulland wrote:
The Massif log indicates that most of the memory is allocated through client.c:init_ssl(), by libssl and zlib. I haven't looked too much at the code yet, but could this be related to the high rate of connection resets/timeouts, combined with connection/session reuse?
I guess you're right. A trivial workaround would be to build OpenSSL without zlib. 8-)
BTW: What is your version of OpenSSL?
The initial report was based on OpenSSL 0.9.8g. I then compiled 0.9.8r without zlib, and it ran well and without any obvious leaks. A massif output of that run is attached: It shows memory use of 314MB while idle (after having served a lot of connections). Is it so that the number of ssl/connections allocated by stunnel is always the maximum observed throughout the entire runtime, i.e. it never frees up idle connections? That's not really a problem, I'm just curious.
What I should have done earlier is to check the OpenSSL change log.
""" Changes between 0.9.8n and 1.0.0 [29 Mar 2010] [...] Fix compression algorithm handling: if resuming a session use the compression algorithm of the resumed session instead of determining it from client hello again. Don't allow server to change algorithm. """
I'm not sure if this was backported to 0.9.8r, but there's also this:
""" Changes between 0.9.8l and 0.9.8m [25 Feb 2010] [...] Modify compression code so it frees up structures without using the ex_data callbacks. This works around a problem where some applications call CRYPTO_cleanup_all_ex_data() before application exit (e.g. when restarting) then use compression (e.g. SSL with compression) later. This results in significant per-connection memory leaks and has caused some security issues including CVE-2008-1678 and CVE-2009-4355. """
I recompiled 0.9.8r with zlib enabled again, but it's not clear to me if zlib was actually used in the following run or not. At least there were no zlib or libz strings in the massif output.
I'll assume it's the OpenSSL issues that were at fault, and then continue to run with the new lib version. If there is any new development in the upcoming days, I'll send a follow-up.
sven
Sven Ulland wrote:
Is it so that the number of ssl/connections allocated by stunnel is always the maximum observed throughout the entire runtime, i.e. it never frees up idle connections? That's not really a problem, I'm just curious.
In order to free up an idle connection stunnel would have to close it. In many cases connections supposed to be persistent. The timeout is controlled with TIMEOUTidle option. The default value is 43200 seconds = 12 hours.
I recompiled 0.9.8r with zlib enabled again, but it's not clear to me if zlib was actually used in the following run or not. At least there were no zlib or libz strings in the massif output.
I'll assume it's the OpenSSL issues that were at fault, and then continue to run with the new lib version. If there is any new development in the upcoming days, I'll send a follow-up.
I'll appreciate it.
Mike
This concerns me. Does the leak occur on badly handled disconnects only, or will the leak happen even when the disconnect is handled in an orderly manner?
On 4/12/2011 3:48 PM, Michal Trojnara wrote:
Sven Ulland wrote:
Is it so that the number of ssl/connections allocated by stunnel is always the maximum observed throughout the entire runtime, i.e. it never frees up idle connections? That's not really a problem, I'm just curious.
In order to free up an idle connection stunnel would have to close it. In many cases connections supposed to be persistent. The timeout is controlled with TIMEOUTidle option. The default value is 43200 seconds = 12 hours.
I recompiled 0.9.8r with zlib enabled again, but it's not clear to me if zlib was actually used in the following run or not. At least there were no zlib or libz strings in the massif output.
I'll assume it's the OpenSSL issues that were at fault, and then continue to run with the new lib version. If there is any new development in the upcoming days, I'll send a follow-up.
I'll appreciate it.
Mike
stunnel-users mailing list stunnel-users@stunnel.org http://stunnel.mirt.net/mailman/listinfo/stunnel-users
-
Allex, Paul M (USA E D EA STS) -
Bing Bang -
Michal Trojnara -
Sven Ulland