Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
I isolated the problem to stunnel by checking that the underlying service (a Java web application) would respond from the localhost machine (it did) but an openssl s_client connection to localhost:stunnel-port would connect but not proceed past the CONNECTED(3) state. Ultimately, it would time-out.
stunnel was not logging anything to syslog when these connections came in. Outgoing stunnel connections seemed to be okay.
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
My concern is that a service-restart for stunnel was not sufficient. This suggests a problem which goes deeper than the stunnel service. Is it possible for stunnel to break in such a way that it continues to be broken after a restart?
I'm sorry, in retrospect, I was not 100% sure that the service did indeed stop and launch a new process when running /etc/init.d/stunnel restart, but there were no errors and the service-runner did report that the service was restarted.
My (simplified) configuration and version information is posted below. Please let me know if there is any other information I might be able to provide in order to investigate this.
Note that this is a package-managed version of stunnel, provided by the Debian package-maintainers.
Thanks, -chris
PS Thanks for a wonderful product. I've relied on stunnel for years to proxy these unencrypted AJP connections for me. Cheers for all your great work.
Configuration:
=== CUT ===
cert = /etc/stunnel/stunnel.crt key = /etc/stunnel/stunnel.pem sslVersion = TLSv1.2 options = NO_SSLv3 options = NO_TLSv1 options = NO_TLSv1.1 chroot = /var/lib/stunnel4/ setuid = stunnel4 setgid = stunnel4 pid = /stunnel4-ajp.pid socket = l:TCP_NODELAY=1 socket = r:TCP_NODELAY=1 verify = 4 CAfile = /etc/stunnel/stunnel-ajp-trusted.pem
[now, a series of 4 services, all configured similarly] accept=public-port connect=localhost:private-port
=== CUT ===
Version: stunnel 5.39 on x86_64-pc-linux-gnu platform Compiled with OpenSSL 1.1.0c 10 Nov 2016 Running with OpenSSL 1.1.0j 20 Nov 2018 Update OpenSSL shared libraries or rebuild stunnel Threading:PTHREAD Sockets:POLL,IPv6,SYSTEMD TLS:ENGINE,FIPS,OCSP,PSK,SNI Auth:LIBWRAP
Global options: debug = daemon.notice pid = /var/run/stunnel4.pid RNDbytes = 64 RNDfile = /dev/urandom RNDoverwrite = yes
Service-level options: ciphers = FIPS (with "fips = yes") ciphers = HIGH:+3DES:+DH:!aNULL:!SSLv2 (with "fips = no") curve = prime256v1 debug = notice logId = sequential options = NO_SSLv2 options = NO_SSLv3 sessionCacheSize = 1000 sessionCacheTimeout = 300 seconds stack = 65536 bytes TIMEOUTbusy = 300 seconds TIMEOUTclose = 60 seconds TIMEOUTconnect = 10 seconds TIMEOUTidle = 43200 seconds verify = none
Linux kernel version:
Linux [hostname] 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux
This is running on a bare metal server.
On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
[snip]
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
Hi,
Unfortunately this is a known problem with the Debian package of stunnel; I did not get around to backporting the fixes to the stunnel version that is in Debian 9.x (stretch); I'm really sorry about that.
I could make a stretch backport of the stunnel version that is in unstable/testing right now - the one that will be in the upcoming Debian 10.x (buster). Actually I'll build a package later tonight and let you know where you can download it from, then I'll upload it to the stretch-backports suite and wait for the backports admins to accept it. This package will not have the memory exhaustion problem that the version in stretch does have.
Could you just confirm that you are indeed running Debian 9.x (stretch)? (it seems this way from the kernel version that you are running and from the symptoms; stunnel 5.06 in jessie does not have this problem)
Sorry again, and thanks for your understanding!
G'luck, Peter
Peter,
Thanks for the quick reply.
On 2/13/19 13:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
[snip]
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
Hi,
Unfortunately this is a known problem with the Debian package of stunnel; I did not get around to backporting the fixes to the stunnel version that is in Debian 9.x (stretch); I'm really sorry about that.
Confirmed: I'm running Debian Stretch:
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.7 (stretch) Release: 9.7 Codename: stretch
I could make a stretch backport of the stunnel version that is in unstable/testing right now - the one that will be in the upcoming Debian 10.x (buster). Actually I'll build a package later tonight and let you know where you can download it from, then I'll upload it to the stretch-backports suite and wait for the backports admins to accept it. This package will not have the memory exhaustion problem that the version in stretch does have.
That would be great. Will I (eventually) end up getting it via the normal apt update mechanism? I only have these apt sources configured:
deb http://ftp.us.debian.org/debian/ stretch main deb http://security.debian.org/ stretch/updates main
Is there a safe mitigation I can put into place before a patch? I believe Debian's stunnel service scripts completely shut-down one stunnel process and then start up a new one, so I might have a (very short) interruption. I can schedule that via cron if necessary, but I'd prefer to avoid even a small interruption if possible.
Could you just confirm that you are indeed running Debian 9.x (stretch)? (it seems this way from the kernel version that you are running and from the symptoms; stunnel 5.06 in jessie does not have this problem)
I have 5.39. Is this older than 5.06? Or is that a typo?
Sorry again, and thanks for your understanding!
No, it's great to get this kind of feedback right away.
I *am* curious about why a service-restart did not seem to fix things. With the above memory-exhaustion, would you have expected a service-restart to fix everything? (I certainly did.) Is it likely that the "restart" didn't actually restart the service, and I was just repeatedly testing a borked server process?
Thanks, -chris
On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
Peter,
Thanks for the quick reply.
On 2/13/19 13:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
[snip]
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
Hi,
Unfortunately this is a known problem with the Debian package of stunnel; I did not get around to backporting the fixes to the stunnel version that is in Debian 9.x (stretch); I'm really sorry about that.
Confirmed: I'm running Debian Stretch:
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.7 (stretch) Release: 9.7 Codename: stretch
I could make a stretch backport of the stunnel version that is in unstable/testing right now - the one that will be in the upcoming Debian 10.x (buster). Actually I'll build a package later tonight and let you know where you can download it from, then I'll upload it to the stretch-backports suite and wait for the backports admins to accept it. This package will not have the memory exhaustion problem that the version in stretch does have.
That would be great. Will I (eventually) end up getting it via the normal apt update mechanism? I only have these apt sources configured:
deb http://ftp.us.debian.org/debian/ stretch main deb http://security.debian.org/ stretch/updates main
Well, to be honest, I'm not sure how easy it will be to backport the change that fixes the memory exhaustion without bringing in a lot of other changes in later versions of stunnel.
I believe your best bet would be to add the stretch-backports suite as described on https://backports.debian.org/Instructions/ and then install only the stunnel4 package from that suite:
apt-get install -t stretch-backports stunnel4
This will bring in this package, and only this package, from the backports suite. The package there is exactly the version that has been in the testing distribution for two months now (it was accepted almost immediately after I uploaded it, it didn't really need to go through human approval), and there have been no reports of trouble with it since then.
Is there a safe mitigation I can put into place before a patch? I believe Debian's stunnel service scripts completely shut-down one stunnel process and then start up a new one, so I might have a (very short) interruption. I can schedule that via cron if necessary, but I'd prefer to avoid even a small interruption if possible.
Could you just confirm that you are indeed running Debian 9.x (stretch)? (it seems this way from the kernel version that you are running and from the symptoms; stunnel 5.06 in jessie does not have this problem)
I have 5.39. Is this older than 5.06? Or is that a typo?
No, it was not a typo; I'm sorry if I was unclear. When I said "jessie", I meant Debian 8.x (jessie) - the previous release of Debian that had a much older version of stunnel that did not have the changes that, unfortunately, led to the memory leak. To be totally honest, the leak itself is partly my fault, since it was introduced in a patch for an issue that was fixed in a more elaborate way in later versions of stunnel.
Sorry again, and thanks for your understanding!
No, it's great to get this kind of feedback right away.
I *am* curious about why a service-restart did not seem to fix things. With the above memory-exhaustion, would you have expected a service-restart to fix everything? (I certainly did.) Is it likely that the "restart" didn't actually restart the service, and I was just repeatedly testing a borked server process?
I think that it is possible that this might have been the case; there was actually a bug in the stunnel4 init script that was fixed in a later Debian upload (the one that was in the stretch-backports suite until now). The bug was that the script did not wait for the old process to really end before attempting to start a new one - and the new one would not really start, since the kernel would not allow it to listen on the same ports that the old one was still listening on. I didn't think of this bug at once when I read your first message, but it makes sense now.
So, yeah, it seems that I'm guilty of not fixing not one, but two serious problems with the version of stunnel in the stable Debian release... I could try to figure out a way to fix the memory exhaustion one, but it may take me a couple of days. Right now, your best bet (and the course of action that I recommend) is to install stunnel 5.50 from the stretch-backports suite.
G'luck, Peter
Peter,
Thanks again for your quick reply.
On 2/13/19 15:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
Peter, On 2/13/19 13:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
[snip]
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
Hi,
Unfortunately this is a known problem with the Debian package of stunnel; I did not get around to backporting the fixes to the stunnel version that is in Debian 9.x (stretch); I'm really sorry about that.
Confirmed: I'm running Debian Stretch:
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.7 (stretch) Release: 9.7 Codename: stretch
I could make a stretch backport of the stunnel version that is in unstable/testing right now - the one that will be in the upcoming Debian 10.x (buster). Actually I'll build a package later tonight and let you know where you can download it from, then I'll upload it to the stretch-backports suite and wait for the backports admins to accept it. This package will not have the memory exhaustion problem that the version in stretch does have.
That would be great. Will I (eventually) end up getting it via the normal apt update mechanism? I only have these apt sources configured:
deb http://ftp.us.debian.org/debian/ stretch main deb http://security.debian.org/ stretch/updates main
Well, to be honest, I'm not sure how easy it will be to backport the change that fixes the memory exhaustion without bringing in a lot of other changes in later versions of stunnel.
I believe your best bet would be to add the stretch-backports suite as described on https://backports.debian.org/Instructions/ and then install only the stunnel4 package from that suite:
apt-get install -t stretch-backports stunnel4
This will bring in this package, and only this package, from the backports suite. The package there is exactly the version that has been in the testing distribution for two months now (it was accepted almost immediately after I uploaded it, it didn't really need to go through human approval), and there have been no reports of trouble with it since then.
I can look into that. I don't exactly need an act of congress to add an apt source and update a package. I might want to do it on a "slow day", though :)
Is there a safe mitigation I can put into place before a patch? I believe Debian's stunnel service scripts completely shut-down one stunnel process and then start up a new one, so I might have a (very short) interruption. I can schedule that via cron if necessary, but I'd prefer to avoid even a small interruption if possible.
Could you just confirm that you are indeed running Debian 9.x (stretch)? (it seems this way from the kernel version that you are running and from the symptoms; stunnel 5.06 in jessie does not have this problem)
I have 5.39. Is this older than 5.06? Or is that a typo?
No, it was not a typo; I'm sorry if I was unclear. When I said "jessie", I meant Debian 8.x (jessie) - the previous release of Debian that had a much older version of stunnel that did not have the changes that, unfortunately, led to the memory leak.
Duh. I didn't even read "jessie", there. Thanks for the clarification.
To be totally honest, the leak itself is partly my fault, since it was introduced in a patch for an issue that was fixed in a more elaborate way in later versions of stunnel.>
Sorry again, and thanks for your understanding!
No, it's great to get this kind of feedback right away.
I *am* curious about why a service-restart did not seem to fix things. With the above memory-exhaustion, would you have expected a service-restart to fix everything? (I certainly did.) Is it likely that the "restart" didn't actually restart the service, and I was just repeatedly testing a borked server process?
I think that it is possible that this might have been the case; there was actually a bug in the stunnel4 init script that was fixed in a later Debian upload (the one that was in the stretch-backports suite until now). The bug was that the script did not wait for the old process to really end before attempting to start a new one - and the new one would not really start, since the kernel would not allow it to listen on the same ports that the old one was still listening on. I didn't think of this bug at once when I read your first message, but it makes sense now.
I almost want to wait until it fails again and then inspect it. If a failing-restart does not explain it, then it suggests some kind of corruption of the kernel's TCP/IP stack which doesn't sound like the kind of thing that should be possible.
So, yeah, it seems that I'm guilty of not fixing not one, but two serious problems with the version of stunnel in the stable Debian release... I could try to figure out a way to fix the memory exhaustion one, but it may take me a couple of days. Right now, your best bet (and the course of action that I recommend) is to install stunnel 5.50 from the stretch-backports suite.
I'll give that a shot and see how things go. If all else fails, I do know that a server reboot will fix the issue. Of course, if I wanted to reboot servers to get them to work, I'd have built everything on Windows :P
Thanks, -chris
Peter,
On 2/13/19 15:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
Peter,
Thanks for the quick reply.
On 2/13/19 13:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
[snip]
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
Hi,
Unfortunately this is a known problem with the Debian package of stunnel; I did not get around to backporting the fixes to the stunnel version that is in Debian 9.x (stretch); I'm really sorry about that.
Confirmed: I'm running Debian Stretch:
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.7 (stretch) Release: 9.7 Codename: stretch
I could make a stretch backport of the stunnel version that is in unstable/testing right now - the one that will be in the upcoming Debian 10.x (buster). Actually I'll build a package later tonight and let you know where you can download it from, then I'll upload it to the stretch-backports suite and wait for the backports admins to accept it. This package will not have the memory exhaustion problem that the version in stretch does have.
That would be great. Will I (eventually) end up getting it via the normal apt update mechanism? I only have these apt sources configured:
deb http://ftp.us.debian.org/debian/ stretch main deb http://security.debian.org/ stretch/updates main
Well, to be honest, I'm not sure how easy it will be to backport the change that fixes the memory exhaustion without bringing in a lot of other changes in later versions of stunnel.
I believe your best bet would be to add the stretch-backports suite as described on https://backports.debian.org/Instructions/ and then install only the stunnel4 package from that suite:
apt-get install -t stretch-backports stunnel4
This will bring in this package, and only this package, from the backports suite. The package there is exactly the version that has been in the testing distribution for two months now (it was accepted almost immediately after I uploaded it, it didn't really need to go through human approval), and there have been no reports of trouble with it since then.
Is there a safe mitigation I can put into place before a patch? I believe Debian's stunnel service scripts completely shut-down one stunnel process and then start up a new one, so I might have a (very short) interruption. I can schedule that via cron if necessary, but I'd prefer to avoid even a small interruption if possible.
Could you just confirm that you are indeed running Debian 9.x (stretch)? (it seems this way from the kernel version that you are running and from the symptoms; stunnel 5.06 in jessie does not have this problem)
I have 5.39. Is this older than 5.06? Or is that a typo?
No, it was not a typo; I'm sorry if I was unclear. When I said "jessie", I meant Debian 8.x (jessie) - the previous release of Debian that had a much older version of stunnel that did not have the changes that, unfortunately, led to the memory leak. To be totally honest, the leak itself is partly my fault, since it was introduced in a patch for an issue that was fixed in a more elaborate way in later versions of stunnel.
Sorry again, and thanks for your understanding!
No, it's great to get this kind of feedback right away.
I *am* curious about why a service-restart did not seem to fix things. With the above memory-exhaustion, would you have expected a service-restart to fix everything? (I certainly did.) Is it likely that the "restart" didn't actually restart the service, and I was just repeatedly testing a borked server process?
I think that it is possible that this might have been the case; there was actually a bug in the stunnel4 init script that was fixed in a later Debian upload (the one that was in the stretch-backports suite until now). The bug was that the script did not wait for the old process to really end before attempting to start a new one - and the new one would not really start, since the kernel would not allow it to listen on the same ports that the old one was still listening on. I didn't think of this bug at once when I read your first message, but it makes sense now.
So, yeah, it seems that I'm guilty of not fixing not one, but two serious problems with the version of stunnel in the stable Debian release... I could try to figure out a way to fix the memory exhaustion one, but it may take me a couple of days. Right now, your best bet (and the course of action that I recommend) is to install stunnel 5.50 from the stretch-backports suite.
So, my application servers are evidently quite regular and predictable. And they stay up for a long time. Therefore, last night just before bedtime, another application server's stunnel service stopped responding, having suffered the same fate as the one earlier in the day.
This time I was prepared for it and ready for some forensics.
Shutting down the service via `/etc/init.d/stunnel4 stop` appeared to succeed (systemd reported "service stopped") but then I did a `ps` and I could plainly see that the process was still running. I have two different stunnel configuration files, so I usually get more than one stunnel process listening, and only one of them had stopped (the one that did NOT run out of memory).
Performing a `kill -9` on the process did indeed kill it, and then a standard service-start for stunnel got it working again.
One of my application servers has been upgraded to the backports version (5.44) and I'll let that one run for a bit before upgrading the others, just to Make Sure :)
Thanks for your help with this. I really do think that the stable package needs to be fixed. I use Debian primarily because it's known to be rock-solid and I'd like it to stay that way. :) Not everyone will be upgrading to Buster as soon as it's released, so it would be good to get that memory fix back-ported to Stretch.
Thanks again for your work on this.
-chris
-
Christopher Schultz -
Peter Pentchev