Peter,
On 2/13/19 15:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
Peter,
Thanks for the quick reply.
On 2/13/19 13:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
[snip]
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
Hi,
Unfortunately this is a known problem with the Debian package of stunnel; I did not get around to backporting the fixes to the stunnel version that is in Debian 9.x (stretch); I'm really sorry about that.
Confirmed: I'm running Debian Stretch:
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.7 (stretch) Release: 9.7 Codename: stretch
I could make a stretch backport of the stunnel version that is in unstable/testing right now - the one that will be in the upcoming Debian 10.x (buster). Actually I'll build a package later tonight and let you know where you can download it from, then I'll upload it to the stretch-backports suite and wait for the backports admins to accept it. This package will not have the memory exhaustion problem that the version in stretch does have.
That would be great. Will I (eventually) end up getting it via the normal apt update mechanism? I only have these apt sources configured:
deb http://ftp.us.debian.org/debian/ stretch main deb http://security.debian.org/ stretch/updates main
Well, to be honest, I'm not sure how easy it will be to backport the change that fixes the memory exhaustion without bringing in a lot of other changes in later versions of stunnel.
I believe your best bet would be to add the stretch-backports suite as described on https://backports.debian.org/Instructions/ and then install only the stunnel4 package from that suite:
apt-get install -t stretch-backports stunnel4
This will bring in this package, and only this package, from the backports suite. The package there is exactly the version that has been in the testing distribution for two months now (it was accepted almost immediately after I uploaded it, it didn't really need to go through human approval), and there have been no reports of trouble with it since then.
Is there a safe mitigation I can put into place before a patch? I believe Debian's stunnel service scripts completely shut-down one stunnel process and then start up a new one, so I might have a (very short) interruption. I can schedule that via cron if necessary, but I'd prefer to avoid even a small interruption if possible.
Could you just confirm that you are indeed running Debian 9.x (stretch)? (it seems this way from the kernel version that you are running and from the symptoms; stunnel 5.06 in jessie does not have this problem)
I have 5.39. Is this older than 5.06? Or is that a typo?
No, it was not a typo; I'm sorry if I was unclear. When I said "jessie", I meant Debian 8.x (jessie) - the previous release of Debian that had a much older version of stunnel that did not have the changes that, unfortunately, led to the memory leak. To be totally honest, the leak itself is partly my fault, since it was introduced in a patch for an issue that was fixed in a more elaborate way in later versions of stunnel.
Sorry again, and thanks for your understanding!
No, it's great to get this kind of feedback right away.
I *am* curious about why a service-restart did not seem to fix things. With the above memory-exhaustion, would you have expected a service-restart to fix everything? (I certainly did.) Is it likely that the "restart" didn't actually restart the service, and I was just repeatedly testing a borked server process?
I think that it is possible that this might have been the case; there was actually a bug in the stunnel4 init script that was fixed in a later Debian upload (the one that was in the stretch-backports suite until now). The bug was that the script did not wait for the old process to really end before attempting to start a new one - and the new one would not really start, since the kernel would not allow it to listen on the same ports that the old one was still listening on. I didn't think of this bug at once when I read your first message, but it makes sense now.
So, yeah, it seems that I'm guilty of not fixing not one, but two serious problems with the version of stunnel in the stable Debian release... I could try to figure out a way to fix the memory exhaustion one, but it may take me a couple of days. Right now, your best bet (and the course of action that I recommend) is to install stunnel 5.50 from the stretch-backports suite.
So, my application servers are evidently quite regular and predictable. And they stay up for a long time. Therefore, last night just before bedtime, another application server's stunnel service stopped responding, having suffered the same fate as the one earlier in the day.
This time I was prepared for it and ready for some forensics.
Shutting down the service via `/etc/init.d/stunnel4 stop` appeared to succeed (systemd reported "service stopped") but then I did a `ps` and I could plainly see that the process was still running. I have two different stunnel configuration files, so I usually get more than one stunnel process listening, and only one of them had stopped (the one that did NOT run out of memory).
Performing a `kill -9` on the process did indeed kill it, and then a standard service-start for stunnel got it working again.
One of my application servers has been upgraded to the backports version (5.44) and I'll let that one run for a bit before upgrading the others, just to Make Sure :)
Thanks for your help with this. I really do think that the stable package needs to be fixed. I use Debian primarily because it's known to be rock-solid and I'd like it to stay that way. :) Not everyone will be upgrading to Buster as soon as it's released, so it would be good to get that memory fix back-ported to Stretch.
Thanks again for your work on this.
-chris