[stunnel-users] stunnel stopped working today, had to reboot server

[Christopher Schultz]

Peter,

On 2/13/19 15:09, Peter Pentchev wrote:
> On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
>> Peter,
>>
>> Thanks for the quick reply.
>>
>> On 2/13/19 13:09, Peter Pentchev wrote:
>>> On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
>>>> Hello,
>>>>
>>>> I'm new to the list so I'm sorry if this isn't the right place to report
>>>> this.
>>>>
>>>> I had a server stop responding to stunnel connections sometime yesterday
>>>> and the resolution was ultimately to reboot the server and everything
>>>> was okay. Restarting the stunnel service was not enough to get things
>>>> working again.
>>> [snip]
>>>>
>>>> Looking through the log file, I could see that there were some odd
>>>> messages coming from stunnel in the daemon.log file suggesting that
>>>> there might be a memory leak. I won't post them here unless requested,
>>>> as they may represent a potential security issue.
>>>
>>> Hi,
>>>
>>> Unfortunately this is a known problem with the Debian package of
>>> stunnel; I did not get around to backporting the fixes to the stunnel
>>> version that is in Debian 9.x (stretch); I'm really sorry about that.
>>
>> Confirmed: I'm running Debian Stretch:
>>
>> $ lsb_release -a
>> No LSB modules are available.
>> Distributor ID:	Debian
>> Description:	Debian GNU/Linux 9.7 (stretch)
>> Release:	9.7
>> Codename:	stretch
>>
>>> I could make a stretch backport of the stunnel version that is in
>>> unstable/testing right now - the one that will be in the upcoming
>>> Debian 10.x (buster).  Actually I'll build a package later tonight and
>>> let you know where you can download it from, then I'll upload it
>>> to the stretch-backports suite and wait for the backports admins to
>>> accept it.  This package will not have the memory exhaustion problem
>>> that the version in stretch does have.
>>
>> That would be great. Will I (eventually) end up getting it via the
>> normal apt update mechanism? I only have these apt sources configured:
>>
>> deb http://ftp.us.debian.org/debian/ stretch main
>> deb http://security.debian.org/ stretch/updates main
> 
> Well, to be honest, I'm not sure how easy it will be to backport
> the change that fixes the memory exhaustion without bringing in
> a lot of other changes in later versions of stunnel.
> 
> I believe your best bet would be to add the stretch-backports suite
> as described on https://backports.debian.org/Instructions/ and
> then install only the stunnel4 package from that suite:
> 
>   apt-get install -t stretch-backports stunnel4
> 
> This will bring in this package, and only this package, from
> the backports suite.  The package there is exactly the version that
> has been in the testing distribution for two months now (it was
> accepted almost immediately after I uploaded it, it didn't really
> need to go through human approval), and there have been no reports of
> trouble with it since then.
> 
>> Is there a safe mitigation I can put into place before a patch? I
>> believe Debian's stunnel service scripts completely shut-down one
>> stunnel process and then start up a new one, so I might have a (very
>> short) interruption. I can schedule that via cron if necessary, but I'd
>> prefer to avoid even a small interruption if possible.
>>
>>> Could you just confirm that you are indeed running Debian 9.x (stretch)?
>>> (it seems this way from the kernel version that you are running and
>>>  from the symptoms; stunnel 5.06 in jessie does not have this problem)
>>
>> I have 5.39. Is this older than 5.06? Or is that a typo?
> 
> No, it was not a typo; I'm sorry if I was unclear.  When I said "jessie",
> I meant Debian 8.x (jessie) - the previous release of Debian that had
> a much older version of stunnel that did not have the changes that,
> unfortunately, led to the memory leak.  To be totally honest, the leak
> itself is partly my fault, since it was introduced in a patch for
> an issue that was fixed in a more elaborate way in later versions of
> stunnel.
> 
>>> Sorry again, and thanks for your understanding!
>>
>> No, it's great to get this kind of feedback right away.
>>
>> I *am* curious about why a service-restart did not seem to fix things.
>> With the above memory-exhaustion, would you have expected a
>> service-restart to fix everything? (I certainly did.) Is it likely that
>> the "restart" didn't actually restart the service, and I was just
>> repeatedly testing a borked server process?
> 
> I think that it is possible that this might have been the case;
> there was actually a bug in the stunnel4 init script that was fixed
> in a later Debian upload (the one that was in the stretch-backports
> suite until now).  The bug was that the script did not wait for
> the old process to really end before attempting to start a new one -
> and the new one would not really start, since the kernel would not
> allow it to listen on the same ports that the old one was still
> listening on.  I didn't think of this bug at once when I read your
> first message, but it makes sense now.
> 
> So, yeah, it seems that I'm guilty of not fixing not one, but two
> serious problems with the version of stunnel in the stable Debian
> release...  I could try to figure out a way to fix the memory
> exhaustion one, but it may take me a couple of days.  Right now,
> your best bet (and the course of action that I recommend) is to
> install stunnel 5.50 from the stretch-backports suite.

So, my application servers are evidently quite regular and predictable.
And they stay up for a long time. Therefore, last night just before
bedtime, another application server's stunnel service stopped
responding, having suffered the same fate as the one earlier in the day.

This time I was prepared for it and ready for some forensics.

Shutting down the service via `/etc/init.d/stunnel4 stop` appeared to
succeed (systemd reported "service stopped") but then I did a `ps` and I
could plainly see that the process was still running. I have two
different stunnel configuration files, so I usually get more than one
stunnel process listening, and only one of them had stopped (the one
that did NOT run out of memory).

Performing a `kill -9` on the process did indeed kill it, and then a
standard service-start for stunnel got it working again.

One of my application servers has been upgraded to the backports version
(5.44) and I'll let that one run for a bit before upgrading the others,
just to Make Sure :)

Thanks for your help with this. I really do think that the stable
package needs to be fixed. I use Debian primarily because it's known to
be rock-solid and I'd like it to stay that way. :) Not everyone will be
upgrading to Buster as soon as it's released, so it would be good to get
that memory fix back-ported to Stretch.

Thanks again for your work on this.

-chris

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 899 bytes
Desc: OpenPGP digital signature
URL: <http://www.stunnel.org/pipermail/stunnel-users/attachments/20190214/1e3c8036/attachment.sig>