[stunnel-users] stunnel stopped working today, had to reboot server

[Christopher Schultz]

Peter,

Thanks again for your quick reply.

On 2/13/19 15:09, Peter Pentchev wrote:
> On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
>> Peter,
>> On 2/13/19 13:09, Peter Pentchev wrote:
>>> On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
>>>> Hello,
>>>>
>>>> I'm new to the list so I'm sorry if this isn't the right place to report
>>>> this.
>>>>
>>>> I had a server stop responding to stunnel connections sometime yesterday
>>>> and the resolution was ultimately to reboot the server and everything
>>>> was okay. Restarting the stunnel service was not enough to get things
>>>> working again.
>>> [snip]
>>>>
>>>> Looking through the log file, I could see that there were some odd
>>>> messages coming from stunnel in the daemon.log file suggesting that
>>>> there might be a memory leak. I won't post them here unless requested,
>>>> as they may represent a potential security issue.
>>>
>>> Hi,
>>>
>>> Unfortunately this is a known problem with the Debian package of
>>> stunnel; I did not get around to backporting the fixes to the stunnel
>>> version that is in Debian 9.x (stretch); I'm really sorry about that.
>>
>> Confirmed: I'm running Debian Stretch:
>>
>> $ lsb_release -a
>> No LSB modules are available.
>> Distributor ID:	Debian
>> Description:	Debian GNU/Linux 9.7 (stretch)
>> Release:	9.7
>> Codename:	stretch
>>
>>> I could make a stretch backport of the stunnel version that is in
>>> unstable/testing right now - the one that will be in the upcoming
>>> Debian 10.x (buster).  Actually I'll build a package later tonight and
>>> let you know where you can download it from, then I'll upload it
>>> to the stretch-backports suite and wait for the backports admins to
>>> accept it.  This package will not have the memory exhaustion problem
>>> that the version in stretch does have.
>>
>> That would be great. Will I (eventually) end up getting it via the
>> normal apt update mechanism? I only have these apt sources configured:
>>
>> deb http://ftp.us.debian.org/debian/ stretch main
>> deb http://security.debian.org/ stretch/updates main
> 
> Well, to be honest, I'm not sure how easy it will be to backport
> the change that fixes the memory exhaustion without bringing in
> a lot of other changes in later versions of stunnel.
> 
> I believe your best bet would be to add the stretch-backports suite
> as described on https://backports.debian.org/Instructions/ and
> then install only the stunnel4 package from that suite:
> 
>   apt-get install -t stretch-backports stunnel4
> 
> This will bring in this package, and only this package, from
> the backports suite.  The package there is exactly the version that
> has been in the testing distribution for two months now (it was
> accepted almost immediately after I uploaded it, it didn't really
> need to go through human approval), and there have been no reports of
> trouble with it since then.

I can look into that. I don't exactly need an act of congress to add an
apt source and update a package. I might want to do it on a "slow day",
though :)

>> Is there a safe mitigation I can put into place before a patch? I
>> believe Debian's stunnel service scripts completely shut-down one
>> stunnel process and then start up a new one, so I might have a (very
>> short) interruption. I can schedule that via cron if necessary, but I'd
>> prefer to avoid even a small interruption if possible.
>>
>>> Could you just confirm that you are indeed running Debian 9.x (stretch)?
>>> (it seems this way from the kernel version that you are running and
>>>  from the symptoms; stunnel 5.06 in jessie does not have this problem)
>>
>> I have 5.39. Is this older than 5.06? Or is that a typo?
> 
> No, it was not a typo; I'm sorry if I was unclear.  When I said "jessie",
> I meant Debian 8.x (jessie) - the previous release of Debian that had
> a much older version of stunnel that did not have the changes that,
> unfortunately, led to the memory leak.

Duh. I didn't even read "jessie", there. Thanks for the clarification.

> To be totally honest, the leak itself is partly my fault, since it
> was introduced in a patch for an issue that was fixed in a more
> elaborate way in later versions of stunnel.>
>>> Sorry again, and thanks for your understanding!
>>
>> No, it's great to get this kind of feedback right away.
>>
>> I *am* curious about why a service-restart did not seem to fix things.
>> With the above memory-exhaustion, would you have expected a
>> service-restart to fix everything? (I certainly did.) Is it likely that
>> the "restart" didn't actually restart the service, and I was just
>> repeatedly testing a borked server process?
> 
> I think that it is possible that this might have been the case;
> there was actually a bug in the stunnel4 init script that was fixed
> in a later Debian upload (the one that was in the stretch-backports
> suite until now).  The bug was that the script did not wait for
> the old process to really end before attempting to start a new one -
> and the new one would not really start, since the kernel would not
> allow it to listen on the same ports that the old one was still
> listening on.  I didn't think of this bug at once when I read your
> first message, but it makes sense now.

I almost want to wait until it fails again and then inspect it. If a
failing-restart does not explain it, then it suggests some kind of
corruption of the kernel's TCP/IP stack which doesn't sound like the
kind of thing that should be possible.

> So, yeah, it seems that I'm guilty of not fixing not one, but two
> serious problems with the version of stunnel in the stable Debian
> release...  I could try to figure out a way to fix the memory
> exhaustion one, but it may take me a couple of days.  Right now,
> your best bet (and the course of action that I recommend) is to
> install stunnel 5.50 from the stretch-backports suite.

I'll give that a shot and see how things go. If all else fails, I do
know that a server reboot will fix the issue. Of course, if I wanted to
reboot servers to get them to work, I'd have built everything on Windows :P

Thanks,
-chris

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 899 bytes
Desc: OpenPGP digital signature
URL: <http://www.stunnel.org/pipermail/stunnel-users/attachments/20190213/efb7cfe9/attachment.sig>