[stunnel-users] stunnel stopped working today, had to reboot server

[Peter Pentchev]

On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
> Peter,
> 
> Thanks for the quick reply.
> 
> On 2/13/19 13:09, Peter Pentchev wrote:
> > On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
> >> Hello,
> >>
> >> I'm new to the list so I'm sorry if this isn't the right place to report
> >> this.
> >>
> >> I had a server stop responding to stunnel connections sometime yesterday
> >> and the resolution was ultimately to reboot the server and everything
> >> was okay. Restarting the stunnel service was not enough to get things
> >> working again.
> > [snip]
> >>
> >> Looking through the log file, I could see that there were some odd
> >> messages coming from stunnel in the daemon.log file suggesting that
> >> there might be a memory leak. I won't post them here unless requested,
> >> as they may represent a potential security issue.
> > 
> > Hi,
> > 
> > Unfortunately this is a known problem with the Debian package of
> > stunnel; I did not get around to backporting the fixes to the stunnel
> > version that is in Debian 9.x (stretch); I'm really sorry about that.
> 
> Confirmed: I'm running Debian Stretch:
> 
> $ lsb_release -a
> No LSB modules are available.
> Distributor ID:	Debian
> Description:	Debian GNU/Linux 9.7 (stretch)
> Release:	9.7
> Codename:	stretch
> 
> > I could make a stretch backport of the stunnel version that is in
> > unstable/testing right now - the one that will be in the upcoming
> > Debian 10.x (buster).  Actually I'll build a package later tonight and
> > let you know where you can download it from, then I'll upload it
> > to the stretch-backports suite and wait for the backports admins to
> > accept it.  This package will not have the memory exhaustion problem
> > that the version in stretch does have.
> 
> That would be great. Will I (eventually) end up getting it via the
> normal apt update mechanism? I only have these apt sources configured:
> 
> deb http://ftp.us.debian.org/debian/ stretch main
> deb http://security.debian.org/ stretch/updates main

Well, to be honest, I'm not sure how easy it will be to backport
the change that fixes the memory exhaustion without bringing in
a lot of other changes in later versions of stunnel.

I believe your best bet would be to add the stretch-backports suite
as described on https://backports.debian.org/Instructions/ and
then install only the stunnel4 package from that suite:

  apt-get install -t stretch-backports stunnel4

This will bring in this package, and only this package, from
the backports suite.  The package there is exactly the version that
has been in the testing distribution for two months now (it was
accepted almost immediately after I uploaded it, it didn't really
need to go through human approval), and there have been no reports of
trouble with it since then.

> Is there a safe mitigation I can put into place before a patch? I
> believe Debian's stunnel service scripts completely shut-down one
> stunnel process and then start up a new one, so I might have a (very
> short) interruption. I can schedule that via cron if necessary, but I'd
> prefer to avoid even a small interruption if possible.
> 
> > Could you just confirm that you are indeed running Debian 9.x (stretch)?
> > (it seems this way from the kernel version that you are running and
> >  from the symptoms; stunnel 5.06 in jessie does not have this problem)
> 
> I have 5.39. Is this older than 5.06? Or is that a typo?

No, it was not a typo; I'm sorry if I was unclear.  When I said "jessie",
I meant Debian 8.x (jessie) - the previous release of Debian that had
a much older version of stunnel that did not have the changes that,
unfortunately, led to the memory leak.  To be totally honest, the leak
itself is partly my fault, since it was introduced in a patch for
an issue that was fixed in a more elaborate way in later versions of
stunnel.

> > Sorry again, and thanks for your understanding!
> 
> No, it's great to get this kind of feedback right away.
> 
> I *am* curious about why a service-restart did not seem to fix things.
> With the above memory-exhaustion, would you have expected a
> service-restart to fix everything? (I certainly did.) Is it likely that
> the "restart" didn't actually restart the service, and I was just
> repeatedly testing a borked server process?

I think that it is possible that this might have been the case;
there was actually a bug in the stunnel4 init script that was fixed
in a later Debian upload (the one that was in the stretch-backports
suite until now).  The bug was that the script did not wait for
the old process to really end before attempting to start a new one -
and the new one would not really start, since the kernel would not
allow it to listen on the same ports that the old one was still
listening on.  I didn't think of this bug at once when I read your
first message, but it makes sense now.

So, yeah, it seems that I'm guilty of not fixing not one, but two
serious problems with the version of stunnel in the stable Debian
release...  I could try to figure out a way to fix the memory
exhaustion one, but it may take me a couple of days.  Right now,
your best bet (and the course of action that I recommend) is to
install stunnel 5.50 from the stretch-backports suite.

G'luck,
Peter

-- 
Peter Pentchev  roam@{ringlet.net,debian.org,FreeBSD.org} pp at storpool.com
PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://www.stunnel.org/pipermail/stunnel-users/attachments/20190213/60a223fc/attachment-0001.sig>