Hi there,
After upgrading to stunnel 5.07 I've seen a lot of truncated http responses. I've tested different versions and 5.04 is the last one to work fine. There seems to be fixes to a similar issue in 5.05, but at least for me the result seems to be the opposite. I'm on Ubuntu 12.04.
Looking at the logs, the log entries are identical except for one part, the amount of data sent to SSL:
stunnel >= 5.05, not working: LOG3[31153]: transfer: s_poll_wait: TIMEOUTclose exceeded: closing LOG5[31153]: Connection closed: 67584 byte(s) sent to SSL, 184 byte(s) sent to socket stunnel 5.04, works: LOG3[6248]: transfer: s_poll_wait: TIMEOUTclose exceeded: closing LOG5[6248]: Connection closed: 113051 byte(s) sent to SSL, 184 byte(s) sent to socket
It's worth noting that in both cases the closing happens after TIMEOUTclose has exceeded (probably because of a missing close notify?).
In any case, apparently less data is sent to SSL and that's how it seems like; the end of the http responses get truncated. If everything's very fast (e.g. localhost connections only) the problem seems not to happen, not very often at least. With connections between two servers I've been able to reproduce it pretty much every time in my setup (stunnel -> haproxy -> apache -> unicorn).
Undefining the POLLRDHUP or reverting the changes in 5.05 to src/client.c seem to fix the problem for me.
Any ideas what would be a proper fix?
Thanks!
Best regards,
Tuomas Silen
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Tuomas Silen wrote:
LOG3[31153]: transfer: s_poll_wait: TIMEOUTclose exceeded: closing
Could you please share your stunnel.conf and some debug logs? I need all the logs generated by stunnel for the problematic connection.
Please help me reproduce this problem: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
Best regards, Michal Trojnara
On 07/11/14 08:48, Michal Trojnara wrote:
Tuomas Silen wrote:
LOG3[31153]: transfer: s_poll_wait: TIMEOUTclose exceeded: closing
Could you please share your stunnel.conf and some debug logs? I need all the logs generated by stunnel for the problematic connection.
Sure, the config and debug level log of both working and non-working connection are available at https://gist.github.com/anonymous/44c47b3893c3ebf45018
Also worth noting is that if the client uses Connection: keep-alive (instead of Connection: close), the problem does not appear, possibly because the connection isn't closed right away.
It also only seems to appear if the response body is large enough.
I'll see if I can come up with some minimal setup to reliably reproduce it.
Best regards,
-Tuomas
Hmmm... at first sight this sounds like exactly the scenario I was concerned about back when the change was made (but got no response).
https://www.stunnel.org/pipermail/stunnel-users/2014-September/004765.html
Graham
----- Original Message ----- From: "Tuomas Silen" tuomas@silen.fi To: stunnel-users@stunnel.org Sent: Friday, November 07, 2014 11:43 AM Subject: Re: [stunnel-users] Truncated responses with stunnel >= 5.05
On 07/11/14 08:48, Michal Trojnara wrote:
Tuomas Silen wrote:
LOG3[31153]: transfer: s_poll_wait: TIMEOUTclose exceeded: closing
Could you please share your stunnel.conf and some debug logs? I need all the logs generated by stunnel for the problematic connection.
Sure, the config and debug level log of both working and non-working connection are available at https://gist.github.com/anonymous/44c47b3893c3ebf45018
Also worth noting is that if the client uses Connection: keep-alive (instead of Connection: close), the problem does not appear, possibly because the connection isn't closed right away.
It also only seems to appear if the response body is large enough.
I'll see if I can come up with some minimal setup to reliably reproduce it.
Best regards,
-Tuomas _______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Tuomas Silen wrote:
Sure, the config and debug level log of both working and non-working connection are available at https://gist.github.com/anonymous/44c47b3893c3ebf45018
Thank you very much. I checked the kernel source again: http://lxr.free-electrons.com/source/net/ipv4/tcp.c#L466 It looks like POLLRDHUP is set regardless of the amount of data left in the kernel buffers.
Please try: https://www.stunnel.org/downloads/beta/stunnel-5.08b2.tar.gz
Mike
On 07/11/14 16:16, Michal Trojnara wrote:
Thank you very much. I checked the kernel source again: http://lxr.free-electrons.com/source/net/ipv4/tcp.c#L466 It looks like POLLRDHUP is set regardless of the amount of data left in the kernel buffers.
Please try: https://www.stunnel.org/downloads/beta/stunnel-5.08b2.tar.gz
Thanks Mike! I've installed the new version and I'm no longer able to reproduce the problem. Haven't encountered any new problems either, so looking good so far! I'll do some more extensive testing a bit later today and over the weekend and let you know if I see any issues.
Thanks! -Tuomas
-
Graham Nayler (work) -
Michal Trojnara -
Tuomas Silen