- stunnel-users - stunnel.org

[Bug][5.73] stunnel segfault and coredump due to OCSP handling
by stunnel＠benary.org 15 Dec '24

15 Dec '24

Hi everyone, I've recently upgraded from NixOS 24.05 to 24.11 which went from stunnel 5.72 to 5.73 and have since had stunnel crash on me multiple times on different servers (pretty much identical setup however) due to what looks like a memory allocation race? Since 5.73 moved OCSP fetches to a thread, and the problem hasn't ever happened with 5.72 it sounds like an issue specific to that. Someone familiar with the codebase probably has a better idea of what's going on. Given that my updates happened somewhen mid last month and there have been no certificate renewals since (except for one that wouldn't affect the systems that were affected, thanks so much to crt.sh and CT existing) the error message of "OCSP routines::status expired" also seems out of place at least for the certificate itself (the staple may be expired, but stunnel should probably get a new one in that case which may have to do with the error in question). ``` Dec 06 04:43:02 shell stunnel[1124111]: LOG5[per-minute]: OCSP: Certificate accepted Dec 06 04:43:02 shell stunnel[1124111]: 2024.12.06 04:43:02 LOG5[per-minute]: OCSP: Certificate accepted Dec 06 04:43:02 shell stunnel[1124111]: LOG5[per-minute]: OCSP: Certificate accepted Dec 06 04:43:02 shell stunnel[1124111]: 2024.12.06 04:43:02 LOG5[per-minute]: OCSP: Certificate accepted Dec 06 04:43:02 shell stunnel[1124111]: LOG5[per-minute]: OCSP: Certificate accepted Dec 06 04:43:02 shell stunnel[1124111]: 2024.12.06 04:43:02 LOG5[per-minute]: OCSP: Certificate accepted Dec 06 04:43:56 shell stunnel[1124111]: LOG5[ui]: Log file reopened Dec 06 04:43:56 shell stunnel[1124111]: 2024.12.06 04:43:56 LOG5[ui]: Log file reopened Dec 06 04:44:02 shell stunnel[1124111]: LOG3[per-minute]: OCSP: OCSP_check_validity: crypto/ocsp/ocsp_cl.c:351: error:1380007D:OCSP routines::status expired Dec 06 04:44:02 shell stunnel[1124111]: 2024.12.06 04:44:02 LOG3[per-minute]: OCSP: OCSP_check_validity: crypto/ocsp/ocsp_cl.c:351: error:1380007D:OCSP routines::status expired Dec 06 04:44:02 shell stunnel[1124111]: 2024.12.06 04:44:02 LOG4[per-minute]: OCSP: Unknown verification status Dec 06 04:44:02 shell stunnel[1124111]: INTERNAL ERROR: Memory allocated in a different thread at ocsp.c, line 412 Dec 06 04:44:02 shell stunnel[1124111]: LOG4[per-minute]: OCSP: Unknown verification status Dec 06 04:44:02 shell stunnel[1124111]: INTERNAL ERROR: Memory allocated in a different thread at ocsp.c, line 412 Dec 06 04:44:02 shell systemd-coredump[282485]: Process 1124111 (stunnel) of user 988 terminated abnormally with signal 6/ABRT, processing... Dec 06 04:44:02 shell systemd[1]: Started Process Core Dump (PID 282485/UID 0). Dec 06 04:44:02 shell systemd-coredump[282486]: [🡕] Process 1124111 (stunnel) of user 988 dumped core. Module libgcc_s.so.1 without build-id. Module libcap.so.2 without build-id. Module stunnel without build-id. Stack trace of thread 1124120: #0 0x00007f9191c99a9c __pthread_kill_implementation (libc.so.6 + 0x92a9c) #1 0x00007f9191c47576 raise (libc.so.6 + 0x40576) #2 0x00007f9191c2f935 abort (libc.so.6 + 0x28935) #3 0x000056142e23e435 fatal_debug.cold (stunnel + 0xc435) #4 0x000056142e23eebc get_alloc_list_ptr (stunnel + 0xcebc) #5 0x000056142e23f0b9 str_detach_debug.part.0 (stunnel + 0xd0b9) #6 0x000056142e23f856 str_free_debug (stunnel + 0xd856) #7 0x000056142e259635 ocsp_stapling (stunnel + 0x27635) #8 0x000056142e25a18f per_minute_thread (stunnel + 0x2818f) #9 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #10 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) Stack trace of thread 676245: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e241f82 client_run (stunnel + 0xff82) #3 0x000056142e2438fc client_thread (stunnel + 0x118fc) #4 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #5 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) Stack trace of thread 1124111: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e25b5f6 daemon_loop (stunnel + 0x295f6) #3 0x000056142e23e680 main (stunnel + 0xc680) #4 0x00007f9191c3127e __libc_start_call_main (libc.so.6 + 0x2a27e) #5 0x00007f9191c31339 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a339) #6 0x000056142e23e875 _start (stunnel + 0xc875) Stack trace of thread 1124119: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e250acd s_poll_sleep (stunnel + 0x1eacd) #3 0x000056142e25a21c per_second_thread (stunnel + 0x2821c) #4 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #5 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) Stack trace of thread 676190: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e241f82 client_run (stunnel + 0xff82) #3 0x000056142e2438fc client_thread (stunnel + 0x118fc) #4 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #5 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) Stack trace of thread 1124121: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e250acd s_poll_sleep (stunnel + 0x1eacd) #3 0x000056142e259fab per_day_thread (stunnel + 0x27fab) #4 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #5 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) Stack trace of thread 676445: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e241f82 client_run (stunnel + 0xff82) #3 0x000056142e2438fc client_thread (stunnel + 0x118fc) #4 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #5 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) Stack trace of thread 676300: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e241f82 client_run (stunnel + 0xff82) #3 0x000056142e2438fc client_thread (stunnel + 0x118fc) #4 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #5 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) Stack trace of thread 676491: #0 0x00007f9191d090af __poll (libc.so.6 + 0x1020af) #1 0x000056142e2509f1 s_poll_wait (stunnel + 0x1e9f1) #2 0x000056142e241f82 client_run (stunnel + 0xff82) #3 0x000056142e2438fc client_thread (stunnel + 0x118fc) #4 0x00007f9191c97d02 start_thread (libc.so.6 + 0x90d02) #5 0x00007f9191d173ac __clone3 (libc.so.6 + 0x1103ac) ELF object binary architecture: AMD x86-64 Dec 06 04:44:02 shell systemd[1]: stunnel.service: Main process exited, code=dumped, status=6/ABRT Dec 06 04:44:02 shell systemd[1]: stunnel.service: Failed with result 'core-dump'. Dec 06 04:44:02 shell systemd[1]: stunnel.service: Consumed 2min 11.812s CPU time, 21.6M memory peak, 19.6M memory swap peak, 50.9M read from disk, 47.1M written to disk, 587.2M incoming IP traffic, 588.8M outgoing IP traffic. ``` Note that stunnel is handling mTLS for my prometheus, which means that it is indeed getting hammered by one scrape per exporter at a time, which in this case would make exactly five scrapes per minute but all of them happening at the same time, which, if this is a race, does make me a lot more likely to hit this issue. Also a huge Thank You for building what has been an integral part of my infrastructure for almost a decade now.

3 4

stunnel 5.74 released
by Michał Trojnara 13 Dec '24

13 Dec '24

Dear Users, I have released version 5.74 of stunnel. ### Version 5.74, 2024.12.13, urgency: HIGH * Bugfixes - Fixed a stapling cache deallocation crash. - Fixed "redirect" with protocol negotiation. * Features - "protocolHost" support for "socks" protocol clients. - More detailed logs in OpenSSL 3.0 or later. Home page: https://www.stunnel.org/ Download: https://www.stunnel.org/downloads.html SHA-256 hashes: 9bef235ab5d24a2a8dff6485dfd782ed235f4407e9bc8716deb383fc80cd6230 stunnel-5.74.tar.gz 8ce19cc782a64b0cacf32356249265ed16b3888e3578454853f5497726778d76 stunnel-5.74-win64-installer.exe 68c3395c1bebcc8b4eee2c7f58190aa2fe39b835dd7a4cce5ad37d5dfc1c5542 stunnel-5.74-android.zip Best regards, Mike

1 0

Stunnel on a windwos 11 pro installation not working
by ron.reijgwart＠gmail.com 09 Dec '24

09 Dec '24

I'm trying to use Stunnel on a windwos 11 pro installation. I tried version 5.56, 5.73 and 5.74b1. On my old pc all was working fine but now I get the following errors: [ ] Initializing inetd mode configuration [ ] Running on Windows 6.2 [ ] No limit detected for the number of clients [.] stunnel 5.73 on x64-pc-mingw32-gnu platform [.] Compiled/running with OpenSSL 3.3.2 3 Sep 2024 [.] Threading:WIN32 Sockets:SELECT,IPv6 TLS:ENGINE,FIPS,OCSP,PSK,SNI [ ] errno: (*_errno()) [ ] Initializing inetd mode configuration [ ] Running on Windows 6.2 [.] Reading configuration from file C:\Program Files (x86)\stunnel\config\stunnel.conf [.] UTF-8 byte order mark detected [.] FIPS mode disabled [ ] Compression disabled [ ] No PRNG seeding was required [ ] Initializing service [ssmtp] [ ] Initializing context [ssmtp] [ ] OpenSSL security level is used: 2 [ ] Ciphers: HIGH:!aNULL:!SSLv2:!DH:!kDHEPSK [ ] TLSv1.3 ciphersuites: TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256 [ ] TLS options: 0x2100000 (+0x0, -0x0) [ ] Session resumption enabled [ ] Loading certificate from file: C:\Program Files (x64)\stunnel\config\stunnel.pem [!] error queue: ssl/ssl_rsa.c:472: error:0A080002:SSL routines::system lib [!] error queue: crypto/bio/bss_file.c:300: error:10080002:BIO routines::system lib [!] SSL_CTX_use_certificate_chain_file: crypto/bio/bss_file.c:297: error:80000003:system library::No such process [!] Service [ssmtp]: Failed to initialize TLS context [!] Configuration failed [ ] Deallocating temporary section defaults [ ] Cleaning up context [(null)] [ ] Deallocating section [ssmtp] [ ] Cleaning up context [ssmtp] On a windows 10 Education installation it works fine with this logging: 2024.12.05 13:38:58 LOG5[service]: Reading configuration from file C:\Program Files (x86)\stunnel\config\stunnel.conf 2024.12.05 13:38:58 LOG5[service]: UTF-8 byte order mark detected 2024.12.05 13:38:58 LOG5[service]: FIPS mode disabled 2024.12.05 13:38:58 LOG4[service]: Service [ssmtp] needs authentication to prevent MITM attacks 2024.12.05 13:38:58 LOG5[service]: Configuration successful My conf-file looks like this: ;use it for client mode client = yes [ssmtp] accept = 1925 cert = C:\Program Files (x64)\stunnel\config\stunnel.pem connect = smtp.bhosted.nl:465 Can you please give me some advice?

2 2

Is it possible for stunnel to reject client connections if the server end cannot connect?
by mohmmad170720＠gmail.com 05 Dec '24

05 Dec '24

Hi all Say we have two machines set up, ClientPC and ServerPC. On ClientPC I run stunnel accepting a connection on a local port and connecting to an stunnel port on ServerPC. On ServerPC I run stunnel accepting a connection on the local stunnel port and connecting to the 'encryption unaware' server on another port on the same ServerPC machine. OK, pretty simple stuff so far. The problem is, when I connect to ClientPC with the 'encryption unaware' client app. then I always get a successful connection, even if there is no server running on ServerPC. Of course once the client has a successful connection, it then performs a write(), and that fails with 'Connection reset by peer'. Really, since there is no server running on ServerPC, I want the connect() to stunnel on ClientPC to fail. Is there a way to do this?

1 0

Router + Cudy mesh system
by trashrap22＠gmail.com 26 Nov '24

26 Nov '24

Hi, My stunnel setup was working. I since moved and had to add a Cudy mesh system / extension to the router to extend the internet through the building. Now I can't get stunnel to work again - I have done port forwarding on the Cudy system too. Has anyone done this extentions of router setup and got stunnel to work?

1 0

Requests to cloud server that requires host header
by Lorne Kates 13 Nov '24

13 Nov '24

(related to Akamai message from before-- but I have better troubleshooting information). I'm tying to route traffic through stunnel to a "cloud" based-endpoint. That endpoint has a static server name-- test.authorize.net. (This is the dev sandbox for auth.net) But if you do an nslookup on test.authorize.net, you'll get back a different servername and IP, because it's so wonderfully "cloud". Stunnel apparently tries to connect to the nslookup value. The server rejects the request because it can't route it back to test.authorize.net. I've tried adding "delay = yes" and "sni = test.authorize.net", but neither work. To see this in action, a simple setup with any accept, then connect to test.authorize.net:443 in client = yes mode. This is what a valid response looks like (13 -- give me the darn merchant ID in a POST): https://test.authorize.net/gateway/transact.dll This is what you'll get if you try to use stunnel (400 invalid url) : https://23.195.204.150/gateway/transact.dll So how can I get stunnel to send the proper Request Header (host: test.authorize.net) make sure it's using http/1.1, etc?

4 3

Is it possible to hide "Acceptable client certificate CA names" during SSL handshaking?
by hshh 12 Nov '24

12 Nov '24

Since version 5.68, when stunnel is configured with verifyPeer=yes, CApath=/path and CAfile=/path/file, the 'Acceptable client certificate CA names' and required certificates are displayed during SSL handshaking. Is it possible to hide this information?

3 4

s_ssl_read_eof: SSL_ERROR_WANT_READ: Retrying
by stavultras＠gmail.com 07 Nov '24

07 Nov '24

Hello, i've encountered a very strange issue when you use stunnel 7.53 (doesn't matter actually, happens in 5.68 as well) like this: [server] accept = 443 protocol = socks [client] accept = 1080 protocol = socks connect = server:443 with iptables -A OUTPUT -p tcp -d 10.8.0.1 -j REDIRECT --to-ports 1080, I see in 95% cases in logs for every request from curl: curl http://10.8.0.1 (server's internal IP), however, it never happens if you use client like this: [client] accept = 1080 connect = server:443 and curl -x socks5://127.0.0.1:1080 http://10.8.0.1 (internal server's IP). Technically it affects nothing, but I still guess there's a small time overhead due to retry. Maybe i'm wrong. Please advise. Thanks.

1 0

Re: [stunnel-users] CPU 100%
by Eric Eberhard 06 Nov '24

06 Nov '24

Long ago I had a server with a bad network card. Every once in a while it would spew uncontrollable bytes onto the network. If you open a stunnel connection and it does that this could very well be the problem. If it cheap, try changing the network card. If not, you should be able to look in your firewall/router and it will tell you who is sending what bytes to whomever. See if that machine is indeed streaming the bytes in massive quantities. If not, I would suspect your stunnel is getting stuck in it's own loop. There are a million reasons this can happen (well a few less than a million) - such as maybe a bad library in the FreeBSD or a bad choice in the Makefile - such as doing threads or something in a way your O/S does not like (I use threads=fork or whatever it is, instead of actual threading). You might tinker with some of the options that make the stunnel build and try different ones. Especially if stunnel was compiled on a different version of the O/S - could be differences in bytes in things and who knows what. That is all I can suggest beyond setting the debug level to 7 (I think is highest) and then watching the log file. Eric From: stunnel-users [mailto:stunnel-users-bounces@stunnel.org] On Behalf Of Murrey, Brian J. Sent: Wednesday, October 17, 2018 10:57 AM To: stunnel-users(a)stunnel.org Subject: [stunnel-users] CPU 100% We are running stunnel 5.49 on FreeBSD 11.2 and we're running into a problem once in a while. CPU pegs at 100% when we get an stunnel connection from one of our external devices. This affects 100% of all cores and we can't even log in to console. To mitigate this temporarily, we have a script running in the background to watch when stunnel begins to spike and restarts the daemon. It doesn't happen 100% of the time, and so far I have been unable to distinguish what makes the connection do this to the CPU. Have any of you run in to this issue? Sincerely, Brian Murrey System Engineer II, IT Infrastructure _____ NOTICE: This message may contain privileged and confidential information and/or protected health information intended solely for the use of the named recipient and may be privileged or otherwise protected by law. If you are not the intended recipient of this message, you should immediately notify the sender and delete this message. Do not disseminate, reproduce, or review this message or attachments if you are not the intended recipient. The sender or others may have legal rights restricting the dissemination of the information contained in this message and, as a result, remedies against you in the event of the improper dissemination of confidential information, trade secrets, personal information or privileged communications. This message is the work of the sender and does not necessarily reflect the position, views, or policies of TriMedx LLC or its affiliates. WARNING: The integrity and security of this message cannot be guaranteed and may contain or transmit a virus or other illicit code. Neither TriMedx LLC or its affiliates accept liability for any damage attributable to viruses or illicit code transmitted through this message or an attachment.

2 1

socks mode doesn't verify peer
by stavultras＠gmail.com 05 Nov '24

05 Nov '24

Hello. I'm trying to configure "protocol = socks" with "redirect = 127.0.0.1:80" and "verifyPeer = yes", but accordingly to the debug logs stunnel doesn't verify peer when I'm trying to connect without client's certificate, for an example, from browser: https://192.168.200.131:9013/, where 192.168.200.131 is stunnel server. Config file: [socks] cert = /usr/local/etc/stunnel/server.pem CAfile = /usr/local/etc/stunnel/client.crt verifyPeer = yes redirect = 127.0.0.1:80 accept = 0.0.0.0:9013 protocol = socks Debug log: 2024.10.02 07:24:19 LOG6[9]: No peer certificate received 2024.10.02 07:24:19 LOG6[9]: Session id: AFA5F9CE2997328612F76E3B3572FE00C71C53ED31A98567E56DCC084EF580AD 2024.10.02 07:24:19 LOG7[9]: TLS state (accept): SSLv3/TLS write session ticket 2024.10.02 07:24:19 LOG6[9]: TLS accepted: new session negotiated 2024.10.02 07:24:19 LOG6[9]: TLSv1.3 ciphersuite: TLS_AES_128_GCM_SHA256 (128-bit encryption) 2024.10.02 07:24:19 LOG6[9]: Peer temporary key: X25519, 253 bits 2024.10.02 07:24:19 LOG7[9]: Compression: null, expansion: null 2024.10.02 07:24:19 LOG7[9]: Waiting for the SOCKS request 2024.10.02 07:24:19 LOG3[9]: Unsupported SOCKS version 0x47 2024.10.02 07:24:19 LOG5[9]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket It tries to accept SOCKS info even without certificate and doesn't redirect to 127.0.0.1:80. In browser, I'm getting: Secure Connection Failed An error occurred during a connection to 192.168.200.131:9013. PR_CONNECT_RESET_ERROR Error code: PR_CONNECT_RESET_ERROR Can you check why it happens? Thank you.

1 9