Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections
I’ve been digging around the code and I am no longer sure whether stunnel is at fault or it is actually an issue in OpenSSL! load_key_engine() in ctx.c shoves the key into the SSL context – SSL_CTX_use_PrivateKey(section->ctx, pkey). So the key should be freed when that context is disposed of. The SIGHUP ends up in case SIGNAL_RELOAD_CONFIG of signal_pipe_dispatch(), which calls options_free() [stunnel.c line 830]. Eventually there is an SSL_CTX_free() call, but it looks like this does not actually free the private key. Ephemeral keys are freed and loads of other stuff, but not the private key associated with the certificate. I may try to build a hacked OpenSSL on Monday to see what happens if SSL_CTX_free also frees the private key… Regards, Andrew. From: Brent Kimberley [mailto:brent_kimberley@rogers.com] Sent: Friday, September 27, 2019 4:29 PM To: stunnel-users@stunnel.org; Lynch, Andrew Subject: Re: stunnel-users Digest, Vol 182, Issue 14
Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded.
Where does the signal dispatcher free keys on SIG_HUP? https://github.com/mtrojnar/stunnel/blob/master/src/stunnel.c Date: Fri, 27 Sep 2019 07:23:29 +0000 From: "Lynch, Andrew" <andrew.lynch@atos.net<mailto:andrew.lynch@atos.net>> To: Eric Eberhard <flash@vicsmba.com<mailto:flash@vicsmba.com>> Cc: "stunnel-users@stunnel.org<mailto:stunnel-users@stunnel.org>" <stunnel-users@stunnel.org<mailto:stunnel-users@stunnel.org>> Subject: Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections Hi Eric, Thank you for your suggestion. It is certainly worth looking into, although I suspect it may be impractical in our environment. Now we have a single stunnel.conf with 70 services which would translate into 70 additional entries in inetd.conf. Each of those entries then needs to have its own reduced stunnel.conf so that it will only load the cert and key for its endpoint. The stunnel HOWTO recommends daemon mode over inetd mode due to the overhead for forking and SSL initialization. We would have additional overhead for engine init and HSM connection. Regards, Andrew. -----Original Message----- From: Eric Eberhard [mailto:flash@vicsmba.com<mailto:flash@vicsmba.com>] Sent: Thursday, September 26, 2019 7:20 PM To: Lynch, Andrew; stunnel-users@stunnel.org<mailto:stunnel-users@stunnel.org> Subject: RE: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Try running in inetd mode -- even if you don't like this you will learn something. Inetd will close connections as needed. E -----Original Message----- From: stunnel-users [mailto:stunnel-users-bounces@stunnel.org<mailto:stunnel-users-bounces@stunnel.org>] On Behalf Of Lynch, Andrew Sent: Thursday, September 26, 2019 5:56 AM To: stunnel-users@stunnel.org<mailto:stunnel-users@stunnel.org> Subject: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Hi, We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored in an HSM and accessed using an OpenSSL engine. Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the current version 5.55. Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates. Currently our workaround is to perform a full restart instead of a reload. This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled maintenance windows. Regards, Andrew. _______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org<mailto:stunnel-users@stunnel.org> https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ Subject: Digest Footer _______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org<mailto:stunnel-users@stunnel.org> https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ End of stunnel-users Digest, Vol 182, Issue 14 **********************************************
Sounds slightly related to "After generating a new key-pair to HSM, ENGINE_load_private_key() still returns the old key."https://stackoverflow.com/questions/40676573/how-to-reload-key-from-hsm-by-u... Suggestions at the time included calling ENGINE_cleanup(), ENGINE_finish() and ENGINE_init() They cleaned-up the engine and then re-opened openssl/crypto. It was slow and affected TLS connections using other key-pairs Consider running one service per port, or better yet, one service per port-connection. On Friday, September 27, 2019, 12:57:48 p.m. EDT, Lynch, Andrew <andrew.lynch@atos.net> wrote: <!--#yiv5227285775 _filtered #yiv5227285775 {font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv5227285775 {font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;} _filtered #yiv5227285775 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv5227285775 {font-family:Tahoma;panose-1:2 11 6 4 3 5 4 4 2 4;}#yiv5227285775 #yiv5227285775 p.yiv5227285775MsoNormal, #yiv5227285775 li.yiv5227285775MsoNormal, #yiv5227285775 div.yiv5227285775MsoNormal {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:"Times New Roman", "serif";}#yiv5227285775 a:link, #yiv5227285775 span.yiv5227285775MsoHyperlink {color:blue;text-decoration:underline;}#yiv5227285775 a:visited, #yiv5227285775 span.yiv5227285775MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv5227285775 span.yiv5227285775E-MailFormatvorlage17 {font-family:"Calibri", "sans-serif";color:#1F497D;}#yiv5227285775 .yiv5227285775MsoChpDefault {font-size:10.0pt;} _filtered #yiv5227285775 {margin:70.85pt 70.85pt 2.0cm 70.85pt;}#yiv5227285775 div.yiv5227285775WordSection1 {}--> I’ve been digging around the code and I am no longer sure whether stunnel is at fault or it is actually an issue in OpenSSL! load_key_engine() in ctx.c shoves the key into the SSL context – SSL_CTX_use_PrivateKey(section->ctx, pkey). So the key should be freed when that context is disposed of. The SIGHUP ends up in case SIGNAL_RELOAD_CONFIG of signal_pipe_dispatch(), which calls options_free() [stunnel.c line 830]. Eventually there is an SSL_CTX_free() call, but it looks like this does not actually free the private key. Ephemeral keys are freed and loads of other stuff, but not the private key associated with the certificate. I may try to build a hacked OpenSSL on Monday to see what happens if SSL_CTX_free also frees the private key… Regards, Andrew. | | | From: Brent Kimberley [mailto:brent_kimberley@rogers.com] Sent: Friday, September 27, 2019 4:29 PM To: stunnel-users@stunnel.org; Lynch, Andrew Subject: Re: stunnel-users Digest, Vol 182, Issue 14
Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full.
Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded.
Where does the signal dispatcher free keys on SIG_HUP? https://github.com/mtrojnar/stunnel/blob/master/src/stunnel.c Date: Fri, 27 Sep 2019 07:23:29 +0000 From: "Lynch, Andrew" <andrew.lynch@atos.net> To: Eric Eberhard <flash@vicsmba.com> Cc: "stunnel-users@stunnel.org" <stunnel-users@stunnel.org> Subject: Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections Hi Eric, Thank you for your suggestion. It is certainly worth looking into, although I suspect it may be impractical in our environment. Now we have a single stunnel.conf with 70 services which would translate into 70 additional entries in inetd.conf. Each of those entries then needs to have its own reduced stunnel.conf so that it will only load the cert and key for its endpoint. The stunnel HOWTO recommends daemon mode over inetd mode due to the overhead for forking and SSL initialization. We would have additional overhead for engine init and HSM connection. Regards, Andrew. -----Original Message----- From: Eric Eberhard [mailto:flash@vicsmba.com] Sent: Thursday, September 26, 2019 7:20 PM To: Lynch, Andrew;stunnel-users@stunnel.org Subject: RE: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Try running in inetd mode -- even if you don't like this you will learn something. Inetd will close connections as needed. E -----Original Message----- From: stunnel-users [mailto:stunnel-users-bounces@stunnel.org] On Behalf Of Lynch, Andrew Sent: Thursday, September 26, 2019 5:56 AM To:stunnel-users@stunnel.org Subject: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Hi, We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored in an HSM and accessed using an OpenSSL engine. Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the current version 5.55. Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates. Currently our workaround is to perform a full restart instead of a reload. This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled maintenance windows. Regards, Andrew. _______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ Subject: Digest Footer _______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ End of stunnel-users Digest, Vol 182, Issue 14 **********************************************
participants (2)
-
Brent Kimberley -
Lynch, Andrew