<html><head></head><body><div class="ydp25dd2242yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div></div>
<div dir="ltr" data-setdir="false">Sounds slightly related to "<span><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif; font-size: 15px;">After generating a new key-pair to HSM, </span><code style="margin: 0px; padding: 1px 5px; border: 0px; font-stretch: inherit; line-height: inherit; font-family: Consolas, Menlo, Monaco, Lucida Console, Liberation Mono, DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace, sans-serif; vertical-align: baseline; background-color: rgb(239, 240, 241); white-space: pre-wrap; color: rgb(36, 39, 41);">ENGINE_load_private_key()</code><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif; font-size: 15px;"> still returns the old key."</span></span></div><div dir="ltr" data-setdir="false"><a href="https://stackoverflow.com/questions/40676573/how-to-reload-key-from-hsm-by-using-openssl" rel="nofollow" target="_blank">https://stackoverflow.com/questions/40676573/how-to-reload-key-from-hsm-by-using-openssl</a><br></div><div dir="ltr" data-setdir="false"><span><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif; font-size: 15px;"><br></span></span></div><div dir="ltr" data-setdir="false"><span><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif; font-size: 15px;"><span><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif;">Suggestions at the time included calling </span></span></span></span><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif; font-size: 15px;">ENGINE_cleanup(), ENGINE_finish() and ENGINE_init() </span></div><div dir="ltr" data-setdir="false"><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif; font-size: 15px;">They cleaned-up the engine and then re-opened openssl/crypto. </span><span style="color: rgb(36, 39, 41); font-family: Arial, Helvetica Neue, Helvetica, sans-serif; font-size: 15px;">It was slow and affected TLS connections using other key-pairs</span></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Consider running one service per port, or better yet, one service per port-connection.</div><div dir="ltr" data-setdir="false"><br><div><br></div><div><br></div><div><br></div><br></div><div><br></div>
</div><div id="yahoo_quoted_0512947318" class="yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Friday, September 27, 2019, 12:57:48 p.m. EDT, Lynch, Andrew <andrew.lynch@atos.net> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="yiv5227285775">
<style><!--
#yiv5227285775
_filtered #yiv5227285775 {font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
_filtered #yiv5227285775 {font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
_filtered #yiv5227285775 {font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
_filtered #yiv5227285775 {font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
#yiv5227285775
#yiv5227285775 p.yiv5227285775MsoNormal, #yiv5227285775 li.yiv5227285775MsoNormal, #yiv5227285775 div.yiv5227285775MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman", "serif";}
#yiv5227285775 a:link, #yiv5227285775 span.yiv5227285775MsoHyperlink
{
color:blue;
text-decoration:underline;}
#yiv5227285775 a:visited, #yiv5227285775 span.yiv5227285775MsoHyperlinkFollowed
{
color:purple;
text-decoration:underline;}
#yiv5227285775 span.yiv5227285775E-MailFormatvorlage17
{
font-family:"Calibri", "sans-serif";
color:#1F497D;}
#yiv5227285775 .yiv5227285775MsoChpDefault
{
font-size:10.0pt;}
_filtered #yiv5227285775 {
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
#yiv5227285775 div.yiv5227285775WordSection1
{}
--></style>
<div>
<div class="yiv5227285775WordSection1">
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;">I’ve been digging around the code and I am no longer sure whether stunnel is at fault or it is actually an issue in OpenSSL!</span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;"> </span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;">load_key_engine() in ctx.c shoves the key into the SSL context – SSL_CTX_use_PrivateKey(section->ctx, pkey). So the key should be freed when that
context is disposed of.</span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;"> </span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;">The SIGHUP ends up in case SIGNAL_RELOAD_CONFIG of signal_pipe_dispatch(), which calls options_free() [stunnel.c line 830].</span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;"> </span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;">Eventually there is an SSL_CTX_free() call, but it looks like this does not actually free the private key. Ephemeral keys are freed and loads
of other stuff, but not the private key associated with the certificate.</span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;"> </span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;">I may try to build a hacked OpenSSL on Monday to see what happens if SSL_CTX_free also frees the private key…</span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;"> </span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;">Regards,</span></p>
<p class="yiv5227285775MsoNormal"><span lang="EN-US" style="font-size:11.0pt;color:#1F497D;">Andrew.</span></p>
<div>
<table class="yiv5227285775MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;border-collapse:collapse;">
<tbody>
<tr>
<td width="20" style="width:15.3pt;padding:0cm 15.0pt 0cm 0cm;"></td>
<td width="1177" style="width:882.45pt;border:none;border-left:solid gray 1.5pt;padding:0cm 0cm 0cm 18.75pt;">
</td>
</tr>
</tbody>
</table>
<p class="yiv5227285775MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></p>
</div>
<p class="yiv5227285775MsoNormal"> </p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm;">
<p class="yiv5227285775MsoNormal"><b><span style="font-size:10.0pt;">From:</span></b><span style="font-size:10.0pt;"> Brent Kimberley [mailto:brent_kimberley@rogers.com]
<br>
<b>Sent:</b> Friday, September 27, 2019 4:29 PM<br>
<b>To:</b> stunnel-users@stunnel.org; Lynch, Andrew<br>
<b>Subject:</b> Re: stunnel-users Digest, Vol 182, Issue 14</span></p>
</div>
</div>
<p class="yiv5227285775MsoNormal"> </p>
<div>
<div>
<p class="yiv5227285775MsoNormal"><span style="">>> <span style="color:#26282A;">
Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. </span></span></p>
</div>
<div>
<div>
<div>
<p class="yiv5227285775MsoNormal"><span style="color:#26282A;">>> Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload
opens 70 new sessions until eventually the HSM's configured limit is exceeded.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="color:#26282A;"> </span></p>
</div>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="color:#26282A;">Where does the signal dispatcher free keys on SIG_HUP?</span></p>
</div>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style=""><a rel="nofollow" target="_blank" href="https://github.com/mtrojnar/stunnel/blob/master/src/stunnel.c">https://github.com/mtrojnar/stunnel/blob/master/src/stunnel.c</a></span></p>
</div>
<div>
<div>
<p class="yiv5227285775MsoNormal"><span style=""> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style=""> </span></p>
</div>
</div>
</div>
<div id="yiv5227285775ydpaf216824yahoo_quoted_0356605184">
<div>
<div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Date: Fri, 27 Sep 2019 07:23:29 +0000</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">From: "Lynch, Andrew" <<a rel="nofollow" ymailto="mailto:andrew.lynch@atos.net" target="_blank" href="mailto:andrew.lynch@atos.net">andrew.lynch@atos.net</a>></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">To: Eric Eberhard <<a rel="nofollow" ymailto="mailto:flash@vicsmba.com" target="_blank" href="mailto:flash@vicsmba.com">flash@vicsmba.com</a>></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Cc: "<a rel="nofollow" ymailto="mailto:stunnel-users@stunnel.org" target="_blank" href="mailto:stunnel-users@stunnel.org">stunnel-users@stunnel.org</a>" <<a rel="nofollow" ymailto="mailto:stunnel-users@stunnel.org" target="_blank" href="mailto:stunnel-users@stunnel.org">stunnel-users@stunnel.org</a>></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Subject: Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Hi Eric,</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Thank you for your suggestion. It is certainly worth looking into, although I suspect it may be impractical in our environment.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Now we have a single stunnel.conf with 70 services which would translate into 70 additional entries in inetd.conf. Each of those entries then needs to have
its own reduced stunnel.conf so that it will only load the cert and key for its endpoint.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">The stunnel HOWTO recommends daemon mode over inetd mode due to the overhead for forking and SSL initialization. We would have additional overhead for engine
init and HSM connection.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Regards,</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Andrew.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">-----Original Message-----</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">From: Eric Eberhard [mailto:<a rel="nofollow" ymailto="mailto:flash@vicsmba.com" target="_blank" href="mailto:flash@vicsmba.com">flash@vicsmba.com</a>]
</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Sent: Thursday, September 26, 2019 7:20 PM</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">To: Lynch, Andrew;
<a rel="nofollow" ymailto="mailto:stunnel-users@stunnel.org" target="_blank" href="mailto:stunnel-users@stunnel.org">stunnel-users@stunnel.org</a></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Subject: RE: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections?</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Try running in inetd mode -- even if you don't like this you will learn something. Inetd will close connections as needed. E</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">-----Original Message-----</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">From: stunnel-users [mailto:<a rel="nofollow" ymailto="mailto:stunnel-users-bounces@stunnel.org" target="_blank" href="mailto:stunnel-users-bounces@stunnel.org">stunnel-users-bounces@stunnel.org</a>] On Behalf Of Lynch,
Andrew</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Sent: Thursday, September 26, 2019 5:56 AM</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">To:
<a rel="nofollow" ymailto="mailto:stunnel-users@stunnel.org" target="_blank" href="mailto:stunnel-users@stunnel.org">stunnel-users@stunnel.org</a></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Subject: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections?</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Hi,</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored
in an HSM and accessed using an OpenSSL engine.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is
never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the
current version 5.55.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Currently our workaround is to perform a full restart instead of a reload.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled
maintenance windows.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Regards,</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Andrew.</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">_______________________________________________</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">stunnel-users mailing list</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"><a rel="nofollow" ymailto="mailto:stunnel-users@stunnel.org" target="_blank" href="mailto:stunnel-users@stunnel.org">stunnel-users@stunnel.org</a></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"><a rel="nofollow" target="_blank" href="https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users">https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users</a></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">------------------------------</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">Subject: Digest Footer</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">_______________________________________________</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">stunnel-users mailing list</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"><a rel="nofollow" ymailto="mailto:stunnel-users@stunnel.org" target="_blank" href="mailto:stunnel-users@stunnel.org">stunnel-users@stunnel.org</a></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"><a rel="nofollow" target="_blank" href="https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users">https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users</a></span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">------------------------------</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;"> </span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">End of stunnel-users Digest, Vol 182, Issue 14</span></p>
</div>
<div>
<p class="yiv5227285775MsoNormal"><span style="font-size:10.0pt;color:#26282A;">**********************************************</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div></div>
</div>
</div></body></html>