Somewhere there is a physical network card - checking the firewall/router for traffic is a good idea no matter what it is running on. VMs have lots of clever software to isolate each instance (including resources) and you may be being streamed at a rate the VM as whole can handle, and your isolated VM cannot. And setting the debug level up is still a good idea. And forcing core files and using a debugger is still a good idea. And trying inetd mode is still a good idea. You need to narrow your focus and each of these suggestions will tell you something that narrows the problem. Eric
From: Murrey, Brian J. [mailto:Brian.Murrey@trimedx.com] Sent: Thursday, October 18, 2018 7:14 AM To: Eric Eberhard flash@vicsmba.com; stunnel-users@stunnel.org Subject: RE: [stunnel-users] CPU 100%
Thank you Eric, this server is a VM running under vSphere 6.5 and has no physical network card.
From: Eric Eberhard <flash@vicsmba.com mailto:flash@vicsmba.com > Sent: Wednesday, October 17, 2018 7:56 PM To: Murrey, Brian J. <Brian.Murrey@trimedx.com mailto:Brian.Murrey@trimedx.com >; stunnel-users@stunnel.org mailto:stunnel-users@stunnel.org Subject: RE: [stunnel-users] CPU 100%
Long ago I had a server with a bad network card. Every once in a while it would spew uncontrollable bytes onto the network. If you open a stunnel connection and it does that this could very well be the problem. If it cheap, try changing the network card. If not, you should be able to look in your firewall/router and it will tell you who is sending what bytes to whomever. See if that machine is indeed streaming the bytes in massive quantities. If not, I would suspect your stunnel is getting stuck in it's own loop. There are a million reasons this can happen (well a few less than a million) - such as maybe a bad library in the FreeBSD or a bad choice in the Makefile - such as doing threads or something in a way your O/S does not like (I use threads=fork or whatever it is, instead of actual threading). You might tinker with some of the options that make the stunnel build and try different ones. Especially if stunnel was compiled on a different version of the O/S - could be differences in bytes in things and who knows what.
That is all I can suggest beyond setting the debug level to 7 (I think is highest) and then watching the log file.
Eric
From: stunnel-users [mailto:stunnel-users-bounces@stunnel.org] On Behalf Of Murrey, Brian J. Sent: Wednesday, October 17, 2018 10:57 AM To: stunnel-users@stunnel.org mailto:stunnel-users@stunnel.org Subject: [stunnel-users] CPU 100%
We are running stunnel 5.49 on FreeBSD 11.2 and we're running into a problem once in a while.
CPU pegs at 100% when we get an stunnel connection from one of our external devices.
This affects 100% of all cores and we can't even log in to console.
To mitigate this temporarily, we have a script running in the background to watch when stunnel begins to spike and restarts the daemon.
It doesn't happen 100% of the time, and so far I have been unable to distinguish what makes the connection do this to the CPU.
Have any of you run in to this issue?
Sincerely,
Brian Murrey System Engineer II, IT Infrastructure
_____
NOTICE: This message may contain privileged and confidential information and/or protected health information intended solely for the use of the named recipient and may be privileged or otherwise protected by law. If you are not the intended recipient of this message, you should immediately notify the sender and delete this message. Do not disseminate, reproduce, or review this message or attachments if you are not the intended recipient. The sender or others may have legal rights restricting the dissemination of the information contained in this message and, as a result, remedies against you in the event of the improper dissemination of confidential information, trade secrets, personal information or privileged communications. This message is the work of the sender and does not necessarily reflect the position, views, or policies of TriMedx LLC or its affiliates.
WARNING: The integrity and security of this message cannot be guaranteed and may contain or transmit a virus or other illicit code. Neither TriMedx LLC or its affiliates accept liability for any damage attributable to viruses or illicit code transmitted through this message or an attachment.
_____
CAUTION: This email originated from outside of the organization. Do not open links or attachments you were not expecting, even from known senders, and use caution when responding. Please contact the Service Desk if in doubt regarding the legitimacy of this email.
_____
NOTICE: This message may contain privileged and confidential information and/or protected health information intended solely for the use of the named recipient and may be privileged or otherwise protected by law. If you are not the intended recipient of this message, you should immediately notify the sender and delete this message. Do not disseminate, reproduce, or review this message or attachments if you are not the intended recipient. The sender or others may have legal rights restricting the dissemination of the information contained in this message and, as a result, remedies against you in the event of the improper dissemination of confidential information, trade secrets, personal information or privileged communications. This message is the work of the sender and does not necessarily reflect the position, views, or policies of TriMedx LLC or its affiliates.
WARNING: The integrity and security of this message cannot be guaranteed and may contain or transmit a virus or other illicit code. Neither TriMedx LLC or its affiliates accept liability for any damage attributable to viruses or illicit code transmitted through this message or an attachment.
Thanks again Eric...we are looking in to all these ideas today.
From: Eric Eberhard flash@vicsmba.com Sent: Thursday, October 18, 2018 3:07 PM To: Murrey, Brian J. Brian.Murrey@trimedx.com; stunnel-users@stunnel.org Subject: RE: [stunnel-users] CPU 100%
Somewhere there is a physical network card - checking the firewall/router for traffic is a good idea no matter what it is running on. VMs have lots of clever software to isolate each instance (including resources) and you may be being streamed at a rate the VM as whole can handle, and your isolated VM cannot. And setting the debug level up is still a good idea. And forcing core files and using a debugger is still a good idea. And trying inetd mode is still a good idea. You need to narrow your focus and each of these suggestions will tell you something that narrows the problem. Eric
From: Murrey, Brian J. [mailto:Brian.Murrey@trimedx.com] Sent: Thursday, October 18, 2018 7:14 AM To: Eric Eberhard <flash@vicsmba.commailto:flash@vicsmba.com>; stunnel-users@stunnel.orgmailto:stunnel-users@stunnel.org Subject: RE: [stunnel-users] CPU 100%
Thank you Eric, this server is a VM running under vSphere 6.5 and has no physical network card.
From: Eric Eberhard <flash@vicsmba.commailto:flash@vicsmba.com> Sent: Wednesday, October 17, 2018 7:56 PM To: Murrey, Brian J. <Brian.Murrey@trimedx.commailto:Brian.Murrey@trimedx.com>; stunnel-users@stunnel.orgmailto:stunnel-users@stunnel.org Subject: RE: [stunnel-users] CPU 100%
Long ago I had a server with a bad network card. Every once in a while it would spew uncontrollable bytes onto the network. If you open a stunnel connection and it does that this could very well be the problem. If it cheap, try changing the network card. If not, you should be able to look in your firewall/router and it will tell you who is sending what bytes to whomever. See if that machine is indeed streaming the bytes in massive quantities. If not, I would suspect your stunnel is getting stuck in it's own loop. There are a million reasons this can happen (well a few less than a million) - such as maybe a bad library in the FreeBSD or a bad choice in the Makefile - such as doing threads or something in a way your O/S does not like (I use threads=fork or whatever it is, instead of actual threading). You might tinker with some of the options that make the stunnel build and try different ones. Especially if stunnel was compiled on a different version of the O/S - could be differences in bytes in things and who knows what.
That is all I can suggest beyond setting the debug level to 7 (I think is highest) and then watching the log file.
Eric
From: stunnel-users [mailto:stunnel-users-bounces@stunnel.org] On Behalf Of Murrey, Brian J. Sent: Wednesday, October 17, 2018 10:57 AM To: stunnel-users@stunnel.orgmailto:stunnel-users@stunnel.org Subject: [stunnel-users] CPU 100%
We are running stunnel 5.49 on FreeBSD 11.2 and we're running into a problem once in a while.
CPU pegs at 100% when we get an stunnel connection from one of our external devices. This affects 100% of all cores and we can't even log in to console.
To mitigate this temporarily, we have a script running in the background to watch when stunnel begins to spike and restarts the daemon.
It doesn't happen 100% of the time, and so far I have been unable to distinguish what makes the connection do this to the CPU.
Have any of you run in to this issue?
Sincerely,
Brian Murrey System Engineer II, IT Infrastructure
________________________________
NOTICE: This message may contain privileged and confidential information and/or protected health information intended solely for the use of the named recipient and may be privileged or otherwise protected by law. If you are not the intended recipient of this message, you should immediately notify the sender and delete this message. Do not disseminate, reproduce, or review this message or attachments if you are not the intended recipient. The sender or others may have legal rights restricting the dissemination of the information contained in this message and, as a result, remedies against you in the event of the improper dissemination of confidential information, trade secrets, personal information or privileged communications. This message is the work of the sender and does not necessarily reflect the position, views, or policies of TriMedx LLC or its affiliates.
WARNING: The integrity and security of this message cannot be guaranteed and may contain or transmit a virus or other illicit code. Neither TriMedx LLC or its affiliates accept liability for any damage attributable to viruses or illicit code transmitted through this message or an attachment.
________________________________
CAUTION: This email originated from outside of the organization. Do not open links or attachments you were not expecting, even from known senders, and use caution when responding. Please contact the Service Desk if in doubt regarding the legitimacy of this email.
________________________________
NOTICE: This message may contain privileged and confidential information and/or protected health information intended solely for the use of the named recipient and may be privileged or otherwise protected by law. If you are not the intended recipient of this message, you should immediately notify the sender and delete this message. Do not disseminate, reproduce, or review this message or attachments if you are not the intended recipient. The sender or others may have legal rights restricting the dissemination of the information contained in this message and, as a result, remedies against you in the event of the improper dissemination of confidential information, trade secrets, personal information or privileged communications. This message is the work of the sender and does not necessarily reflect the position, views, or policies of TriMedx LLC or its affiliates.
WARNING: The integrity and security of this message cannot be guaranteed and may contain or transmit a virus or other illicit code. Neither TriMedx LLC or its affiliates accept liability for any damage attributable to viruses or illicit code transmitted through this message or an attachment.
________________________________
CAUTION: This email originated from outside of the organization. Do not open links or attachments you were not expecting, even from known senders, and use caution when responding. Please contact the Service Desk if in doubt regarding the legitimacy of this email.
________________________________
NOTICE: This message may contain privileged and confidential information and/or protected health information intended solely for the use of the named recipient and may be privileged or otherwise protected by law. If you are not the intended recipient of this message, you should immediately notify the sender and delete this message. Do not disseminate, reproduce, or review this message or attachments if you are not the intended recipient. The sender or others may have legal rights restricting the dissemination of the information contained in this message and, as a result, remedies against you in the event of the improper dissemination of confidential information, trade secrets, personal information or privileged communications. This message is the work of the sender and does not necessarily reflect the position, views, or policies of TriMedx LLC or its affiliates.
WARNING: The integrity and security of this message cannot be guaranteed and may contain or transmit a virus or other illicit code. Neither TriMedx LLC or its affiliates accept liability for any damage attributable to viruses or illicit code transmitted through this message or an attachment.
-
Eric Eberhard -
Murrey, Brian J.