Hello,
I have noticed that stunnel will periodically core dump on AIX 5.3.
My configuration: stunnel 4.34 AIX 5.3 OpenSSL 0.9.8m
After some initial investigation, I did find the following post about stunnel core dumps on AIX 5.3:
http://www.stunnel.org/pipermail/stunnel-users/2007-April/001528.html
Some of the core dumps were definitely due to this issue, so I rebuilt stunnel using the steps outlined in the post. Now I do not see the stack overflow core dumps, however I am still seeing other cores.
These cores have little to no valuable information in them, so I do not have a lot to go off of. The one thing I have noticed is that stunnel seems to be consuming more and more memory over time. I set up several tests to see what would be causing this.
Tests:
1. Send large amounts of data through stunnel using a simple client and server. This seemed to have no affect on the long term memory usage of stunnel.
2. Repeatedly create and drop connections to stunnel. For this I wrote a short script that simply creates a connection to stunnel using telnet in a a tight loop. I did not start anything for the stunnel instance to connect to, so telnet would create a connection and get it closed immediately.
telnet --> stunnel --> nothing
The process started with a size of 836 KB and seems to increase by about .1 to .2 KB per connection.
$ count=1; while true; do tn localhost 10500; echo $count; count=$(( count + 1 )); done Trying... Escape character is '^T'. Connection closed. 1 ... Trying... Connected to loopback. Escape character is '^T'. Connection closed. 15475 Trying... Connected to loopback. Escape character is '^T'. Connection closed. 15476 Trying... Connected to loopback. Escape character is '^T'. Connection closed. 15477
$ monitor_process test_stunnel Size: 836 ... Size: 2556 Size: 2564 Size: 2564 Size: 2564 Size: 2568 Size: 2572 Size: 2572 Size: 2592
I have not dug into the code yet, so I suppose that will be my next task. If anyone has any ideas or suggestions on what else to look at, I would really appreciate it.
Thanks,
Paul Allex
Sorry for posting my follow up incorrectly.
Anyways, I believe I have located the memory leak. In the pthread version of create_client (sthreads.c), there is the following code:
int create_client(int ls, int s, CLI *arg, void *(*cli)(void *)) { pthread_attr_t pth_attr; ... pthread_attr_init(&pth_attr); ... if(pthread_create(&thread, &pth_attr, cli, arg)) { ... return 0; }
The issue here is that pthread_attr_destroy is not being called for pth_attr after pthread_create is done. This leaks memory allocated into pth_attr by pthread_attr_init.
http://www.kernel.org/doc/man-pages/online/pages/man3/pthread_attr_init. 3.html
I corrected the code locally and tested, and it appears to no longer leak memory for the test I described earlier.
Patch:
*** sthreads.orig.c Thu Mar 10 17:50:03 2011 --- sthreads.c Thu Mar 10 17:50:43 2011 *************** *** 313,318 **** --- 313,319 ---- if(pthread_create(&thread, &pth_attr, cli, arg)) { #ifdef HAVE_PTHREAD_SIGMASK pthread_sigmask(SIG_SETMASK, &oldmask, NULL); /* restore the mask */ + pthread_attr_destroy(&pth_attr); #endif /* HAVE_PTHREAD_SIGMASK */ if(arg) free(arg); *************** *** 322,327 **** --- 323,329 ---- } #ifdef HAVE_PTHREAD_SIGMASK pthread_sigmask(SIG_SETMASK, &oldmask, NULL); /* restore the mask */ + pthread_attr_destroy(&pth_attr); #endif /* HAVE_PTHREAD_SIGMASK */ return 0; }
Thanks,
Paul Allex
-----Original Message----- From: stunnel-users-bounces@stunnel.org [mailto:stunnel-users-bounces@stunnel.org] On Behalf Of Allex, Paul M (USA E D EA STS) Sent: Wednesday, March 09, 2011 3:30 PM To: stunnel-users@stunnel.org Subject: [stunnel-users] Periodic stunnel 4.34 core dumps on AIX 5.3
Hello,
I have noticed that stunnel will periodically core dump on AIX 5.3.
My configuration: stunnel 4.34 AIX 5.3 OpenSSL 0.9.8m
After some initial investigation, I did find the following post about stunnel core dumps on AIX 5.3:
http://www.stunnel.org/pipermail/stunnel-users/2007-April/001528.html
Some of the core dumps were definitely due to this issue, so I rebuilt stunnel using the steps outlined in the post. Now I do not see the stack overflow core dumps, however I am still seeing other cores.
These cores have little to no valuable information in them, so I do not have a lot to go off of. The one thing I have noticed is that stunnel seems to be consuming more and more memory over time. I set up several tests to see what would be causing this.
Tests:
1. Send large amounts of data through stunnel using a simple client and server. This seemed to have no affect on the long term memory usage of stunnel.
2. Repeatedly create and drop connections to stunnel. For this I wrote a short script that simply creates a connection to stunnel using telnet in a a tight loop. I did not start anything for the stunnel instance to connect to, so telnet would create a connection and get it closed immediately.
telnet --> stunnel --> nothing
The process started with a size of 836 KB and seems to increase by about .1 to .2 KB per connection.
$ count=1; while true; do tn localhost 10500; echo $count; count=$(( count + 1 )); done Trying... Escape character is '^T'. Connection closed. 1 ... Trying... Connected to loopback. Escape character is '^T'. Connection closed. 15475 Trying... Connected to loopback. Escape character is '^T'. Connection closed. 15476 Trying... Connected to loopback. Escape character is '^T'. Connection closed. 15477
$ monitor_process test_stunnel Size: 836 ... Size: 2556 Size: 2564 Size: 2564 Size: 2564 Size: 2568 Size: 2572 Size: 2572 Size: 2592
I have not dug into the code yet, so I suppose that will be my next task. If anyone has any ideas or suggestions on what else to look at, I would really appreciate it.
Thanks,
Paul Allex _______________________________________________ stunnel-users mailing list stunnel-users@stunnel.org http://stunnel.mirt.net/mailman/listinfo/stunnel-users
On Thu, Mar 10, 2011 at 06:02:19PM -0600, Allex, Paul M (USA E D EA STS) wrote:
Sorry for posting my follow up incorrectly.
Anyways, I believe I have located the memory leak. In the pthread version of create_client (sthreads.c), there is the following code:
int create_client(int ls, int s, CLI *arg, void *(*cli)(void *)) { pthread_attr_t pth_attr; ... pthread_attr_init(&pth_attr); ... if(pthread_create(&thread, &pth_attr, cli, arg)) { ... return 0; }
The issue here is that pthread_attr_destroy is not being called for pth_attr after pthread_create is done. This leaks memory allocated into pth_attr by pthread_attr_init.
http://www.kernel.org/doc/man-pages/online/pages/man3/pthread_attr_init. 3.html
I corrected the code locally and tested, and it appears to no longer leak memory for the test I described earlier.
Patch:
*** sthreads.orig.c Thu Mar 10 17:50:03 2011 --- sthreads.c Thu Mar 10 17:50:43 2011
*** 313,318 **** --- 313,319 ---- if(pthread_create(&thread, &pth_attr, cli, arg)) { #ifdef HAVE_PTHREAD_SIGMASK pthread_sigmask(SIG_SETMASK, &oldmask, NULL); /* restore the mask */
pthread_attr_destroy(&pth_attr);
*** 322,327 **** --- 323,329 ---- } #ifdef HAVE_PTHREAD_SIGMASK pthread_sigmask(SIG_SETMASK, &oldmask, NULL); /* restore the mask */
pthread_attr_destroy(&pth_attr);Thanks,
Paul Allex
Hmmm, doesn't this point to another little buglet? Shouldn't both your new pthread_attr_destroy() and the pthread_attr_create() that is in the current stunnel code be *outside* the #ifdefs?
Michal, what do you think of the two attached patches? The first one moves pthread_attr_create() out of the ifdefs, and the second one is the modified memory leak patch of Paul's with the destroying happening outside of the #ifdefs.
G'luck, Peter
-
Allex, Paul M (USA E D EA STS) -
Peter Pentchev