Hi, there.
Base server details: - Using the latest stunnel 5.37 - Ubuntu Trusty - The computers communicating with each other use bridged interface addresses that are configured by the clustering software (i.e. to ensure that at least one of the nodes is master and host to some common addresses) - For discussion purposes, let's say that each server has an eth0 and eth1, and that the master node has two additional bridged address (ie. eth0:shared-ip1, eth1:shared-ip2) - The slave nodes will try to connect to eth0:shared-ip1:20001 and eth1:shared-ip2:20001 that are hosted on the master - The master node will also try to connect to these as well, while listening on port 20001 - Not running in a chroot and am running stunnel as stunnel4 user - I have created a working solution using 4 services in the stunnel config and using hard-coded destinations in the iptables rules and the stunnel config. I want to create something more dynamic and versatile.
I am trying to insert a stunnel instance between my server application that attempts to connect to its twin on another server, using the bridging addresses. I am trying to avoid hard-coding destination IP addresses in my stunnel configuration file and the iptables rules. I also don't want to have separate rules for each interface.
For example, from server A's perspective (i.e. the master node), the desirable sequence of events are:
- Attempt to connect to port 20001 using the bridge addresses as destinations (in this case, they would be local interfaces). - Intercept the requests to port 20001 on server A and divert them to port 40000 for encryption (i.e. the local stunnel instance). - Send unencrypted data from stunnel directly to local 20001.
At the same time, server B is doing the same, except that the bridge addresses reside on server A and are not local. So, server A would also need to do this:
- Intercept external request for port 20001 and send it to stunnel port 40000 for encryption - Send decrypted packets to local port 20001
This same type of sequence would also be happening
My stunnel configuration looks like this: fips=yes pid=/var/run/stunnel4/testsrv.pid debug=7 output=/var/log/stunnel4/testsrv.log sslVersion = TLSv1.2 key = <private-key-location> cert = <cert-location> CAfile = <ca-file-location> ciphers = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:AES256-GCM-SHA384:AES256-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:AES128-GCM-SHA256:AES128-SHA256
[test-srv] client=yes accept=40000 transparent=destination My iptables rules, base on what I read in the man page, are this:
STUNNEL_UID=$(id -u stunnel4)
# Redirect any incoming requests to port 20001 iptables -t nat -A PREROUTING -p tcp --dport 20001 ! -i lo -j DNAT --to-destination 127.0.0.1:40000 iptables -A INPUT -p tcp --dport 40000 -j ACCEPT
# Redirect any outgoing locally generated requests for 20001 to stunnel port 40000 iptables -t nat -A OUTPUT -p tcp --dport 20001 -m owner ! --uid-owner $STUNNEL_UID -j DNAT --to-destination 127.0.0.1:40000
# Redirect the output of stunnel to the local 20001 port iptables -t nat -A OUTPUT -p tcp --dport 20001 -m owner --uid-owner $STUNNEL_UID -j REDIRECT --to-ports 20001 iptables -t nat -A INPUT -p tcp -d 127.0.0.1 --dport 20001 -j ACCEPT These rules are not completely adequate. For now, I'm just trying to get server A's connection to port 20001 working without having to worry about incoming from server B.
When I do this and start stunnel and my process, server A's stuff appears to loop endlessly. I think this is happening because the packets that come out of stunnel, due to the transparency configuration, look exactly the same as the original. If I log stuff using iptables, I see repeat entries like this: nat-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=150.150.242.27 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=20001 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 nat-POSTROUTING: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=30542 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=30542 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 nat-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=150.150.242.27 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=20001 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 nat-POSTROUTING: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48828 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48828 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=269 TOS=0x00 PREC=0x00 TTL=64 ID=48829 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK PSH URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=269 TOS=0x00 PREC=0x00 TTL=64 ID=48829 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK PSH URGP=0 ... As can be seen, line 1 and line 7 are virtually identical.
In the stunnel log, I can see that stunnel is receiving multiple requests for port 20001 in rapid succession but is not actually receiving any reply from the final destination.
So, if anyone has any ideas, tweaks or suggestions for how to solve this problem or where I am going wrong, I would appreciate it. I can use a static configuration with hard-coded destination IP's but if there is an alternative, I would like to know. I have tried for several days to figure this out and have searched Google extensively, including stunnel-users postings, to figure out how to do this without success.
Thanks,
Rodney Lott
Okay, I found a solution to my problem. I'm just posting it for completeness and maybe it will be helpful to someone else. Here the pertinent parts of my stunnel config:
... [eth-out] client=yes accept=127.0.0.1:40000 transparent=destination
[eth-in] accept=127.0.0.1:40001 transparent=destination
[lo-out] client=yes accept=127.0.0.1:40002 connect=127.0.0.1:40003
[lo-in] accept=127.0.0.1:40003 connect=127.0.0.1:20001
My iptables rules:
# in rules: iptables -t nat -A PREROUTING -p tcp --dport 20001 -j DNAT --to-destination 127.0.0.1:40001
# out rules: iptables -t nat -A OUTPUT -p tcp ! -o lo --dport 20001 -m owner ! --uid-owner $STUNNEL_UID -j DNAT --to-destination 127.0.0.1:40000 iptables -t nat -A OUTPUT -p tcp -o lo --dport 20001 -m owner ! --uid-owner $STUNNEL_UID -j REDIRECT --to-ports 40002 iptables -t nat -A OUTPUT -p tcp --dport 20001 -m owner --uid-owner $STUNNEL_UID -j ACCEPT
# Allow local access to the stunnels port and reject anything else iptables -A INPUT -d 127.0.0.0/8 -p tcp -m multiport --dports 40000:40005 -j ACCEPT iptables -A INPUT -p tcp -m multiport --dports 40000:40005 -j REJECT
# Restrict input access to port 20001 to local iptables -A INPUT -i lo -p tcp --dport 20001 -j ACCEPT iptables -A INPUT -p tcp --dport 20001 -j REJECT
sysctl -w net.ipv4.conf.all.route_localnet=1 sysctl -w net.ipv4.conf.all.rp_filter=2
Regards,
Rodney
On 2016-11-21 03:47 PM, Rodney Lott wrote: Hi, there.
Base server details: - Using the latest stunnel 5.37 - Ubuntu Trusty - The computers communicating with each other use bridged interface addresses that are configured by the clustering software (i.e. to ensure that at least one of the nodes is master and host to some common addresses) - For discussion purposes, let's say that each server has an eth0 and eth1, and that the master node has two additional bridged address (ie. eth0:shared-ip1, eth1:shared-ip2) - The slave nodes will try to connect to eth0:shared-ip1:20001 and eth1:shared-ip2:20001 that are hosted on the master - The master node will also try to connect to these as well, while listening on port 20001 - Not running in a chroot and am running stunnel as stunnel4 user - I have created a working solution using 4 services in the stunnel config and using hard-coded destinations in the iptables rules and the stunnel config. I want to create something more dynamic and versatile.
I am trying to insert a stunnel instance between my server application that attempts to connect to its twin on another server, using the bridging addresses. I am trying to avoid hard-coding destination IP addresses in my stunnel configuration file and the iptables rules. I also don't want to have separate rules for each interface.
For example, from server A's perspective (i.e. the master node), the desirable sequence of events are:
- Attempt to connect to port 20001 using the bridge addresses as destinations (in this case, they would be local interfaces). - Intercept the requests to port 20001 on server A and divert them to port 40000 for encryption (i.e. the local stunnel instance). - Send unencrypted data from stunnel directly to local 20001.
At the same time, server B is doing the same, except that the bridge addresses reside on server A and are not local. So, server A would also need to do this:
- Intercept external request for port 20001 and send it to stunnel port 40000 for encryption - Send decrypted packets to local port 20001
This same type of sequence would also be happening
My stunnel configuration looks like this: fips=yes pid=/var/run/stunnel4/testsrv.pid debug=7 output=/var/log/stunnel4/testsrv.log sslVersion = TLSv1.2 key = <private-key-location> cert = <cert-location> CAfile = <ca-file-location> ciphers = ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:AES256-GCM-SHA384:AES256-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:AES128-GCM-SHA256:AES128-SHA256
[test-srv] client=yes accept=40000 transparent=destination My iptables rules, base on what I read in the man page, are this:
STUNNEL_UID=$(id -u stunnel4)
# Redirect any incoming requests to port 20001 iptables -t nat -A PREROUTING -p tcp --dport 20001 ! -i lo -j DNAT --to-destination 127.0.0.1:40000 iptables -A INPUT -p tcp --dport 40000 -j ACCEPT
# Redirect any outgoing locally generated requests for 20001 to stunnel port 40000 iptables -t nat -A OUTPUT -p tcp --dport 20001 -m owner ! --uid-owner $STUNNEL_UID -j DNAT --to-destination 127.0.0.1:40000
# Redirect the output of stunnel to the local 20001 port iptables -t nat -A OUTPUT -p tcp --dport 20001 -m owner --uid-owner $STUNNEL_UID -j REDIRECT --to-ports 20001 iptables -t nat -A INPUT -p tcp -d 127.0.0.1 --dport 20001 -j ACCEPT These rules are not completely adequate. For now, I'm just trying to get server A's connection to port 20001 working without having to worry about incoming from server B.
When I do this and start stunnel and my process, server A's stuff appears to loop endlessly. I think this is happening because the packets that come out of stunnel, due to the transparency configuration, look exactly the same as the original. If I log stuff using iptables, I see repeat entries like this: nat-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=150.150.242.27 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=20001 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 nat-POSTROUTING: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30541 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=30542 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=30542 DF PROTO=TCP SPT=50101 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 nat-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=150.150.242.27 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=20001 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 nat-POSTROUTING: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48827 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=43690 RES=0x00 SYN URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48828 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48828 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK URGP=0 filter-OUTPUT: IN= OUT=lo SRC=150.150.242.26 DST=127.0.0.1 LEN=269 TOS=0x00 PREC=0x00 TTL=64 ID=48829 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK PSH URGP=0 filter-INPUT: IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=150.150.242.26 DST=127.0.0.1 LEN=269 TOS=0x00 PREC=0x00 TTL=64 ID=48829 DF PROTO=TCP SPT=50102 DPT=40000 WINDOW=342 RES=0x00 ACK PSH URGP=0 ... As can be seen, line 1 and line 7 are virtually identical.
In the stunnel log, I can see that stunnel is receiving multiple requests for port 20001 in rapid succession but is not actually receiving any reply from the final destination.
So, if anyone has any ideas, tweaks or suggestions for how to solve this problem or where I am going wrong, I would appreciate it. I can use a static configuration with hard-coded destination IP's but if there is an alternative, I would like to know. I have tried for several days to figure this out and have searched Google extensively, including stunnel-users postings, to figure out how to do this without success.
Thanks,
Rodney Lott
_______________________________________________ stunnel-users mailing list stunnel-users@stunnel.orgmailto:stunnel-users@stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users
-
Rodney Lott