running initial pings clientQA ping -------------------- nmap found 10.233.54.10 IS UP fd00:6b13:d07c:4acf::a IS UP 10.233.54.4 IS DOWN fd00:6b13:d07c:4acf::4 IS DOWN 10.233.54.11 IS UP fd00:6b13:d07c:4acf::b IS UP 10.233.54.2 IS DOWN fd00:6b13:d07c:4acf::2 IS DOWN 10.233.54.12 IS UP fd00:6b13:d07c:4acf::c IS UP 10.233.54.13 IS UP fd00:6b13:d07c:4acf::d IS UP 10.233.54.6 IS DOWN fd00:6b13:d07c:4acf::6 IS DOWN 10.116.0.2 IS DOWN 10.116.0.3 IS DOWN 10.116.0.4 IS DOWN 10.116.0.5 IS DOWN 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS DOWN 10.116.0.9 IS DOWN 10.233.54.5 IS DOWN fd00:6b13:d07c:4acf::5 IS DOWN 10.233.54.1 IS UP fd00:6b13:d07c:4acf::1 IS UP 10.233.54.254 IS DOWN fd00:6b13:d07c:4acf:ffff:ffff:ffff:fffe IS DOWN docker-netclient ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.5 IS UP arm-redhat ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP userspace-wireguard ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP clientqa02 ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP clientqa03 ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP clientqa04 ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP checking routing clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 10.233.54.0/24 dev netmaker proto kernel scope link src 10.233.54.8 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:6b13:d07c:4acf::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium docker-netclient ip route -------------------- default via 170.64.224.1 dev eth0 proto static 10.49.0.0/16 dev eth0 proto kernel scope link src 10.49.0.7 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.3 10.126.0.0/20 dev eth1 proto kernel scope link src 10.126.0.2 170.64.224.0/20 dev eth0 proto kernel scope link src 170.64.238.167 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev docker0 proto kernel metric 256 linkdown pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.2 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.4 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.5 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.6 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.7 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium getting initial wg configs for clients clientQA wg -------------------- interface: netmaker public key: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= private key: (hidden) listening port: 51821 peer: soNvLQwnyzYmpHlFPlXFIMRkyZ0EOoSS3sNyxc/LnSg= endpoint: 98.216.56.86:51821 allowed ips: 10.233.54.10/32, fd00:6b13:d07c:4acf::a/128 latest handshake: 9 seconds ago transfer: 218.96 KiB received, 5.43 MiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.233.54.6/32, fd00:6b13:d07c:4acf::6/128, 10.116.0.0/16 latest handshake: 30 seconds ago transfer: 418.57 KiB received, 7.13 MiB sent persistent keepalive: every 20 seconds peer: oUsjI8JQOEr2dZ5dtLiNODVmA/MTo0NsjWkHJVg75xM= endpoint: 167.172.163.172:64645 allowed ips: 10.233.54.11/32, fd00:6b13:d07c:4acf::b/128 latest handshake: 58 seconds ago transfer: 17.01 MiB received, 20.84 MiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 10.106.0.10:51821 allowed ips: 10.233.54.5/32, fd00:6b13:d07c:4acf::5/128 latest handshake: 1 minute, 13 seconds ago transfer: 416.16 KiB received, 7.13 MiB sent persistent keepalive: every 20 seconds peer: D+/om9ZCRx+WkluPt/LwUm9+PA4PAWrvyh7po2TYRSU= endpoint: 67.205.143.140:51821 allowed ips: 10.233.54.1/32, fd00:6b13:d07c:4acf::1/128, 10.233.54.254/32, fd00:6b13:d07c:4acf:ffff:ffff:ffff:fffe/128 latest handshake: 1 minute, 24 seconds ago transfer: 17.03 MiB received, 21.01 MiB sent persistent keepalive: every 20 seconds peer: aX/jf5+CBewOOzOn7EWz3skEIIv9qMiNjVGUUNOnDQQ= endpoint: 167.172.163.172:62619 allowed ips: 10.233.54.12/32, fd00:6b13:d07c:4acf::c/128 latest handshake: 1 minute, 47 seconds ago transfer: 16.95 MiB received, 20.85 MiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.233.54.4/32, fd00:6b13:d07c:4acf::4/128 latest handshake: 1 minute, 49 seconds ago transfer: 415.34 KiB received, 7.13 MiB sent persistent keepalive: every 20 seconds peer: EZTM7nXo4ZPnoDmVovTX0yuNmeNcFNG+wXkznSOzr1Y= endpoint: 146.190.147.149:62394 allowed ips: 10.233.54.13/32, fd00:6b13:d07c:4acf::d/128 latest handshake: 1 minute, 55 seconds ago transfer: 17.19 MiB received, 21.13 MiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.233.54.2/32, fd00:6b13:d07c:4acf::2/128 latest handshake: 1 minute, 58 seconds ago transfer: 248.53 KiB received, 7.25 MiB sent persistent keepalive: every 20 seconds docker-netclient wg interface -------------------- interface: netmaker public key: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= private key: (hidden) listening port: 51821 peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 36 seconds ago transfer: 2.26 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 36 seconds ago transfer: 2.27 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 37 seconds ago transfer: 2.25 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 44 seconds ago transfer: 2.26 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 1 minute, 35 seconds ago transfer: 2.25 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 37 seconds ago transfer: 2.24 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 transfer: 0 B received, 430.56 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= private key: (hidden) listening port: 51822 peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 23 seconds ago transfer: 2.28 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 1 second ago transfer: 2.29 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 38 seconds ago transfer: 2.26 MiB received, 2.24 MiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 1 minute, 39 seconds ago transfer: 2.27 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 1 minute, 41 seconds ago transfer: 2.25 MiB received, 2.23 MiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 1 minute, 54 seconds ago transfer: 2.25 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 2 minutes, 3 seconds ago transfer: 30.19 KiB received, 614.29 KiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= private key: (hidden) listening port: 51821 peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 25 seconds ago transfer: 2.24 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 40 seconds ago transfer: 2.25 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 10.124.0.28:51821 allowed ips: 10.101.0.1/32 latest handshake: 58 seconds ago transfer: 2.29 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 1 minute, 3 seconds ago transfer: 2.26 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 1 minute, 56 seconds ago transfer: 31.01 KiB received, 615.68 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 56 seconds ago transfer: 2.25 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 2 minutes, 3 seconds ago transfer: 2.26 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= private key: (hidden) listening port: 51821 peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: Now transfer: 2.27 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 6 seconds ago transfer: 2.26 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 40 seconds ago transfer: 2.26 MiB received, 2.27 MiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 48 seconds ago transfer: 2.25 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 10.106.0.9:51821 allowed ips: 10.101.0.8/32 latest handshake: 1 minute, 21 seconds ago transfer: 31.00 KiB received, 616.69 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 42 seconds ago transfer: 2.25 MiB received, 2.27 MiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 2 minutes, 4 seconds ago transfer: 2.25 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= private key: (hidden) listening port: 51821 peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 8 seconds ago transfer: 2.25 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 28 seconds ago transfer: 2.25 MiB received, 2.28 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 40 seconds ago transfer: 29.78 KiB received, 618.50 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 42 seconds ago transfer: 2.25 MiB received, 2.27 MiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 48 seconds ago transfer: 2.25 MiB received, 2.27 MiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 5 seconds ago transfer: 2.29 MiB received, 2.27 MiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 1 minute, 6 seconds ago transfer: 2.25 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= private key: (hidden) listening port: 51825 peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 28 seconds ago transfer: 2.25 MiB received, 2.24 MiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 42 seconds ago transfer: 2.27 MiB received, 2.26 MiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 48 seconds ago transfer: 2.27 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 10 seconds ago transfer: 2.27 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 41 seconds ago transfer: 2.26 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 46 seconds ago transfer: 2.23 MiB received, 2.25 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 transfer: 0 B received, 430.41 KiB sent persistent keepalive: every 20 seconds checking initial logs for errors clientQA log errors -------------------- docker-netclient log errors -------------------- arm-redhat log errors -------------------- May 12 20:41:50 ip-172-31-23-197.ec2.internal netclient[118321]: {"time":"2024-05-12T20:41:50.468372785Z","level":"ERROR","source":"daemon.go 217}","msg":"fail to pull config from server","error":"Post \"https://api.scripttest.clustercat.com:443/api/hosts/adm/authenticate\": dial tcp 147.182.203.170:443: connect: connection refused"} userspace-wireguard log errors -------------------- May 12 20:41:49 userspace-wireguard netclient[406293]: {"time":"2024-05-12T20:41:49.931729473Z","level":"ERROR","source":"daemon.go 217}","msg":"fail to pull config from server","error":"Post \"https://api.scripttest.clustercat.com:443/api/hosts/adm/authenticate\": dial tcp 147.182.203.170:443: connect: connection refused"} clientqa02 log errors -------------------- May 12 20:41:50 clientQA-02 netclient[3313250]: {"time":"2024-05-12T20:41:50.207923446Z","level":"ERROR","source":"daemon.go 217}","msg":"fail to pull config from server","error":"Post \"https://api.scripttest.clustercat.com:443/api/hosts/adm/authenticate\": dial tcp 147.182.203.170:443: connect: connection refused"} May 12 20:42:33 clientQA-02 netclient[3313250]: {"time":"2024-05-12T20:42:33.014402597Z","level":"ERROR","source":"mqpublish.go 250}","msg":"could not connect to broker at","server":"scripttest.clustercat.com","error":"connection timeout"} May 12 20:42:33 clientQA-02 netclient[3313250]: [netclient] 2024-05-12 20:42:33 error occurred during publishing of metrics on node clientQA-02 connection timeout clientqa03 log errors -------------------- May 12 20:41:49 clientQA-03 netclient[1542027]: {"time":"2024-05-12T20:41:49.953722173Z","level":"ERROR","source":"daemon.go 217}","msg":"fail to pull config from server","error":"Post \"https://api.scripttest.clustercat.com:443/api/hosts/adm/authenticate\": dial tcp 147.182.203.170:443: connect: connection refused"} clientqa04 log errors -------------------- pulling clients and checking config pulling clientQA -------------------- completed pull for server nm.67-205-143-140.nip.io pulling arm-redhat -------------------- completed pull for server scripttest.clustercat.com pulling userspace-wireguard -------------------- completed pull for server scripttest.clustercat.com pulling clientqa02 -------------------- completed pull for server scripttest.clustercat.com pulling clientqa03 -------------------- completed pull for server scripttest.clustercat.com pulling clientqa04 -------------------- completed pull for server scripttest.clustercat.com [netclient] 2024-05-13 01:09:23 [pull.go-80] Pull(): restarting daemon running pings after pull clientQA ping -------------------- nmap found 10.233.54.1 IS UP fd00:6b13:d07c:4acf::1 IS UP 10.233.54.254 IS DOWN fd00:6b13:d07c:4acf:ffff:ffff:ffff:fffe IS DOWN 10.233.54.10 IS UP fd00:6b13:d07c:4acf::a IS UP 10.233.54.4 IS DOWN fd00:6b13:d07c:4acf::4 IS DOWN 10.233.54.11 IS UP fd00:6b13:d07c:4acf::b IS UP 10.233.54.2 IS DOWN fd00:6b13:d07c:4acf::2 IS DOWN 10.233.54.6 IS DOWN fd00:6b13:d07c:4acf::6 IS DOWN 10.116.0.2 IS DOWN 10.116.0.3 IS DOWN 10.116.0.4 IS DOWN 10.116.0.5 IS DOWN 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS DOWN 10.116.0.9 IS DOWN 10.233.54.5 IS DOWN fd00:6b13:d07c:4acf::5 IS DOWN 10.233.54.12 IS UP fd00:6b13:d07c:4acf::c IS UP 10.233.54.13 IS UP fd00:6b13:d07c:4acf::d IS UP docker-netclient ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.5 IS UP arm-redhat ping ----------------- nmap found 10.101.0.5 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.7 IS UP 10.101.0.3 IS UP userspace-wireguard ping ----------------- nmap found 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.8 IS DOWN clientqa02 ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP clientqa03 ping ----------------- nmap found 10.101.0.5 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP clientqa04 ping ----------------- nmap found 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN checking routing after pull clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 10.233.54.0/24 dev netmaker proto kernel scope link src 10.233.54.8 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:6b13:d07c:4acf::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium docker-netclient ip route -------------------- default via 170.64.224.1 dev eth0 proto static 10.49.0.0/16 dev eth0 proto kernel scope link src 10.49.0.7 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.3 10.126.0.0/20 dev eth1 proto kernel scope link src 10.126.0.2 170.64.224.0/20 dev eth0 proto kernel scope link src 170.64.238.167 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev docker0 proto kernel metric 256 linkdown pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.2 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.4 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.5 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.6 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.7 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium getting wg configs after pull clientQA wg -------------------- interface: netmaker public key: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= private key: (hidden) listening port: 51821 peer: aX/jf5+CBewOOzOn7EWz3skEIIv9qMiNjVGUUNOnDQQ= endpoint: 167.172.163.172:62619 allowed ips: 10.233.54.12/32, fd00:6b13:d07c:4acf::c/128 latest handshake: 12 seconds ago transfer: 35.53 KiB received, 43.02 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.233.54.4/32, fd00:6b13:d07c:4acf::4/128 latest handshake: 21 seconds ago transfer: 1.31 KiB received, 15.39 KiB sent persistent keepalive: every 20 seconds peer: EZTM7nXo4ZPnoDmVovTX0yuNmeNcFNG+wXkznSOzr1Y= endpoint: 146.190.147.149:62394 allowed ips: 10.233.54.13/32, fd00:6b13:d07c:4acf::d/128 latest handshake: 26 seconds ago transfer: 36.30 KiB received, 43.95 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.233.54.6/32, fd00:6b13:d07c:4acf::6/128, 10.116.0.0/16 latest handshake: 27 seconds ago transfer: 1.22 KiB received, 15.98 KiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 10.106.0.10:51821 allowed ips: 10.233.54.5/32, fd00:6b13:d07c:4acf::5/128 latest handshake: 29 seconds ago transfer: 1.31 KiB received, 17.11 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.233.54.2/32, fd00:6b13:d07c:4acf::2/128 latest handshake: 33 seconds ago transfer: 1.22 KiB received, 16.61 KiB sent persistent keepalive: every 20 seconds peer: oUsjI8JQOEr2dZ5dtLiNODVmA/MTo0NsjWkHJVg75xM= endpoint: 167.172.163.172:64645 allowed ips: 10.233.54.11/32, fd00:6b13:d07c:4acf::b/128 latest handshake: 33 seconds ago transfer: 35.45 KiB received, 42.91 KiB sent persistent keepalive: every 20 seconds peer: soNvLQwnyzYmpHlFPlXFIMRkyZ0EOoSS3sNyxc/LnSg= endpoint: 98.216.56.86:51821 allowed ips: 10.233.54.10/32, fd00:6b13:d07c:4acf::a/128 latest handshake: 45 seconds ago transfer: 59.88 KiB received, 54.25 KiB sent persistent keepalive: every 20 seconds peer: D+/om9ZCRx+WkluPt/LwUm9+PA4PAWrvyh7po2TYRSU= endpoint: 67.205.143.140:51821 allowed ips: 10.233.54.1/32, fd00:6b13:d07c:4acf::1/128, 10.233.54.254/32, fd00:6b13:d07c:4acf:ffff:ffff:ffff:fffe/128 latest handshake: 48 seconds ago transfer: 38.02 KiB received, 46.08 KiB sent persistent keepalive: every 20 seconds docker-netclient wg interface -------------------- interface: netmaker public key: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= private key: (hidden) listening port: 51821 peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 29 seconds ago transfer: 2.33 MiB received, 2.31 MiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 35 seconds ago transfer: 2.31 MiB received, 2.33 MiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 35 seconds ago transfer: 2.33 MiB received, 2.31 MiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 36 seconds ago transfer: 2.31 MiB received, 2.32 MiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 41 seconds ago transfer: 2.31 MiB received, 2.31 MiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 50 seconds ago transfer: 2.34 MiB received, 2.31 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 transfer: 0 B received, 442.41 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= private key: (hidden) listening port: 51822 peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 28 seconds ago transfer: 59.27 KiB received, 60.23 KiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 29 seconds ago transfer: 59.06 KiB received, 58.46 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 33 seconds ago transfer: 58.66 KiB received, 58.58 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 36 seconds ago transfer: 58.55 KiB received, 59.49 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 38 seconds ago transfer: 560 B received, 15.85 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 41 seconds ago transfer: 57.88 KiB received, 57.88 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 43 seconds ago transfer: 62.22 KiB received, 60.82 KiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= private key: (hidden) listening port: 51821 peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 25 seconds ago transfer: 56.30 KiB received, 57.11 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 28 seconds ago transfer: 656 B received, 15.85 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 39 seconds ago transfer: 59.16 KiB received, 57.19 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 10.124.0.28:51821 allowed ips: 10.101.0.1/32 latest handshake: 39 seconds ago transfer: 58.95 KiB received, 58.13 KiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 42 seconds ago transfer: 59.49 KiB received, 59.70 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 43 seconds ago transfer: 57.64 KiB received, 57.88 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 44 seconds ago transfer: 58.19 KiB received, 58.44 KiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= private key: (hidden) listening port: 51821 peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 27 seconds ago transfer: 59.35 KiB received, 59.59 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 31 seconds ago transfer: 59.91 KiB received, 59.27 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 10.106.0.9:51821 allowed ips: 10.101.0.8/32 latest handshake: 37 seconds ago transfer: 688 B received, 16.57 KiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 39 seconds ago transfer: 59.79 KiB received, 60.11 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 39 seconds ago transfer: 59.22 KiB received, 59.68 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 43 seconds ago transfer: 59.38 KiB received, 59.49 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 45 seconds ago transfer: 59.12 KiB received, 58.79 KiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= private key: (hidden) listening port: 51821 peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 28 seconds ago transfer: 59.36 KiB received, 59.35 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 34 seconds ago transfer: 58.55 KiB received, 59.33 KiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 35 seconds ago transfer: 56.44 KiB received, 56.78 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 37 seconds ago transfer: 592 B received, 16.79 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 37 seconds ago transfer: 57.89 KiB received, 58.57 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 41 seconds ago transfer: 57.39 KiB received, 59.44 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 44 seconds ago transfer: 60.97 KiB received, 61.00 KiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= private key: (hidden) listening port: 51825 peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 28 seconds ago transfer: 56.73 KiB received, 56.30 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 33 seconds ago transfer: 60.42 KiB received, 59.60 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 34 seconds ago transfer: 58.33 KiB received, 60.29 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 36 seconds ago transfer: 56.55 KiB received, 56.44 KiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 41 seconds ago transfer: 60.75 KiB received, 60.44 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 42 seconds ago transfer: 58.58 KiB received, 57.85 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 transfer: 0 B received, 11.27 KiB sent persistent keepalive: every 20 seconds checking logs for errors after a pull clientQA log errors -------------------- docker-netclient log errors -------------------- arm-redhat log errors -------------------- May 12 20:41:50 ip-172-31-23-197.ec2.internal netclient[118321]: {"time":"2024-05-12T20:41:50.468372785Z","level":"ERROR","source":"daemon.go 217}","msg":"fail to pull config from server","error":"Post \"https://api.scripttest.clustercat.com:443/api/hosts/adm/authenticate\": dial tcp 147.182.203.170:443: connect: connection refused"} userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- May 12 20:42:33 clientQA-02 netclient[3313250]: {"time":"2024-05-12T20:42:33.014402597Z","level":"ERROR","source":"mqpublish.go 250}","msg":"could not connect to broker at","server":"scripttest.clustercat.com","error":"connection timeout"} May 12 20:42:33 clientQA-02 netclient[3313250]: [netclient] 2024-05-12 20:42:33 error occurred during publishing of metrics on node clientQA-02 connection timeout clientqa03 log errors -------------------- May 12 20:41:49 clientQA-03 netclient[1542027]: {"time":"2024-05-12T20:41:49.953722173Z","level":"ERROR","source":"daemon.go 217}","msg":"fail to pull config from server","error":"Post \"https://api.scripttest.clustercat.com:443/api/hosts/adm/authenticate\": dial tcp 147.182.203.170:443: connect: connection refused"} clientqa04 log errors -------------------- restarting clients and checking config restarting clientQA -------------------- restarting arm-redhat -------------------- restarting userspace-wireguard -------------------- restarting clientqa02 -------------------- restarting clientqa03 -------------------- restarting clientqa04 -------------------- running pings after restart clientQA ping -------------------- nmap found 10.233.54.6 IS DOWN fd00:6b13:d07c:4acf::6 IS DOWN 10.116.0.2 IS DOWN 10.116.0.3 IS DOWN 10.116.0.4 IS DOWN 10.116.0.5 IS DOWN 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS DOWN 10.116.0.9 IS DOWN 10.233.54.5 IS DOWN fd00:6b13:d07c:4acf::5 IS DOWN 10.233.54.12 IS UP fd00:6b13:d07c:4acf::c IS UP 10.233.54.13 IS UP fd00:6b13:d07c:4acf::d IS UP 10.233.54.1 IS UP fd00:6b13:d07c:4acf::1 IS UP 10.233.54.254 IS DOWN fd00:6b13:d07c:4acf:ffff:ffff:ffff:fffe IS DOWN 10.233.54.10 IS UP fd00:6b13:d07c:4acf::a IS UP 10.233.54.4 IS DOWN fd00:6b13:d07c:4acf::4 IS DOWN 10.233.54.11 IS UP fd00:6b13:d07c:4acf::b IS UP 10.233.54.2 IS DOWN fd00:6b13:d07c:4acf::2 IS DOWN docker-netclient ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.5 IS UP arm-redhat ping ----------------- nmap found 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN userspace-wireguard ping ----------------- nmap found 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP clientqa02 ping ----------------- nmap found 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP clientqa03 ping ----------------- nmap found 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.7 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.101.0.1 IS UP clientqa04 ping ----------------- nmap found 10.101.0.5 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.4 IS UP 10.101.0.8 IS DOWN 10.101.0.2 IS UP 10.101.0.3 IS UP checking routing after restart clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 10.233.54.0/24 dev netmaker proto kernel scope link src 10.233.54.8 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:6b13:d07c:4acf::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium docker-netclient ip route -------------------- default via 170.64.224.1 dev eth0 proto static 10.49.0.0/16 dev eth0 proto kernel scope link src 10.49.0.7 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.3 10.126.0.0/20 dev eth1 proto kernel scope link src 10.126.0.2 170.64.224.0/20 dev eth0 proto kernel scope link src 170.64.238.167 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev docker0 proto kernel metric 256 linkdown pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.2 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.4 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.5 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.6 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.7 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium getting wg configs after restart clientQA wg -------------------- interface: netmaker public key: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= private key: (hidden) listening port: 51821 peer: oUsjI8JQOEr2dZ5dtLiNODVmA/MTo0NsjWkHJVg75xM= endpoint: 167.172.163.172:64645 allowed ips: 10.233.54.11/32, fd00:6b13:d07c:4acf::b/128 latest handshake: 5 seconds ago transfer: 43.36 KiB received, 52.64 KiB sent persistent keepalive: every 20 seconds peer: aX/jf5+CBewOOzOn7EWz3skEIIv9qMiNjVGUUNOnDQQ= endpoint: 167.172.163.172:62619 allowed ips: 10.233.54.12/32, fd00:6b13:d07c:4acf::c/128 latest handshake: 6 seconds ago transfer: 43.96 KiB received, 52.82 KiB sent persistent keepalive: every 20 seconds peer: D+/om9ZCRx+WkluPt/LwUm9+PA4PAWrvyh7po2TYRSU= endpoint: 67.205.143.140:51821 allowed ips: 10.233.54.1/32, fd00:6b13:d07c:4acf::1/128, 10.233.54.254/32, fd00:6b13:d07c:4acf:ffff:ffff:ffff:fffe/128 latest handshake: 8 seconds ago transfer: 45.84 KiB received, 55.46 KiB sent persistent keepalive: every 20 seconds peer: EZTM7nXo4ZPnoDmVovTX0yuNmeNcFNG+wXkznSOzr1Y= endpoint: 146.190.147.149:62394 allowed ips: 10.233.54.13/32, fd00:6b13:d07c:4acf::d/128 latest handshake: 8 seconds ago transfer: 43.98 KiB received, 52.50 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.233.54.6/32, fd00:6b13:d07c:4acf::6/128, 10.116.0.0/16 latest handshake: 51 seconds ago transfer: 1.34 KiB received, 19.94 KiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 10.106.0.10:51821 allowed ips: 10.233.54.5/32, fd00:6b13:d07c:4acf::5/128 latest handshake: 1 minute, 12 seconds ago transfer: 1.38 KiB received, 19.64 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.233.54.4/32, fd00:6b13:d07c:4acf::4/128 latest handshake: 1 minute, 16 seconds ago transfer: 1.25 KiB received, 18.23 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.233.54.2/32, fd00:6b13:d07c:4acf::2/128 latest handshake: 1 minute, 35 seconds ago transfer: 1.07 KiB received, 20.17 KiB sent persistent keepalive: every 20 seconds peer: soNvLQwnyzYmpHlFPlXFIMRkyZ0EOoSS3sNyxc/LnSg= endpoint: 98.216.56.86:51821 allowed ips: 10.233.54.10/32, fd00:6b13:d07c:4acf::a/128 latest handshake: 2 minutes, 5 seconds ago transfer: 73.67 KiB received, 65.91 KiB sent persistent keepalive: every 20 seconds docker-netclient wg interface -------------------- interface: netmaker public key: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= private key: (hidden) listening port: 51821 peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 17 seconds ago transfer: 2.42 MiB received, 2.39 MiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 24 seconds ago transfer: 2.38 MiB received, 2.40 MiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 47 seconds ago transfer: 2.40 MiB received, 2.38 MiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 1 minute, 16 seconds ago transfer: 2.40 MiB received, 2.38 MiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 1 minute, 36 seconds ago transfer: 2.38 MiB received, 2.39 MiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 54 seconds ago transfer: 2.38 MiB received, 2.40 MiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 transfer: 0 B received, 457.15 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= private key: (hidden) listening port: 51822 peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 22 seconds ago transfer: 63.83 KiB received, 65.77 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 53 seconds ago transfer: 64.80 KiB received, 64.72 KiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 1 minute, 15 seconds ago transfer: 66.81 KiB received, 67.79 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 1 minute, 15 seconds ago transfer: 67.61 KiB received, 68.67 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 1 minute, 41 seconds ago transfer: 720 B received, 18.60 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 55 seconds ago transfer: 70.73 KiB received, 70.60 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 56 seconds ago transfer: 72.30 KiB received, 69.74 KiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= private key: (hidden) listening port: 51821 peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 33 seconds ago transfer: 65.47 KiB received, 65.57 KiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 35 seconds ago transfer: 64.08 KiB received, 64.29 KiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 58 seconds ago transfer: 65.19 KiB received, 66.86 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 18 seconds ago transfer: 67.82 KiB received, 67.79 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 1 minute, 24 seconds ago transfer: 624 B received, 18.04 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 10.124.0.28:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 29 seconds ago transfer: 69.81 KiB received, 69.04 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 39 seconds ago transfer: 68.58 KiB received, 68.50 KiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= private key: (hidden) listening port: 51821 peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 15 seconds ago transfer: 61.70 KiB received, 62.59 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 44 seconds ago transfer: 61.97 KiB received, 62.56 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 59 seconds ago transfer: 66.19 KiB received, 65.19 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 16 seconds ago transfer: 65.12 KiB received, 63.91 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 18 seconds ago transfer: 64.78 KiB received, 64.72 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 20 seconds ago transfer: 64.34 KiB received, 63.63 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 10.106.0.9:51821 allowed ips: 10.101.0.8/32 latest handshake: 1 minute, 21 seconds ago transfer: 560 B received, 17.38 KiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= private key: (hidden) listening port: 51821 peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 36 seconds ago transfer: 61.61 KiB received, 62.54 KiB sent persistent keepalive: every 20 seconds peer: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= endpoint: 209.38.230.81:51825 allowed ips: 10.101.0.7/32 latest handshake: 45 seconds ago transfer: 57.66 KiB received, 59.00 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 45 seconds ago transfer: 61.45 KiB received, 61.33 KiB sent persistent keepalive: every 20 seconds peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 46 seconds ago transfer: 62.27 KiB received, 62.69 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 53 seconds ago transfer: 61.30 KiB received, 62.39 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 58 seconds ago transfer: 60.69 KiB received, 61.80 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 latest handshake: 1 minute, 2 seconds ago transfer: 784 B received, 17.22 KiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: 5AGf5qHBqyCPsWpAXZ+H6IRJKxdaN7ovHe//DcCN7Uw= private key: (hidden) listening port: 51825 peer: n6ACDC7UROTux167JMVaK59VqUYcjLjLFqGpm6Nhnn4= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.5/32 latest handshake: 18 seconds ago transfer: 59.02 KiB received, 58.74 KiB sent persistent keepalive: every 20 seconds peer: SyXgjHtBeKEiErJBpysBHwMIRLkCTA2w2syr0yyFPHo= endpoint: 54.91.197.18:51822 allowed ips: 10.101.0.2/32 latest handshake: 28 seconds ago transfer: 58.09 KiB received, 57.88 KiB sent persistent keepalive: every 20 seconds peer: YoX9U7SHMQ8RwsieXXtuRKnfL/ghWIvq7UqK++h16Ek= endpoint: 170.64.238.167:51821 allowed ips: 10.101.0.3/32 latest handshake: 31 seconds ago transfer: 56.81 KiB received, 56.97 KiB sent persistent keepalive: every 20 seconds peer: fY4N7/hYwH4afoPrSTqN3EyB5PTmnuBj3t+Be1jbNW0= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.4/32 latest handshake: 39 seconds ago transfer: 60.50 KiB received, 60.00 KiB sent persistent keepalive: every 20 seconds peer: u83XVaKFduF73Hr8EH21wh0Mb/RL8kbuVlvkmGn/5Hg= endpoint: 147.182.203.170:51821 allowed ips: 10.101.0.1/32 latest handshake: 43 seconds ago transfer: 59.00 KiB received, 57.97 KiB sent persistent keepalive: every 20 seconds peer: i2JsLBKGD+APnhQn2L9gMZcq2pn2aw/6HMchAdzH1EM= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.6/32 latest handshake: 46 seconds ago transfer: 58.33 KiB received, 57.66 KiB sent persistent keepalive: every 20 seconds peer: qDhyUxTb2M9NVv6caIsjwbGr0zBJ23oYfg5Q0AOYLAg= endpoint: 165.22.115.179:51821 allowed ips: 10.101.0.8/32 transfer: 0 B received, 11.42 KiB sent persistent keepalive: every 20 seconds checking logs for errors after a restart clientQA log errors -------------------- docker-netclient log errors -------------------- arm-redhat log errors -------------------- userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- clientqa03 log errors -------------------- clientqa04 log errors --------------------