running initial pings clientQA ping -------------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP arm-redhat ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.8 IS UP userspace-wireguard ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP clientqa02 ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.116.0.2 IS DOWN 10.116.0.3 IS DOWN 10.116.0.4 IS DOWN 10.116.0.5 IS DOWN 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS DOWN 10.116.0.9 IS DOWN 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.8 IS UP clientqa03 ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP clientqa04 ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.7 IS UP ipv4-test ping ----------------- nmap found 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP checking routing clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.8 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.2 10.116.0.0/16 dev netmaker src 10.101.0.2 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.3 10.116.0.0/16 dev netmaker src 10.101.0.3 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.4 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.5 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.6 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 10.116.0.0/16 dev netmaker src 10.101.0.6 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium ipv4-test ip route -------------------- default via 209.38.128.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.5 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.7 10.116.0.0/16 dev netmaker src 10.101.0.7 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.2 209.38.128.0/19 dev eth0 proto kernel scope link src 209.38.128.43 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium getting initial wg configs for clients clientQA wg -------------------- interface: netmaker public key: EM1ST/n1fwIsNVLMRuwMu7QDdHdiC7zHzxaaAZwXny4= private key: (hidden) listening port: 51821 peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 10.106.0.10:51821 allowed ips: 10.101.0.4/32, 10.101.0.1/32, 10.101.0.2/32, 10.101.0.3/32, 10.101.0.5/32, 10.116.0.0/16, 10.101.0.6/32, 10.101.0.7/32 latest handshake: 18 seconds ago transfer: 38.96 KiB received, 39.00 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= private key: (hidden) listening port: 51821 peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 11 seconds ago transfer: 5.21 MiB received, 5.16 MiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute ago transfer: 5.20 MiB received, 5.21 MiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 1 minute, 6 seconds ago transfer: 5.18 MiB received, 5.15 MiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 20 seconds ago transfer: 5.23 MiB received, 5.17 MiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 24 seconds ago transfer: 5.22 MiB received, 5.19 MiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 2 minutes ago transfer: 5.26 MiB received, 5.23 MiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= private key: (hidden) listening port: 51821 peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 13 seconds ago transfer: 5.20 MiB received, 5.18 MiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 10.124.0.2:51821 allowed ips: 10.101.0.7/32 latest handshake: 52 seconds ago transfer: 5.24 MiB received, 5.21 MiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 10.124.0.4:51821 allowed ips: 10.101.0.1/32 latest handshake: 59 seconds ago transfer: 5.18 MiB received, 5.21 MiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 18 seconds ago transfer: 5.22 MiB received, 5.20 MiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 26 seconds ago transfer: 5.19 MiB received, 5.23 MiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 1 minute, 42 seconds ago transfer: 5.19 MiB received, 5.17 MiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= private key: (hidden) listening port: 51821 peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 14 seconds ago transfer: 35.19 KiB received, 34.87 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 14 seconds ago transfer: 36.91 KiB received, 35.89 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 15 seconds ago transfer: 35.88 KiB received, 35.06 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 16 seconds ago transfer: 35.57 KiB received, 35.18 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 20 seconds ago transfer: 33.60 KiB received, 34.81 KiB sent persistent keepalive: every 20 seconds peer: EM1ST/n1fwIsNVLMRuwMu7QDdHdiC7zHzxaaAZwXny4= endpoint: 10.106.0.9:51821 allowed ips: 10.101.0.8/32 latest handshake: 22 seconds ago transfer: 40.84 KiB received, 40.35 KiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 23 seconds ago transfer: 33.66 KiB received, 34.31 KiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= private key: (hidden) listening port: 51821 peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 16 seconds ago transfer: 41.10 KiB received, 42.02 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 20 seconds ago transfer: 44.75 KiB received, 46.04 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 24 seconds ago transfer: 43.64 KiB received, 44.40 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 31 seconds ago transfer: 40.25 KiB received, 42.21 KiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 1 minute, 33 seconds ago transfer: 44.99 KiB received, 46.78 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 1 minute, 33 seconds ago transfer: 44.24 KiB received, 45.92 KiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= private key: (hidden) listening port: 51821 peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 17 seconds ago transfer: 5.17 MiB received, 5.20 MiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 1 minute ago transfer: 5.21 MiB received, 5.23 MiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 10 seconds ago transfer: 5.15 MiB received, 5.18 MiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 33 seconds ago transfer: 5.21 MiB received, 5.17 MiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 44 seconds ago transfer: 5.17 MiB received, 5.19 MiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 47 seconds ago transfer: 5.17 MiB received, 5.19 MiB sent persistent keepalive: every 20 seconds ipv4-test wg interface -------------------- interface: netmaker public key: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= private key: (hidden) listening port: 51821 peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 6 seconds ago transfer: 5.22 MiB received, 5.25 MiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 26 seconds ago transfer: 5.16 MiB received, 5.19 MiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 10.124.0.13:51821 allowed ips: 10.101.0.3/32 latest handshake: 56 seconds ago transfer: 5.20 MiB received, 5.24 MiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 1 minute, 2 seconds ago transfer: 5.23 MiB received, 5.21 MiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 10.124.0.4:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 5 seconds ago transfer: 5.21 MiB received, 5.23 MiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 35 seconds ago transfer: 5.24 MiB received, 5.21 MiB sent persistent keepalive: every 20 seconds checking initial logs for errors clientQA log errors -------------------- arm-redhat log errors -------------------- userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- clientqa03 log errors -------------------- clientqa04 log errors -------------------- ipv4-test log errors -------------------- pulling clients and checking config pulling clientQA -------------------- completed pull for server nm.64-23-184-110.nip.io pulling arm-redhat -------------------- completed pull for server nm.64-23-184-110.nip.io pulling userspace-wireguard -------------------- completed pull for server nm.64-23-184-110.nip.io pulling clientqa02 -------------------- completed pull for server nm.64-23-184-110.nip.io pulling clientqa03 -------------------- completed pull for server nm.64-23-184-110.nip.io pulling clientqa04 -------------------- completed pull for server nm.64-23-184-110.nip.io pulling ipv4-test -------------------- completed pull for server nm.64-23-184-110.nip.io running pings after pull clientQA ping -------------------- nmap found 10.101.0.4 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP arm-redhat ping ----------------- nmap found 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.1 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP userspace-wireguard ping ----------------- nmap found 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP clientqa02 ping ----------------- nmap found 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.8 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP clientqa03 ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP clientqa04 ping ----------------- nmap found 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.7 IS UP 10.101.0.1 IS UP ipv4-test ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP checking routing after pull clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.8 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 10.116.0.0/16 dev netmaker src 10.101.0.8 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.2 10.116.0.0/16 dev netmaker src 10.101.0.2 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.3 10.116.0.0/16 dev netmaker src 10.101.0.3 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.4 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 10.116.0.0/16 dev netmaker src 10.101.0.4 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.5 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.6 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 10.116.0.0/16 dev netmaker src 10.101.0.6 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium ipv4-test ip route -------------------- default via 209.38.128.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.5 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.7 10.116.0.0/16 dev netmaker src 10.101.0.7 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.2 209.38.128.0/19 dev eth0 proto kernel scope link src 209.38.128.43 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium getting wg configs after pull clientQA wg -------------------- interface: netmaker public key: EM1ST/n1fwIsNVLMRuwMu7QDdHdiC7zHzxaaAZwXny4= private key: (hidden) listening port: 51821 peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 10.106.0.10:51821 allowed ips: 10.101.0.4/32, 10.101.0.2/32, 10.101.0.3/32, 10.101.0.5/32, 10.116.0.0/16, 10.101.0.6/32, 10.101.0.7/32, 10.101.0.1/32 latest handshake: 7 seconds ago transfer: 47.59 KiB received, 48.02 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= private key: (hidden) listening port: 51821 peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 1 second ago transfer: 36.42 KiB received, 36.43 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 4 seconds ago transfer: 42.29 KiB received, 39.96 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 4 seconds ago transfer: 36.52 KiB received, 37.54 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 6 seconds ago transfer: 35.72 KiB received, 36.82 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 11 seconds ago transfer: 38.82 KiB received, 39.27 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 1 minute, 53 seconds ago transfer: 36.71 KiB received, 36.82 KiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= private key: (hidden) listening port: 51821 peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 10.124.0.4:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 second ago transfer: 35.11 KiB received, 35.06 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 3 seconds ago transfer: 35.42 KiB received, 35.71 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 3 seconds ago transfer: 38.14 KiB received, 38.87 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 6 seconds ago transfer: 44.92 KiB received, 43.39 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 8 seconds ago transfer: 36.58 KiB received, 35.72 KiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 10.124.0.2:51821 allowed ips: 10.101.0.7/32 latest handshake: 2 minutes ago transfer: 37.18 KiB received, 38.21 KiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= private key: (hidden) listening port: 51821 peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 2 seconds ago transfer: 38.70 KiB received, 39.90 KiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 2 seconds ago transfer: 37.82 KiB received, 39.76 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 3 seconds ago transfer: 35.47 KiB received, 35.42 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 6 seconds ago transfer: 45.72 KiB received, 45.58 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 7 seconds ago transfer: 37.30 KiB received, 36.52 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 9 seconds ago transfer: 36.52 KiB received, 35.53 KiB sent persistent keepalive: every 20 seconds peer: EM1ST/n1fwIsNVLMRuwMu7QDdHdiC7zHzxaaAZwXny4= endpoint: 10.106.0.9:51821 allowed ips: 10.101.0.8/32 latest handshake: 10 seconds ago transfer: 47.44 KiB received, 48.72 KiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= private key: (hidden) listening port: 51821 peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 8 seconds ago transfer: 42.54 KiB received, 44.52 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 8 seconds ago transfer: 46.66 KiB received, 47.81 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 8 seconds ago transfer: 43.16 KiB received, 44.92 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 50 seconds ago transfer: 39.12 KiB received, 40.96 KiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 1 minute, 54 seconds ago transfer: 39.15 KiB received, 39.77 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 2 minutes ago transfer: 38.43 KiB received, 38.63 KiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= private key: (hidden) listening port: 51821 peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: Now transfer: 38.39 KiB received, 38.77 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 5 seconds ago transfer: 39.86 KiB received, 39.13 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 6 seconds ago transfer: 36.93 KiB received, 36.70 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 7 seconds ago transfer: 39.21 KiB received, 39.52 KiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 1 minute, 57 seconds ago transfer: 35.07 KiB received, 35.55 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 58 seconds ago transfer: 36.16 KiB received, 36.53 KiB sent persistent keepalive: every 20 seconds ipv4-test wg interface -------------------- interface: netmaker public key: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= private key: (hidden) listening port: 51821 peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 10.124.0.4:51821 allowed ips: 10.101.0.1/32 latest handshake: 3 seconds ago transfer: 37.00 KiB received, 38.53 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 10.124.0.13:51821 allowed ips: 10.101.0.3/32 latest handshake: 6 seconds ago transfer: 39.96 KiB received, 40.30 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 8 seconds ago transfer: 38.43 KiB received, 37.92 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 9 seconds ago transfer: 36.74 KiB received, 38.03 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 58 seconds ago transfer: 39.57 KiB received, 39.15 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 2 minutes ago transfer: 35.32 KiB received, 35.11 KiB sent persistent keepalive: every 20 seconds checking logs for errors after a pull clientQA log errors -------------------- arm-redhat log errors -------------------- userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- clientqa03 log errors -------------------- clientqa04 log errors -------------------- ipv4-test log errors -------------------- restarting clients and checking config restarting clientQA -------------------- restarting arm-redhat -------------------- restarting userspace-wireguard -------------------- restarting clientqa02 -------------------- restarting clientqa03 -------------------- restarting clientqa04 -------------------- restarting ipv4-test -------------------- running pings after restart clientQA ping -------------------- nmap found 10.101.0.4 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.5 IS UP 10.116.0.2 IS DOWN 10.116.0.3 IS DOWN 10.116.0.4 IS DOWN 10.116.0.5 IS DOWN 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS DOWN 10.116.0.9 IS DOWN 10.101.0.6 IS UP 10.101.0.7 IS UP arm-redhat ping ----------------- nmap found 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.1 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP userspace-wireguard ping ----------------- nmap found 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP clientqa02 ping ----------------- nmap found 10.101.0.5 IS UP 10.116.0.2 IS DOWN 10.116.0.3 IS DOWN 10.116.0.4 IS DOWN 10.116.0.5 IS DOWN 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS DOWN 10.116.0.9 IS DOWN 10.101.0.6 IS UP 10.101.0.7 IS UP 10.101.0.8 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP clientqa03 ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.6 IS UP 10.101.0.7 IS UP clientqa04 ping ----------------- nmap found 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.7 IS UP ipv4-test ping ----------------- nmap found 10.101.0.4 IS UP 10.101.0.8 IS UP 10.101.0.5 IS UP 10.116.0.2 IS UP 10.116.0.3 IS UP 10.116.0.4 IS UP 10.116.0.5 IS UP 10.116.0.6 IS DOWN 10.116.0.7 IS DOWN 10.116.0.8 IS UP 10.116.0.9 IS UP 10.101.0.6 IS UP 10.101.0.1 IS UP 10.101.0.2 IS UP 10.101.0.3 IS UP checking routing after restart clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.8 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.2 10.116.0.0/16 dev netmaker src 10.101.0.2 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.3 10.116.0.0/16 dev netmaker src 10.101.0.3 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.4 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.5 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.6 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 10.116.0.0/16 dev netmaker src 10.101.0.6 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium ipv4-test ip route -------------------- default via 209.38.128.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.5 10.101.0.0/16 dev netmaker proto kernel scope link src 10.101.0.7 10.116.0.0/16 dev netmaker src 10.101.0.7 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.2 209.38.128.0/19 dev eth0 proto kernel scope link src 209.38.128.43 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium getting wg configs after restart clientQA wg -------------------- interface: netmaker public key: EM1ST/n1fwIsNVLMRuwMu7QDdHdiC7zHzxaaAZwXny4= private key: (hidden) listening port: 51821 peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 10.106.0.10:51821 allowed ips: 10.101.0.4/32, 10.101.0.5/32, 10.116.0.0/16, 10.101.0.6/32, 10.101.0.7/32, 10.101.0.1/32, 10.101.0.2/32, 10.101.0.3/32 latest handshake: 1 minute, 33 seconds ago transfer: 82.86 KiB received, 83.12 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= private key: (hidden) listening port: 51821 peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 8 seconds ago transfer: 73.03 KiB received, 72.52 KiB sent persistent keepalive: every 20 seconds peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 24 seconds ago transfer: 69.64 KiB received, 70.90 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 54 seconds ago transfer: 68.23 KiB received, 69.11 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 16 seconds ago transfer: 72.48 KiB received, 71.97 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 1 minute, 18 seconds ago transfer: 67.83 KiB received, 67.72 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 49 seconds ago transfer: 69.36 KiB received, 70.53 KiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= private key: (hidden) listening port: 51821 peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 10.124.0.2:51821 allowed ips: 10.101.0.7/32 latest handshake: 24 seconds ago transfer: 67.66 KiB received, 68.32 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 53 seconds ago transfer: 65.16 KiB received, 66.46 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 12 seconds ago transfer: 68.50 KiB received, 69.45 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 1 minute, 35 seconds ago transfer: 66.29 KiB received, 67.56 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 10.124.0.4:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 51 seconds ago transfer: 69.66 KiB received, 69.54 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 51 seconds ago transfer: 69.86 KiB received, 69.36 KiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= private key: (hidden) listening port: 51821 peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 32 seconds ago transfer: 65.92 KiB received, 65.66 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 53 seconds ago transfer: 65.58 KiB received, 65.66 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 1 minute, 9 seconds ago transfer: 63.66 KiB received, 63.77 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 21 seconds ago transfer: 66.53 KiB received, 66.69 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 24 seconds ago transfer: 66.81 KiB received, 66.45 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 36 seconds ago transfer: 66.89 KiB received, 66.23 KiB sent persistent keepalive: every 20 seconds peer: EM1ST/n1fwIsNVLMRuwMu7QDdHdiC7zHzxaaAZwXny4= endpoint: 10.106.0.9:51821 allowed ips: 10.101.0.8/32 latest handshake: 1 minute, 37 seconds ago transfer: 74.92 KiB received, 74.45 KiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= private key: (hidden) listening port: 51821 peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 39 seconds ago transfer: 64.81 KiB received, 65.30 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 55 seconds ago transfer: 63.64 KiB received, 65.65 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 1 minute, 11 seconds ago transfer: 62.61 KiB received, 63.42 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 1 minute, 15 seconds ago transfer: 66.42 KiB received, 67.05 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute, 21 seconds ago transfer: 66.78 KiB received, 67.66 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 1 minute, 22 seconds ago transfer: 62.97 KiB received, 64.20 KiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= private key: (hidden) listening port: 51821 peer: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= endpoint: 209.38.128.43:51821 allowed ips: 10.101.0.7/32 latest handshake: 37 seconds ago transfer: 59.21 KiB received, 59.73 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 64.23.184.110:51821 allowed ips: 10.101.0.1/32 latest handshake: 48 seconds ago transfer: 61.87 KiB received, 62.20 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 56 seconds ago transfer: 61.77 KiB received, 61.55 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 56 seconds ago transfer: 64.98 KiB received, 63.64 KiB sent persistent keepalive: every 20 seconds peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 24.144.84.150:51821 allowed ips: 10.101.0.3/32 latest handshake: 57 seconds ago transfer: 60.91 KiB received, 60.75 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 1 minute ago transfer: 61.73 KiB received, 61.74 KiB sent persistent keepalive: every 20 seconds ipv4-test wg interface -------------------- interface: netmaker public key: AAypHedUuihvfpuajNRAXFhDDV/pPaVdnIqrJ0N8908= private key: (hidden) listening port: 51821 peer: OlQOMB1SVej+CW5yQsAVuAqh4ckl5NHjre+KpjrKbgY= endpoint: 10.124.0.13:51821 allowed ips: 10.101.0.3/32 latest handshake: 30 seconds ago transfer: 60.92 KiB received, 60.54 KiB sent persistent keepalive: every 20 seconds peer: X37mUPaiPAT3clMr2xMImZKS8K3kyVlubWfVWUcc4jI= endpoint: 54.91.197.18:51821 allowed ips: 10.101.0.2/32 latest handshake: 32 seconds ago transfer: 59.80 KiB received, 59.43 KiB sent persistent keepalive: every 20 seconds peer: uI0P/jj8Rfijy58BGekwnxUiOQ8prZshtlMUXb5NuXY= endpoint: 165.22.115.252:51821 allowed ips: 10.101.0.4/32, 10.101.0.8/32 latest handshake: 37 seconds ago transfer: 60.92 KiB received, 60.86 KiB sent persistent keepalive: every 20 seconds peer: 11b836EyyjmeTd1F4Z5ZhvNHctaDAs6MK2cz4FYdDV0= endpoint: 209.38.230.81:51821 allowed ips: 10.101.0.6/32 latest handshake: 39 seconds ago transfer: 59.06 KiB received, 59.21 KiB sent persistent keepalive: every 20 seconds peer: 2j6tcAOJI81PioYgLYksuOI2ML4XhCTIdSfQf5RvFRQ= endpoint: 159.223.160.241:51821 allowed ips: 10.101.0.5/32, 10.116.0.0/16 latest handshake: 42 seconds ago transfer: 62.19 KiB received, 61.66 KiB sent persistent keepalive: every 20 seconds peer: KWPq7MH+g/6ynfiBNXARA6e3W5fJ1wXsQLEkwYI5RgU= endpoint: 10.124.0.4:51821 allowed ips: 10.101.0.1/32 latest handshake: 42 seconds ago transfer: 61.17 KiB received, 60.13 KiB sent persistent keepalive: every 20 seconds checking logs for errors after a restart clientQA log errors -------------------- arm-redhat log errors -------------------- userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- clientqa03 log errors -------------------- clientqa04 log errors -------------------- ipv4-test log errors --------------------