First of all, I would want to list the most important tools for the debugging process:
hostname – the program that allows to find out the host name of the server
ping – the program that allows you to send an IP-packet to another server. It's useful because it allows to check multiple things - if a simple command ping google.com executes successfully, then it signifies that the following things are configured correctly:
traceroute – (can be found in a package with the same name – traceroute) – displays the list of servers that happen to be in between the server the comand is executed on, and the one that is supplied as an argument to this. However, is not very useful nowadays because many modern routers configured not to expose information about themselves
dig – lets you test the configuration of DNS – sends a request to translate a hostname to an IP-address to a DNS-server
netstat or, a newer one ss – lists all listening ports, very useful instrument – one of the first when debugging problems with services. For netstat, the command is netstat -tulpen – easy to remember because I learned it together with a mnemonic rule from Sander van Vugt, who is Dutch, and the Netherlands is the country of tulpen (tulips).
With ss, to list all listening TCP ports: ss -tlpn.
On macOS, where ss doesn't exist, the equivalent is lsof -nP -iTCP -sTCP:LISTEN. I remember it mnemonically as "lsof no problem internet tcp search tcp:listen" – or, flag by flag:
lsof
-nP
-iTCP
-sTCP:LISTENList open files → don't resolve names → Internet TCP → state: LISTEN.
macOS does have netstat, but it's the BSD version with different flags, and I didn't bother to learn it.
nmap – an extremely powerful and versatile tool. Can be used in many circumstances, but one of the most interesting ones is that it allows to list open ports on a remote host (nmap -p- <hostname>)
ip – the last one in the list of tools for debugging, but the first one in the list of the network configuration, and it means that it is important to begin your debugging process from understanding whether the current network configuration is correct. To edit connections should be used nmcli (see next)
It's worth to begin with ip address show to understand, like it was said in the last bullet of the previous list, that the current network configuration is in order
ip route show - check what host plays the role of a router
cat /etc/resolv.conf - check the configured DNS servers. But beware that one should not edit this file directly, because NetworkManager will overwrite it after it restarts or a connection is activated. Consider an example:
[root@workstation ~]# nmcli connection edit System\ eth0
nmcli> set ipv4.ignore-auto-dns [yes|no]
nmcli> set ipv4.dns 8.8.8.8
nmcli> save
nmcli> print
nmcli> quit
[root@workstation ~]# nmcli c down System\ eth0; nmcli c up System\ eth0The last commands should be executed together, chained with ; in order not to get locked out of your remote server.
ping google.com, because it allows to check three symptoms at once (see the previous list). If the command does not succeed, then check if the router is accessible – ping <output from ip route show>. If this doesn't work either – check routes configurations, for example, check whether the specified IP address really belongs to a router or not, and if it does then understand why it doesn't accept any packets - is it turned on? And if it can be pinged, then there may be two conclusions: it is either not a router, or the problem is in-between the router and the target host => try using ping on a server in the range
To get the route from A to B use traceroute. For instance, the output from a virtual machine within a network (192.168.121.0/24) on my server:
[root@workstation ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 gateway (192.168.121.1) 0.207 ms 0.152 ms 0.105 ms
2 static.249.206.9.5.clients.your-server.de (5.9.206.249) 0.326 ms 0.304 ms 0.298 ms
3 core22.fsn1.hetzner.com (213.239.229.85) 0.511 ms 0.471 ms core21.fsn1.hetzner.com (213.239.229.81) 0.237 ms
4 core1.fra.hetzner.com (213.239.245.177) 4.998 ms core4.fra.hetzner.com (213.239.245.14) 4.973 ms 4.901 ms
5 72.14.218.94 (72.14.218.94) 6.452 ms 72.14.218.176 (72.14.218.176) 4.991 ms 72.14.218.94 (72.14.218.94) 6.374 ms
6 108.170.251.129 (108.170.251.129) 5.147 ms 108.170.252.1 (108.170.252.1) 6.166 ms 108.170.251.129 (108.170.251.129) 5.219 ms
7 216.239.59.185 (216.239.59.185) 5.261 ms 108.170.236.91 (108.170.236.91) 5.215 ms 108.170.226.145 (108.170.226.145) 6.143 ms
8 google-public-dns-a.google.com (8.8.8.8) 5.141 ms 5.146 ms 4.864 ms
[root@workstation ~]# ip route show
default via 192.168.121.1 dev eth0 proto static metric 100
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.120 metric 100When traceroute does not show the route, dig can be used instead. dig's output consists of multiple sections:
; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63272
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com. IN A
;; ANSWER SECTION:
google.com. 250 IN A 216.58.214.110
;; Query time: 1 msec
;; SERVER: 192.168.121.1#53(192.168.121.1)
;; WHEN: Mon Apr 23 21:03:16 UTC 2018
;; MSG SIZE rcvd: 55The sections are neatly separated with a blank line. The most interesting sections for us are:
status field, which in the example above is NOERROR – which is good, the hostname got resolvedFor the domains that the DNS server is not aware of, dig returns NXDOMAIN status.