How can I identify and fix network issues without an interactive or ssh login

Question

I am not sure if this is an appropriate place for this question, so please suggest an alternative and move if required...

So I was trying to update a Bufallo Linkstation Pro v2 NAS from Debian Squeeze to Debian Lenny. I followed the Debian upgrade guide and everything seemed to work okay, except for an apt error with libav_tools, but that doesn't seem essential. I chose not to upgrade the kernel at this time (this was to be a later step).

Now, this NAS has no display device and no serial connection. The only access to this machine is via SSH. So, all things looking good I restarted my machine. The boot seemed like it worked, the correct lights were flashing, but now I can't connect via SSH.

The only way I have to access this NAS is by taking the disk out and inspecting it in another machine. Looking at the syslog etc, I can see that the machine is booting into Debian fine. However, something is up with the network. I manually edited the crontab to run the following diagnostics script:

/sbin/ifconfig
echo "---------------"
cat /etc/resolv.conf
echo "-----------------"
cat /etc/hosts
echo "-----------------"
cat /etc/netconfig
echo "----------------"
cat /etc/networks
echo "-----------------"
cat /etc/network/interfaces
echo "-----------------"
/usr/sbin/sshd -d
echo "-----------------"
netstat -tulpn

Here's the output of the script:

---------------
domain Home
search Home
nameserver 192.168.1.1
-----------------
127.0.0.1   localhost
127.0.1.1   debian.Home debian

# The following lines are desirable for IPv6 capable hosts
#::1     ip6-localhost ip6-loopback
#fe00::0 ip6-localnet
#ff00::0 ip6-mcastprefix
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters
-----------------
#
# The network configuration file. This file is currently only used in
# conjunction with the TI-RPC code in the libtirpc library.
#
# Entries consist of:
#
#       <network_id> <semantics> <flags> <protofamily> <protoname> \
#               <device> <nametoaddr_libs>
#
# The <device> and <nametoaddr_libs> fields are always empty in this
# implementation.
#
udp        tpi_clts      v     inet     udp     -       -
tcp        tpi_cots_ord  v     inet     tcp     -       -
udp6       tpi_clts      v     inet6    udp     -       -
tcp6       tpi_cots_ord  v     inet6    tcp     -       -
rawip      tpi_raw       -     inet      -      -       -
local      tpi_cots_ord  -     loopback  -      -       -
unix       tpi_cots_ord  -     loopback  -      -       -
----------------
default     0.0.0.0
loopback    127.0.0.0
link-local  169.254.0.0

-----------------
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#allow-hotplug eth0
#iface eth0 inet dhcp
auto eth0
iface eth0 inet static
    address 192.168.1.150
    netmask 255.255.255.0
    gateway 192.168.1.1
-----------------
debug1: sshd version OpenSSH_5.5p1 Debian-6+squeeze5
debug1: could not open key file '/etc/ssh/ssh_host_rsa_key': Permission denied
Could not load host key: /etc/ssh/ssh_host_rsa_key
debug1: could not open key file '/etc/ssh/ssh_host_dsa_key': Permission denied
Could not load host key: /etc/ssh/ssh_host_dsa_key
debug1: setgroups() failed: Operation not permitted
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_adj from 0 to -17
socket: Address family not supported by protocol
debug1: Bind to port 22 on 0.0.0.0.
Bind to port 22 on 0.0.0.0 failed: Permission denied.
Cannot bind any address.
-----------------
(No info could be read for "-p": geteuid()=1000 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:60756           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -               
udp        0      0 0.0.0.0:40009           0.0.0.0:*                           -               
udp        0      0 0.0.0.0:111             0.0.0.0:*                           -               
udp        0      0 0.0.0.0:764             0.0.0.0:*                           -               
udp        0      0 127.0.0.1:161           0.0.0.0:*                           -               
udp        0      0 0.0.0.0:39720           0.0.0.0:*                           -               
udp        0      0 127.0.0.1:809           0.0.0.0:*                           -   

Why questions are twofold: - Why does ifconfig not print any output? - What other information can I obtain via a cronjob that will help diagnose network issues and what other files can I look at without an interactive shell?

Answer 1

(Turning my comments into an answer)

Try /sbin/ifconfig -a. I've had issues in some cases where ifconfig won't display any data until 1) I'm root or 2) I supply the -a flag. It's counter intuitive to what I know as a sys admin, but it seems to be the case in some instances.

You can also look at netstat -rn to look at the routing table -- I'm not extremely familiar with Debian, but I didn't see what I'd expect from that command in your output.

Another command is sudo /sbin/ip a. It (at least on my system) requires root access to execute, but gives a similar set of information than ifconfig.