SSH troubleshooting
Posted on Fri, Feb 16, 2024 at 11:40 by Hubertus A. Haniel
When SSH issues are reported it is all to tempting to jump on a box make changes to the config file to fix the suspected issues and restarting sshd.
This may not always be the best way because:
- Error messages in syslog may be misleading as it is difficult to track down an individual session and debug messages may be filtered out in syslog
- On a busy system other users that still work may get disconnected/locked out while the problem is being worked on
- In the worst case you will get disconnected and will not get back into the system other than via the console
The way to avoid this is to start ssh with the "-d" option which will start sshd in debugging mode and it will listen for one session only. If required multiple -dd (up to three) can be specified to increase the debugging level. Obviously the running ssh session is already listening on port 22 so we do not interfere with that so we need to specify a different port to listen on that is not in use with the -p option. For security reasons build into sshd you must run sshd with the full path of where it is installed.
# /usr/sbin/sshd -ddd -p 2222 debug2: load_server_config: filename /etc/ssh/sshd_config debug2: load_server_config: done config len = 595 debug2: parse_server_config: config /etc/ssh/sshd_config len 595 debug3: /etc/ssh/sshd_config:21 setting Protocol 2 debug3: /etc/ssh/sshd_config:36 setting SyslogFacility AUTHPRIV debug3: /etc/ssh/sshd_config:66 setting PasswordAuthentication yes debug3: /etc/ssh/sshd_config:70 setting ChallengeResponseAuthentication no debug3: /etc/ssh/sshd_config:80 setting GSSAPIAuthentication yes debug3: /etc/ssh/sshd_config:82 setting GSSAPICleanupCredentials yes debug3: /etc/ssh/sshd_config:96 setting UsePAM yes debug3: /etc/ssh/sshd_config:99 setting AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES debug3: /etc/ssh/sshd_config:100 setting AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT debug3: /etc/ssh/sshd_config:101 setting AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE debug3: /etc/ssh/sshd_config:102 setting AcceptEnv XMODIFIERS debug3: /etc/ssh/sshd_config:108 setting X11Forwarding yes debug3: /etc/ssh/sshd_config:131 setting Subsystem sftp /usr/libexec/openssh/sftp-server debug3: /etc/ssh/sshd_config:138 setting PermitRootLogin without-password debug1: sshd version OpenSSH_5.3p1 debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key. debug1: read PEM private key done: type RSA debug1: private host key: #0 type 1 RSA debug3: Not a RSA1 key file /etc/ssh/ssh_host_dsa_key. debug1: read PEM private key done: type DSA debug1: private host key: #1 type 2 DSA debug1: rexec_argv[0]='/usr/sbin/sshd' debug1: rexec_argv[1]='-ddd' debug1: rexec_argv[2]='-p' debug1: rexec_argv[3]='2222' debug2: fd 3 setting O_NONBLOCK debug1: Bind to port 2222 on 0.0.0.0. Server listening on 0.0.0.0 port 2222. debug2: fd 4 setting O_NONBLOCK debug1: Bind to port 2222 on ::. Server listening on :: port 2222.
Now the user can connect to that port with something like "ssh -p 2222 user@host" which will then give us detailed information of what is happening with that connection.
To make changes to the config file and to debug/test these changes it is best to make a copy of the existing config file and edit this file instead so we copy the config with something like:
# cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.TEST
Then we can start a session using this config file with:
# /usr/sbin/sshd -ddd -p 2222 -f /etc/ssh/sshd_config.TEST debug2: load_server_config: filename /etc/ssh/sshd_config.TEST
Once confident that our changes are safe and they will not break anything else we can copy the changes to the real config file and restart the main ssh daemon on the system.
Edited on: Fri, Feb 16, 2024 12:36Posted in HowTo (RSS), System - AIX (RSS), System - Linux (RSS), System - Solaris (RSS)
