Capture SSH and sshd strace Logs for Troubleshooting

When ssh -vvv and sudo journalctl -u sshd still leave gaps, strace records the actual system calls OpenSSH makes: files opened, sockets created, DNS lookups, and errno values such as ENOENT, EACCES, or ECONNREFUSED. On Debian and Ubuntu, the service is commonly named ssh, so use journalctl -u ssh if sshd.service does not exist. This guide shows how to collect client and server evidence without turning the page into a general SSH troubleshooting walkthrough.

Quick reference: SSH and sshd debug commands

Start with OpenSSH's own debug output before strace:

Escalation path:

ssh -vvv / journalctl
        ↓
sshd -ddd on alternate port
        ↓
strace on client and/or server
        ↓
tcpdump on the test port (optional)

Use strace when OpenSSH debug lines name a symptom but not the failing file, permission, or syscall.

When should you use strace for SSH?

strace helps when the failure involves:

• Missing or unreadable files (sshd_config, host keys, authorized_keys, PAM modules)
• Permission or ownership problems on key paths
• DNS or NSS lookups that hang or fail
• Socket bind, listen, accept, or connect errors
• Unexpected syscall errno values during authentication

It is not the first step. Run ssh -vvv, inspect sudo journalctl -u sshd, and validate config with sudo /usr/sbin/sshd -t first. strace is for the cases where those logs stop short of the root cause.

Install strace

On RHEL-family systems:

sudo dnf install strace

On Debian or Ubuntu:

sudo apt install strace

Confirm the binary is available:

strace -V

Sample output:

strace -- version 6.12

Capture sshd strace safely on an alternate port

The preferred method keeps production sshd on port 22 running and starts a one-off debug server on a high port such as 2222. Debug-mode sshd handles a single connection, which makes the trace easier to read than attaching to a busy master process.

Pick a hostname label for log files:

HOST=$(hostname -s)

On the server, start debug sshd under strace and send OpenSSH debug output to a separate file:

sudo strace -ftttTvyyo "/tmp/sshd-${HOST}.strace" -s 4096 /usr/sbin/sshd -ddd -p 2222 2>"/tmp/sshd-${HOST}.debug"

Flag meanings:

• -f — follow child processes created during the session
• -ttt — Unix epoch timestamps with microsecond precision
• -T — time spent in each syscall
• -v — less abbreviated arguments
• -yy — decode file descriptors and socket endpoints
• -s 4096 — print longer strings
• -o — write the trace to a file instead of the terminal
• -ddd — maximum sshd debug level
• -p 2222 — listen on the test port only

Leave that terminal open. In another session, confirm the listener:

ss -lntp 'sport = :2222'

Sample output:

LISTEN 0 128 0.0.0.0:2222 0.0.0.0:* users:(("sshd",pid=11651,fd=7))
LISTEN 0 128    [::]:2222       [::]:* users:(("sshd",pid=11651,fd=8))

If clients connect from another host, open 2222/tcp in the host firewall before testing. Local ssh -p 2222 localhost tests do not need a firewall change.

Inspect the server debug file for config loading at startup:

head -3 "/tmp/sshd-${HOST}.debug"

Sample output:

debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 3657
debug2: parse_server_config_depth: config /etc/ssh/sshd_config len 3657

After you finish one test connection, the debug sshd exits and returns you to the shell prompt.

Connect with SSH client debug and strace

From the client, run ssh at maximum verbosity under strace. Use the same test port and capture debug and syscall output separately:

HOST=$(hostname -s)

Wrap the client connection with strace and save verbose SSH output separately:

strace -ftttTvyyo "/tmp/ssh-${HOST}.strace" -s 4096 ssh -vvv -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null user@server exit 2>"/tmp/ssh-${HOST}.debug"

Replace user@server with the account and host you are testing. The trailing exit closes the session immediately after login so the capture ends cleanly.

Client debug output starts similarly to:

OpenSSH_9.9p1, OpenSSL 3.5.5 27 Jan 2026
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0

Useful syscall lines in the client trace often include connect, openat, read, and write:

grep -E 'connect|openat' "/tmp/ssh-${HOST}.strace" | head -2

Sample output:

connect(3<TCPv6:[49602]>, {sa_family=AF_INET6, sin6_port=htons(2222), ...}, 28) = 0 <0.004973>
openat(AT_FDCWD, "/etc/ssh/ssh_config", O_RDONLY) = 3</etc/ssh/ssh_config> <0.000891>

Errors appear as negative return values, for example = -1 ENOENT or = -1 EACCES.

Capture network packets at the same time

For stubborn connection or handshake failures, record packets on the test port while the strace and debug captures run. On the server:

sudo tcpdump -i any -s 0 -nn -w /tmp/ssh-debug.pcap port 2222

Stop tcpdump after the test. You then have five artifacts for one attempt:

client debug (/tmp/ssh-HOST.debug)
client strace (/tmp/ssh-HOST.strace)
server debug (/tmp/sshd-HOST.debug)
server strace (/tmp/sshd-HOST.strace)
packet capture (/tmp/ssh-debug.pcap)

Align them by timestamp when you compare client, server, and wire behavior.

Analyze SSH and sshd strace logs

strace files are large. Filter before reading line by line.

List syscalls that returned an error:

grep '= -1 ' /tmp/sshd-vm1.strace | head

Sample output:

11651 ... access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000271>
11651 ... openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) <0.000386>

Not every ENOENT matters. ENOENT on optional paths such as /etc/ld.so.preload is normal. Focus on failures at the time of your failed login.

Find config and authentication file access:

grep -E 'openat|stat|access' /tmp/sshd-vm1.strace | grep -E 'sshd_config|authorized_keys|passwd|pam' | head

Sample output:

11651 ... openat(AT_FDCWD, "/etc/ssh/sshd_config", O_RDONLY) = 7</etc/ssh/sshd_config> <0.000515>

Inspect socket activity:

grep -E 'socket|bind|listen|accept|connect' /tmp/sshd-vm1.strace | head

Sample output:

11651 ... socket(AF_INET6, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 7<UDPv6:[47564]> <0.000460>
11651 ... bind(7<NETLINK:[47563]>, {sa_family=AF_NETLINK, ...}, 12) = 0 <0.000510>
11651 ... connect(7<UDPv6:[47564]>, {sa_family=AF_INET6, sin6_port=htons(2222), ...}, 28) = 0 <0.000432>

Cross-check the same timestamp window in the matching .debug file and in journalctl -u sshd if production sshd was also involved.

Attach strace to the existing sshd process

When you cannot use a test port, attach to the running master process. Read the PID:

cat /var/run/sshd.pid

Sample output:

1234

Attach with fork following enabled:

sudo strace -f -p 1234 -o /tmp/sshd-live.strace

Without -f, strace watches only the master process and misses per-connection children that handle logins. Press Ctrl+C to detach.

Sample live trace lines during a connection:

accept(8, {sa_family=AF_INET6, sin6_port=htons(33590), ...}, [128 => 28]) = 9
socketpair(AF_UNIX, SOCK_STREAM, 0, [12, 13]) = 0
clone(...) = 11888

This method is secondary because:

• Production sessions mix into the same trace
• Multiple simultaneous logins are harder to separate
• Debug output is not at -ddd unless you also changed logging elsewhere

Use it when port 22 must stay the only listener and you need a short live sample.

Protect sensitive data in strace logs

Traces are verbose because they record real paths and arguments. Before attaching logs to a support case:

• Prefer a dedicated test account instead of a privileged production user
• Search for home-directory paths, key filenames, and environment strings you do not want exported
• Compress and encrypt archives when your policy requires it
• Delete /tmp/ssh-* and /tmp/sshd-* files after the investigation

Do not paste full strace files into public forums or tickets without redaction.

Summary

Collect SSH evidence in layers: ssh -vvv and journalctl -u sshd first, then a one-off sshd -ddd on an alternate port wrapped with strace -ftttTvyyo, and a matching client strace + ssh -vvv run against that port. Add tcpdump on the test port when the failure is still ambiguous on the wire.

Filter traces with grep '= -1 ', path greps for sshd_config and authorized_keys, and socket-related syscalls. Treat strace -f -p on the production master PID as a fallback, not the default. Review and delete captures when finished because they can contain authentication-sensitive data.

References

• Red Hat Customer Portal — How can I gather straces from ssh and sshd?
• Linux strace(1) manual — https://man7.org/linux/man-pages/man1/strace.1.html
• OpenSSH ssh(1) manual — https://man.openbsd.org/ssh.1
• OpenSSH sshd(8) manual — https://man.openbsd.org/sshd.8