Linux intrusion triage
Triage a compromised Linux server: auth.log and the systemd journal, auditd rules and records, what bash history can and cannot tell you, the persistence locations attackers actually use (cron, systemd units, authorized_keys, LD_PRELOAD, PAM), and live response in order of volatility.
01Authentication logs and the journal
- Read sshd, sudo and PAM lines in auth.log and the journal, and pivot on key fingerprints
- Write auditd rules and decode SYSCALL, EXECVE and PATH records, including
auid - Know exactly what shell history can prove and how attackers blind it
- Hunt the five common persistence families and run live response in the right order
Most Linux intrusions start with a login: a stolen SSH key, a guessed password on an exposed service account, or a web shell that later becomes an interactive session. The first evidence is in the authentication log, written by sshd, sudo, su and PAM (Pluggable Authentication Modules, the library every login path calls to authenticate and open sessions).
Nov 13 02:09:41 web-03 sshd[2187]: Failed password for invalid user admin from 203.0.113.7 port 50412 ssh2
Nov 13 02:09:44 web-03 sshd[2187]: Failed password for invalid user admin from 203.0.113.7 port 50412 ssh2
Nov 13 02:14:33 web-03 sshd[2201]: Accepted publickey for deploy from 198.51.100.23 port 51744 ssh2: ED25519 SHA256:q3Vt9mXo2nLk8rTb5yWc1dFh7jGs4aPe6uIz0xKvR2E
Nov 13 02:14:33 web-03 sshd[2201]: pam_unix(sshd:session): session opened for user deploy(uid=1001) by (uid=0)
Nov 13 02:16:02 web-03 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/bash
Nov 13 02:16:02 web-03 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by deploy(uid=1001)Read it as a sequence. The failed passwords from 203.0.113.7 are background noise (internet-wide guessing). The real event is a publickey login for deploy from a different IP, followed two minutes later by sudo to a root shell. A key login means the attacker holds a private key whose public half is in deploy's authorized_keys. The SHA256: fingerprint tells you which one: run ssh-keygen -lf ~deploy/.ssh/authorized_keys and match it.
| Source | Where | What it holds |
|---|---|---|
| Auth log | /var/log/auth.log (Debian, Ubuntu), /var/log/secure (RHEL, Fedora) | sshd, sudo, su, PAM session open and close |
| wtmp / btmp | /var/log/wtmp, /var/log/btmp (binary) | Successful and failed logins; read with last -Faiw and lastb -Faiw |
| lastlog | /var/log/lastlog (binary, sparse) | Last login per account; lastlog prints it |
| systemd journal | /var/log/journal (persistent) or /run/log/journal (RAM) | Everything above plus service output, structured fields such as _PID, _COMM, _SYSTEMD_UNIT |
| Audit log | /var/log/audit/audit.log | Kernel audit records from auditd rules: execs, file writes, syscalls |
# sshd unit is ssh.service on Debian/Ubuntu, sshd.service on RHEL
journalctl -u ssh.service --since "2026-11-13 02:00" --until "2026-11-13 03:00" --utc -o short-iso-precise
journalctl _COMM=sudo --since "2026-11-13" -o json | jq -r '[.__REALTIME_TIMESTAMP, .MESSAGE] | @tsv'
journalctl --list-boots # reboots during the window are findings
last -Faiw | head -20 # full times, IPs, untruncated namesThe classic syslog format has no year and no time zone, so Nov 13 02:14:33 is local time on that box; newer distros write RFC 3339 timestamps with an offset. And with Storage=auto in journald.conf, the journal is only persistent if /var/log/journal exists; otherwise it lives in /run and a reboot erases it.
This is an advanced module
Investigations beyond Windows is part of the Pro pass. 4 more sections — the deep dives, the final design, follow-ups and the test — unlock with it.
Already a cyart.io or studio.cyart.io member? Your pass code is in your Studio account.