Linux logging basics
Where Linux records security events (journald, syslog, wtmp), how auditd watches files and syscalls, what is worth alerting on, and why logs must leave the host before an attacker with root edits them.
01Where Linux writes security events
- Find authentication, privilege and service events in journald and syslog files
- Write auditd rules that watch sensitive files and record command execution
- Choose host events worth alerting on
- Explain why logs must be forwarded off-host and timestamps synchronised
| Source | Location | What you find there | Read with |
|---|---|---|---|
| systemd journal | /var/log/journal/ (binary) | Everything from services, the kernel and sshd, with structured fields | journalctl |
| Auth log | /var/log/auth.log (Debian/Ubuntu), /var/log/secure (RHEL) | SSH logins, sudo, su, PAM failures | grep, less |
| Login records | /var/log/wtmp, btmp, lastlog | Successful logins, failed logins, last login per user | last, lastb, lastlog |
| Kernel ring buffer | Memory, also in the journal | Module loads, OOM kills, segfaults, USB devices | dmesg, journalctl -k |
| Audit log | /var/log/audit/audit.log | Kernel audit records: syscalls, file watches, logins | ausearch, aureport |
# Failed and successful SSH logins today
journalctl -u ssh --since today | grep -E 'Failed|Accepted'
# Accepted publickey for alice from 203.0.113.7 port 51234 ssh2: ED25519 SHA256:Vb3...
# Failed password for invalid user admin from 198.51.100.23 port 40022 ssh2
# Every sudo command, with the invoking user and working directory
journalctl _COMM=sudo --since "-24h"
# alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/systemctl restart nginxOn some distributions the journal lives only in /run and disappears at reboot. Set Storage=persistent in /etc/systemd/journald.conf (or create /var/log/journal) and size it with SystemMaxUse=.
02auditd: watching files and syscalls
The Linux audit subsystem records events inside the kernel and hands them to the auditd daemon. It sees what syslog cannot: every execve, every write to a watched file, every module load, along with the audit UID (auid). The auid is set at login and survives sudo and su, so you can tell that "root" running a command was really alice.
# /etc/audit/rules.d/50-security.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/ssh/sshd_config -p wa -k sshd
-w /etc/cron.d/ -p wa -k cron
# Commands run as root by a real logged-in user (auid 4294967295 means "unset")
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=4294967295 -k root_cmd
# Kernel module loading
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module -k modules
# Lock the rules until reboot (an attacker must reboot to change them, which is noisy)
-e 2sudo augenrules --load
sudo ausearch -k identity -i --start today # -i turns numeric UIDs into names
sudo aureport --summary --start this-weekRecording all command execution on a busy build server can produce gigabytes a day and measurable CPU load. Scope rules with filters such as euid=0 or auid>=1000, exclude noisy service accounts, and test volume before rolling out fleet-wide. eBPF-based tools (Falco, Tetragon) and osquery are alternatives with richer context.
03What is worth alerting on
Host logs are only useful if someone looks at the right events. Start with a small set of high-signal detections, each tied to an attacker behaviour, then tune them against a few weeks of normal activity.
| Behaviour | Signal | Source |
|---|---|---|
| Credential attack | Many failed SSH logins, then a success from the same IP | auth log or journal |
| Unusual access | Successful login from a new country, ASN or at an odd hour for that user | auth log plus enrichment |
| Persistence | New user, new authorized_keys, cron or systemd unit created | auditd file watches |
| Privilege escalation | sudo by a user who never uses it; new setuid binary | journal, auditd, file integrity scan |
| Defence evasion | auditd stopped, rules changed, log files truncated, history -c | auditd, gaps in forwarded logs |
| Rootkit or driver | Kernel module loaded outside a patch window | auditd modules key, dmesg |
- ACTOR
- EDGE
- SERVICE
- DATA
- STREAM
- Live flow
- Lookup / async
04Forward logs off the host
An attacker who gets root can edit or delete anything on the host, including logs. Shipping events to a separate system as they happen means the record of the break-in survives the break-in. The receiving side should be write-only from the host's point of view: hosts can append, never delete.
# /etc/rsyslog.d/90-forward.conf: forward everything over TLS with a disk-backed queue
global(DefaultNetstreamDriverCAFile="/etc/ssl/certs/log-ca.pem")
action(type="omfwd" target="logs.internal.cyart.io" port="6514" protocol="tcp"
StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name"
queue.type="LinkedList" queue.filename="fwd" queue.maxDiskSpace="1g"
queue.saveOnShutdown="on" action.resumeRetryCount="-1")- Agents: Fluent Bit, Vector, the Elastic or Splunk forwarders, or
systemd-journal-uploadall work. Pick one, run it everywhere, and alert when a host stops sending. - Time: run chrony or another NTP client and log in UTC. Correlating an SSH login on one host with a database query on another is impossible with clocks minutes apart.
- Silence is a signal: a host that stops forwarding is either broken or compromised. Treat missing heartbeats as an alert, not a dashboard curiosity.
- Do not log secrets: tokens in URLs, passwords in failed-login usernames (users type them into the wrong field) and environment dumps all end up in the SIEM, which then needs protecting like a secrets store.
05Bringing it together
A frequent scenario: "you suspect a server was compromised last week, what do you look at?". Good answers go from auth logs (who logged in, from where) to sudo and auditd (what they ran, with auid), to persistence locations, and immediately note that local logs may have been altered so the forwarded copy in the SIEM is the source of truth.
"On each host I keep the journal persistent, add auditd rules for identity files, sudoers, sshd config, cron and module loads, and record root commands from real users using the audit UID so sudo does not hide who acted. Everything forwards over TLS to an append-only store as it happens, because an attacker with root can rewrite local files. Detections start small: brute force followed by success, new persistence, auditd stopping, and hosts that go silent."
Alice logs in over SSH and runs `sudo -i`, then deletes a file as root. Which field lets you attribute the deletion to alice in audit logs?
Q1Why lock audit rules with `-e 2`?+
It makes the rule set immutable until reboot. An attacker with root cannot quietly remove the watches; they must reboot the host, which is itself visible.
Q2Is the local log ever trustworthy after a compromise?+
Treat it as possibly altered. Compare it with the forwarded copy; differences are evidence of tampering in their own right.
Q3Why might a password appear in auth logs?+
Users sometimes type their password into the username prompt. The failed attempt is then logged with the password as the username, so auth logs need access control and retention limits.