CYART // SCHEMA · SIEM-PIPELINE9 NODES · 9 LINKSENRICHEDALERTACTOR · Sources — cloud, EDR, auth, netACTOR01Sourcescloud, EDR, auth, netACTOR · Agents / APIs — Fluent Bit, pullACTOR02Agents / APIsFluent Bit, pullEDGE · Ingest gateway — auth, backpressureEDGE03Ingest gatewayauth, backpressureSTREAM · Stream — Kafka, per-tenant topicsSTREAM04StreamKafka, per-tenant topicsSERVICE · Normaliser — parse → OCSF/ECSSERVICE05Normaliserparse → OCSF/ECSSERVICE · Detection — rules, correlation, MLSERVICE06Detectionrules, correlation, MLDATA · Hot store — 7 to 30 d, searchDATA07Hot store7 to 30 d, searchDATA · Cold store — object store, 1 y+DATA08Cold storeobject store, 1 y+DATA · Cases / alerts — dedupe, SOARDATA09Cases / alertsdedupe, SOAR

Linux logging basics

Where Linux records security events (journald, syslog, wtmp), how auditd watches files and syscalls, what is worth alerting on, and why logs must leave the host before an attacker with root edits them.

FREEBy ·Lesson·11 min read·Beginner → Intermediate
journaldauditdauth.logLinux audit ruleslog forwardinghost detection

01Where Linux writes security events

✓What you will learn
  • Find authentication, privilege and service events in journald and syslog files
  • Write auditd rules that watch sensitive files and record command execution
  • Choose host events worth alerting on
  • Explain why logs must be forwarded off-host and timestamps synchronised
SourceLocationWhat you find thereRead with
systemd journal/var/log/journal/ (binary)Everything from services, the kernel and sshd, with structured fieldsjournalctl
Auth log/var/log/auth.log (Debian/Ubuntu), /var/log/secure (RHEL)SSH logins, sudo, su, PAM failuresgrep, less
Login records/var/log/wtmp, btmp, lastlogSuccessful logins, failed logins, last login per userlast, lastb, lastlog
Kernel ring bufferMemory, also in the journalModule loads, OOM kills, segfaults, USB devicesdmesg, journalctl -k
Audit log/var/log/audit/audit.logKernel audit records: syscalls, file watches, loginsausearch, aureport
Reading authentication eventsbash
# Failed and successful SSH logins today
journalctl -u ssh --since today | grep -E 'Failed|Accepted'
# Accepted publickey for alice from 203.0.113.7 port 51234 ssh2: ED25519 SHA256:Vb3...
# Failed password for invalid user admin from 198.51.100.23 port 40022 ssh2

# Every sudo command, with the invoking user and working directory
journalctl _COMM=sudo --since "-24h"
# alice : TTY=pts/0 ; PWD=/home/alice ; USER=root ; COMMAND=/usr/bin/systemctl restart nginx
ijournald persistence

On some distributions the journal lives only in /run and disappears at reboot. Set Storage=persistent in /etc/systemd/journald.conf (or create /var/log/journal) and size it with SystemMaxUse=.

02auditd: watching files and syscalls

The Linux audit subsystem records events inside the kernel and hands them to the auditd daemon. It sees what syslog cannot: every execve, every write to a watched file, every module load, along with the audit UID (auid). The auid is set at login and survives sudo and su, so you can tell that "root" running a command was really alice.

A starter audit rulesetbash
# /etc/audit/rules.d/50-security.rules
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
-w /etc/ssh/sshd_config -p wa -k sshd
-w /etc/cron.d/ -p wa -k cron

# Commands run as root by a real logged-in user (auid 4294967295 means "unset")
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid>=1000 -F auid!=4294967295 -k root_cmd

# Kernel module loading
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module -k modules

# Lock the rules until reboot (an attacker must reboot to change them, which is noisy)
-e 2
Loading and queryingbash
sudo augenrules --load
sudo ausearch -k identity -i --start today      # -i turns numeric UIDs into names
sudo aureport --summary --start this-week
!Auditing every execve is expensive

Recording all command execution on a busy build server can produce gigabytes a day and measurable CPU load. Scope rules with filters such as euid=0 or auid>=1000, exclude noisy service accounts, and test volume before rolling out fleet-wide. eBPF-based tools (Falco, Tetragon) and osquery are alternatives with richer context.

03What is worth alerting on

Host logs are only useful if someone looks at the right events. Start with a small set of high-signal detections, each tied to an attacker behaviour, then tune them against a few weeks of normal activity.

BehaviourSignalSource
Credential attackMany failed SSH logins, then a success from the same IPauth log or journal
Unusual accessSuccessful login from a new country, ASN or at an odd hour for that userauth log plus enrichment
PersistenceNew user, new authorized_keys, cron or systemd unit createdauditd file watches
Privilege escalationsudo by a user who never uses it; new setuid binaryjournal, auditd, file integrity scan
Defence evasionauditd stopped, rules changed, log files truncated, history -cauditd, gaps in forwarded logs
Rootkit or driverKernel module loaded outside a patch windowauditd modules key, dmesg
FigFrom host to SIEM
CYART // SCHEMA · SIEM-PIPELINE9 NODES · 9 LINKSENRICHEDALERTACTOR · Sources — cloud, EDR, auth, netACTOR01Sourcescloud, EDR, auth, netACTOR · Agents / APIs — Fluent Bit, pullACTOR02Agents / APIsFluent Bit, pullEDGE · Ingest gateway — auth, backpressureEDGE03Ingest gatewayauth, backpressureSTREAM · Stream — Kafka, per-tenant topicsSTREAM04StreamKafka, per-tenant topicsSERVICE · Normaliser — parse → OCSF/ECSSERVICE05Normaliserparse → OCSF/ECSSERVICE · Detection — rules, correlation, MLSERVICE06Detectionrules, correlation, MLDATA · Hot store — 7 to 30 d, searchDATA07Hot store7 to 30 d, searchDATA · Cold store — object store, 1 y+DATA08Cold storeobject store, 1 y+DATA · Cases / alerts — dedupe, SOARDATA09Cases / alertsdedupe, SOAR
  • ACTOR
  • EDGE
  • SERVICE
  • DATA
  • STREAM
  • Live flow
  • Lookup / async
//Agents collect journal, syslog and audit events on each host, forward them over TLS to a pipeline that parses and enriches them, and land them in a SIEM where detections run. The host keeps only a short local buffer.

04Forward logs off the host

An attacker who gets root can edit or delete anything on the host, including logs. Shipping events to a separate system as they happen means the record of the break-in survives the break-in. The receiving side should be write-only from the host's point of view: hosts can append, never delete.

rsyslog forwarding over TLSbash
# /etc/rsyslog.d/90-forward.conf: forward everything over TLS with a disk-backed queue
global(DefaultNetstreamDriverCAFile="/etc/ssl/certs/log-ca.pem")
action(type="omfwd" target="logs.internal.cyart.io" port="6514" protocol="tcp"
       StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name"
       queue.type="LinkedList" queue.filename="fwd" queue.maxDiskSpace="1g"
       queue.saveOnShutdown="on" action.resumeRetryCount="-1")
  • Agents: Fluent Bit, Vector, the Elastic or Splunk forwarders, or systemd-journal-upload all work. Pick one, run it everywhere, and alert when a host stops sending.
  • Time: run chrony or another NTP client and log in UTC. Correlating an SSH login on one host with a database query on another is impossible with clocks minutes apart.
  • Silence is a signal: a host that stops forwarding is either broken or compromised. Treat missing heartbeats as an alert, not a dashboard curiosity.
  • Do not log secrets: tokens in URLs, passwords in failed-login usernames (users type them into the wrong field) and environment dumps all end up in the SIEM, which then needs protecting like a secrets store.

05Bringing it together

▭On the whiteboard · Minimum viable host logging4 points
01
collect
journal persistent | auth events | auditd watches on identity, sudoers, sshd, cron | module loads
02
protect
forward over TLS in near real time | append-only store | audit rules locked with -e 2
03
detect
brute force then success | new persistence | auditd stopped | host silent
04
operate
NTP everywhere | UTC | retention agreed with legal | volume budget per host
◎Interview radar

A frequent scenario: "you suspect a server was compromised last week, what do you look at?". Good answers go from auth logs (who logged in, from where) to sudo and auditd (what they ran, with auid), to persistence locations, and immediately note that local logs may have been altered so the forwarded copy in the SIEM is the source of truth.

“How to say it in the room

"On each host I keep the journal persistent, add auditd rules for identity files, sudoers, sshd config, cron and module loads, and record root commands from real users using the audit UID so sudo does not hide who acted. Everything forwards over TLS to an append-only store as it happens, because an attacker with root can rewrite local files. Detections start small: brute force followed by success, new persistence, auditd stopping, and hosts that go silent."

Test yourself

Alice logs in over SSH and runs `sudo -i`, then deletes a file as root. Which field lets you attribute the deletion to alice in audit logs?

Q1Why lock audit rules with `-e 2`?+

It makes the rule set immutable until reboot. An attacker with root cannot quietly remove the watches; they must reboot the host, which is itself visible.

Q2Is the local log ever trustworthy after a compromise?+

Treat it as possibly altered. Compare it with the forwarded copy; differences are evidence of tampering in their own right.

Q3Why might a password appear in auth logs?+

Users sometimes type their password into the username prompt. The failed attempt is then logged with the password as the username, so auth logs need access control and retention limits.

Finished this one?
Tick it and it shows as done on the roadmap and the course.