Reading Linux, auth and web server logs

Where Linux keeps its evidence and how to read it: auth.log and secure, the systemd journal, sudo and SSH lines, wtmp and btmp, then Apache, Nginx and IIS access logs, and how to spot scanning, brute force, injection attempts and web shells in them.

FREEBy ·Lesson·13 min read·Beginner → Intermediate
Linux logsauth.logjournalctlApache access logweb shellIIS logs

01Where Linux logs live

✓What you will learn
  • Find authentication evidence on Debian, Ubuntu and Red Hat family systems
  • Query the systemd journal by unit, time and boot
  • Read SSH and sudo lines and the binary login records
  • Read web access logs and spot scanning, brute force, injection and web shells

Linux logging is split between the syslog text files under /var/log (written by rsyslog or syslog-ng) and the binary systemd journal. Many modern distributions keep both; some minimal and container images keep only the journal, or nothing persistent at all.

EvidenceDebian / UbuntuRHEL / Fedora / Amazon LinuxNotes
Authentication (sshd, sudo, PAM)/var/log/auth.log/var/log/securePlain text, rotated by logrotate (auth.log.1, .2.gz)
General system/var/log/syslog/var/log/messagesService starts, kernel messages, cron
JournaljournalctljournalctlPersistent only if /var/log/journal exists or Storage=persistent
Logins and reboots/var/log/wtmp (read with last)sameBinary; btmp holds failed logins (lastb)
Last login per user/var/log/lastlogsameBinary; read with lastlog
Audit/var/log/audit/audit.logsameOnly if auditd is installed and has rules
journalctl, last and lastbbash
# Journal: sshd for a time window, on this boot and the previous one
journalctl -u ssh -u sshd --since "2026-11-14 01:00" --until "2026-11-14 04:00" -o short-iso
journalctl --list-boots
journalctl -b -1 -p warning

# Binary login records
last -F -i | head -20      # full timestamps, IPs instead of names
lastb -F -i | head -20     # failed logins (root only)
iTimestamps without a year or zone

Classic syslog lines (Nov 14 02:09:41) have no year and no time zone; they are in the host's local zone. Check timedatectl on the host and record the offset before you merge Linux lines with UTC sources. The journal stores UTC internally and displays local time unless you pass --utc.

02SSH and sudo lines

auth.log: brute force, a success, sudo to a root shell, and a UID 0 accounttext
Nov 14 02:09:41 web-03 sshd[2187]: Failed password for invalid user admin from 203.0.113.7 port 50412 ssh2
Nov 14 02:09:43 web-03 sshd[2191]: Failed password for root from 203.0.113.7 port 50420 ssh2
Nov 14 02:31:12 web-03 sshd[2402]: Accepted password for deploy from 203.0.113.7 port 51882 ssh2
Nov 14 02:31:12 web-03 sshd[2402]: pam_unix(sshd:session): session opened for user deploy(uid=1001) by (uid=0)
Nov 14 02:33:05 web-03 sudo:   deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/bash
Nov 14 02:33:05 web-03 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by deploy(uid=1001)
Nov 14 02:35:40 web-03 useradd[2511]: new user: name=backup2, UID=0, GID=0, home=/root, shell=/bin/bash

This is a full compromise in seven lines. A source guessed passwords, then Accepted password for deploy from the same address. Twenty minutes later deploy used sudo to start a root shell, and a new account was created with UID 0, which makes it a second root regardless of its name. Each line has a field to pivot on: the source IP, the sshd PID (2402) that ties the session lines together, and the TTY.

Line containsMeaningFollow-up
Failed password for invalid userUsername does not exist on the hostGuessing from a list; check whether any real names succeeded
Accepted publickey ... SHA256:...Key-based login; the fingerprint names the keyFind which authorized_keys entry matches; was it added recently?
Connection closed by authenticating user ... [preauth]Gave up before authenticatingScanners and brute force tools produce lots of these
sudo: ... COMMAND=A command run with elevated rightsShells (bash, sh), editors on /etc/sudoers, chmod u+s
sudo: ... incorrect password attemptsFailed sudoA user or attacker trying to escalate
useradd, usermod, passwdAccount changesUID 0, new members of sudo or wheel groups
SPL: sources that failed many times and then succeededspl
index=linux sourcetype=linux_secure earliest=-24h
| rex "sshd\[\d+\]: (?<outcome>Accepted|Failed) (?<method>\S+) for (?:invalid user )?(?<user>\S+) from (?<src>\S+)"
| stats count(eval(outcome="Failed")) AS failed count(eval(outcome="Accepted")) AS accepted values(user) AS users BY host src
| where failed > 20 AND accepted > 0
!Missing lines are evidence too

An attacker with root can edit text logs. Gaps in time, a log that starts mid-day, auth.log smaller than its rotated predecessor, or wtmp with no entries for a period when you know people logged in all suggest tampering. Forwarded copies in the SIEM are why you ship logs off the host.

03Web access log formats

Web servers write one line per request. Apache and Nginx default to variants of the combined format; IIS writes W3C extended logs with a header that names the fields. Learn to read both by eye.

Apache combined: client, ident, user, time, request, status, bytes, referer, user agenttext
203.0.113.50 - - [14/Nov/2026:03:02:11 +0000] "GET /index.php?id=1%27%20OR%20%271%27=%271 HTTP/1.1" 500 512 "-" "sqlmap/1.8#stable (https://sqlmap.org)"
203.0.113.50 - - [14/Nov/2026:03:04:47 +0000] "POST /wp-content/uploads/2026/11/thumb.php HTTP/1.1" 200 64 "-" "Mozilla/5.0"
203.0.113.50 - - [14/Nov/2026:03:04:52 +0000] "GET /wp-content/uploads/2026/11/thumb.php?c=id HTTP/1.1" 200 54 "-" "Mozilla/5.0"
IIS W3C: the #Fields header defines the columns, times are UTC by defaulttext
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2026-11-14 03:10:02 10.20.1.40 POST /owa/auth/errorFE.aspx - 443 - 203.0.113.51 Mozilla/5.0 200 0 0 31
2026-11-14 03:10:09 10.20.1.40 GET /aspnet_client/system_web/x.aspx cmd=whoami 443 - 203.0.113.51 curl/8.4.0 200 0 0 15
  • Status and bytes together: a 200 with a tiny, constant size for many different payloads often means a generic error page; a 200 whose size changes with the payload means the server is processing it.
  • URL encoding: %27 is a single quote and %20 a space. Decode before you judge, and watch for double encoding (%2527) aimed at filters.
  • Apache logs the query string but not the POST body. A web shell driven by POST shows only the path and size; you need the file on disk, a WAF log, or full packet capture for the commands.
  • Behind a load balancer or CDN, the client column is the proxy. The real client is in X-Forwarded-For only if the server is configured to log it.

04Spotting attacks in web logs

PatternWhat it looks likeReading
Scanning / content discoveryHundreds of 404s from one client in minutes, paths like /.env, /.git/config, /wp-login.php, /phpmyadmin/Reconnaissance; matters only if something returned 200
Credential attacksMany POSTs to one login path, mostly 401 or 200 with the same small sizeBrute force or stuffing; a different size or a 302 to a dashboard is a success
Injection attemptsQuotes, UNION SELECT, ../, ${jndi:, <script> in query strings; 500sAttempts; confirm impact from application or database logs
Web shellA new script under an uploads or static folder, a few IPs, requests with cmd=, c= or POST onlyPost-exploitation; isolate and preserve the file
Shell one-liners for a first pass over an access logbash
# Top talkers producing 404s, then everything they got a 200 for
awk '$9 == 404 {print $1}' access.log | sort | uniq -c | sort -rn | head
awk '$1 == "203.0.113.50" && $9 == 200 {print $4, $6, $7, $10}' access.log

# Scripts requested in folders that should only hold static files
grep -E '"(GET|POST) /(wp-content/uploads|images|static)/[^ ]*\.(php|aspx?|jsp)' access.log
KQL: candidate web shells on IISkql
// Sentinel W3CIISLog: script files under static paths answered with 200
W3CIISLog
| where TimeGenerated > ago(7d)
| where csUriStem matches regex @"(?i)/(aspnet_client|images|uploads)/.*\.(aspx|ashx|asp)$"
| where scStatus == 200
| summarize hits = count(), clients = make_set(cIP, 10), queries = make_set(csUriQuery, 10) by sComputerName, csUriStem
| order by hits desc
▭On the whiteboard · Web shell triage5 points
01
find
Requests to scripts in upload, static or aspnet_client folders; first request time
02
confirm
Hash and copy the file; check its creation time against the first request
03
entry
What happened just before: an upload request, an exploit pattern, a vulnerable plugin
04
scope
Every client IP that requested it; any other new script files
05
contain
Block the path at the WAF, preserve logs and disk, then remove

05Practice and pitfalls

Test yourself

An Apache log shows `POST /wp-content/uploads/2026/11/thumb.php` returning 200 to the same IP several times, with no query string. What can you conclude, and what is missing?

“How to say it in the room

"On Linux I go to auth.log or secure for sshd and sudo, the journal for services, and wtmp for sessions, and I note the host's time zone before merging anything. In web logs I look for scripts in places that should be static, status and size patterns that show the server processing a payload, and the request just before the first hit on anything suspicious."

◎Interview radar

Interviewers like to paste a few log lines and ask what happened. Narrate source, outcome and the pivot field for each line, call out the moment of success (Accepted, 200 on a shell path, UID 0), and say what log you would check next.

Q1Why might auth.log be empty on a server?+

The distribution may log only to the journal, rsyslog may not be installed (common in containers), or logs may have been deleted. Check journalctl and the SIEM copy.

Q2How do you find who added an SSH key?+

authorized_keys changes are not logged by default. Use file timestamps (and their limits), auditd watches on the file if configured, shell history, and the first Accepted publickey line for that fingerprint.

Q3What does a 304 mean in a web log?+

Not Modified: the client had a cached copy. It is normal browsing behaviour and rarely interesting for attack analysis.

Finished this one?
Tick it and it shows as done on the roadmap and the course.