Reading Linux, auth and web server logs
Where Linux keeps its evidence and how to read it: auth.log and secure, the systemd journal, sudo and SSH lines, wtmp and btmp, then Apache, Nginx and IIS access logs, and how to spot scanning, brute force, injection attempts and web shells in them.
01Where Linux logs live
- Find authentication evidence on Debian, Ubuntu and Red Hat family systems
- Query the systemd journal by unit, time and boot
- Read SSH and sudo lines and the binary login records
- Read web access logs and spot scanning, brute force, injection and web shells
Linux logging is split between the syslog text files under /var/log (written by rsyslog or syslog-ng) and the binary systemd journal. Many modern distributions keep both; some minimal and container images keep only the journal, or nothing persistent at all.
| Evidence | Debian / Ubuntu | RHEL / Fedora / Amazon Linux | Notes |
|---|---|---|---|
| Authentication (sshd, sudo, PAM) | /var/log/auth.log | /var/log/secure | Plain text, rotated by logrotate (auth.log.1, .2.gz) |
| General system | /var/log/syslog | /var/log/messages | Service starts, kernel messages, cron |
| Journal | journalctl | journalctl | Persistent only if /var/log/journal exists or Storage=persistent |
| Logins and reboots | /var/log/wtmp (read with last) | same | Binary; btmp holds failed logins (lastb) |
| Last login per user | /var/log/lastlog | same | Binary; read with lastlog |
| Audit | /var/log/audit/audit.log | same | Only if auditd is installed and has rules |
# Journal: sshd for a time window, on this boot and the previous one
journalctl -u ssh -u sshd --since "2026-11-14 01:00" --until "2026-11-14 04:00" -o short-iso
journalctl --list-boots
journalctl -b -1 -p warning
# Binary login records
last -F -i | head -20 # full timestamps, IPs instead of names
lastb -F -i | head -20 # failed logins (root only)Classic syslog lines (Nov 14 02:09:41) have no year and no time zone; they are in the host's local zone. Check timedatectl on the host and record the offset before you merge Linux lines with UTC sources. The journal stores UTC internally and displays local time unless you pass --utc.
02SSH and sudo lines
Nov 14 02:09:41 web-03 sshd[2187]: Failed password for invalid user admin from 203.0.113.7 port 50412 ssh2
Nov 14 02:09:43 web-03 sshd[2191]: Failed password for root from 203.0.113.7 port 50420 ssh2
Nov 14 02:31:12 web-03 sshd[2402]: Accepted password for deploy from 203.0.113.7 port 51882 ssh2
Nov 14 02:31:12 web-03 sshd[2402]: pam_unix(sshd:session): session opened for user deploy(uid=1001) by (uid=0)
Nov 14 02:33:05 web-03 sudo: deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/bash
Nov 14 02:33:05 web-03 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by deploy(uid=1001)
Nov 14 02:35:40 web-03 useradd[2511]: new user: name=backup2, UID=0, GID=0, home=/root, shell=/bin/bashThis is a full compromise in seven lines. A source guessed passwords, then Accepted password for deploy from the same address. Twenty minutes later deploy used sudo to start a root shell, and a new account was created with UID 0, which makes it a second root regardless of its name. Each line has a field to pivot on: the source IP, the sshd PID (2402) that ties the session lines together, and the TTY.
| Line contains | Meaning | Follow-up |
|---|---|---|
Failed password for invalid user | Username does not exist on the host | Guessing from a list; check whether any real names succeeded |
Accepted publickey ... SHA256:... | Key-based login; the fingerprint names the key | Find which authorized_keys entry matches; was it added recently? |
Connection closed by authenticating user ... [preauth] | Gave up before authenticating | Scanners and brute force tools produce lots of these |
sudo: ... COMMAND= | A command run with elevated rights | Shells (bash, sh), editors on /etc/sudoers, chmod u+s |
sudo: ... incorrect password attempts | Failed sudo | A user or attacker trying to escalate |
useradd, usermod, passwd | Account changes | UID 0, new members of sudo or wheel groups |
index=linux sourcetype=linux_secure earliest=-24h
| rex "sshd\[\d+\]: (?<outcome>Accepted|Failed) (?<method>\S+) for (?:invalid user )?(?<user>\S+) from (?<src>\S+)"
| stats count(eval(outcome="Failed")) AS failed count(eval(outcome="Accepted")) AS accepted values(user) AS users BY host src
| where failed > 20 AND accepted > 0An attacker with root can edit text logs. Gaps in time, a log that starts mid-day, auth.log smaller than its rotated predecessor, or wtmp with no entries for a period when you know people logged in all suggest tampering. Forwarded copies in the SIEM are why you ship logs off the host.
03Web access log formats
Web servers write one line per request. Apache and Nginx default to variants of the combined format; IIS writes W3C extended logs with a header that names the fields. Learn to read both by eye.
203.0.113.50 - - [14/Nov/2026:03:02:11 +0000] "GET /index.php?id=1%27%20OR%20%271%27=%271 HTTP/1.1" 500 512 "-" "sqlmap/1.8#stable (https://sqlmap.org)"
203.0.113.50 - - [14/Nov/2026:03:04:47 +0000] "POST /wp-content/uploads/2026/11/thumb.php HTTP/1.1" 200 64 "-" "Mozilla/5.0"
203.0.113.50 - - [14/Nov/2026:03:04:52 +0000] "GET /wp-content/uploads/2026/11/thumb.php?c=id HTTP/1.1" 200 54 "-" "Mozilla/5.0"#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken
2026-11-14 03:10:02 10.20.1.40 POST /owa/auth/errorFE.aspx - 443 - 203.0.113.51 Mozilla/5.0 200 0 0 31
2026-11-14 03:10:09 10.20.1.40 GET /aspnet_client/system_web/x.aspx cmd=whoami 443 - 203.0.113.51 curl/8.4.0 200 0 0 15- Status and bytes together: a 200 with a tiny, constant size for many different payloads often means a generic error page; a 200 whose size changes with the payload means the server is processing it.
- URL encoding:
%27is a single quote and%20a space. Decode before you judge, and watch for double encoding (%2527) aimed at filters. - Apache logs the query string but not the POST body. A web shell driven by POST shows only the path and size; you need the file on disk, a WAF log, or full packet capture for the commands.
- Behind a load balancer or CDN, the client column is the proxy. The real client is in
X-Forwarded-Foronly if the server is configured to log it.
04Spotting attacks in web logs
| Pattern | What it looks like | Reading |
|---|---|---|
| Scanning / content discovery | Hundreds of 404s from one client in minutes, paths like /.env, /.git/config, /wp-login.php, /phpmyadmin/ | Reconnaissance; matters only if something returned 200 |
| Credential attacks | Many POSTs to one login path, mostly 401 or 200 with the same small size | Brute force or stuffing; a different size or a 302 to a dashboard is a success |
| Injection attempts | Quotes, UNION SELECT, ../, ${jndi:, <script> in query strings; 500s | Attempts; confirm impact from application or database logs |
| Web shell | A new script under an uploads or static folder, a few IPs, requests with cmd=, c= or POST only | Post-exploitation; isolate and preserve the file |
# Top talkers producing 404s, then everything they got a 200 for
awk '$9 == 404 {print $1}' access.log | sort | uniq -c | sort -rn | head
awk '$1 == "203.0.113.50" && $9 == 200 {print $4, $6, $7, $10}' access.log
# Scripts requested in folders that should only hold static files
grep -E '"(GET|POST) /(wp-content/uploads|images|static)/[^ ]*\.(php|aspx?|jsp)' access.log// Sentinel W3CIISLog: script files under static paths answered with 200
W3CIISLog
| where TimeGenerated > ago(7d)
| where csUriStem matches regex @"(?i)/(aspnet_client|images|uploads)/.*\.(aspx|ashx|asp)$"
| where scStatus == 200
| summarize hits = count(), clients = make_set(cIP, 10), queries = make_set(csUriQuery, 10) by sComputerName, csUriStem
| order by hits desc05Practice and pitfalls
An Apache log shows `POST /wp-content/uploads/2026/11/thumb.php` returning 200 to the same IP several times, with no query string. What can you conclude, and what is missing?
"On Linux I go to auth.log or secure for sshd and sudo, the journal for services, and wtmp for sessions, and I note the host's time zone before merging anything. In web logs I look for scripts in places that should be static, status and size patterns that show the server processing a payload, and the request just before the first hit on anything suspicious."
Interviewers like to paste a few log lines and ask what happened. Narrate source, outcome and the pivot field for each line, call out the moment of success (Accepted, 200 on a shell path, UID 0), and say what log you would check next.
Q1Why might auth.log be empty on a server?+
The distribution may log only to the journal, rsyslog may not be installed (common in containers), or logs may have been deleted. Check journalctl and the SIEM copy.
Q2How do you find who added an SSH key?+
authorized_keys changes are not logged by default. Use file timestamps (and their limits), auditd watches on the file if configured, shell history, and the first Accepted publickey line for that fingerprint.
Q3What does a 304 mean in a web log?+
Not Modified: the client had a cached copy. It is normal browsing behaviour and rarely interesting for attack analysis.