Hardening and patching
Reducing a Linux host's attack surface: fewer services, systemd sandboxing, SSH configuration, kernel sysctls, mandatory access control and benchmarks, then a patching process that actually closes the window, including reboots and live patching.
01Start by removing things
- Shrink a host's attack surface by removing services and packages
- Sandbox services with systemd directives and measure the result
- Harden SSH and apply the kernel sysctls that matter
- Explain SELinux and AppArmor and the role of CIS benchmarks
- Run a patching process that includes reboots, restarts and live patching
Hardening is mostly subtraction. Every listening port, installed package and running service is code an attacker can reach or reuse. A minimal image with only what the workload needs beats any amount of tuning on a general-purpose install.
# What is listening, and which process owns it?
ss -tulpn
# What starts at boot?
systemctl list-unit-files --state=enabled
# Remove what the role does not need
sudo apt purge telnetd rsh-server avahi-daemon cups # Debian/Ubuntu example
sudo systemctl disable --now rpcbind.socket rpcbind.service- One role per host or image. A database server does not need a compiler, a mail server or a desktop.
- Prefer immutable images rebuilt from a pipeline over long-lived pets edited by hand: drift is where forgotten services come from.
- Compilers and network tools on production hosts make an attacker's next step easier. Leave them out of runtime images.
02Sandboxing services with systemd
systemd can apply most of the isolation from the previous lesson to an ordinary service with a few lines of unit configuration. systemd-analyze security <unit> scores each service's exposure from 0 (locked down) to 10 (unprotected) and lists which directives are missing, which makes it a good review tool.
[Service]
User=report
DynamicUser=no
NoNewPrivileges=yes
ProtectSystem=strict # whole filesystem read-only except below
ReadWritePaths=/var/lib/report
ProtectHome=yes # /home, /root and /run/user invisible
PrivateTmp=yes # private /tmp and /var/tmp
PrivateDevices=yes
ProtectKernelTunables=yes # /proc/sys and /sys read-only
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes # no writable and executable mappings
SystemCallFilter=@system-service
SystemCallArchitectures=native
CapabilityBoundingSet=ProtectSystem=strict and MemoryDenyWriteExecute=yes break some software (anything with a JIT, for example). Add directives incrementally, run the service's tests after each, and keep the ones that hold. A score of 2 to 4 is realistic for most services.
03SSH and kernel settings
SSH is the front door of most Linux fleets. Turn off password authentication so credential stuffing and brute force have nothing to attack, keep root logins off, and limit who may log in at all. Larger fleets move to SSH certificates signed by an internal CA with short lifetimes, which avoids distributing authorized_keys files.
# /etc/ssh/sshd_config.d/10-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowGroups ssh-users
MaxAuthTries 3
X11Forwarding no
# validate before reloading, or you can lock yourself out
sudo sshd -t && sudo systemctl reload ssh| sysctl | Value | Effect |
|---|---|---|
kernel.randomize_va_space | 2 | Full ASLR for stack, heap, mmap and libraries (the usual default) |
kernel.kptr_restrict | 2 | Hide kernel pointer addresses from /proc and logs, which helps defeat KASLR |
kernel.dmesg_restrict | 1 | Only privileged users read the kernel log |
kernel.yama.ptrace_scope | 1 or 2 | 1: only ancestors may ptrace; 2: only with CAP_SYS_PTRACE |
kernel.unprivileged_bpf_disabled | 1 | Unprivileged users cannot load BPF programs |
fs.protected_symlinks, fs.protected_hardlinks | 1 | Block classic /tmp symlink and hardlink attacks |
fs.suid_dumpable | 0 | No core dumps from setuid processes, which could leak secrets |
net.ipv4.conf.all.accept_redirects | 0 | Ignore ICMP redirects that could reroute traffic |
Mount options add a little more: nodev,nosuid,noexec on /tmp, /var/tmp and /dev/shm stop device files, setuid binaries and direct execution from world-writable locations. noexec is a speed bump rather than a wall, since an interpreter can still read a script from there, but it breaks a lot of commodity malware.
04Mandatory access control and benchmarks
Ordinary Unix permissions are discretionary: the owner of a file decides who may read it, and root can do anything. Mandatory access control (MAC) adds a policy that even root processes must obey. A web server confined by SELinux cannot read /etc/shadow or open outbound connections to arbitrary ports even if it is compromised and running as root.
| SELinux | AppArmor | |
|---|---|---|
| Default on | RHEL, Fedora, CentOS Stream, Android | Ubuntu, Debian, SUSE |
| Model | Labels (types) on every file and process; rules between types | Path-based profiles per program |
| Check status | getenforce, sestatus | aa-status |
| Learning mode | Permissive (logs, does not block) | Complain mode |
| Strength | Fine-grained, whole-system | Easier to write and read |
Turning SELinux off because one application failed removes protection from everything on the host. Read the denial (ausearch -m AVC -ts recent), then fix the label (restorecon), set a boolean (setsebool -P httpd_can_network_connect on) or write a small policy module.
CIS Benchmarks and DISA STIGs are consensus checklists covering hundreds of settings like the ones above. Scan against them with OpenSCAP or Lynis, apply them through configuration management, and record justified exceptions. The value is less in any single control than in making the baseline explicit and measurable.
05Patching that actually closes the window
Installing a package does not always mean the fix is running. A patched OpenSSL library on disk does nothing for a process that loaded the old one last month; the kernel fix does nothing until reboot. Patch management has three steps, and teams often stop after the first.
- Install: automated security updates (
unattended-upgradeson Debian and Ubuntu,dnf-automaticon RHEL-family) or, better, rebuilt images rolled out by the pipeline. - Restart what uses old code:
needrestartlists services still mapping deleted libraries.lsof +L1or a scan of/proc/*/mapsfor(deleted)does the same by hand. - Reboot or live patch the kernel: kernel fixes need a reboot unless you use live patching (kpatch on RHEL, Canonical Livepatch on Ubuntu), which covers many but not all kernel fixes and still needs periodic reboots.
Interviewers like "we patched OpenSSL, are we safe?". The answer they want: only after every process using it has restarted, and for kernel bugs only after a reboot or live patch. Mentioning regreSSHion (CVE-2024-6387, a signal-handler race in OpenSSH's sshd that reintroduced a 2006 bug) as a reason to patch internet-facing daemons fast, and EPSS or KEV for prioritisation, shows practical experience.
"I harden by removing first: a minimal image, one role per host, then systemd sandboxing scored with systemd-analyze security. SSH is key-only or certificate-based with root login off, the kernel sysctls restrict pointers, ptrace and BPF, and SELinux or AppArmor stays enforcing. For patching I rebuild images from the pipeline, prioritise by KEV and EPSS, restart services still mapping deleted libraries, and track kernel versions so reboots or live patches actually happen."
After `apt upgrade` installed a fixed libssl, a scanner still flags the host as vulnerable. Uptime is 90 days and `needrestart` lists nginx. What is the most likely explanation?
Q1Is `noexec` on /tmp a real control?+
A partial one. It stops direct execution of dropped binaries but not bash /tmp/x.sh or python3 /tmp/x.py. It raises the cost for commodity malware and costs little, so it is worth having, not worth relying on.
Q2Why prefer rebuilding images over patching in place?+
Rebuilt images are reproducible, tested before rollout and free of drift. Patching in place leaves every host slightly different, and you cannot be sure which fixes are actually running.
Q3What does live patching not cover?+
Changes to data structures, some architectural fixes and anything outside the kernel. Vendors publish which CVEs each live patch covers, and you still need regular reboots.