Linux Kernel
The Linux kernel is the central program of every Linux server: a single C program of roughly 27 million lines of source code that mediates every request from userland to the hardware, translating a PHP worker's read() call into an NVMe command and a web server's accept() into a TCP segment.
It is the only software on the box that touches physical memory, and every other layer of the LAMP and ASP.NET stacks, from Apache to the .NET runtime, runs as its tenant. This page maps that kernel onto the web stack, walks the 4 main subsystems, tracks the 3 parallel version lines, and shows how to read the 2 files a kernel panic leaves behind.
Before the internals, three landmarks anchor this page. The kernel is stewarded by The Linux Foundation, the industry body that holds the copyright for a large share of the source and runs the kernel's release cadence. The tree itself lives in public git at git.kernel.org, with the main repository mirrored on GitHub, so any 6.1 patch is one grep away from a server that is still on 4.18. And the kernel is ultimately the thing your shell talks to, which is why a working set of Linux Commands such as uname, dmesg, and sysctl belongs in the same mental drawer as the panic decoder below. The same goes for disk layout: if a service can't see a file, the first move is to Find Files on Linux with find and locate, because "missing" is usually "mounted somewhere I have not looked".
Where the kernel sits in the web stack
The stack runs in two directions. Downward, PHP's FPM workers, nginx's event loop, and the .NET CLR all ask the kernel for the same four things: process time, file data, sockets, and memory. Upward, the kernel answers with the signals those frameworks listen for, so a scheduler change in 5.6's CFS code is visible as a PHP response-time regression before it is visible in any kernel release note. That is why the kernel is the one shared dependency in both a LAMP box and an ASP.NET box, and why its version is the first field on the incident report.
| Stack layer | What it asks of the kernel | What it feels when the kernel changes |
|---|---|---|
| Web server (nginx, Apache, IIS on Linux) | Socket accept, epoll and poll event loops, file sends | Connection-per-core ceilings, TLS session cache sizing |
| PHP (FPM, OPcache) | Process forking, memory maps, file reads | Fork cost, page-cache hit rate on opcache files |
| Database (MySQL, PostgreSQL) | File I/O, shared memory, fsync semantics | Commit latency on a new I/O scheduler or a new filesystem version |
| Runtime (.NET CLR, Mono) | JIT pages, thread scheduling, native sockets | GC pause drift, thread-pool starvation under a new cgroup |
The practical rule follows from the table: when a web service regresses after an OS update, the first 5 minutes of diagnosis go to the kernel, because it is the only row that touches all 4 rows at once.
The 4 main subsystems of the kernel
The kernel splits into 4 main subsystems, and each one owns a different failure class on a web server.
- Process management, the scheduler and the forker, which owns CPU starvation: a PHP-FPM pool that stops returning slots is usually a scheduling or cgroup story, not a PHP story.
- Memory management, the page cache and the swap paths, which owns the out-of-killer: OOM events that take a database down at 2 a.m. are read from /var/log/kern.log, which is where the kernel writes its own findings.
- The VFS and block layer, which owns disk behavior: a filesystem upgrade, an ext4 feature flag, or a new I/O scheduler shows up here before it shows up in an application log.
- The networking stack, the TCP/IP implementation in net/, which owns connection behavior: the 128-entry listen backlog, the TIME_WAIT reaper, and the eBPF hooks that modern web servers lean on all live here.
Every kernel bug report on the public trackers is filed against exactly one of these 4 subsystems, and the maintainer assigned to the report is a maintainer of that directory, which is how the 200 or so kernel developers stay organized. It is also the reason a panic's last lines name a directory: the crash site is the subsystem.
Version lines: stable, LTS, and mainline
Linux kernel versions move on 3 parallel lines, and knowing which line a server is on decides which patches it will ever receive. A version string such as 6.1.100 reads as major.minor.patch, where the major.minor pair names the line and the patch number counts the fixes stacked on it.
| Line | What it is | Support window | When a server runs it |
|---|---|---|---|
| Stable | A released branch with backported fixes, such as 6.6 or 6.12 | About 1 year for the most recent 5 stable branches | Most production workloads |
| Long-term support (LTS) | A stable branch the community has committed to keep patched | Up to about 10 years, with 6.1 and 6.6 as current examples | Fleet infrastructure that updates quarterly |
| Mainline | Linus's integration branch, where new features land first | Not supported; it becomes a stable line after the 2 weekly merge windows | Distro development and feature testing |
The LTS line is the one to watch on a server fleet, because the 10-year window is a promise, and the 6.1 and 6.6 branches are the 2 LTS lines a distro will base the next 10 years of its releases on. A web server that ships on 5.4 today will be offered security fixes for that branch for roughly 8 more years, which is longer than most PHP or .NET support contracts, and it is why "the kernel is old but the distro is supported" is a normal, defensible production state. The stable line, by contrast, turns over: 6.12 is the newest stable line as of the writing, and the previous 5 branches stay alive in step, so a server on 6.6 is on a line that will be patched for about another 12 months.
Reading a kernel panic
A kernel panic is the kernel's last resort: it hits a condition it cannot recover from, it dumps what it can, and it stops, and the server is dead until you reboot it. The 2 artifacts a panic leaves are what make it diagnosable, and both are plain text.
The first artifact is the call trace, and the first line of the trace names the function where the kernel gave up. A line such as "BUG: unable to handle kernel paging request at ffff9c8d3f1a0018" tells you the crash was a memory fault, the address is the one it tried to touch, and the trace below it lists the call path, each frame indented one level deeper, from the faulting function back to the entry point. Read the trace top down: the first frame is the crash site, and the last frame is usually where your process called into the kernel, which is where the subsystem boundary from the architecture section shows up. The second artifact is the log, and on a production box you read the kernel's log with dmesg, and on a box you can still log into you also read /var/log/kern.log, which is the file the kernel writes to and the same file that catches the out-of-killer events described above. The 3 fields worth copying into an incident report are the kernel version from the first line, the function names from the trace, and the last message before the "Kernel panic" line, because together they reproduce the bug for whoever greps the public trackers.
Not every panic is a bug in the kernel itself. A driver that dereferences a freed pointer panics the same way a core kernel bug does, and the fix is a driver update, not a kernel update, which is why the trace's module names, printed in square brackets after each function, decide which of the 2 upgrade paths you take.
Panic and reboot: what the server actually does
The server does exactly one of 3 things after a panic, and the choice is a sysctl knob, not a decision. With panic_on_oop set to 0, the default, a single bad page fault in a kernel thread is tolerated, the offending process is killed, and the box keeps serving. With panic_on_oop set to 1, the same fault becomes a full panic, and administrators set that on a database host on the theory that a corrupted page is worse than a 4-second outage. The third option is the panic timeout, the number of seconds after a panic before the kernel reboots the machine, and setting it to 10 means the box is back in service before a human has finished reading the first log line, which is the right posture for a load-balanced web tier and the wrong posture for a box whose disk you need to image.
How to check and report a kernel
Two commands answer the two daily questions, which version am I on and what is it telling me, and both ship on every Linux server without any package install.
- Run uname -r to print the running kernel's version, the 6.1.100 string that decides the support line, and compare it against the distro's supported list before you plan an upgrade.
- Run dmesg -T to print the kernel's own log with human-readable timestamps, and read the last 50 lines after any unexplained latency spike, because the kernel logs its OOM kills, its I/O errors, and its hung-task warnings there before the application ever sees a symptom.
- Run uname -a to get the architecture, the build date, and the hostname in one line, and paste it into the report, because a bug that reproduces on x86-64 and not on arm64 is a different bug, and the 2 architectures share the same source tree.
The reporting path is public and short: the kernel's bug tracker at bugzilla.kernel.org takes a report with the 3 fields from the panic section, and the mailing list archives at lore.kernel.org are where the 200 or so developers discuss it, so a good report is findable, and a findable report is the one that gets fixed. That is the full loop a web server administrator runs, from the first dmesg line to the version string that will carry the fix, and it is the loop that makes the kernel the most testable piece of software in the entire stack.