1 · What this is#

This is a rebuilt replacement for a 326-page trainer PDF covering Linux, virtualization, system administration and Docker. It is not a summary of that PDF. The PDF was used only as a syllabus — a list of what must be covered — and then every explanation was rewritten from scratch, at greater depth, with the missing prerequisites filled in, the errors corrected, and the diagrams redrawn.

Three things make it different from the notes it replaces.

It teaches, rather than listing. The original gives ls two options and one example. Here, every command carries its purpose, full option table, several worked examples, real output explained field by field, the way professionals actually use it, the mistakes people make, and how it differs on BSD or macOS.

It covers the hidden syllabus. The source PDF ends with 495 multiple-choice questions spanning filesystems, LVM, RAID, NFS, SSH hardening, ACLs, sudoers, LDAP, cron and the boot process — none of which its prose ever teaches. Roughly 40% of that question bank was unanswerable from the notes themselves. Those topics are taught here as full chapters.

It corrects the source. Where the original is wrong, this handbook says so in a bordered callout rather than silently repeating the error. The corrections so far include the claim that macOS and PlayStation OS derive from Linux (they derive from BSD and Mach), the release year of Linux 6.0, a section on rmdir that duplicates mkdir’s text verbatim, the meaning of file -k, du -M, and the description of kill -9 as an “option” rather than a signal.

2 · How to read the reader#

Four kinds of panel appear throughout, and they mean different things:

# A command panel. This is something you type.
uname -r

$ uname -r
6.8.0-45-generic

A terminal panel is darker and shows a transcript — the command and its real output. The $ marks what you typed; everything else came back from the machine.

   ┌──────────────┐         ┌──────────────┐
   │  monospace   │────────▶│  ASCII art   │
   └──────────────┘         └──────────────┘

Mermaid panels render as real diagrams — flowcharts, sequence diagrams, timelines and mind maps.

Callouts are graded, and the grading is meaningful:

3 · The map#

The handbook is built in seven parts, each readable independently. Chapters that are still being written are marked; the rest are complete.

Part I — Foundations

01 Introduction to Linux · what an operating system does, kernel vs distribution, the Unix lineage, POSIX, GPL, and how to fingerprint any machine you are dropped onto.
02 Virtualization & Virtual Machines · hypervisor types, full vs paravirtualization, VM networking modes, VirtualBox/VMware/KVM from the command line, and building the throwaway lab machine that every later chapter assumes.

Part II — Working in the Shell

03 Shells & Terminals · terminal vs shell, TTYs and PTYs, startup file order, aliases, functions, environment variables, history, prompts, redirection and quoting.
04 Files & Directories · the filesystem tree, and ls pwd cd mkdir rmdir rm cp mv touch file ln locate find in full.
05 Text Processing & Searching · streams and pipes, then cat less head tail grep sed awk sort uniq cut tr wc diff tee echo printf xargs, plus survivable vi.
06 Archiving & Compression · archiving vs compression, tar in depth, zip/unzip, and gzip vs bzip2 vs xz vs zstd.

Part III — System Internals

07 Filesystems, Mounting & the FHS · inodes, journalling, filesystem types, mkfs, mount, /etc/fstab, fsck, df vs du, swap and NFS.
08 Disks, Partitioning, LVM & RAID · the storage stack, MBR vs GPT, fdisk/parted, LVM end to end, and every RAID level.
09 Processes, Signals & Services · fork and exec, process states, zombies, the full signal table, ps/top, nice, jobs, and writing your own systemd unit.
10 The Boot Process · firmware to login prompt, GRUB, initramfs, targets and runlevels. In progress.

Part IV — Networking & Remote Access

11 Linux Networking · addressing, subnets, routing, DNS, ports, and the ip/ss/dig/curl toolkit. In progress.
12 SSH & Secure Transfer · key authentication, ssh-agent, hardening sshd_config, tunnels, scp and rsync. In progress.

Part V — System Administration

13 Users, Permissions, ACLs & sudo · the permission model, umask, setuid/setgid/sticky, ACLs and sudoers. In progress.
14 Package Management · apt/dpkg, dnf/yum/rpm, repositories and building from source. In progress.
15 Logs, Monitoring & Scheduling · journalctl, rsyslog, logrotate, cron, at, systemd timers and the monitoring toolkit. In progress.
16 Directory Services with LDAP · DNs, LDIF, objectClasses, ports 389 and 636. In progress.

Part VI — Automation

17 Bash Scripting · from shebang to set -euo pipefail, with the quoting and exit-code discipline that separates working scripts from dangerous ones. In progress.
18 The Automation Script Library · the source PDF's 25 production scripts, rebuilt, explained line by line and hardened. In progress.

Part VII — Containers

19 Containers & Docker · namespaces and cgroups, images and layers, Dockerfiles, networking, volumes, Compose, Podman and production practice. In progress.

4 · Before anything else: build the lab#

You cannot learn this material by reading it. Almost every chapter ends in a hands-on lab, and several of them are destructive on purpose — you will fill a disk, break /etc/fstab, kill the wrong process and recover from it.

So the first practical task is a virtual machine you are free to ruin:

1. Install VirtualBox on your laptop (Chapter 2 covers hypervisor choice, and why a Type 2 hypervisor is right for a learning lab).
2. Create a VM with 2 vCPUs, 4 GB RAM and a 25 GB disk, and install Ubuntu Server LTS — no desktop.
3. Set networking to NAT and forward host port 2222 to guest port 22, so you can ssh -p 2222 user@127.0.0.1 from a real terminal instead of the VirtualBox console window.
4. Take a snapshot immediately and call it clean-install. Every time a lab wrecks the machine, roll back to it in seconds.

Chapter 2 gives the exact commands for all four steps.

5 · Three routes through it#

Pick the route that matches why you are here. All three assume you do the labs.

For a university exam. Read Chapters 1, 2, 3 and 7 closely — definitions, comparison tables and the > [!EXAM] callouts are written for you. Then work the Practice section of every chapter: the MCQs, fill-in-the-blanks and true/false questions are modelled on the source PDF’s own 495-question bank. Memorise the cheat sheets last, the week before the paper.

For an interview. Go straight to the Interview Corner of every chapter and answer each question out loud before opening the model answer. Prioritise Chapters 9 (processes and signals), 7 (filesystems), 13 (permissions), 11 (networking) and 19 (Docker) — in practice that is where the questions come from. The scenario questions matter more than the definitions: being handed an unfamiliar server and knowing your first five commands is the thing that gets people hired.

For real work. Read Part II properly, then use the rest as reference — search is there for exactly this. Read Chapter 17 before you write any script that runs unattended, and Chapter 9’s systemd section before you deploy anything that must survive a reboot.

6 · A note on honesty#

The source PDF was described as covering Kubernetes, Cloud and DevOps across hundreds of pages. It does not. Kubernetes appears six times, AWS three, and Jenkins thirteen — all as short shell snippets inside its automation-scripts chapter. There is no Terraform, Ansible, Helm, Prometheus, Git or CI/CD content in it, and no cloud-architecture material. Docker is the only container technology it teaches properly.

This handbook covers that syllabus faithfully and improves it. Where a topic is genuinely absent from the source, it says so rather than pretending otherwise, and points you at what to study next.

1 · The Big Picture#

Why this topic exists#

You already know how to program. You can write a function, call an API, loop over an array. But every line of code you have ever written was executed by something, on something, and that something made thousands of decisions on your behalf: which CPU core ran your loop, where in physical memory your array lives, how your print reached a screen, how your HTTP request reached a network card.

That something is the operating system. For roughly 96% of the world’s servers, ~85% of smartphones, all 500 of the world’s fastest supercomputers, and effectively 100% of cloud compute, that operating system is Linux.

Learning Linux is not learning “another OS.” It is learning the environment in which all professional software actually runs.

The real problem it solves#

Imagine you had no operating system. To write a program that saves a file, you would need to know:

• the exact model of disk controller in this machine, and its register layout
• which physical sectors are free
• how to avoid two programs writing the same sector simultaneously
• how to keep a malicious program from reading another program’s data

Now ship that program to a laptop with a different disk. It breaks. An operating system solves this by inserting one layer that does two jobs, forever:

                     Your program says: open("/etc/passwd")
                                    │
              ┌─────────────────────┴─────────────────────┐
              │                                           │
      1. ABSTRACTION                              2. ARBITRATION
      Turn thousands of different                 Decide who gets the CPU,
      hardware devices into a few                 the memory, the disk, and
      uniform ideas: "file",                      the network — and stop
      "process", "socket".                        anyone taking it all or
                                                  reading someone else's data.
              │                                           │
              └─────────────────────┬─────────────────────┘
                                    │
                     One stable interface: the system call

Every operating system feature you will ever learn — permissions, processes, mounts, cgroups, namespaces, containers — is a variation on abstraction or arbitration.

Where you will encounter it#

Why companies care#

• Cost — no per-socket licence. A startup can run 200 servers for the price of the hardware.
• Control — source is available, so a vendor cannot end-of-life your platform out from under you.
• Automatability — everything is a file or a command, so everything can be scripted. This is the precondition for DevOps, Infrastructure as Code, and immutable infrastructure.
• Talent & tooling — the entire cloud-native ecosystem (Docker, Kubernetes, Terraform, Prometheus) is written for Linux first.

2 · Intuition First#

Do not start with definitions. Start with a picture.

Analogy 1: the OS as a building manager#

You rent an office in a large building. You do not negotiate with the electricity grid, you do not install your own plumbing, and you cannot walk into another tenant’s office.

• The building is the hardware: power, water, lifts, floor space.
• The building manager is the kernel. Small team, enormous authority.
• The tenants are your programs.
• Requesting a service desk visit is a system call: you fill a specific form (read, write, open), hand it over, and wait.
• Your keycard only opens your floor — that is memory protection and file permissions.
• The manager decides lift priority at 9 a.m. — that is the CPU scheduler.

Notice what the analogy predicts correctly: tenants cannot see each other’s rooms (process isolation), a tenant hogging the lift is a management failure (scheduler), and if the building manager crashes, everything stops (kernel panic).

Analogy 2: kernel versus distribution#

This confusion costs people interview points constantly.

    THE LINUX KERNEL                    A LINUX DISTRIBUTION
    (one program, ~40M lines)           (kernel + several thousand programs)

    ┌───────────────┐                   ┌──────────────────────────────────────┐
    │               │                   │  ╔════════════╗  installer, desktop, │
    │    ENGINE     │        →          │  ║   ENGINE   ║  package manager,    │
    │               │                   │  ╚════════════╝  shell, coreutils,   │
    └───────────────┘                   │  wheels, seats, dashboard, paint      │
                                        └──────────────────────────────────────┘
    You cannot drive an engine.         Ubuntu, Debian, Fedora, RHEL, Arch —
    Linus Torvalds ships this.          different cars, same kind of engine.

Linux is the engine. Ubuntu is a car. Both use the word “Linux” in conversation, and that is fine — but in an exam or interview, say precisely which one you mean.

Analogy 3: GNU + Linux#

In 1983 Richard Stallman started building a complete free operating system called GNU. By 1991 GNU had the compiler, the shell, the text editor, the libraries, the utilities — everything except a working kernel. In 1991 a Finnish student published a kernel and no userland.

Two half-systems, perfectly complementary. That is why purists insist on the name GNU/Linux: the commands you will type in Chapter 4 (ls, cp, grep) are GNU programs, not Linux ones.

3 · Technical Definitions#

Now the precise versions.

Operating system. A program that manages hardware resources and provides services to application software through a defined interface. It consists of a kernel (privileged, always resident) plus userspace components (shell, libraries, daemons, utilities).

Kernel. The core of the OS. It is the only code that runs with full hardware privilege (on x86-64, CPU privilege ring 0). It owns memory management, process scheduling, device drivers, filesystems, and the network stack. Application code runs in ring 3 and must ask the kernel for anything privileged.

Linux. Specifically, a monolithic, modular, preemptive, multi-user, multitasking Unix-like kernel, first released by Linus Torvalds in 1991, licensed under GPLv2. Unpack that:

POSIX (IEEE 1003.1). A standard from the 1980s defining the interface between applications and a Unix-like OS: the C library, system interfaces and headers, plus a set of commands and utilities. POSIX is why a script written on macOS mostly runs on Linux.

Linux distribution (“distro”). A packaged, tested, installable operating system built around the Linux kernel. Every distribution ships:

• the Linux kernel — hardware, processes, memory, peripherals
• libraries — shared code for I/O, maths, cryptography (e.g. glibc, openssl)
• system daemons — background services started at boot: logging, task scheduling, networking, sshd
• development and packaging tools — compilers and the package manager (dpkg/apt, rpm/dnf)
• life-cycle management utilities — updates, configuration, health monitoring

Crucially, these components are tested together for compatibility and interoperability before release. That integration testing is the actual product a distribution sells.

Free / open-source software (the prerequisite the PDF assumes you know). “Free” as in freedom, not price. The GNU project defines four freedoms: to run the program, to study it, to modify it, and to redistribute it (modified or not). The GPL enforces these with copyleft: if you distribute a modified version, you must also distribute the source under the same licence.

4 · Internal Working#

The six things an operating system manages#

The source notes list these; here is what each one actually does.

flowchart TB
  subgraph US["USER SPACE — ring 3, unprivileged"]
    A["Your app<br/>python, nginx, bash"]
    L["Libraries<br/>glibc"]
  end
  SC{{"SYSTEM CALL BOUNDARY<br/>open · read · write · fork · execve · socket · mmap"}}
  subgraph KS["KERNEL SPACE — ring 0, full privilege"]
    P["Process management<br/>fork, scheduling, signals"]
    M["Memory management<br/>MMU, virtual memory, paging"]
    V["Storage / VFS<br/>ext4, XFS, Btrfs"]
    D["Device drivers<br/>block, char, net"]
    N["Network stack<br/>TCP/IP, netfilter"]
    S["CPU scheduler<br/>CFS / EEVDF"]
  end
  HW["HARDWARE — CPU · RAM · disks · NIC · GPU"]

  A --> L --> SC
  SC --> P & M & V & N & S
  P --> S
  V --> D
  N --> D
  D --> HW
  M --> HW
  S --> HW

Memory (MMU)

Gives every process its own private illusion of a full address space. The Memory Management Unit in the CPU translates a virtual address your program uses into a physical RAM address, using page tables the kernel maintains. This is why one process cannot read another's memory — the translation simply does not exist.

Processes

Creates, tracks and destroys running programs. Each has a PID, a parent, an owner, an address space and a set of open file descriptors. Linux creates processes by cloning (fork) and then replacing the program image (execve).

Devices (drivers)

Translates uniform kernel requests into device-specific register writes. A driver is why cat /dev/sda works the same whether the disk is NVMe, SATA or virtual.

Storage

The Virtual File System (VFS) layer presents one API (open/read/write/close) over ext4, XFS, Btrfs, NFS, tmpfs and even /proc. Chapter 11 goes deep on this.

CPU (scheduling)

Decides which runnable thread gets a core, and for how long. Linux used the Completely Fair Scheduler (CFS) for years; kernel 6.6 replaced it with EEVDF. You influence it with nice, chrt and cgroup CPU shares.

Networking

Implements TCP/IP, routing, firewalling (netfilter/nftables) and sockets. Chapter 15 covers the packet path end to end.

What actually happens when you run ls#

This single walkthrough explains more about Linux than any definition.

sequenceDiagram
    autonumber
    participant U as You
    participant B as bash (PID 2410)
    participant K as Kernel
    participant D as Disk driver
    U->>B: type "ls" + Enter
    B->>B: parse line, split words
    B->>K: search $PATH, stat("/usr/bin/ls")
    B->>K: fork() — clone myself
    K-->>B: child PID 2411
    Note over B,K: parent waits with wait4()
    B->>K: child calls execve("/usr/bin/ls")
    K->>K: build new address space, load ELF, map glibc
    K-->>B: child now runs ls code
    B->>K: ls calls openat(".") then getdents64()
    K->>D: read directory blocks
    D-->>K: raw bytes
    K-->>B: directory entries
    B->>K: write(1, "file1 file2\n", 12)
    K-->>U: bytes appear on terminal
    B->>K: exit(0) → parent reaped, prompt returns

Six kernel concepts are visible in that trace: PATH lookup, fork, execve, file descriptors (1 = stdout), system calls, and exit codes. You will meet all six again.

Kernel space vs user space, concretely#

  ┌────────────────────────────────────────────────────────────┐
  │  USER SPACE            can NOT: touch hardware directly,   │
  │  ring 3                        read other processes' RAM,  │
  │  bash, nginx, python           disable interrupts          │
  └───────────────────────────┬────────────────────────────────┘
                              │  syscall instruction
                              │  (a deliberate, checked trap)
  ┌───────────────────────────┴────────────────────────────────┐
  │  KERNEL SPACE          can do everything                   │
  │  ring 0                Crash here = kernel panic           │
  └────────────────────────────────────────────────────────────┘

A crash in user space kills one process. A crash in kernel space kills the machine. This is the entire security argument for VMs vs containers in Chapter 2 and Chapter 22 — containers share this kernel box; VMs each get their own.

5 · The History — and Why It Is Not Trivia#

Interviewers ask about Unix history because the design decisions of 1970 explain the commands you type in 2026.

timeline
    title From Multics to the cloud
    1965 : Multics begins — MIT, Bell Labs, GE
    1969 : Thompson and Ritchie write the first Unix on a PDP-7
    1971 : First Edition Unix manual published
    1973 : Unix rewritten in C — becomes portable
    1983 : Richard Stallman announces the GNU project
    1987 : Tanenbaum releases MINIX for teaching
    1988 : POSIX.1 (IEEE 1003.1) standardised
    1991 : Linus Torvalds posts Linux 0.01
    1992 : Linux relicensed under GPLv2
    1994 : Linux 1.0 · Red Hat Linux · Debian 0.91
    2003 : Linux 2.6 — scalable to big SMP servers
    2007 : The Linux Foundation is formed
    2008 : Android ships on the Linux kernel
    2011 : Linux 3.0
    2015 : Linux 4.0 — live kernel patching
    2017 : Linux 4.14 LTS — security and hardware breadth
    2018 : Ubuntu 18.04 LTS — GNOME replaces Unity
    2020 : Linux 5.10 LTS
    2022 : Linux 6.0
    2023 : Rust merged as a second kernel language (6.1+)

The lineage, step by step#

I. Multics (1965–1969) — “Multiplexed Information and Computer Services.” An ambitious MIT/Bell Labs/GE time-sharing system: many users on one machine, each believing they had it to themselves. It worked, but it was enormous, late and slow. Bell Labs withdrew.

II. Unix (1969–1971). Ken Thompson and Dennis Ritchie built a deliberately small alternative on a spare PDP-7 — the name UNICS (“UNiplexed Information and Computing Service”) was a pun on Multics. It contributed the ideas you will use every day:

• a hierarchical file system — one tree from /
• process management — cheap processes you can compose
• a command-line interface — a shell that is itself a program
• a wide range of small utilities — each doing one thing, chained by pipes
• (1973) rewritten in C, making Unix the first portable OS

III. POSIX (1980s). As Unix forked into many incompatible commercial variants (AIX, HP-UX, SunOS, Xenix — the “Unix wars”), the IEEE 1003.1 standard defined the language interface between application programs and the Unix OS to ensure portability: the C library, system interfaces and headers, plus commands and utilities.

IV. GNU (1983) — “GNU’s Not Unix.” Stallman’s response to increasingly proprietary Unix. It promoted the Free Software concept — the freedoms to run, study, modify and redistribute — protected by the GNU General Public License (GPL), and set out to build a complete free OS: the shell (bash), core utilities (ls, cp, grep), compilers (gcc) and the C library (glibc).

V. The Linux kernel (1991). Torvalds, a 21-year-old at the University of Helsinki, wanted a Unix-like system on his 386 and found MINIX too restrictively licensed. He posted:

Hello everybody out there using minix — I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones.

Licensed under GPLv2 from 1992, compiled with GNU GCC, it became exactly the kernel GNU was missing — giving a Unix-like OS with low cost, full control and strong community support.

Why the history shows up in your daily work#

6 · Why Learn Linux — the Honest Version#

The source notes give seven reasons. Here they are with the evidence a hiring manager would want.

1. Three decades of consistent growth. Linux is not a legacy skill or a fad; it has been the default server OS for over 20 years, and cloud made that dominance near-total.
2. Extraordinary versatility. Web servers, supercomputers (all of the TOP500), IoT devices, in-car systems including Tesla, and Android phones. Many other systems are Unix-like relatives, which makes the skills transferable.
3. Vast hardware support. One kernel spans a €4 microcontroller board to a 256-core server, thanks to a huge driver tree and a stable driver model.
4. Rich software availability. A deep native ecosystem, plus most major Windows/macOS applications either ported or replaced by equivalents.
5. Customisability. Open source and modular: strip it to 8 MB for an embedded device, or tune it for 10-million-packet-per-second networking.
6. A strong community and ecosystem. Forums, documentation, tools, conferences, and a genuinely searchable body of knowledge — when something breaks at 2 a.m., someone has already written it down.
7. Cost-effectiveness — especially for startups. Run websites, databases and applications with no licence fees, and easy installation, upgrade, deployment and maintenance.

To that list, add the two reasons that matter most for your career:

8. Linux is the substrate of DevOps. Docker, Kubernetes, Terraform, Ansible, Prometheus and every CI runner assume Linux. You cannot debug a CrashLoopBackOff without understanding processes, exit codes and file permissions.
9. It is the only environment you can fully inspect. /proc, /sys, strace, perf, bpftrace — you can watch the OS think. That builds the mental model that makes you senior.

7 · Understanding a Linux Distribution#

A distribution is an integration product. Same engine, wildly different assembly decisions.

flowchart TD
  K["Linux kernel<br/>kernel.org"] --> DEB["Debian"] & RPM["Red Hat / Fedora"] & ARCH["Arch"] & OTHER["SUSE · Alpine · Gentoo"]
  DEB --> UB["Ubuntu"] --> MINT["Linux Mint"] & POP["Pop!_OS"]
  DEB --> RASP["Raspberry Pi OS"] & KALI["Kali Linux"]
  RPM --> RHEL["RHEL"] --> ROCKY["Rocky / AlmaLinux"] & CENT["CentOS Stream"]
  RPM --> AMZ["Amazon Linux"]
  ARCH --> MANJ["Manjaro"] & ENDEAVOUR["EndeavourOS"]
  OTHER --> ALP["Alpine — the container default"]

The distributions named in the syllabus#

What “release model” means for you#

8 · Practical Demonstration#

Everything below is safe to run on any Linux machine, including a VM you just built. Type each one; do not read passively.

Identify the kernel: uname#

uname -a

$ uname -a
Linux web-prod-01 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Read the output field by field — this is exactly what an interviewer means by “can you read a command’s output?”

Every option:

Identify the distribution: /etc/os-release#

The kernel does not know which distribution it is part of. You have to ask userspace.

cat /etc/os-release

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
UBUNTU_CODENAME=noble

Why this file and not the alternatives:

# The idiom to memorise for scripts that must behave differently per family
. /etc/os-release
case "$ID_LIKE$ID" in
  *debian*) sudo apt-get install -y "$1" ;;
  *rhel*|*fedora*) sudo dnf install -y "$1" ;;
  *) echo "unsupported distro: $ID" >&2; exit 1 ;;
esac

Confirm you are on a VM or a container#

You will need this constantly, because “why is this file missing?” is often “because you are not where you think you are.”

$ hostnamectl
 Static hostname: web-prod-01
       Icon name: computer-vm
         Chassis: vm 🖳
      Machine ID: 9f2c1e4b7a8d4f1e9c3b5a6d7e8f9012
         Boot ID: 4a1b2c3d4e5f6789abcdef0123456789
  Virtualization: kvm
Operating System: Ubuntu 24.04.1 LTS
          Kernel: Linux 6.8.0-45-generic
    Architecture: x86-64

Virtualization: kvm is the giveaway. Other quick checks:

systemd-detect-virt          # prints kvm, vmware, oracle, docker, lxc — or "none" on bare metal
cat /sys/class/dmi/id/product_name   # "VirtualBox", "VMware Virtual Platform", "Standard PC (Q35 ...)"
grep -c ^processor /proc/cpuinfo     # how many logical CPUs the OS can see

Read the machine’s vital signs#

$ uptime
 21:34:52 up 43 days,  2:11,  3 users,  load average: 0.42, 0.55, 0.61

• up 43 days — uptime. A very long uptime on a server is not a badge of honour; it usually means unpatched kernels.
• 3 users — logged-in sessions.
• load average: 0.42, 0.55, 0.61 — runnable+waiting tasks averaged over 1, 5 and 15 minutes. Compare it to your core count: load 4.0 on 4 cores is fully busy; load 4.0 on 32 cores is idle. Get the core count from nproc.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       2.1Gi       0.9Gi        84Mi       4.7Gi       5.3Gi
Swap:          2.0Gi          0B       2.0Gi

lscpu | head -15        # architecture, cores, sockets, model, virtualization flags
nproc                   # just the number of usable logical CPUs
cat /proc/version       # kernel + compiler used to build it
ls /boot/vmlinuz-*      # the actual kernel images installed

Prove that “everything is a file”#

$ cat /proc/uptime
3729055.34 8912304.11

$ cat /proc/sys/kernel/hostname
web-prod-01

$ ls -l /proc/self/fd
lrwx------ 1 dev dev 64 Jul 31 21:09 0 -> /dev/pts/3
lrwx------ 1 dev dev 64 Jul 31 21:09 1 -> /dev/pts/3
lrwx------ 1 dev dev 64 Jul 31 21:09 2 -> /dev/pts/3

/proc is not a real directory on a disk — it is a virtual filesystem the kernel generates on read. You just read live kernel state with cat. That is the Unix philosophy made concrete, and it is why shell scripts can monitor a machine without any special API.

9 · Comparison Tables#

Linux vs Unix#

The single most common opening interview question in this syllabus.

Linux vs Windows Server#

Kernel vs distribution vs GNU#

Free software licences you will be asked about#

10 · Interview Corner#

11 · Common Mistakes#

12 · Summary & Mind Map#

mindmap
  root(("Linux<br/>foundations"))
    ("Operating system")
      ("Two jobs: abstraction + arbitration")
      ("Kernel space ring 0 / user space ring 3")
      ("System calls are the boundary")
    ("Six managed resources")
      ("Memory — MMU, virtual memory")
      ("Processes — fork, exec, signals")
      ("Devices — drivers")
      ("Storage — VFS, ext4/XFS")
      ("CPU — scheduler")
      ("Networking — TCP/IP")
    ("Lineage")
      ("Multics 1965")
      ("Unix 1969-71, C in 1973")
      ("POSIX IEEE 1003.1")
      ("GNU 1983 — four freedoms, GPL")
      ("Linux kernel 1991, GPLv2 1992")
    ("Distribution")
      ("kernel + libs + daemons")
      ("packaging + lifecycle tools")
      ("tested together")
      ("Debian / Red Hat / SUSE / Arch families")
    ("Why it matters")
      ("Cloud and containers run on it")
      ("Free, controllable, scriptable")
      ("Inspectable: /proc, /sys, strace")

Ten sentences that carry the chapter.

1. An OS abstracts hardware and arbitrates access to it; every Linux feature is one of those two things.
2. The kernel is the privileged core; everything else is userspace, and the system call is the border crossing.
3. Linux is a monolithic, modular, preemptive, multi-user, Unix-like kernel under GPLv2, first released in 1991.
4. A distribution is kernel + libraries + daemons + packaging/dev tools + lifecycle utilities, tested together.
5. GNU supplied the userland; Linux supplied the kernel; together they made a complete free OS.
6. POSIX is why Unix skills transfer between Linux, macOS and BSD.
7. uname -r is the kernel; /etc/os-release is the distribution; the kernel does not know its own distro.
8. Judge memory by available, and load average against core count.
9. /proc and /sys are the kernel exposed as files — the purest form of “everything is a file.”
10. The GPL’s copyleft is the legal reason three decades of competing vendors improved one shared kernel.

13 · Cheat Sheet#

IDENTIFY THE SYSTEM                      READ ITS STATE
  uname -r        kernel release           uptime          load + how long up
  uname -m        architecture             nproc           usable CPU count
  uname -a        everything               free -h         memory (read AVAILABLE)
  cat /etc/os-release   distro (script)    df -hT          disk usage + fs type
  hostnamectl     human summary            lscpu           CPU detail
  systemd-detect-virt   vm? container?     ps aux          process snapshot
  cat /proc/version     kernel + gcc       systemctl --failed   broken units

THE SIX THINGS AN OS MANAGES     Memory (MMU) · Processes · Devices (drivers)
                                 Storage (VFS) · CPU (scheduling) · Networking

PRIVILEGE                        user space = ring 3  →  syscall  →  kernel = ring 0
                                 user crash = one process ends
                                 kernel crash = the machine ends (panic)

TIMELINE   Multics 1965 → Unix 1969/71 → C 1973 → GNU 1983 → MINIX 1987
           → POSIX 1988 → Linux 0.01 1991 → GPLv2 1992 → 1.0 + Red Hat 1994
           → 2.6 in 2003 → Foundation 2007 → Android 2008 → 3.0 2011
           → 4.0 live patching 2015 → 5.10 LTS 2020 → 6.0 in 2022 → Rust 6.1

FIVE PARTS OF A DISTRO   kernel · libraries · system daemons ·
                         dev + packaging tools · lifecycle utilities
                         ...all TESTED TOGETHER  (classic exam answer)

FAMILIES   Debian/Ubuntu → apt, .deb        Red Hat/Fedora/RHEL → dnf, .rpm
           SUSE → zypper, .rpm              Arch → pacman       Alpine → apk

ONE-LINERS THAT WIN INTERVIEWS
  "Linux is the engine; Ubuntu is the car."
  "Unix is the ancestor and a certification; Linux is a GPL-licensed
   Unix-like kernel that shares its behaviour but not its code."
  "The kernel doesn't know it's Ubuntu — that's a userspace fact."

14 · Practice#

Flashcards#

Multiple choice#

1. Which is not one of the six resources an OS manages, as listed in the syllabus? (a) Memory (b) Processes (c) Compilation (d) Networking
2. uname -r prints: (a) the distribution name (b) the kernel release (c) the CPU architecture (d) the hostname
3. The Linux kernel is licensed under: (a) MIT (b) Apache 2.0 (c) GPLv2 (d) GPLv3
4. Which is a Debian-family distribution? (a) Fedora (b) openSUSE (c) Ubuntu (d) Rocky Linux
5. Which statement is true? (a) macOS is based on the Linux kernel (b) Android is based on the Linux kernel (c) PlayStation OS is based on Linux (d) Windows uses a Linux kernel
6. POSIX exists to: (a) speed up the kernel (b) standardise the application/OS interface for portability (c) manage packages (d) license free software
7. A crash in kernel space: (a) kills one process (b) kills the machine (c) is caught by the shell (d) restarts the process
8. Which command shows whether you are inside a VM? (a) uname -p (b) systemd-detect-virt (c) nproc (d) history
9. Load average 4.00 is healthy on a host with: (a) 1 core (b) 2 cores (c) 32 cores (d) load average is unrelated to cores
10. “Monolithic kernel” means: (a) it cannot be extended (b) drivers run in kernel address space (c) it is one single file on disk (d) it supports one user

Fill in the blanks#

1. The kernel runs in ring __ ; applications run in ring ____ .
2. The interface applications use to request privileged operations is the __ ____ .
3. GNU was founded in __ by _ ___ .
4. Linux was relicensed under the GPL in ______ .
5. ______ ______ is the standard file to read to identify the distribution.
6. The unit that translates virtual addresses to physical ones is the ______ .
7. The layer presenting one API over many filesystems is the ______ .
8. A Linux distribution’s five components are kernel, __ , system _ , development/packaging tools and ___ -cycle utilities.

True or false#

1. Linux contains original AT&T Unix source code.
2. uname can tell you which distribution you are running.
3. Every Linux distribution is free of charge.
4. Android is built on the Linux kernel.
5. A distribution’s main value is its integration and testing of components.
6. /proc is stored on your disk.
7. macOS is a certified UNIX; Linux generally is not.
8. Adding a driver to Linux always requires recompiling the kernel.

Hands-on lab#

Do these on a throwaway VM (build one in Chapter 2 first if you have not).

1. Fingerprint your machine. Record kernel release, architecture, distribution and version, virtualisation type, core count, total RAM and root filesystem type — one command each. Put the results in a file called fingerprint.txt.
2. Read live kernel state as files. cat five different files under /proc and explain in one line each what they told you. Include /proc/uptime, /proc/meminfo, /proc/cpuinfo, /proc/loadavg and /proc/self/status.
3. Watch a program talk to the kernel. Install strace, then run strace -c ls and strace -f -e trace=openat ls. Identify the execve, the openat calls for shared libraries, and the final write to fd 1.
4. Prove case sensitivity. Create file.txt and File.txt in one directory, put different text in each, and cat both. Explain why this breaks code ported from macOS or Windows.
5. Compare two package families. Without installing anything, work out and write down the command to install nginx, remove it, search for it, and list installed packages — once for apt, once for dnf.
6. Read a distribution’s identity in a script. Write a five-line shell script that sources /etc/os-release and prints Detected: <NAME> <VERSION_ID> (family: <ID_LIKE>).

Challenge problems#

1. Explain, without notes, the difference between a Linux distribution and the Linux kernel, and how they interact within the overall system.
2. Find three trustworthy sources for downloading Linux distributions, and explain how you would verify an ISO you downloaded is genuine (research sha256sum and GPG signature verification).
3. Investigate whether Linux is the same as UNIX: histories, similarities, differences, and the reason Linux was created rather than a Unix being adopted.
4. Determine whether all Linux distributions are free. Compare pricing models, professional support, additional services and enterprise features for Ubuntu, RHEL and Rocky Linux.
5. Investigate how well Linux runs on varied hardware, and list the key considerations before installing it on an unfamiliar device (firmware type, drivers, architecture, secure boot).
6. Map the Linux community and ecosystem: where the resources, forums, tutorials and documentation you would actually rely on live, ranked by trustworthiness.
7. Explain the significance of the GPL and how it affects using, modifying and redistributing Linux. Contrast it with a permissive licence like MIT.
8. Enumerate Linux’s security features and how each maintains system security: user/group permissions, sudo, file ownership, SELinux/AppArmor, seccomp, capabilities, namespaces, ASLR, signed kernel modules.
9. List the basic commands every Linux user should know and group them by purpose. Compare your list against Chapters 4–9 afterwards and note what you missed.
10. Compare Ubuntu, Fedora and Arch Linux on installation ease, package management, community support and target audience. Recommend one for a beginner and defend the choice.

1 · The Big Picture#

Why this topic exists#

In 2001, a typical corporate data centre looked like this: a rack of servers, each one running exactly one application, each one averaging 5 to 15 percent CPU utilisation. The mail server was idle at night. The payroll server was idle for 29 days a month. The web server needed twelve cores at 11 a.m. and none at 3 a.m. Every one of those machines drew power, occupied rack space, required cooling, and had to be bought, patched and eventually replaced.

The reason for one-app-per-server was not stupidity. It was blast radius. If the mail server and the payroll database shared a machine, a memory leak in one starved the other; a security compromise in one exposed the other; a required reboot for one took down both. Operations teams bought isolation the only way they knew how — with separate hardware.

Virtualization broke that trade-off. It lets you keep the isolation and stop wasting the hardware, by inserting a layer that hands each workload a convincing, private, complete-looking computer that does not actually exist.

That is the whole idea. Everything else in this chapter is mechanism.

The real problem it solves#

  BEFORE                                    AFTER

  ┌────────┐ ┌────────┐ ┌────────┐          ┌───────────────────────────┐
  │ mail   │ │ payroll│ │  web   │          │  ┌─────┐ ┌─────┐ ┌─────┐  │
  │  8%    │ │   3%   │ │  11%   │          │  │mail │ │pay  │ │ web │  │
  │  used  │ │  used  │ │  used  │          │  └─────┘ └─────┘ └─────┘  │
  ├────────┤ ├────────┤ ├────────┤          │  ┌─────┐ ┌─────┐ ┌─────┐  │
  │ 1 CPU  │ │ 1 CPU  │ │ 1 CPU  │          │  │ ci  │ │test │ │stage│  │
  │ 1 PSU  │ │ 1 PSU  │ │ 1 PSU  │          │  └─────┘ └─────┘ └─────┘  │
  │ 1 rack │ │ 1 rack │ │ 1 rack │          ├───────────────────────────┤
  │  slot  │ │  slot  │ │  slot  │          │      HYPERVISOR           │
  └────────┘ └────────┘ └────────┘          ├───────────────────────────┤
                                            │  ONE machine · 78% used   │
  3 machines · 92% of the money             └───────────────────────────┘
  spent on idle silicon
                                            Same isolation. One third of
                                            the power, space and capital.

Three separate wins fall out of that single move, and it is worth naming them separately because interviewers ask for them separately:

• Consolidation — many logical machines on one physical machine, so you stop paying for idle silicon.
• Decoupling — the operating system no longer knows or cares what hardware it is on, so a machine becomes a file you can copy, snapshot, move and version.
• Isolation — a crash, a kernel panic, a rogue process or a compromised service is contained inside one VM.

The second one is the quiet revolution. Once a server is a file, everything DevOps does becomes possible: reproducible environments, immutable infrastructure, golden images, automated provisioning, disaster recovery that is a restore rather than a purchase order. Amazon Web Services is, at its foundation, a very large business built on the observation that a computer can be an API call.

Where you will encounter it#

Why companies care#

• Capital and operating cost. Consolidation ratios of 10:1 to 30:1 are routine. Fewer machines means less power, less cooling, less rack space, fewer support contracts.
• Speed of provisioning. A physical server takes weeks — quote, purchase, ship, rack, cable, install. A VM takes 90 seconds. That difference is why “just spin up a test environment” is a sentence that exists.
• Disaster recovery. A VM is a directory of files plus a description. Replicate those to another site and you have a recovery plan that can be tested without buying a second data centre.
• Hardware independence. A VM configured with a virtual virtio disk and a virtual NIC boots on any host, regardless of whether the physical disk is SAS, SATA or NVMe. You can replace a whole generation of hardware underneath a running estate.
• Consolidated maintenance. Live migration moves running VMs off a host so you can patch its firmware at 2 p.m. on a Tuesday instead of 3 a.m. on a Sunday.

2 · Intuition First#

Analogy 1: the subdivided building#

Chapter 1 used a building manager for the operating system. Extend the same building.

You own a large warehouse. You could let one tenant have all of it — that is a bare-metal server. Instead, you build internal walls, run separate electricity meters, fit separate locks, and let out six units. Each tenant:

• believes they have “a building” — their own front door, their own meter, their own space
• cannot walk into another unit
• shares the roof, the foundations and the electricity supply, whether they know it or not
• can be evicted, and their unit re-let, without touching anyone else

The hypervisor is the landlord: it decides how much floor space and power each unit gets, enforces the locks, and is the only party who can see the whole floor plan. The VM’s operating system is a tenant who has never seen the outside of the building.

The analogy predicts real behaviour. If the landlord promises 8 units of power each but the supply only delivers 30, everyone is fine until they all switch on the kettle at once — that is memory and CPU overcommitment, and section 9 covers what happens next. If someone drills through a wall, the isolation was never as absolute as the tenants believed — that is a VM escape, and section 17 covers it.

Analogy 2: the flight simulator#

A flight simulator presents a pilot with a cockpit: throttle, yoke, instruments, the view out of the window. Every control is real to the touch, and every instrument reacts correctly. Nothing behind the panel is an aircraft.

A full virtualization hypervisor does exactly this to an operating system. When Windows asks the hard disk controller for sector 40,000, something answers, and it answers the way an Intel AHCI controller would — but it is software, and the sector lives inside a .vdi file on your laptop’s SSD. Windows never finds out.

Now imagine a simulator where the pilot knows it is a simulator, and instead of moving a physical throttle they type set_thrust(0.8). Far less machinery to build, far more efficient, but it only works with a pilot trained for that interface. That is paravirtualization: the guest knows, cooperates, and calls the hypervisor directly.

Analogy 3: the house, the flat and the hot-desk#

This one settles the VM-versus-container question that Chapter 19 will formalise.

  DETACHED HOUSE           BLOCK OF FLATS           HOT-DESK OFFICE
  = physical server        = virtual machines       = containers

  Your own foundations     Shared foundations,      Shared everything:
  Your own plumbing        shared roof              one kitchen, one
  Your own roof            YOUR OWN kitchen,        bathroom, one set
  Your own everything      bathroom, front door     of plumbing.
                                                    You get a desk and
  Expensive.               Each flat has its        a locker.
  Total isolation.         OWN KERNEL.
                                                    Every container
                           A fire in flat 3         SHARES THE HOST
                           is contained by          KERNEL.
                           concrete.
                                                    Cheap, instant,
                           Boots in 30 s,           dense. Boots in
                           costs ~1 GB RAM          0.1 s, costs ~10 MB.
                           of overhead.
                                                    A plumbing failure
                                                    affects everyone.

The single sentence to remember: a VM virtualizes hardware, a container virtualizes an operating system. A VM has its own kernel; a container borrows the host’s.

3 · Technical Definitions#

Now the precise versions.

Virtual machine (VM)

A software-based representation of a physical computer. It emulates the hardware components a real machine has — CPU, memory, storage devices and network interfaces — presenting them to a guest operating system that runs unmodified and, in general, unaware. A VM is defined by two things on disk: a configuration (how much RAM, which devices) and one or more virtual disk images.

Virtualization

The technique of creating those software-based representations of physical resources, so that one physical resource can be presented as many logical ones (or, less commonly, many as one — that is aggregation, as in a storage pool).

Hypervisor (Virtual Machine Monitor, VMM)

The software layer that creates, runs and manages virtual machines. It allocates physical resources to VMs, schedules their virtual CPUs onto real cores, mediates their I/O, and enforces isolation between them. It is the only component that sees all VMs.

Host

The physical machine, and (for a Type 2 hypervisor) the operating system running directly on it.

Guest

The operating system running inside a VM. "Guest OS", "guest kernel", "guest additions" all refer to this side of the boundary.

vCPU

A virtual CPU: from the guest's point of view a processor core, from the host's point of view a thread that the hypervisor schedules onto a real logical CPU. vCPUs are time-shared, which is why 40 vCPUs can exist on a 16-thread host.

Virtual disk image

A file (or set of files, or a block device) that the guest sees as a hard disk. Formats: .vdi (VirtualBox), .vmdk (VMware), .qcow2 and raw (QEMU/KVM), .vhdx (Hyper-V).

Guest additions / guest tools

Drivers and daemons installed inside the guest that let it cooperate with the hypervisor: paravirtualized device drivers, clipboard and drag-and-drop, shared folders, dynamic screen resize, time synchronisation, graceful-shutdown handling, and reporting the guest's IP address back to the host.

Snapshot

A captured point-in-time state of a VM — disk contents, and optionally RAM and CPU state — that you can revert to. Implemented by freezing the current disk read-only and writing all subsequent changes to a new delta or differencing file.

Unpacking the dense definition#

The sentence “a hypervisor multiplexes physical resources among isolated guests while retaining control” hides five distinct jobs. Interviewers like to pull them apart.

Virtualization is not emulation, and not simulation#

These three get used interchangeably in conversation and must not be in an exam.

QEMU is the confusing one, because it does both: qemu-system-aarch64 on an x86 host is emulation, while qemu-system-x86_64 -enable-kvm on an x86 host is virtualization with QEMU acting only as the device model. Apple’s Rosetta 2 and Microsoft’s x86-on-ARM layer are emulation (specifically, binary translation) at the application rather than machine level.

4 · Types of Virtualization#

Virtualization comes in several forms, each solving a different scope of problem. The three the syllabus names are the three you must be able to distinguish instantly, because the boundary between the first two is the VM-versus-container question.

Hardware-level virtualization#

Emulating an entire physical machine’s hardware: CPU, memory, storage, firmware, buses, network interfaces. Because the illusion is complete, an unmodified operating system runs on it. The guest boots its own bootloader, loads its own kernel, initialises its own drivers, and manages its own memory — all against virtual hardware.

This is what “a VM” means in ordinary usage, and it is what the rest of this chapter is mostly about.

Its defining use case is running software that requires a specific hardware or OS environment you do not have. If you have an application that only runs on Windows but your machine runs Linux, hardware-level virtualization lets you create a VM presenting a complete PC, install Windows in it, and run the application — on the same laptop, at the same time, with no dual-boot and no reboot.

A popular example of hardware-level virtualization is KVM — the Kernel-based Virtual Machine — which transforms the Linux kernel itself into a hypervisor, allowing multiple virtual machines to run unmodified Linux or Windows images. Section 13 goes through it properly; note the phrasing now, because “name an example of hardware-level virtualization” is a one-mark question and “KVM” is the expected answer.

Operating-system-level virtualization (containerization)#

Instead of faking hardware, fake the operating system. Multiple isolated user-space instances — containers — run on a single shared host kernel. Each container gets its own view of the filesystem, process table, network stack, hostname and user IDs, but there is exactly one kernel on the machine, and every container’s system calls go into it.

On Linux this is built from two kernel features you will meet again in Chapter 19:

• namespaces — per-container views of the filesystem mount table, PIDs, network interfaces, hostname, users and IPC. Isolation of what you can see.
• cgroups — per-container limits and accounting for CPU, memory, block I/O and process count. Isolation of what you can use.

Because there is no second kernel, no firmware, no boot process and no duplicated OS in memory, containers are dramatically lighter than VMs: tens of megabytes rather than gigabytes, and start times measured in tens of milliseconds. That efficiency and density is precisely why containers won for microservices and horizontally scalable applications, where you may want 300 instances of a small service on one host.

The cost is the shared kernel. Containers isolate at the application level, not the hardware level. A kernel vulnerability is a shared vulnerability, and you cannot run a Windows container on a Linux kernel — or a different kernel version — no matter how much you want to.

Application virtualization#

Separate a single application from the operating system beneath it, so that it runs in a self-contained environment carrying its own dependencies.

The problem it solves is dependency conflict. Application A needs version 1.8 of a runtime; application B needs 2.4; the OS ships 2.0. Rather than virtualizing a machine or an OS, encapsulate each application with the exact libraries and runtime it expects, so it neither affects nor is affected by the rest of the system.

Real examples, most of which you have used without labelling them this way:

The other types worth naming#

The syllabus stops at three. Production estates use four more, and mentioning them signals breadth.

flowchart TB
  V["Virtualization"] --> HW["Hardware level<br/>emulate a whole machine"]
  V --> OS["OS level<br/>share one kernel"]
  V --> APP["Application level<br/>encapsulate one app"]
  V --> OTH["Also: desktop · network<br/>storage · GPU"]
  HW --> HWE["KVM · VMware ESXi · Hyper-V<br/>Xen · VirtualBox · QEMU"]
  OS --> OSE["Docker · Podman · LXC<br/>containerd · Windows containers"]
  APP --> APE["Snap · Flatpak · AppImage<br/>App-V · venv · JVM"]
  HWE --> HWR["Own kernel · GB of RAM<br/>seconds to boot · strong isolation"]
  OSE --> OSR["Shared kernel · MB of RAM<br/>ms to start · process isolation"]
  APE --> APR["Shared kernel and OS<br/>dependency isolation only"]

5 · Internal Working — how a hypervisor actually does it#

This section is the one the source notes skip entirely, and it is the one that separates someone who has used VirtualBox from someone who can be trusted with a hypervisor estate. It answers a single question: if the guest kernel thinks it owns the machine, what stops it?

The problem: x86 was not built for this#

Recall the privilege rings from Chapter 1. A normal OS kernel runs in ring 0 and applications in ring 3. Ring 0 code may load page tables, mask interrupts, halt the CPU and talk to devices.

Now put two kernels on one machine. Both were written expecting ring 0. Only one thing can actually be in charge. So the hypervisor takes ring 0 and pushes the guest kernel down — historically into ring 1 or ring 3, a trick called ring deprivileging. The guest kernel is now running at a privilege level it was never compiled for.

For a classically virtualizable architecture, that is fine: every privileged instruction the guest attempts would trap — raise an exception the hypervisor catches — and the hypervisor emulates the intended effect. That is trap-and-emulate, and it is the 1974 Popek–Goldberg model.

x86 broke it. In 2000, Robin and Irvine catalogued 17 sensitive but unprivileged instructions on x86: instructions whose behaviour depends on the current privilege level but which fail silently instead of trapping when executed with insufficient privilege. The textbook case is POPF: it pops flags off the stack, and one of those flags is the interrupt-enable flag IF. Executed in ring 0, it changes interrupt masking. Executed in ring 1 or 3, it silently ignores that bit and carries on. No trap, no error — the guest kernel believes it has disabled interrupts and it has not.

Other symptoms of the same disease: PUSHF reveals the real flags, so the guest can see it is deprivileged; SGDT/SIDT/SLDT read descriptor-table registers and leak the hypervisor’s values; LAR/LSL and PUSH CS reveal the true current ring. Collectively: ring aliasing, address-space compression and non-faulting access to privileged state.

So x86 offered three escape routes, and all three matter historically.

Technique 1 — trap-and-emulate#

The clean model, and still the mechanism underneath the modern one.

  Guest kernel (deprivileged)                Hypervisor (ring 0)
  ───────────────────────────                ───────────────────
   mov eax, 1
   add ebx, eax          ← runs natively, full speed
   mov cr3, edx          ← PRIVILEGED
        │
        │ CPU raises #GP  ─────────────────►  1. catch the fault
        │                                     2. decode the instruction
        │                                     3. update the SHADOW page
        │                                        tables to match
        │ ◄─────────────── resume at next ──   4. return to guest
        │                   instruction
   jmp continue           ← guest never knew

Verdict: correct and elegant. On x86 it is incomplete, because the 17 sensitive-unprivileged instructions never generate the fault in step 1.

Technique 2 — binary translation#

VMware’s answer in 1999, and the reason VMware existed as a company before Intel and AMD caught up.

The hypervisor does not run guest kernel code directly. It reads guest kernel instructions, translates them into a safe equivalent sequence, caches the translation, and runs the translation instead. Guest user-space code (ring 3) still runs natively at full speed, because it is already unprivileged and harmless. Only ring-0 guest code goes through the translator.

  GUEST RING 3 code  ────────────────────────►  runs NATIVELY, untouched
  (your app, ~99% of cycles)

  GUEST RING 0 code
  ┌──────────────┐    ┌──────────────────┐    ┌───────────────────────┐
  │ cli          │    │  TRANSLATOR      │    │  TRANSLATION CACHE    │
  │ popf         │───►│  scan basic block│───►│  vcpu->flags.if = 0   │
  │ mov cr3, edx │    │  rewrite unsafe  │    │  vcpu->flags = ...    │
  │ ret          │    │  instrs into      │    │  call shadow_pt_set  │
  └──────────────┘    │  hypervisor calls│    │  ret                  │
                      └──────────────────┘    └───────────┬───────────┘
                        translate ONCE                    │ execute
                        execute MANY times      ◄──────────┘  many times

Because a basic block is translated once and executed thousands of times, the amortised cost is far lower than interpretation. It works on any x86 CPU, including pre-2005 hardware, with an unmodified guest — the guest genuinely cannot tell. The cost is enormous engineering complexity, plus real overhead on system-call-heavy and page-table-heavy workloads.

Technique 3 — hardware-assisted virtualization (the one you actually use)#

In 2005–2006 Intel shipped VT-x (project name Vanderpool) and AMD shipped AMD-V (project name Pacifica). Both solve the problem the same way: rather than squeezing two kernels into four rings, add a whole new dimension of privilege.

The CPU gains two modes of operation:

• VMX root mode — where the hypervisor runs. It has rings 0–3 of its own.
• VMX non-root mode — where guests run. Also has rings 0–3 of its own.

So the guest kernel runs in ring 0 of non-root mode. It is genuinely in ring 0; POPF behaves; PUSHF shows what the guest expects; no deprivileging, no translation, no lying. But the CPU is configured so that a list of specified operations cause a VM exit: an atomic, hardware-assisted transition into the hypervisor with all guest state saved.

The informal name for the hypervisor’s new privilege level is “ring −1”. It is not an architectural term — Intel’s manuals say VMX root operation and AMD’s say host mode — but it is the phrase interviewers use, and it captures the idea exactly: a level of privilege beneath ring 0.

   ┌─────────────────────────────────────────────────────────────┐
   │ VMX NON-ROOT OPERATION  (the guest's own private world)      │
   │                                                              │
   │   ring 3   guest applications        nginx, bash, notepad    │
   │   ring 0   GUEST KERNEL — genuinely ring 0, unmodified       │
   └───────────────────────────┬─────────────────────────────────┘
                               │  VM EXIT      ▲  VM ENTRY
                               │  (hardware,   │  (VMLAUNCH /
                               │   atomic,     │   VMRESUME)
                               ▼   ~1-2 µs)    │
   ┌───────────────────────────┴─────────────────────────────────┐
   │ VMX ROOT OPERATION — informally "RING -1"                    │
   │   ring 0   HYPERVISOR (KVM module / ESXi vmkernel / Hyper-V) │
   │   Owns the VMCS: which events exit, and the saved guest state│
   └───────────────────────────┬─────────────────────────────────┘
   ┌───────────────────────────┴─────────────────────────────────┐
   │ PHYSICAL CPU · RAM · devices                                 │
   └─────────────────────────────────────────────────────────────┘

The control block that stores all of this is the VMCS (Virtual Machine Control Structure) on Intel, or the VMCB (Virtual Machine Control Block) on AMD. It is a 4 KB region per vCPU holding the guest’s saved register state, the host’s saved state, and — critically — the exit controls: bitmaps saying which instructions, exceptions, I/O ports and MSR accesses should cause an exit. Tuning those bitmaps to exit as rarely as possible is what hypervisor performance engineering largely consists of.

New instructions came with it: VMXON/VMXOFF to enter and leave root operation, VMLAUNCH/VMRESUME to run a guest, VMREAD/VMWRITE to manipulate the VMCS, and VMCALL — the hypercall instruction a cooperating guest uses to deliberately call the hypervisor. AMD’s equivalents are VMRUN, VMLOAD/VMSAVE and VMMCALL.

The second half: virtualizing memory with EPT and NPT#

Solving the CPU was only half the problem. Memory was arguably worse.

A normal OS maintains page tables mapping virtual → physical. A guest OS does the same, but its idea of “physical” is a fiction — guest-physical address 0 is not host-physical address 0. So there are two translations needed:

  guest virtual  ──(guest page tables)──►  guest physical
                                                │
                                     ──(hypervisor mapping)──►  host physical

Before hardware support, hypervisors used shadow page tables: the hypervisor secretly built a third set of page tables collapsing both steps into one guest-virtual → host-physical map, and pointed the real CR3 at that. It works, but the hypervisor must intercept every guest page-table modification to keep the shadow in sync. On a fork-heavy or memory-mapping-heavy workload, that is thousands of VM exits per second, and shadow tables consume real memory per process per VM.

The second generation of hardware assist fixed it:

• Intel EPT — Extended Page Tables (Nehalem, 2008)
• AMD NPT / RVI — Nested Page Tables, marketed as Rapid Virtualization Indexing (Barcelona, 2007)

The MMU now walks two page-table hierarchies in hardware. The guest’s own CR3 and page tables handle guest-virtual → guest-physical; a second, hypervisor-owned hierarchy handles guest-physical → host-physical. The guest can rewrite its page tables as often as it likes with no exits at all.

Two more hardware pieces complete the picture, both worth naming:

• IOMMU — Intel VT-d, AMD-Vi. Translates device DMA addresses the way the MMU translates CPU addresses. It is what makes PCI passthrough safe: without it, giving a VM direct control of a NIC or GPU would let that VM’s device DMA anywhere in host memory. Required for GPU passthrough, SR-IOV and secure device assignment.
• SR-IOV — Single Root I/O Virtualization. A physical NIC advertises multiple lightweight virtual functions, each assignable straight to a VM. Near-native network performance with no hypervisor in the data path. This is a large part of how AWS delivers 100 Gbit networking to instances.

What a VM exit looks like, end to end#

Trace one disk read from a guest to physical hardware. This single walkthrough explains more about hypervisors than any definition.

sequenceDiagram
    autonumber
    participant A as App in guest
    participant G as Guest kernel
    participant C as CPU
    participant K as KVM (ring -1)
    participant Q as QEMU device model
    participant H as Host kernel + SSD
    A->>G: read() syscall
    Note over A,G: ordinary syscall, no exit — stays inside the VM
    G->>C: write to virtio queue, then notify the device
    C->>K: VM EXIT (reason: EPT violation on the MMIO page)
    Note over C,K: guest state saved to the VMCS, ~1-2 microseconds
    K->>Q: hand off: guest wants I/O on this virtio device
    Q->>H: pread() on MyVM.qcow2
    H-->>Q: 4 KB of data
    Q->>Q: place data in the guest's DMA buffer
    Q->>K: raise a virtual interrupt for the guest
    K->>C: VM ENTRY (VMRESUME) with interrupt injected
    C->>G: guest sees IRQ, virtio driver completes the request
    G-->>A: read() returns

Three lessons live in that trace, and each is a real interview answer:

1. Compute is nearly free; I/O is where the overhead is. Steps for arithmetic, branches and even system calls inside the guest never leave non-root mode. The exit happens at the device boundary. This is why CPU-bound workloads run at 97–99% of native speed while naive I/O can be far worse.
2. The hypervisor is often two pieces. KVM does the CPU and memory virtualization inside the kernel; QEMU, a userspace process, emulates the devices. That split is why ps aux on a KVM host shows one qemu-system-x86_64 process per VM.
3. Reducing exits is the whole game. VirtIO, vhost-net (moving the network datapath into the host kernel), vhost-user with DPDK, SR-IOV and device passthrough are all strategies for making step 3 happen less often, or not at all.

Checking whether your CPU can do this#

Everything above requires hardware support. Any x86 CPU sold since roughly 2008 has it, but it is frequently disabled in firmware, and this is the single most common reason a beginner’s first VM refuses to start or crawls.

# 1. Does the CPU report the extension at all?
#    vmx = Intel VT-x    svm = AMD-V
grep -Eoc '(vmx|svm)' /proc/cpuinfo

$ grep -Eoc '(vmx|svm)' /proc/cpuinfo
16

That 16 is the number of lines in /proc/cpuinfo whose CPU flags contain vmx or svm — in other words, one per logical CPU on a 16-thread machine. The number itself does not matter. Any value greater than 0 means the extension is present and enabled; 0 means it is absent or switched off in firmware.

# Variants worth knowing
grep -E --color '(vmx|svm)' /proc/cpuinfo | head -1   # see the whole flags line
lscpu | grep -i virtual                               # cleaner, summarised

$ lscpu | grep -i virtual
Virtualization:                       VT-x
Virtualization type:                  full

# 2. The friendliest check on Debian/Ubuntu
sudo apt install cpu-checker
sudo kvm-ok

$ sudo kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Versus the failure you must be able to diagnose:

$ sudo kvm-ok
INFO: /dev/kvm does not exist
HINT:   sudo modprobe kvm_intel
INFO: Your CPU supports KVM extensions
INFO: KVM (vmx) is disabled by your BIOS
HINT: Enter your BIOS setup and enable Virtualization Technology (VT),
      and then hard poweroff/poweron your system
KVM acceleration can NOT be used

Read those two outputs carefully — they are different failures. “CPU does not support KVM extensions” means the silicon lacks it (very old, or a low-end Atom/Celeron). “Disabled by your BIOS” means you can fix it.

# 3. Are the kernel modules loaded, and does the device node exist?
lsmod | grep -E '^kvm'
ls -l /dev/kvm
cat /sys/module/kvm_intel/parameters/nested   # Y if nested virtualisation is on

$ lsmod | grep -E '^kvm'
kvm_intel             376832  6
kvm                  1146880  1 kvm_intel

$ ls -l /dev/kvm
crw-rw----+ 1 root kvm 10, 232 Jul 31 09:14 /dev/kvm

/dev/kvm is a character device owned by group kvm. If your user is not in that group, libvirt and QEMU will fall back to slow emulation or refuse outright — sudo usermod -aG kvm,libvirt $USER, then log out and back in.

Full virtualization#

With the mechanisms understood, the two named technologies in the syllabus make sense.

Full virtualization completely emulates the underlying hardware, allowing unmodified guest operating systems to run in isolation. The hypervisor traps and emulates privileged instructions from the guest OS, ensuring complete isolation and security. The guest operates as if it were running on actual hardware, entirely unaware that it is in a virtual environment.

  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐
  │   GUEST OS A   │  │   GUEST OS B   │  │   GUEST OS C   │
  │  Windows 11    │  │ Ubuntu 24.04   │  │    RHEL 9      │
  │  UNMODIFIED    │  │  UNMODIFIED    │  │  UNMODIFIED    │
  │                │  │                │  │                │
  │ sees: Intel    │  │ sees: Intel    │  │ sees: Intel    │
  │  AHCI disk,    │  │  AHCI disk,    │  │  AHCI disk,    │
  │  e1000 NIC,    │  │  e1000 NIC,    │  │  e1000 NIC,    │
  │  PS/2 mouse,   │  │  PS/2 mouse,   │  │  PS/2 mouse,   │
  │  PIIX3 chipset │  │  PIIX3 chipset │  │  PIIX3 chipset │
  └───────┬────────┘  └───────┬────────┘  └───────┬────────┘
          │                   │                   │
     privileged instruction / device access
     is TRAPPED and EMULATED — the guest is never told
          │                   │                   │
  ┌───────┴───────────────────┴───────────────────┴────────┐
  │  HYPERVISOR + device model                             │
  │  Presents a complete, believable virtual machine:      │
  │  vCPU · vRAM · virtual BIOS/UEFI · virtual PCI bus ·   │
  │  virtual disk controller · virtual NIC · virtual VGA   │
  └────────────────────────┬───────────────────────────────┘
  ┌────────────────────────┴───────────────────────────────┐
  │  PHYSICAL HARDWARE   CPU · RAM · SSD · NIC · GPU       │
  └────────────────────────────────────────────────────────┘

  ✔ Maximum compatibility — install any OS from its normal ISO
  ⚠ Emulation layer costs performance, mostly on I/O

Provides maximum compatibility, but can incur performance overhead due to the emulation layer. That is the trade-off in one line. Examples: VMware ESXi and Workstation, KVM/QEMU, VirtualBox, Hyper-V, Xen in HVM mode.

Paravirtualization#

Paravirtualization provides a software interface similar to the underlying hardware but not identical. The guest operating system is modified to be aware it is virtualized, and interacts with the hypervisor through special hypercalls rather than by pretending to poke hardware. Reducing the complexity of hardware emulation reduces overhead — but it requires modifying the guest OS, which is not always feasible.

  ┌──────────────────────────┐  ┌──────────────────────────┐
  │      GUEST OS A          │  │      GUEST OS B          │
  │   MODIFIED kernel        │  │   MODIFIED kernel        │
  │   + paravirtual drivers  │  │   + paravirtual drivers  │
  │                          │  │                          │
  │  knows it is a VM.       │  │  knows it is a VM.       │
  │  Does NOT pretend to     │  │  Does NOT pretend to     │
  │  write to a disk         │  │  write to a disk         │
  │  controller register.    │  │  controller register.    │
  └────────────┬─────────────┘  └────────────┬─────────────┘
               │                              │
   HYPERCALL: a deliberate, direct call into the hypervisor
   "map this page"          "here is a ring buffer of packets"
   "yield my timeslice"     "block me until this event fires"
               │                              │
   No trap. No instruction decoding. No fake hardware to emulate.
               │                              │
  ┌────────────┴──────────────────────────────┴─────────────┐
  │  HYPERVISOR — exposes an API, not imitation hardware     │
  │  Shared memory ring buffers · event channels · grant     │
  │  tables (Xen) or virtqueues (VirtIO)                     │
  └────────────────────────┬────────────────────────────────┘
  ┌────────────────────────┴────────────────────────────────┐
  │  PHYSICAL HARDWARE                                      │
  └─────────────────────────────────────────────────────────┘

  ✔ Much less overhead — batching replaces per-register traps
  ⚠ Guest kernel must be ported. Closed-source guests need vendor buy-in

Why paravirtualization mattered — and where it went#

This is the piece the source notes leave out, and it is the piece that makes the two technologies stop feeling arbitrary.

Before VT-x, paravirtualization was the only way to get good x86 performance without VMware’s binary translator. Xen, released from Cambridge in 2003, took exactly that route: it modified the Linux kernel to run as a Xen PV guest, replacing every privileged operation with a hypercall. The results were startling for the time — within a few percent of native, at a moment when trap-and-emulate was impossible and binary translation was proprietary. Amazon EC2 launched in 2006 on Xen, and ran on it for over a decade.

The catch was structural: you cannot paravirtualize a guest you cannot modify. Xen PV could run Linux, NetBSD and Solaris. It could not run Windows, because Microsoft was not going to port the NT kernel to Xen’s hypercall interface.

VT-x and AMD-V removed the reason for whole-kernel paravirtualization. With hardware assist, an unmodified guest runs in real ring 0 at near-native speed, so the enormous effort of porting a kernel bought you very little on the CPU side. Xen added HVM mode; the industry moved to hardware-assisted full virtualization; Amazon migrated EC2 from Xen PV to Xen HVM and then to the KVM-derived Nitro hypervisor.

But paravirtualization did not die — it retreated to exactly where it still wins: I/O. Hardware assist made privileged instructions cheap. It did nothing to make emulating an Intel e1000 network card cheap, and emulating a NIC is genuinely awful: for every packet, the guest driver writes several device registers, each write is an exit, and the hypervisor must reconstruct the intent from a sequence of register pokes designed for a chip that does not exist.

So the modern arrangement is a hybrid, and this is the sentence to give in an interview:

Modern virtualization is hardware-assisted full virtualization for CPU and memory, with paravirtualized drivers for I/O.

Those paravirtualized drivers are VirtIO — designed by Rusty Russell in 2007 and now the standard across KVM, and supported by Xen, VirtualBox, Hyper-V guests on Linux and cloud providers generally. A VirtIO device is not an imitation of any real chip. It is a documented, minimal contract: shared-memory virtqueues into which the guest places descriptors, and a single notification to say “there is work”. One exit can carry hundreds of packets or many disk requests.

The three techniques side by side#

6 · Understanding Hypervisors#

At the heart of virtualization lies the hypervisor: the software layer that enables the creation and management of virtual machines, allocates physical resources to them, and ensures isolation between them. Hypervisors are traditionally sorted into two types by what sits underneath them.

Type 1 — bare-metal hypervisors#

A Type 1 hypervisor runs directly on the host’s hardware. There is no general-purpose operating system beneath it; the hypervisor is the operating system, specialised for one job — running VMs. It boots from firmware, initialises the hardware itself, and everything else on the machine is a guest.

  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
  │  VM 1   │ │  VM 2   │ │  VM 3   │ │  VM 4   │ │  VM 5   │
  │ ┌─────┐ │ │ ┌─────┐ │ │ ┌─────┐ │ │ ┌─────┐ │ │ ┌─────┐ │
  │ │ app │ │ │ │ app │ │ │ │ app │ │ │ │ app │ │ │ │ app │ │
  │ ├─────┤ │ │ ├─────┤ │ │ ├─────┤ │ │ ├─────┤ │ │ ├─────┤ │
  │ │guest│ │ │ │guest│ │ │ │guest│ │ │ │guest│ │ │ │guest│ │
  │ │ OS  │ │ │ │ OS  │ │ │ │ OS  │ │ │ │ OS  │ │ │ │ OS  │ │
  │ └─────┘ │ │ └─────┘ │ │ └─────┘ │ │ └─────┘ │ │ └─────┘ │
  └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘
       │           │           │           │           │
  ┌────┴───────────┴───────────┴───────────┴───────────┴─────┐
  │              H Y P E R V I S O R                          │
  │   VMware ESXi · Microsoft Hyper-V · Xen · KVM/Proxmox     │
  │   Nutanix AHV · Citrix Hypervisor · AWS Nitro             │
  │                                                            │
  │   ← THERE IS NO HOST OS. This layer boots from firmware.   │
  │     It owns the hardware, the drivers and the scheduler.    │
  └───────────────────────────┬───────────────────────────────┘
  ┌───────────────────────────┴───────────────────────────────┐
  │   P H Y S I C A L   H A R D W A R E                        │
  │   CPU (VT-x/AMD-V) · RAM · NVMe · NIC · IOMMU             │
  └───────────────────────────────────────────────────────────┘

  Thin privileged layer → high efficiency, small attack surface.
  No desktop, no browser, no package manager, nothing to distract it.

Because they interact directly with the hardware, Type 1 hypervisors offer better performance and are considered more secure, due to the minimal software layer between the hardware and the VMs. They are what enterprise environments and data centres use, where performance, density and scalability are critical.

Concretely, what makes them enterprise-grade is not just speed:

• NUMA awareness — pinning a VM’s vCPUs and memory to the same CPU socket so memory access stays local.
• Live migration and high availability — moving running VMs, and restarting them elsewhere automatically when a host dies.
• Distributed resource scheduling — automatically rebalancing VMs across a cluster.
• Clustered storage — shared datastores (vSAN, Ceph, NFS, iSCSI) so any host can run any VM.
• A tiny footprint — ESXi installs in about 150 MB and can boot from an SD card or USB stick.

Type 2 — hosted hypervisors#

A Type 2 hypervisor runs on top of a host operating system, functioning as an application. You install it the way you install a browser. It asks the host OS for memory, for file access, for network access, and for CPU time, like any other process — with the addition of a kernel driver (vboxdrv, vmmon) that lets it enter VMX root mode.

  ┌─────────┐ ┌─────────┐                ┌──────────────────────┐
  │  VM 1   │ │  VM 2   │                │  ORDINARY HOST APPS  │
  │ ┌─────┐ │ │ ┌─────┐ │                │                      │
  │ │ app │ │ │ │ app │ │                │  Firefox             │
  │ ├─────┤ │ │ ├─────┤ │                │  VS Code             │
  │ │guest│ │ │ │guest│ │                │  Slack               │
  │ │ OS  │ │ │ │ OS  │ │                │  Spotify             │
  │ └─────┘ │ │ └─────┘ │                │                      │
  └────┬────┘ └────┬────┘                └──────────┬───────────┘
       │           │                                │
  ┌────┴───────────┴──────────────┐                 │
  │  H Y P E R V I S O R          │                 │
  │  as an ORDINARY APPLICATION   │  ← same         │
  │  VirtualBox · VMware           │    privilege    │
  │  Workstation/Fusion ·          │    level as     │
  │  Parallels Desktop · QEMU      │    Spotify      │
  │  (+ a kernel driver for VT-x)  │                 │
  └───────────────┬───────────────┘                 │
  ┌───────────────┴─────────────────────────────────┴─────────┐
  │  H O S T   O P E R A T I N G   S Y S T E M                 │
  │  Windows · macOS · Linux — with its own scheduler,          │
  │  its own memory manager, its own drivers, its own updates   │
  └───────────────────────────┬───────────────────────────────┘
  ┌───────────────────────────┴───────────────────────────────┐
  │  P H Y S I C A L   H A R D W A R E                         │
  └───────────────────────────────────────────────────────────┘

  Two schedulers stacked, two memory managers stacked.
  Convenient to install; extra latency; host OS can swap your VM out.

Type 2 hypervisors are generally easier to set up and suitable for desktop virtualization and development environments. Their weakness is structural: the extra layer of the host OS introduces additional latency, making them less suitable for performance-intensive applications.

The reason is worth understanding rather than memorising. There are now two schedulers in series. The guest kernel decides which of its processes should run, then the host kernel decides whether the hypervisor process should run at all. If you open forty browser tabs, the host may preempt your VM’s vCPU thread mid-timeslice — the guest experiences that as a stalled CPU, and its own scheduler has no idea why. The same doubling applies to memory: the host can page your VM’s “physical” RAM out to swap, so a guest memory access that should take 80 nanoseconds takes 8 milliseconds.

The trap: KVM is not cleanly either#

The Type 1 / Type 2 dichotomy is a teaching device from the 1970s. It survives because it is useful, but KVM breaks it, and asking about that is a favourite interview move.

KVM is a Linux kernel module (kvm.ko plus kvm_intel.ko or kvm_amd.ko). Loading it does not create a new operating system — it converts the running Linux kernel into a hypervisor. Afterwards, that Linux kernel is simultaneously:

• a Type 1 hypervisor: it runs on bare metal, it owns the hardware directly, guests execute in VMX non-root mode with no OS layer between them and the silicon, and every VM exit lands in kernel code
• a general-purpose operating system: still running sshd, bash, nginx, cron, apt and your text editor as ordinary processes, with all its usual drivers, filesystems and scheduler

  ┌──────────┐ ┌──────────┐        ┌────────────────────────┐
  │   VM 1   │ │   VM 2   │        │ nginx · sshd · bash    │
  │ guest OS │ │ guest OS │        │ cron · your editor     │
  └────┬─────┘ └────┬─────┘        └───────────┬────────────┘
       │            │                          │
  ┌────┴────┐  ┌────┴────┐                     │  ordinary
  │ qemu-   │  │ qemu-   │  ← userspace         │  syscalls
  │ system  │  │ system  │    processes!        │
  │ (device │  │ (device │    ioctl(/dev/kvm)   │
  │  model) │  │  model) │                      │
  └────┬────┘  └────┬────┘                      │
  ┌────┴────────────┴──────────────────────────┴────────────┐
  │  L I N U X   K E R N E L                                 │
  │  ┌───────────────────┐                                   │
  │  │ kvm.ko            │  ← the hypervisor lives INSIDE    │
  │  │ kvm_intel.ko      │    a general-purpose kernel        │
  │  └───────────────────┘                                   │
  │  scheduler · MM · VFS · ext4 · TCP/IP · drivers          │
  └────────────────────────┬────────────────────────────────┘
  ┌────────────────────────┴────────────────────────────────┐
  │  P H Y S I C A L   H A R D W A R E                       │
  └─────────────────────────────────────────────────────────┘

  Type 1?  Yes — no OS between guest and hardware; exits land in ring 0.
  Type 2?  Yes — it is a module inside an OS that also runs normal apps.
  Correct answer: "Type 1 with a Type 2 heritage — the taxonomy predates it."

The same fuzziness catches other products, and knowing this is a genuine differentiator:

Type 1 versus Type 2#

7 · Networking Methods#

Networking in virtualization is pivotal: it is what lets VMs talk to each other, to the host, and to the outside world. The hypervisor gives the guest a virtual NIC and then attaches that NIC to a virtual switch whose wiring you choose. Different wiring gives you very different degrees of connectivity and isolation — and picking wrongly is the single most common cause of “my VM has no internet” and “I can’t SSH into my VM”.

Four modes matter. Learn them as a spectrum from connected to sealed.

Network Address Translation (NAT)#

NAT allows VMs to access external networks using the host’s IP address. The host acts as a middleman, translating requests from the VM to the outside world and vice versa. It is simple to set up — it is the default in VirtualBox, VMware and libvirt — and it provides a layer of security by hiding the VM’s IP address from external networks.

                                                ┌───────────────┐
  ┌───────────────────────────────────────┐     │   INTERNET    │
  │  HOST      LAN address 192.168.1.10   │     │               │
  │                                        │     └───────▲───────┘
  │   ┌────────────────────────────────┐  │             │
  │   │  NAT engine inside the         │  │   packets leave with
  │   │  hypervisor:                   │  │   SOURCE = 192.168.1.10
  │   │   · gateway     10.0.2.2       │──┼─────────────┘
  │   │   · DNS proxy   10.0.2.3       │  │   (the host's address —
  │   │   · DHCP server 10.0.2.x       │  │    the world never sees
  │   │   · rewrites src IP + port     │  │    10.0.2.15)
  │   └───────────────▲────────────────┘  │
  │                   │                    │
  │        ┌──────────┴──────────┐        │
  │        │  VM   10.0.2.15     │        │
  │        │  gw    10.0.2.2     │        │
  │        └─────────────────────┘        │
  └───────────────────────────────────────┘

  OUTBOUND  ✔ apt update, git clone, curl — always works, zero config
  INBOUND   ✘ nothing on the LAN can initiate a connection to the VM
              …unless you add a PORT FORWARD on the host
  VM ↔ HOST ✔ (reach the host at 10.0.2.2)
  VM ↔ VM   ⚠ each VM gets its OWN private NAT — they cannot see each
              other. VirtualBox's separate "NAT Network" mode fixes this

Exactly as with your home router, NAT complicates incoming connections, because port forwarding must be configured to allow external access. That is the price of the isolation, and section 15 sets up the one port forward you will actually use: host port 2222 to guest port 22, so you can SSH in.

Bridged networking#

Bridged networking connects the VM directly to the host’s physical network. The virtual NIC is joined to a software bridge that also contains the host’s physical interface, so guest frames go out on the wire with the guest’s own MAC address. The VM obtains its own IP address from the network’s DHCP server, or via static configuration, making it appear as a separate physical device on the network.

     ┌──────────── LAN / switch — 192.168.1.0/24 ─────────────┐
     │            │                │                  │
  ┌──┴────┐  ┌────┴─────┐   ┌──────┴───────┐   ┌──────┴───────┐
  │ROUTER │  │  HOST    │   │ VM (bridged) │   │  Colleague's │
  │  .1   │  │  .10     │   │     .21      │   │  laptop  .30 │
  │ DHCP  │  │ 6a:1f:.. │   │  08:00:27:.. │   │  b4:2e:99:.. │
  └───────┘  └────┬─────┘   └──────▲───────┘   └──────────────┘
                  │                 │
                  └── software bridge inside the host ──┘
                      (host NIC in promiscuous mode)

  The VM has its OWN MAC and its OWN DHCP lease from the router.
  VM ↔ HOST      ✔    VM ↔ VM        ✔
  VM ↔ INTERNET  ✔    LAN → VM       ✔  ssh 192.168.1.21 just works

This method allows full network functionality but exposes the VM to the same security risks as any other network device. That sentence deserves emphasis: a bridged VM is on your office or home network with no firewall between it and everything else. If you bridge a deliberately vulnerable practice VM, you have put a vulnerable machine on a network you do not control. Bridge for realism; do not bridge for malware analysis.

Host-only networking#

Host-only networking creates a private network between the host and the VMs. The hypervisor creates a virtual interface on the host (vboxnet0, vmnet1, virbr1) and a virtual switch that only VMs and the host are attached to. The VMs cannot access external networks, nor can external devices access the VMs. It is ideal for testing and development environments where internet access is unnecessary.

                    ┌───────────────┐
                    │   INTERNET    │        ✘ no route out
                    └───────▲───────┘
                            │  ╳ blocked
  ┌─────────────────────────┴───────────────────────────────┐
  │  HOST                                                    │
  │    eth0     192.168.1.10   ──── to the real LAN          │
  │                                                           │
  │    vboxnet0 192.168.56.1   ──┐  a virtual interface       │
  └──────────────────────────────┼───────────────────────────┘
                                 │   virtual switch
           ┌─────────────────────┼─────────────────────┐
           │                     │                     │
   ┌───────┴────────┐   ┌────────┴───────┐   ┌─────────┴──────┐
   │ VM 1           │   │ VM 2           │   │ VM 3           │
   │ 192.168.56.101 │   │ 192.168.56.102 │   │ 192.168.56.103 │
   └────────────────┘   └────────────────┘   └────────────────┘

  VM ↔ HOST      ✔  ssh 192.168.56.101 straight from the host
  VM ↔ VM        ✔  build a cluster
  VM ↔ INTERNET  ✘  no apt update
  LAN → VM       ✘  invisible to everything else

This configuration enhances security by isolating VMs, but limits connectivity.

Internal networking#

Internal networking allows VMs to communicate with each other on a private network, but not with the host or external networks. It is the strictest mode: a virtual switch that the host itself has no interface on. Useful when you need VMs to interact without exposing them externally at all.

  ┌───────────────────────────────────────────────────────────┐
  │  HOST — has NO interface on this switch.                   │
  │         Cannot ping the VMs. Cannot be pinged by them.     │
  └───────────────────────────────────────────────────────────┘

      ┌──────── virtual switch  "intnet-lab"  ────────┐
      │                    │                    │
 ┌────┴──────┐      ┌──────┴─────┐       ┌──────┴──────┐
 │ VM  app   │      │ VM  db     │       │ VM  target  │
 │ 10.10.0.2 │      │ 10.10.0.3  │       │ 10.10.0.4   │
 └───────────┘      └────────────┘       └─────────────┘

  VM ↔ VM  ✔ everything else ✘
  No DHCP server exists on this switch unless you run one yourself,
  so configure static addresses (or put a DHCP server in one VM).

  Use for: malware analysis · penetration-testing ranges ·
           air-gapped multi-tier application labs ·
           reproducing a network fault with zero risk

Comparison of networking methods#

This is the source table, corrected in one place and expanded with the rows that matter in practice.

flowchart TD
  Q1{"Does the VM need<br/>internet access?"}
  Q1 -->|"No"| Q2{"Does the host need<br/>to reach the VM?"}
  Q2 -->|"Yes"| HO["HOST-ONLY"]
  Q2 -->|"No"| IN["INTERNAL"]
  Q1 -->|"Yes"| Q3{"Must other machines<br/>on the LAN reach<br/>the VM directly?"}
  Q3 -->|"No"| NAT["NAT<br/>+ port forward if you<br/>need SSH from the host"]
  Q3 -->|"Yes"| Q4{"Wired Ethernet?"}
  Q4 -->|"Yes"| BR["BRIDGED"]
  Q4 -->|"No, Wi-Fi"| NAT2["NAT + port forwarding<br/>bridging is unreliable<br/>over 802.11"]

8 · Creating Virtual Machines#

Setting up a VM involves five steps. Each matters, and each has a decision inside it that beginners get wrong.

I. Install a hypervisor#

Choose and install a hypervisor compatible with your host operating system and hardware. For beginners, Type 2 hypervisors like VirtualBox or VMware Workstation are user-friendly and widely supported.

Before you install anything, run the checks in section 5. Hardware virtualization must be enabled.

II. Create a new VM#

Use the hypervisor’s interface to create a new VM. You typically provide a name, select the guest operating system type, and choose the version.

That OS-type selection is not cosmetic. It sets defaults for the whole machine: chipset, firmware (BIOS or UEFI), default disk controller, default NIC model, whether the RTC runs in UTC or local time, pointing-device type, and how much RAM and disk to suggest. Choosing “Other/Unknown” when you meant “Ubuntu (64-bit)” produces a VM that boots to a black screen or fails to see its disk, and the error message will not tell you why.

III. Allocate resources#

Assign hardware to the VM:

• CPU — decide the number of processor cores.
• Memory — allocate RAM based on the guest OS requirements.
• Storage — create or assign a virtual hard disk.
• Network — choose a networking mode (NAT, bridged, host-only, internal).

It is important to balance the resources to avoid overloading the host system. Concrete rules for a laptop:

IV. Install the guest operating system#

Mount the installation media — an ISO file or physical disk — and boot the VM. Follow the standard installation process of the chosen operating system. Practical notes the PDF omits:

• Download the ISO from the vendor and verify its checksum: sha256sum ubuntu-24.04.1-live-server-amd64.iso compared against the published SHA256SUMS. A corrupted ISO produces installer failures that look like hardware problems.
• Choose Ubuntu Server, not Desktop, for Linux practice. No GUI means less RAM, faster boot, and you learn the shell rather than avoiding it.
• Install and enable OpenSSH during setup. On the Ubuntu Server installer this is one checkbox, and skipping it means your first task is enabling SSH from a console you find awkward.
• Keep the username and password simple and memorable. This machine is disposable.

V. Install hypervisor tools#

After installing the OS, install the hypervisor’s guest additions or tools. These enhance performance and enable features like shared folders, clipboard sharing and better graphics support.

What they actually install, so you know what you lose without them:

# VirtualBox on an Ubuntu guest — the reliable route
sudo apt update
sudo apt install -y build-essential dkms linux-headers-$(uname -r)
# then: Devices → Insert Guest Additions CD image…
sudo mount /dev/cdrom /mnt
sudo /mnt/VBoxLinuxAdditions.run
sudo reboot

# Or, from the distro packages — simpler, slightly older
sudo apt install -y virtualbox-guest-utils          # headless server
sudo apt install -y virtualbox-guest-x11            # graphical desktop

# VMware
sudo apt install -y open-vm-tools                   # server
sudo apt install -y open-vm-tools-desktop           # desktop

# KVM — VirtIO drivers are already in the Linux kernel; add the agent
sudo apt install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

flowchart LR
  A["1 · Install<br/>hypervisor"] --> B["2 · Create VM<br/>name, OS type,<br/>version"]
  B --> C["3 · Allocate<br/>CPU · RAM ·<br/>disk · network"]
  C --> D["4 · Mount ISO,<br/>install guest OS"]
  D --> E["5 · Install guest<br/>additions / tools"]
  E --> F["6 · SNAPSHOT<br/>immediately"]
  F --> G["Now break it<br/>freely"]

Step 6 is not in the source notes and is the most valuable step in the list. Section 15 explains why.

9 · Managing Virtual Machines#

Effective VM management ensures optimal performance and resource utilisation. Six activities cover it.

Lifecycle management#

You can start, pause, resume and stop VMs as needed. Pausing a VM saves its state, allowing you to resume later without rebooting.

The distinction between the several ways to “stop” a VM is a real interview question and a real production hazard.

stateDiagram-v2
    [*] --> PoweredOff: defined
    PoweredOff --> Running: start
    Running --> Paused: pause / freeze vCPUs
    Paused --> Running: resume
    Running --> Saved: save state / suspend to disk
    Saved --> Running: restore
    Running --> PoweredOff: graceful shutdown (ACPI)
    Running --> PoweredOff: FORCED power off
    Running --> Running: reboot / reset
    PoweredOff --> [*]: delete / undefine

Snapshots#

Snapshots capture the VM’s state at a specific point in time. They are invaluable for testing or before making significant changes. Before installing new software, take a snapshot; if something goes wrong, revert to the previous state.

How they work is the part that predicts every problem they cause:

  BEFORE the snapshot
  ┌────────────────────────────┐
  │  MyVM.vdi   (read-write)   │ ← guest writes go here
  └────────────────────────────┘

  AFTER taking snapshot "clean-install"
  ┌────────────────────────────┐
  │  MyVM.vdi   READ-ONLY      │ ← frozen. This IS the snapshot.
  │  (the base image)          │
  └─────────────┬──────────────┘
                │ backing / parent
  ┌─────────────┴──────────────┐
  │  {uuid}.vdi  (read-write)  │ ← every NEW write goes here.
  │  DELTA / differencing disk │   Reads check here first, then
  │  starts at 0 bytes         │   fall back to the parent.
  └────────────────────────────┘

  AFTER two more snapshots — a CHAIN
  base (RO) ──► delta1 (RO) ──► delta2 (RO) ──► delta3 (RW, live)

  Reading one block may now traverse FOUR files.
  Reverting = throw away the deltas after the chosen point.

Consequences of that design, all of which you will meet:

• Taking a snapshot is instant. Nothing is copied; a file is frozen and a new empty one created.
• Deleting a snapshot is slow. The delta must be merged back into its parent — gigabytes of I/O.
• Reverting discards everything after it, silently and permanently.
• Performance degrades with chain depth. Each read may walk the chain. Four or five snapshots deep, a VM feels noticeably sluggish; twenty deep, it is unusable.
• Deltas grow without limit. A snapshot on a busy database VM can grow larger than the original disk within days, because every changed block is stored again.

Cloning#

Cloning creates an exact copy of a VM. It is useful when deploying multiple VMs with the same configuration. Two kinds, and picking wrongly costs either disk space or reliability:

Whichever you choose, regenerate the machine’s identity afterwards — the step that catches everyone:

# VirtualBox: --mode machine copies the current state only,
# and this flag is essential
VBoxManage clonevm "MyVM" --name "MyVM-clone" --register \
  --mode machine --options keepdisknames=off

# The MAC address must be new, or two clones fight on the network.
# VirtualBox reassigns MACs on clone by default; verify:
VBoxManage showvminfo "MyVM-clone" | grep -i "NIC 1"

# Inside a cloned Linux guest, reset the things that must be unique
sudo hostnamectl set-hostname web-02
sudo truncate -s0 /etc/machine-id && sudo systemd-machine-id-setup
sudo rm -f /etc/ssh/ssh_host_*                # then:
sudo dpkg-reconfigure openssh-server          # regenerate host keys
sudo rm -f /etc/netplan/*.yaml.orig

Migration#

VMs can be moved between hosts, sometimes even while running — live migration. This facilitates load balancing and hardware maintenance without downtime.

Live migration is worth understanding because it sounds impossible:

1. The destination host creates an empty VM shell with the same configuration.
2. The hypervisor copies the source VM’s memory pages to the destination while the VM keeps running.
3. Pages the guest modifies during the copy are marked dirty and re-sent. This repeats, converging as the dirty rate falls below the transfer rate.
4. When the remaining dirty set is small enough to move in a few milliseconds, the source VM is paused, the final pages, CPU registers and device state are transferred, and the destination is resumed.
5. An ARP/RARP announcement tells the network switch that this MAC address now lives on a different port. Existing TCP connections survive; a ping may lose one packet.

# KVM/libvirt live migration, one command
virsh migrate --live --persistent --undefinesource \
  MyVM qemu+ssh://host02.example.com/system

Configuration adjustments#

You can modify VM settings after creation:

• Increase memory — allocate more RAM if the guest OS needs it.
• Add storage — expand the virtual disk or add new ones.
• Change networking — switch between NAT, bridged or host-only.

Almost all of these require the VM to be powered off in a Type 2 hypervisor. Enterprise hypervisors support hot-add of CPU and memory (and hot-plug of disks and NICs) if the guest OS supports it — Linux does, and Windows Server does for memory. Note that hot-removal is much harder than hot-add and is often unsupported.

Growing a disk is a two-stage operation and forgetting the second stage is a classic:

# 1. Grow the virtual disk (hypervisor side)
VBoxManage modifymedium disk "/home/user/VirtualBox VMs/MyVM/MyVM.vdi" --resize 40000
# or:  qemu-img resize /var/lib/libvirt/images/MyVM.qcow2 +20G

# 2. THEN, inside the guest, grow the partition and the filesystem
sudo growpart /dev/sda 3          # extend partition 3 to fill the disk
sudo resize2fs /dev/sda3          # ext4 — or: xfs_growfs /
# with LVM (the Ubuntu Server default):
sudo pvresize /dev/sda3
sudo lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv
sudo resize2fs /dev/ubuntu-vg/ubuntu-lv

Monitoring#

Regularly monitor VM performance using tools provided by the hypervisor. Keep an eye on CPU usage, memory consumption and disk I/O to detect and resolve bottlenecks.

The crucial idea, and the one the source notes miss: the guest’s own metrics lie about the host. A guest at “100% CPU” may be waiting on a contended physical core; a guest reporting plenty of free RAM may be having its pages swapped out by the host. You must look from both sides.

Resource overcommitment: the thing that actually bites#

You may allocate more resources to VMs than the host physically has. That is not a bug — it is the point, because most VMs are idle most of the time. It is also how you turn a working host into a stalled one.

Memory overcommit and ballooning. The hypervisor promises 4 GB each to ten VMs on a 32 GB host. As long as the guests do not all use their full allocation, everyone is fine. When they do, the hypervisor must reclaim, and it has four increasingly unpleasant tools:

1. Ballooning — a virtio-balloon (or VMware Tools balloon) driver inside the guest is asked to inflate: it allocates guest memory and pins it, which makes the guest’s own kernel evict its caches and page out its least-used data through its own, well-informed reclaim logic. The hypervisor then takes those physical pages back. This is the good option, because the guest chooses what to give up.
2. Page sharing / KSM — Kernel Samepage Merging scans for identical pages across VMs and collapses them to one copy-on-write page. Excellent when you run twenty identical Ubuntu VMs; costs CPU to scan, and has known side-channel implications, so it is off by default in some distributions.
3. Compression — compress cold guest pages rather than writing them out.
4. Host swapping — the hypervisor pages guest “physical” memory to host disk. This is the disaster case. The guest has no idea; from inside, memory access latency jumps from ~80 ns to milliseconds at random, and the guest’s own tuning is worthless because it cannot see what is happening. A guest whose memory is being host-swapped looks inexplicably slow.

# Balloon a running KVM guest down from 4 GB to 2 GB
virsh setmem MyVM 2G --live
virsh dommemstat MyVM

$ virsh dommemstat MyVM
actual 2097152
swap_in 0
swap_out 0
major_fault 137
minor_fault 8842119
unused 421356
available 2043912
usable 1201884
last_update 1753951200
rss 2103296

CPU overcommit and steal time. vCPUs are time-shared, so 60 vCPUs on 16 threads is legal. What the guest experiences when it loses that race is steal time: the %st column in top.

$ top
top - 14:22:09 up 12 days,  4:31,  1 user,  load average: 3.91, 3.72, 3.55
Tasks: 148 total,   4 running, 144 sleeping,   0 stopped,   0 zombie
%Cpu(s): 41.2 us,  6.1 sy,  0.0 ni, 24.7 id,  0.3 wa,  0.0 hi,  0.4 si, 27.3 st
MiB Mem :   3908.4 total,    204.1 free,   2611.7 used,   1092.6 buff/cache

27.3 st means 27% of the time this vCPU was ready to run and the hypervisor gave the physical CPU to somebody else. Read that carefully, because of what it implies:

• It is not your workload’s fault, and no amount of application tuning will fix it.
• It is measured from inside the guest, and it is your only window onto host contention.
• The other 27% of your CPU is being spent by someone else — a noisy neighbour on a shared cloud host, or your own overcommitted hypervisor.

Nested virtualization#

Nested virtualization is running a hypervisor inside a VM — an L1 guest that is itself a hypervisor for L2 guests. It requires the physical CPU’s virtualization extensions to be exposed to the L1 guest, which the hypervisor must explicitly permit.

  ┌──────────────────────────────────────────────────────────┐
  │  L0 — PHYSICAL HOST                    KVM / ESXi / Nitro│
  │  ┌────────────────────────────────────────────────────┐  │
  │  │  L1 — GUEST that is ALSO a hypervisor              │  │
  │  │       sees vmx/svm in /proc/cpuinfo                │  │
  │  │  ┌──────────────┐  ┌──────────────┐                │  │
  │  │  │ L2 guest     │  │ L2 guest     │                │  │
  │  │  │ (a container │  │ (a Windows   │                │  │
  │  │  │  runtime VM, │  │  test VM)    │                │  │
  │  │  │  a K8s node) │  │              │                │  │
  │  │  └──────────────┘  └──────────────┘                │  │
  │  └────────────────────────────────────────────────────┘  │
  └──────────────────────────────────────────────────────────┘
   Every L2 VM exit must be handled by L1, which itself exits
   to L0. Overhead compounds — expect a real performance cost.

When you actually need it:

• Running Docker Desktop, WSL2 or Hyper-V inside a Windows VM — because those are hypervisors.
• CI runners in the cloud that build or test VM images, run minikube/kind with a VM driver, or execute integration tests against a full VM.
• Kubernetes on VMs where the container runtime uses microVMs — Kata Containers, Firecracker, gVisor with a VM platform.
• Training and demonstration environments — teaching virtualization on cloud instances.
• Android emulators inside a cloud VM, which need KVM to be usable.

How to enable it:

# --- Check whether you have it (inside the L1 guest) ---
grep -Eoc '(vmx|svm)' /proc/cpuinfo      # non-zero → nesting is exposed
systemd-detect-virt                       # confirms you are in a VM at all

# --- KVM host (L0): expose VMX/SVM to guests ---
cat /sys/module/kvm_intel/parameters/nested        # want: Y or 1
echo "options kvm_intel nested=1" | sudo tee /etc/modprobe.d/kvm-nested.conf
# AMD: options kvm_amd nested=1
sudo modprobe -r kvm_intel && sudo modprobe kvm_intel

# Then give the guest a host-passthrough CPU model:
virsh edit MyVM      #   <cpu mode='host-passthrough'/>
# or with virt-install:  --cpu host-passthrough

# --- VirtualBox (6.1+) ---
VBoxManage modifyvm "MyVM" --nested-hw-virt on

# --- VMware Workstation/Fusion ---
# VM settings → Processors → "Virtualize Intel VT-x/EPT or AMD-V/RVI"
# or add to the .vmx:   vhv.enable = "TRUE"

10 · Benefits — and the Honest Costs#

Virtual machines offer numerous advantages that make them indispensable in modern computing. The five the syllabus names, each with the mechanism that delivers it and the caveat nobody mentions.

Isolation#

Each VM operates independently, ensuring that crashes or security breaches in one VM do not affect others or the host system. The mechanism is real and hardware-enforced: separate kernels, separate address spaces enforced by EPT/NPT, and separate virtual device sets. A kernel panic in one VM is a panic in that VM; the host and its siblings never notice.

Caveat: isolation of fault, not of resource. A VM that saturates the shared disk or the shared NIC degrades every VM on the host — the “noisy neighbour” problem. And isolation is not absolute; see section 17.

Efficient resource utilisation#

By running multiple VMs on a single physical machine, you maximise hardware usage, reducing costs and energy consumption. A host at 70% utilisation instead of 8% is roughly nine machines’ worth of work done by one. Overcommit (section 9) pushes this further by exploiting the fact that VMs are idle at different times.

Caveat: virtualization has real overhead — roughly 2–5% CPU, more on I/O-heavy workloads — and each guest duplicates a full OS in memory. You trade a little efficiency per workload for an enormous gain in aggregate.

Flexibility#

VMs can be easily cloned, moved or backed up. This flexibility simplifies testing, development and disaster recovery processes. This is the decoupling win: because a VM is a file plus a description, everything you can do to a file you can do to a server.

Caveat: flexibility invites sprawl. “VM sprawl” — hundreds of forgotten VMs consuming licences, patching effort and storage — is a genuine operational disease. Tag everything with an owner and an expiry date.

Scalability#

Adding more VMs to handle increased workloads is straightforward. Cloud providers leverage this to offer scalable services. Autoscaling groups, managed instance groups and Kubernetes node pools are all “create more VMs from a template, automatically”.

Caveat: scalability is bounded by the host. Vertical scaling stops at one machine’s capacity, and horizontal scaling requires the application to tolerate it.

Legacy software support#

Run outdated or unsupported software within a VM without affecting the host system. This is one of virtualization’s most valuable and least glamorous uses: the 2004 manufacturing control application whose vendor no longer exists, running on Windows Server 2003 in a VM, on hardware that did not exist when it was written, because the virtual hardware it sees has not changed.

Caveat: an unpatched legacy OS is an unpatched legacy OS. Put it on an isolated network segment (host-only or internal, or a locked-down VLAN) and treat it as compromised by default.

The benefits the syllabus omits#

And the honest costs#

An answer that lists only benefits is a weak answer. Name the trade-offs:

11 · Practical Demonstration — VirtualBox#

VirtualBox is a free and open-source hypervisor developed by Oracle. It is popular for its ease of use and cross-platform support — the same product, the same interface and the same command line on Windows, macOS, Linux and Solaris hosts. It is the right first hypervisor, and it is what section 15 builds your lab with.

Features of VirtualBox#

• Supports multiple guest OSes — run Windows, Linux, macOS, BSD, Solaris and others.
• Snapshots and cloning — easily save VM states and duplicate VMs.
• Shared folders and clipboard — seamless file sharing between host and VM.
• Extensive networking options — configure NAT, NAT Network, bridged, host-only and internal networks.

Worth adding to that list:

• VBoxManage — a complete command-line interface. Every GUI action has a CLI equivalent, which means every VM operation is scriptable.
• Headless mode — run VMs with no window at all, controlled entirely over SSH or RDP.
• Guest Additions — the driver and integration package described in section 8.
• Seamless mode and shared clipboard — genuine desktop integration.
• The Extension Pack — USB 2.0/3.0 passthrough, VirtualBox RDP, disk encryption, PXE boot. Note the licensing: the base product is GPLv2, but the Extension Pack is under the PUEL licence and is free only for personal, educational and evaluation use. Commercial use requires a licence. This has caught real companies out in audits.

Networking in VirtualBox#

VirtualBox offers flexible networking options to suit different needs, and by default VMs use NAT networking, allowing internet access through the host.

Setting up port forwarding (GUI)#

To reach a service inside a NAT’d VM from the host, add a port-forwarding rule:

1. Open VM settings.
2. Go to the Network section.
3. Click on Advanced.
4. Set up port forwarding rules.

In the Port Forwarding dialog, each rule has five fields, and understanding them removes all the guesswork:

Everything in that dialog is one CLI command, which is why professionals never open it:

VBoxManage modifyvm "MyVM" --natpf1 "ssh,tcp,127.0.0.1,2222,,22"
#                            ^      ^   ^         ^    ^ ^
#                            name  proto hostIP  hostPort│guestPort
#                                                    guestIP (blank)

# List the rules currently configured
VBoxManage showvminfo "MyVM" --machinereadable | grep -i natpf

# Remove a rule
VBoxManage modifyvm "MyVM" --natpf1 delete "ssh"

$ VBoxManage showvminfo "MyVM" --machinereadable | grep -i natpf
Forwarding(0)="ssh,tcp,127.0.0.1,2222,,22"

--natpf1 refers to adapter 1. For a rule on adapter 2 use --natpf2, and so on.

Retrieving the VM’s IP address#

You can obtain the IP address of a running VM using:

VBoxManage guestproperty get "MyVM" "/VirtualBox/GuestInfo/Net/0/V4/IP"

$ VBoxManage guestproperty get "MyVM" "/VirtualBox/GuestInfo/Net/0/V4/IP"
Value: 10.0.2.15

The VM named “MyVM” has an IP address of 10.0.2.15 on its first network interface. Reading that output precisely:

# See every guest property the additions report — genuinely useful
VBoxManage guestproperty enumerate "MyVM"

$ VBoxManage guestproperty enumerate "MyVM"
Name: /VirtualBox/GuestInfo/OS/Product, value: Linux, timestamp: 1753949100123, flags:
Name: /VirtualBox/GuestInfo/OS/Release, value: 6.8.0-45-generic, timestamp: ...
Name: /VirtualBox/GuestInfo/Net/Count, value: 2, timestamp: ...
Name: /VirtualBox/GuestInfo/Net/0/V4/IP, value: 10.0.2.15, timestamp: ...
Name: /VirtualBox/GuestInfo/Net/1/V4/IP, value: 192.168.56.101, timestamp: ...
Name: /VirtualBox/GuestInfo/Net/0/MAC, value: 080027A1B2C3, timestamp: ...
Name: /VirtualBox/GuestInfo/OS/LoggedInUsers, value: 1, timestamp: ...

That output shows the two-adapter lab pattern from section 7: Net/0 is the NAT address, Net/1 is the host-only address you would actually SSH to.

Changing networking mode#

Set adapter 1 to bridged networking:

VBoxManage modifyvm "MyVM" --nic1 bridged --bridgeadapter1 eth0

The first network adapter of “MyVM” is now bridged to the host’s eth0 interface. No output is produced; silence means success.

Every value --nicN accepts:

# What host interfaces can I bridge to? Do not guess the name.
VBoxManage list bridgedifs | grep -E '^(Name|Status)'

$ VBoxManage list bridgedifs | grep -E '^(Name|Status)'
Name:            enp0s31f6
Status:          Up
Name:            wlp2s0
Status:          Up

Managing VMs with VBoxManage#

VBoxManage is the command-line utility for controlling VirtualBox. Everything the GUI does, it does — and it is how you script, automate and work over SSH. What follows builds a complete VM from nothing, in the order you would actually run it.

Creating a VM#

VBoxManage createvm --name "MyVM" --register

$ VBoxManage createvm --name "MyVM" --register
Virtual machine 'MyVM' is created and registered.
UUID: 12345678-1234-1234-1234-123456789abc
Settings file: '/home/user/VirtualBox VMs/MyVM/MyVM.vbox'

A VM named “MyVM” is created and ready for configuration. Three things in that output, each worth understanding:

# Better practice: declare the OS type at creation so sane defaults apply
VBoxManage list ostypes | grep -i ubuntu
VBoxManage createvm --name "MyVM" --ostype "Ubuntu_64" --register

Modifying VM settings#

Allocate memory and CPUs:

VBoxManage modifyvm "MyVM" --memory 2048 --cpus 2

No output is returned for this command, indicating success. That silence is the Unix convention — Chapter 1’s “no news is good news”. A failure would print a VBoxManage: error: line and set a non-zero exit code, which is what you check in scripts (if ! VBoxManage modifyvm …; then).

--memory is in megabytes, so 2048 is 2 GB. Writing --memory 2 gives the VM 2 MB and it will not boot.

The options you will actually use:

Attaching storage#

Create a virtual hard disk:

VBoxManage createhd --filename "/home/user/VirtualBox VMs/MyVM/MyVM.vdi" --size 20000

$ VBoxManage createhd --filename "/home/user/VirtualBox VMs/MyVM/MyVM.vdi" --size 20000
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Medium created. UUID: 8f3c1a72-4b9e-4d2a-9f1c-6a7b8c9d0e1f
Disk image created: /home/user/VirtualBox VMs/MyVM/MyVM.vdi

A 20 GB virtual disk is created for “MyVM”. Two details:

• --size is in megabytes, so 20000 is roughly 20 GB (19.5 GiB, precisely). A common error is --size 20, producing a 20 MB disk.
• The 0%...100% progress indicator appears even for a dynamically allocated disk, because VirtualBox still writes the image header and metadata. For a fixed-size disk (--variant Fixed) that progress bar is the full 20 GB being zeroed, and it takes minutes.

# Explicitly dynamic (the default) — grows on demand
VBoxManage createhd --filename ".../MyVM.vdi" --size 20000 --variant Standard

# In VirtualBox 6+ the modern spelling is `createmedium`;
# `createhd` remains as a supported alias
VBoxManage createmedium disk --filename ".../MyVM.vdi" --size 20000

Attach the disk to the VM. This is two steps, because a disk needs a controller to plug into:

VBoxManage storagectl "MyVM" --name "SATA Controller" --add sata --controller IntelAHCI

VBoxManage storageattach "MyVM" --storagectl "SATA Controller" \
  --port 0 --device 0 --type hdd \
  --medium "/home/user/VirtualBox VMs/MyVM/MyVM.vdi"

Attach an ISO for OS installation:

VBoxManage storageattach "MyVM" --storagectl "SATA Controller" \
  --port 1 --device 0 --type dvddrive \
  --medium "/path/to/os.iso"

Note the two differences from the disk attachment: --type dvddrive instead of hdd, and --port 1 so it does not collide with the hard disk on port 0.

# Eject the ISO after installation — otherwise the VM may boot the
# installer again instead of the system you just installed
VBoxManage storageattach "MyVM" --storagectl "SATA Controller" \
  --port 1 --device 0 --type dvddrive --medium none

Starting the VM#

VBoxManage startvm "MyVM" --type headless

$ VBoxManage startvm "MyVM" --type headless
Waiting for VM "MyVM" to power on...
VM "MyVM" has been successfully started.

“MyVM” is now running in headless mode (without a GUI). Reading it:

• Waiting for VM "MyVM" to power on... — VBoxManage has asked the VirtualBox service to start the VM and is blocking until the VM process reports that it reached the running state. If this line appears and then hangs or errors, the failure is in the VM’s configuration or in VT-x availability, not in your command.
• has been successfully started — the VM process is alive and the virtual power button has been pressed. It does not mean the guest OS has booted; that takes another 10–30 seconds.

--type values:

# Headless does not mean inaccessible. Options for getting in:
ssh -p 2222 user@127.0.0.1                    # via the NAT port forward
VBoxManage controlvm "MyVM" screenshotpng /tmp/vm.png   # look at the console
VBoxManage startvm "MyVM" --type separate     # attach a window later

Stopping the VM#

VBoxManage controlvm "MyVM" acpipowerbutton

No output is returned; the VM will begin a graceful shutdown. What actually happens: VirtualBox signals an ACPI power-button event to the guest’s virtual chipset. The guest’s ACPI daemon (systemd-logind on modern Linux, or the Windows power service) receives it and runs a normal shutdown — unmounting filesystems, stopping services, flushing buffers. It is the software equivalent of briefly pressing the power button on a physical machine.

This only works if the guest is listening for ACPI events. A guest with no ACPI support, sitting at a BIOS prompt, or hung in a kernel panic will ignore it entirely and stay running.

Every controlvm power state, in order of increasing violence:

# Wait for a graceful shutdown to complete, with a fallback.
# The pattern to use in scripts.
VBoxManage controlvm "MyVM" acpipowerbutton
for i in $(seq 1 30); do
  VBoxManage showvminfo "MyVM" --machinereadable | grep -q 'VMState="poweroff"' && break
  sleep 2
done
VBoxManage showvminfo "MyVM" --machinereadable | grep -q 'VMState="poweroff"' \
  || VBoxManage controlvm "MyVM" poweroff

The rest of the VBoxManage vocabulary#

# --- Inventory ---
VBoxManage list vms                      # all registered VMs, with UUIDs
VBoxManage list runningvms               # only the running ones
VBoxManage list ostypes                  # valid --ostype identifiers
VBoxManage list hostonlyifs              # host-only networks available
VBoxManage list bridgedifs               # host interfaces you can bridge to
VBoxManage showvminfo "MyVM"             # everything about one VM
VBoxManage showvminfo "MyVM" --machinereadable   # the same, greppable

# --- Snapshots ---
VBoxManage snapshot "MyVM" take "clean-install" --description "OS + additions, nothing else"
VBoxManage snapshot "MyVM" list --details
VBoxManage snapshot "MyVM" restore "clean-install"     # VM must be powered off
VBoxManage snapshot "MyVM" restorecurrent
VBoxManage snapshot "MyVM" delete "clean-install"      # merges the delta back

# --- Cloning ---
VBoxManage clonevm "MyVM" --name "MyVM-clone" --register --mode machine

# --- Import / export (OVF/OVA — the portable, cross-hypervisor format) ---
VBoxManage export "MyVM" -o MyVM.ova
VBoxManage import MyVM.ova

# --- Disks ---
VBoxManage list hdds
VBoxManage modifymedium disk "/path/MyVM.vdi" --resize 40000      # in MB
VBoxManage closemedium disk "/path/old.vdi" --delete

# --- Guest control (needs Guest Additions) ---
VBoxManage guestcontrol "MyVM" run --exe /bin/systemctl \
  --username user --password secret -- systemctl is-system-running

12 · Practical Demonstration — VMware#

VMware provides robust virtualization solutions suitable for both desktop and enterprise environments. It is worth being precise about the product family, because they are very different things sharing a brand:

Features of VMware#

• High performance — optimised for efficient resource utilisation.
• Advanced networking — supports complex network configurations.
• Snapshot and cloning — easy VM state management.
• Cross-platform support — run various guest OSes.

In practice, what VMware is genuinely best at: mature and reliable snapshot management, the best 3D/graphics acceleration among desktop hypervisors, Unity/seamless mode, and — at the enterprise tier — vMotion, the live migration implementation that made the technique mainstream.

Networking in VMware#

VMware offers flexible networking similar to VirtualBox, implemented as named virtual switches on the host:

# Linux host: inspect and configure VMware's virtual networks
ip -brief addr show | grep vmnet
sudo vmware-netcfg               # the GUI network editor

Managing VMs in VMware Workstation#

VMware Workstation provides a GUI for VM management, but command-line tools are also available.

Starting a VM#

Use the vmrun command:

vmrun start "/path/to/MyVM.vmx"

$ vmrun start "/path/to/MyVM.vmx"
Started VM successfully

“MyVM” has started successfully.

Note what vmrun is addressed by: the path to the .vmx file, not a VM name. The .vmx is the VM’s identity as far as VMware is concerned — a plain text file listing every setting. This differs fundamentally from VirtualBox (which keeps a registry of named VMs) and libvirt (which keeps named domains), and it is why vmrun commands are always long.

# You will often need -T to say which VMware product to talk to
vmrun -T ws start "/path/to/MyVM.vmx"          # Workstation on Linux/Windows
vmrun -T fusion start "/path/to/MyVM.vmx"      # Fusion on macOS
vmrun -T player start "/path/to/MyVM.vmx"      # Player

# Start with no window — the equivalent of VirtualBox's --type headless
vmrun -T ws start "/path/to/MyVM.vmx" nogui

# What is running right now?
vmrun list

$ vmrun list
Total running VMs: 2
/home/user/vmware/ubuntu-24.04/ubuntu-24.04.vmx
/home/user/vmware/win11-test/win11-test.vmx

Stopping a VM#

vmrun stop "/path/to/MyVM.vmx" soft

$ vmrun stop "/path/to/MyVM.vmx" soft
Stopped VM successfully

“MyVM” is shutting down gracefully.

The soft argument is doing the important work here, and the source notes do not explain it:

# The rest of the vmrun vocabulary
vmrun suspend "/path/to/MyVM.vmx" soft        # save state to disk
vmrun reset   "/path/to/MyVM.vmx" soft        # restart
vmrun pause   "/path/to/MyVM.vmx"
vmrun unpause "/path/to/MyVM.vmx"

vmrun snapshot     "/path/to/MyVM.vmx" "clean-install"
vmrun listSnapshots "/path/to/MyVM.vmx"
vmrun revertToSnapshot "/path/to/MyVM.vmx" "clean-install"
vmrun deleteSnapshot   "/path/to/MyVM.vmx" "clean-install"

vmrun clone "/path/to/MyVM.vmx" "/path/to/Clone.vmx" full -cloneName="Clone"
vmrun clone "/path/to/MyVM.vmx" "/path/to/Linked.vmx" linked -snapshot="clean-install"

vmrun getGuestIPAddress "/path/to/MyVM.vmx" -wait    # ← the good one
vmrun runProgramInGuest "/path/to/MyVM.vmx" /bin/uname -a
vmrun copyFileFromHostToGuest "/path/to/MyVM.vmx" ./setup.sh /tmp/setup.sh

Changing VM settings#

Modifying memory and CPUs typically requires editing the VM’s .vmx file or using the GUI. Example .vmx configuration changes:

memsize = "4096"
numvcpus = "2"

Reading those two lines:

A .vmx is worth seeing in full, because it makes the “a VM is just a file” point concretely:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "21"
displayName = "MyVM"
guestOS = "ubuntu-64"

memsize = "4096"
numvcpus = "2"
cpuid.coresPerSocket = "2"

scsi0.present = "TRUE"
scsi0.virtualDev = "pvscsi"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "MyVM.vmdk"

ethernet0.present = "TRUE"
ethernet0.connectionType = "nat"          # nat | bridged | hostonly | custom
ethernet0.virtualDev = "vmxnet3"
ethernet0.addressType = "generated"

ide1:0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
ide1:0.fileName = "/isos/ubuntu-24.04.1-live-server-amd64.iso"

vhv.enable = "TRUE"                        # nested virtualization
tools.syncTime = "TRUE"

Retrieving the VM’s IP address#

Inside the VM, use:

ifconfig

$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.20  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::20c:29ff:fe4b:8a1c  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:4b:8a:1c  txqueuelen 1000  (Ethernet)
        RX packets 18422  bytes 21094883 (21.0 MB)
        TX packets 9137   bytes 1204411 (1.2 MB)

The VM’s IP address is 192.168.1.20. Read the whole output field by field, because you will see it constantly:

# Better ways to find a VMware guest's address
vmrun getGuestIPAddress "/path/to/MyVM.vmx" -wait   # from the HOST, via VMware Tools
ip -brief addr show                                  # from inside the guest, modern
hostname -I                                          # from inside, quickest

13 · Summary & Mind Map#

The virtualization ecosystem#

                              ╔════════════════════════════╗
                              ║   VIRTUALIZATION           ║
                              ║   One physical machine     ║
                              ║   becomes many logical     ║
                              ╚════════════╤═══════════════╝
                                           │
                ┌──────────────────────────┼──────────────────────────┐
                │                          │                          │
        ┌───────▼────────┐      ┌──────────▼──────────┐     ┌────────▼─────────┐
        │  HYPERVISORS   │      │   NETWORKING       │     │  VM MANAGEMENT   │
        └───────┬────────┘      └──────────┬──────────┘     └────────┬─────────┘
                │                          │                         │
        ┌───────┴────────────┐    ┌────────┴────────────┐   ┌───────┴──────────┐
        │                    │    │                     │   │                  │
    ┌───▼──┐            ┌───▼──┐ │  NAT               │  Create & Config   Snapshots
    │Type1 │            │Type2 │ │  ├─ Port forward   │  ├─ CPU, RAM, Disk  ├─ Point-in-time
    │ Bare │            │Host  │ │  └─ Private 10.x   │  ├─ Network attach  └─ Revert & test
    │metal │            │ed OS │ │                    │  └─ BIOS/UEFI      
    └───┬──┘            └───┬──┘ │  Bridged           │                     
        │                   │    │  ├─ Own LAN IP     │  Cloning            Monitoring
        │                   │    │  └─ Full access    │  ├─ Full clone      ├─ CPU %st
  ESXi  │  Proxmox   KVM   │    │                    │  └─ Linked clone    ├─ RAM %used
  Hyper-V│ Nutanix  Xen    │    │  Host-only         │                     ├─ Disk I/O
        │                   │    │  ├─ Isolated 192.x │  Migration          └─ Network
  Performance│ VirtualBox   │    │  └─ Lab networks   │  ├─ Cold            
  Density│ VMware WS       │    │                    │  ├─ Live            
        │ UTM              │    │  Internal          │  └─ Storage         
        │                  │    │  ├─ VM-to-VM only  │                     
        └──────────────────┘    │  └─ Air-gapped     │                     
                                │                    │                     
                                └────────────────────┘                     

Virtualization in context#

Virtualization is the foundational technology of modern infrastructure. It solves one core problem: isolation without waste. Before virtualization, you bought physical hardware isolation at massive cost. After virtualization, you buy logical isolation at a tenth the price, while still running the same hardware at high utilization.

The three types of virtualization each solve a different scope problem:

• Hardware-level virtualization — complete emulation of a machine, allowing unmodified guest OSes to run in isolation. The foundation of cloud computing. Examples: KVM, ESXi, Hyper-V, Xen HVM.
• OS-level virtualization — sharing one kernel with isolated user-space views via namespaces and cgroups. Lightweight, dense, fast. Examples: Docker, Podman, LXC.
• Application virtualization — bundling a single app with its dependencies. Solves dependency conflicts. Examples: venv, Snap, Flatpak, the JVM.

The key distinction that predicts everything else: VMs have their own kernel; containers share the host’s. This is why containers boot in milliseconds and VMs in seconds, why a container escape compromises the host while a VM escape does not, and why you cannot run Windows containers on a Linux kernel.

At the technical level, virtualization on x86 required hardware assistance after 2005 (Intel VT-x, AMD-V). Without it, the CPU was fundamentally broken for virtualization — 17 sensitive but unprivileged instructions would fail silently instead of trapping to the hypervisor. VT-x and AMD-V introduced a whole new privilege level — “ring −1” — where the hypervisor sits, while guests run in a de-privileged ring 0. This solved the CPU problem. Memory was solved separately via EPT (Extended Page Tables) and NPT (Nested Page Tables), allowing the MMU to walk two page-table hierarchies in hardware instead of the hypervisor doing it in software.

The production pattern is hybrid: hardware-assisted full virtualization for CPU and memory, with paravirtualized I/O drivers (VirtIO) for performance. That split exists because emulating a network card or disk controller is expensive — every register write traps to the hypervisor. A paravirtualized device is a simpler contract: a shared-memory ring buffer, a single notification, and the hypervisor handles the I/O. That one change moves network throughput from ~1 Gbit/s to 10–100 Gbit/s.

Why it matters for the rest of this handbook#

Every cloud instance you will ever launch is a virtual machine. Every Docker container on macOS or Windows runs inside a Linux VM. Kubernetes nodes are almost always VMs. CI/CD runners are VMs created on demand and destroyed when the job finishes. Understanding virtualization is understanding the foundation that everything else in this handbook sits on.

The practitioner’s insight: virtualization is nearly free in the CPU and memory sense, but I/O is where you feel it. CPU-bound workloads run at 97–99% of native speed inside a VM. Naive disk I/O can be an order of magnitude slower because every operation exits the VM to the hypervisor. That is why paravirtualized drivers matter so much, why virtio-blk and virtio-net are standard on every cloud image, and why a production hypervisor estate spends more time optimizing I/O paths than anything else.

14 · Cheat Sheet#

Hypervisor selection quick reference#

VM creation workflow (VirtualBox)#

# 1. Create the VM shell
VBoxManage createvm --name "MyVM" --ostype "Ubuntu_64" --register

# 2. Allocate resources
VBoxManage modifyvm "MyVM" --memory 2048 --cpus 2 --vram 128

# 3. Create the virtual disk (20 GB, dynamic)
VBoxManage createmedium disk --filename "/path/MyVM.vdi" --size 20000

# 4. Attach storage controller
VBoxManage storagectl "MyVM" --name "SATA" --add sata --controller IntelAHCI

# 5. Attach the disk
VBoxManage storageattach "MyVM" --storagectl "SATA" \
  --port 0 --device 0 --type hdd --medium "/path/MyVM.vdi"

# 6. Attach the ISO
VBoxManage storageattach "MyVM" --storagectl "SATA" \
  --port 1 --device 0 --type dvddrive --medium "/path/ubuntu-24.04.iso"

# 7. Configure networking (two adapters: NAT + host-only)
VBoxManage modifyvm "MyVM" --nic1 nat --nic2 hostonly --hostonlyadapter2 vboxnet0

# 8. Add SSH port forwarding on NAT adapter
VBoxManage modifyvm "MyVM" --natpf1 "ssh,tcp,127.0.0.1,2222,,22"

# 9. Start the VM
VBoxManage startvm "MyVM" --type headless

# 10. Boot, install OS, install guest additions, snapshot
VBoxManage snapshot "MyVM" take "clean-install" --description "OS + additions"

VM lifecycle — the precise terms#

The rule: use acpipowerbutton for normal shutdown; use poweroff only if the guest is hung. Use pause when you need to freeze a VM instantly; use savestate when you want to release host memory.

Networking modes quick decision tree#

Does the VM need internet access?
├─ NO
│  └─ Does the HOST need to reach the VM?
│     ├─ YES → HOST-ONLY (clean addressing, 192.168.56.x)
│     └─ NO  → INTERNAL (maximum isolation, VMs only)
│
└─ YES
   └─ Must other machines on the LAN reach it directly?
      ├─ NO (it reaches them, not vice versa)
      │  └─ NAT (default, simple, + port fwd if needed)
      │
      └─ YES (it must be a "real" server)
         └─ Wired Ethernet?
            ├─ YES → BRIDGED (gets own LAN IP)
            └─ NO (Wi-Fi) → NAT + port forward (bridging unreliable)

Common guest additions commands#

# === VirtualBox ===
sudo apt update
sudo apt install -y build-essential dkms linux-headers-$(uname -r)
# Then: Devices → Insert Guest Additions CD image… (GUI)
sudo mount /dev/cdrom /mnt && sudo /mnt/VBoxLinuxAdditions.run
# Or from repos (simpler):
sudo apt install -y virtualbox-guest-utils virtualbox-guest-x11

# === VMware (open-vm-tools, maintained by VMware) ===
sudo apt install -y open-vm-tools open-vm-tools-desktop

# === KVM (QEMU Guest Agent) ===
sudo apt install -y qemu-guest-agent
sudo systemctl enable --now qemu-guest-agent

Disk sizing quick reference#

Always use dynamically allocated disks on a laptop. A “20 GB” dynamic disk costs ~3 GB on your SSD until you fill it. Fixed disks pre-allocate the full size and are slow to create.

Snapshots: when to use, when to avoid#

The rule: snapshots are undo buttons for the next few hours. Delete them after. Snapshot chains deeper than 5 layers degrade performance visibly. Snapshots older than 72 hours in production trigger alert pages for a reason.

Live migration requirements (KVM example)#

# Requires: shared storage, compatible CPU features, fast network
# Single command:
virsh migrate --live --persistent --undefinesource \
  MyVM qemu+ssh://dest-host.example.com/system

# Gotchas:
# 1. CPU features must be compatible (EVC / <cpu mode='custom'>)
# 2. If guest dirties memory faster than the link can carry,
#    it won't converge — use:
virsh migrate-setmaxdowntime MyVM 30000
# 3. Watch for "post-copy" mode if available — trades
#    a small downtime for guaranteed convergence

Performance tuning flags (by hypervisor)#

Troubleshooting matrix#

File locations and paths#

Essential CLI one-liners#

# === Inventory ===
VBoxManage list vms                                    # all registered VMs
virsh list --all                                       # all libvirt domains
vmrun list                                             # running VMware VMs

# === Snapshots ===
VBoxManage snapshot "MyVM" take "name"                 # create snapshot
VBoxManage snapshot "MyVM" restore "name"              # revert (power off first)
virsh snapshot-create-as MyVM name --disk-only         # KVM disk snapshot

# === Networking ===
VBoxManage showvminfo "MyVM" --machinereadable | grep -i natpf     # show port forwards
virsh domiflist MyVM                                   # guest interfaces in libvirt
VBoxManage guestproperty enumerate "MyVM"              # all guest properties (requires additions)

# === Disk management ===
VBoxManage modifymedium disk "MyVM.vdi" --resize 40000 # resize in MB
VBoxManage list hdds                                   # all registered disks
qemu-img info /var/lib/libvirt/images/MyVM.qcow2       # QEMU/KVM disk info

# === Graceful shutdown (always prefer over poweroff) ===
VBoxManage controlvm "MyVM" acpipowerbutton
vmrun stop "/path/MyVM.vmx" soft
virsh shutdown MyVM

# === Performance check ===
top                                                    # look for `%st` (steal time) ≤5%
virsh dommemstat MyVM                                  # memory balloon state
VBoxManage metrics query "MyVM" CPU/Load/Average       # CPU metrics

Remember: the first place to troubleshoot a slow VM is the host, not the guest. Check top for steal time, free -h for host swapping, and datastore space for snapshot decay. The VM’s top output lies about what the hypervisor is doing to it.

1 · The Big Picture#

Why this topic exists#

You can type ls and see files. Nothing mysterious about that. But between your keystroke and the output, at least three layers of software have made decisions: the terminal caught your key and sent it somewhere, a shell interpreted the command you typed, and the shell started a program and displayed its output. If any one of those layers is misconfigured, the experience breaks in ways that seem like magic.

Understanding the shell is not about memorising commands. It is about understanding what happens between the cursor and the filesystem — how environment variables propagate, why aliases work in the interactive shell but not in scripts, why your .bashrc runs 100 times a day but you only edited it once, and how to read a terminal that has 40 panes because someone is using tmux.

This is also the chapter where most people lock themselves out of their machines. Reach the end, and you will know the recovery steps.

The real problem it solves#

Without a shell, you would invoke a program like this:

fork();           // Create a new process
execve("/bin/ls", args, env);  // Replace it with /bin/ls
// wait for it to finish
// read its output from... where?
// read its exit code

A shell is the abstraction layer that lets you do this:

ls | grep hosts

The shell parses the line, creates two processes, attaches a pipe between them, waits for both, and gives you the result — all from one line of text. It also handles the messy details: storing your password in memory so you don’t type it 50 times, expanding *.txt to a list of files without the program knowing about wildcards, providing history so you can re-run the command you typed yesterday, and letting you define functions that work like programs but are written in the shell’s own language.

Where you will encounter it#

Why companies care#

• Universality — every Linux machine has /bin/sh, even minimal containers. Your script will run.
• Scriptability — automation happens in shell scripts. Terraform uses HCL, but Terraform launches provisioners via shell. Kubernetes runs exec probes via shell.
• Debuggability — when a service is broken, you SSH in and run commands interactively. The shell is your microscope.
• Reproducibility — a shell script documents what you did. Running it again reproduces it; logging into the GUI and clicking buttons does not.

2 · Intuition First#

Analogy 1: the terminal and shell are not the same#

You can use a shell without a terminal (over SSH, in a container, in a script). You can use a terminal without a shell (opening a telnet connection, writing bytes to /dev/ttyUSB0). But together, in the interactive login session on your laptop, they create the illusion that you are directly commanding the machine.

Analogy 2: startup files as waking up#

Imagine you get up every morning and do the same tasks: put on glasses, drink coffee, check email. You don’t want to think about this every time, so you write them down as a checklist.

A shell’s startup files are that checklist. When you open a terminal:

1. If you are logging in (like ssh user@host), the shell reads /etc/profile, then your ~/.bash_profile — these are the “getting ready for the day” tasks.
2. If you are opening a sub-shell (like running a script), the shell skips the login steps and just reads ~/.bashrc — these are the “every time I’m here” tasks.

Most people never touch their login startup files; they live in ~/.bashrc, adding aliases and functions that they use every day.

Analogy 3: environment variables as inherited traits#

When you start a program, the shell passes it a list of variables — things like PATH (where to find programs), HOME (your home directory), USER (your username). These are environment variables — every program born from this shell inherits them.

If you then run a command that starts another process, that grandchild inherits the same environment, unless you explicitly unset a variable. This is why knowing which variables are exported matters: set a wrong PATH, and every program run from that shell onwards fails silently.

3 · Technical Definitions#

Terminal. The user interface that multiplexes your keyboard and screen to one or more shell sessions. In the past, a physical piece of hardware (VT100 terminal); now, a software application (Terminal.app on macOS, GNOME Terminal on Linux, Windows Terminal on Windows 11). The terminal speaks to the kernel via a TTY (teletypewriter) device — a character device at /dev/tty* or /dev/pts/* that handles the low-level protocol.

Shell. A command-line interpreter: a program that reads a command, parses it into words and operators, expands variable references and globs, and executes the resulting program(s). Examples: bash (Bourne-Again Shell), sh (the POSIX minimal shell), zsh (Z shell), fish (Friendly Interactive Shell), ksh (Korn shell).

TTY and PTY. A TTY (teletypewriter, or terminal) is a character device representing a physical or virtual terminal. A PTY (pseudo-terminal) is a software pair: a master half (held by the terminal emulator) and a slave half (given to the shell as file descriptors 0/1/2, stdin/stdout/stderr). When you type in the terminal, the characters go to the slave; when the shell writes to file descriptor 1, the bytes come out of the master to your screen.

Line discipline. The kernel module between the terminal driver and the PTY that handles backspace, Ctrl+C, Ctrl+Z, and other raw terminal features — the reason you can hit backspace to erase a character even though the shell has not seen it yet. There are different line disciplines for raw vs. canonical mode, which is why the screen goes weird when you cat a binary file and you have to reset it afterwards.

Login shell. A shell that starts when you log in (SSH, su -, physical login). It reads startup files designed for setting up your environment once: /etc/profile, /etc/profile.d/*, ~/.bash_profile, ~/.bash_login, or ~/.profile.

Interactive shell. A shell connected to a terminal (you can type commands). It reads startup files designed for interactive use: /etc/bashrc, ~/.bashrc. A non-interactive shell (running a script) skips these.

Environment variable. A key-value pair passed from parent to child process. Defined with export NAME=value; inherited by all subshells unless unset. Examples: PATH, HOME, USER, SHELL, PWD.

Shell variable. A variable that exists only in the current shell, not passed to subprocesses. Defined with NAME=value (no export). Examples: loop counters, temporary strings.

4 · Internal Working#

The TTY/PTY/line discipline stack#

┌─────────────────────────────────────────────────────┐
│  Terminal emulator (Terminal.app, GNOME Terminal)    │
│  Captures keystrokes; renders output                 │
└─────────────────────────────────┬───────────────────┘
                                  │
                          PTY master ↔ master end
                                  │
        ┌─────────────────────────┴────────────────────┐
        │  KERNEL                                      │
        │  ┌────────────────────────────────────────┐  │
        │  │  PTY (pseudo-terminal) slave          │  │
        │  │  /dev/pts/0, /dev/pts/1, etc          │  │
        │  └────────────────────────────────────────┘  │
        │  ┌────────────────────────────────────────┐  │
        │  │  Line discipline                       │  │
        │  │  • Handle backspace, Ctrl+C, Ctrl+Z    │  │
        │  │  • Cooked (canonical) vs raw mode      │  │
        │  │  • Echo handling                       │  │
        │  └────────────────────────────────────────┘  │
        │  ┌────────────────────────────────────────┐  │
        │  │  Terminal driver (tty_io)              │  │
        │  │  • fd 0, 1, 2 (stdin/stdout/stderr)    │  │
        │  └────────────────────────────────────────┘  │
        └─────────────────────────────────────────────┘
                                  │
        ┌─────────────────────────┴─────────────────────┐
        │  Shell (bash, zsh, fish, sh)                   │
        │  Reads fd 0 (your keystrokes)                  │
        │  Writes to fd 1 and 2 (output)                 │
        └─────────────────────────────────────────────────┘

When you type l s Enter, this happens:

1. Terminal emulator sees keystroke L, sends ASCII 108 to the PTY master.
2. PTY slave receives it; line discipline echoes it back to the master (so you see it) and buffers it.
3. When you hit Enter, line discipline sends the whole line to the shell on fd 0.
4. Shell reads “ls”, parses it, calls execve("/bin/ls").
5. ls writes to fd 1 (stdout).
6. Bytes flow through the PTY slave back to the master, to the terminal emulator, to your screen.

This is why a shell always runs as the controlling process of the PTY: it is the program responsible for reading from that device. If the shell exits, the PTY becomes orphaned and the terminal window closes.

Login vs non-login, interactive vs non-interactive#

Not all shells are the same. The shell’s startup sequence depends on how it was invoked:

Start shell
    │
    ├─ Invoked with -i flag (interactive)?
    │  If no: non-interactive
    │         → skip all startup files
    │         → read only BASH_ENV (if sh compatibility mode)
    │         → execute script and exit
    │
    └─ If yes: interactive shell
       │
       ├─ Invoked as login shell?
       │  (SSH, su -, /bin/login, --login flag)
       │
       ├─ If yes: LOGIN INTERACTIVE
       │  → read /etc/profile
       │  → read /etc/profile.d/* (if it exists)
       │  → read ~/.bash_profile (or ~/.bash_login, or ~/.profile)
       │  → read ~/.bashrc (if bash and interactive)
       │  → prompt appears
       │
       └─ If no: NON-LOGIN INTERACTIVE
          (running bash again inside bash)
          → skip /etc/profile and ~/.bash_profile
          → read ~/.bashrc only
          → prompt appears

Which startup file runs where:

The Bash startup sequence in full#

When you SSH in or run su - user, this is what actually happens:

flowchart TD
    A["Shell invoked<br/>(login + interactive)"] --> B["Read /etc/profile"]
    B --> C["Loop: read each file in /etc/profile.d/*"]
    C --> D["Read ~/.bash_profile"]
    D --> E["If ~/.bash_profile doesn't exist,<br/>try ~/.bash_login"]
    E --> F["If that doesn't exist,<br/>try ~/.profile"]
    F --> G["(bash also reads ~/.bashrc<br/>from within ~/.bash_profile)"]
    G --> H["Prompt appears"]
    H --> I["You type command"]
    I --> J["Shell parses, expands, executes"]
    J --> K["Before exit: read ~/.bash_logout"]
    K --> L["Shell exits"]

The typical ~/.bash_profile contains:

# ~/.bash_profile: login-shell setup
# Sourced by login shells (SSH, su -)

# Load system-wide defaults and bash login scripts
if [ -f /etc/profile ]; then
  source /etc/profile
fi

# NOW source ~/.bashrc
# This is the key idiom: login shells source the interactive setup
if [ -f ~/.bashrc ]; then
  source ~/.bashrc
fi

Why? Because you almost certainly have interactive setup (aliases, functions, prompt) in ~/.bashrc. If ~/.bash_profile did not source it, those would not exist on login. This is the idiom to memorise.

The login process from /etc/passwd#

When you SSH and provide a password, this happens:

sequenceDiagram
    participant S as sshd
    participant K as Kernel
    participant L as Login process
    participant B as bash (your shell)
    S->>K: fork + execve(/etc/passwd entry)
    K->>L: load login/su/whatever
    L->>L: authenticate user
    L->>K: fork
    K->>B: execve shell from /etc/passwd
    Note over B: /etc/passwd has argv[0] = "-bash"
    Note over B: dash prefix = login shell
    B->>B: read /etc/profile
    B->>B: read ~/.bash_profile
    B->>B: source ~/.bashrc (idiom)
    B-->>S: send prompt

5 · A Shell is a Language — the Operators and Expansions#

One reason people find the shell confusing is that it is a full language with operators, conditionals, loops, and variable expansion — but the syntax is terse and full of special characters.

The expanding you need to know#

The order matters:

1. { } expansion
2. ~ expansion
3. Parameter, command, arithmetic expansion (left to right)
4. Word splitting
5. Pathname expansion (globbing)
6. Quote removal

This is why echo *.txt might print 50 filenames, but echo "*.txt" prints the literal string — quotes prevent pathname expansion.

Redirections#

6 · Practical Demonstration#

Three ways to find your current shell#

# Method 1: check the SHELL variable
echo $SHELL

# Method 2: use the ps command — which process are you inside
ps -p $$

# Method 3: read the tty
tty

$ echo $SHELL
/bin/bash

$ ps -p $$
    PID TTY      STAT   TIME COMMAND
   2410 pts/0    Ss     0:00 bash

$ tty
/dev/pts/0

$ who
user     pts/0        Aug  2 15:23 (192.168.1.50)

The ps -p $$ line tells you the process ID of the running shell ($$), and the COMMAND field shows /bin/bash (or /bin/sh, or /bin/zsh). The tty command shows which device file your terminal is connected to; if you run it over SSH, it says /dev/pts/0 or similar (a pseudo-terminal, because it is software). If you were on a physical machine with a physical serial cable, it might say /dev/ttyS0.

Switching shells — temporary and permanent#

Temporary switch (lasts only this session):

# Just type the new shell
zsh
# You are now in zsh; exit to go back
exit

Permanent switch (becomes your login shell):

# List available shells
cat /etc/shells

# Change your login shell (in /etc/passwd)
chsh -s /bin/zsh
# You must log out and back in for this to take effect

# Verify it worked (on next login)
echo $SHELL

$ cat /etc/shells
# /etc/shells: valid login shells.
/bin/sh
/bin/bash
/bin/zsh
/usr/bin/zsh
/bin/ksh
/bin/fish

$ chsh -s /bin/zsh
Password:
Changing the login shell for user
Enter the new value, or press ENTER for the default
	Login Shell [/bin/bash]: /bin/zsh

After chsh, SSH into a fresh session to confirm your login shell is now in the SHELL variable.

The startup files and their roles#

/etc/profile — system-wide login shell setup:

# /etc/profile — read by all login shells (SSH, su -)
# Run by: sh, bash, zsh (if invoked as sh or login)

# Set umask, PATH, timezone
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export LANG=en_US.UTF-8

# Source all files in /etc/profile.d/
if [ -d /etc/profile.d ]; then
  for i in /etc/profile.d/*.sh; do
    if [ -r "$i" ]; then
      . "$i"
    fi
  done
  unset i
fi

~/.bash_profile — your login shell setup (SSH, console login):

# ~/.bash_profile — read by login bash shells
# Load interactive setup from ~/.bashrc
if [ -f ~/.bashrc ]; then
  source ~/.bashrc
fi

# Add anything login-specific below
export EDITOR=vim

~/.bashrc — your interactive shell setup (every shell, login or not):

# ~/.bashrc — read by all interactive bash shells
# Run by: every shell you open on this system

# Aliases
alias ll='ls -lh'
alias grep='grep --color=auto'

# Functions
extract() {
  case "$1" in
    *.tar.bz2)   tar xvjf "$1" ;;
    *.tar.gz)    tar xvzf "$1" ;;
    *.tar)       tar xvf "$1" ;;
    *.zip)       unzip "$1" ;;
    *.rar)       unrar x "$1" ;;
    *)           echo "Usage: extract <file>" ;;
  esac
}

# Prompt
PS1='[\u@\h \W]\$ '

# History
HISTSIZE=1000
HISTFILESIZE=2000

# Colors
export CLICOLOR=1
export LS_COLORS='di=34:ln=31'

~/.bash_logout — cleanup on shell exit:

# ~/.bash_logout — run when login shell exits
# Use for cleanup: clearing screen, backing up history, etc.

clear
echo "Goodbye $USER!"

Aliases — the full lifecycle#

Define an alias:

alias ll='ls -lh'

This creates a command named ll that expands to ls -lh before execution.

Use it:

$ ll
total 48K
-rw-r--r-- 1 user group 1.5K Aug  2 15:00 file.txt
-rw-r--r-- 1 user group 4.2K Aug  2 14:00 script.sh

List all aliases:

$ alias
alias ll='ls -lh'
alias grep='grep --color=auto'
alias mkdir='mkdir -pv'

Temporarily override an alias:

# Prefix with backslash
\ls    # calls /bin/ls, not the alias

# Or quote it
'ls'   # calls /bin/ls

Remove an alias:

unalias ll

Make it permanent:

Add it to ~/.bashrc:

# ~/.bashrc
alias ll='ls -lh'
alias grep='grep --color=auto'
alias mkdir='mkdir -pv'

Shell functions vs aliases#

Both define reusable commands, but they differ:

# ALIAS — simple text replacement
alias myls='ls -lh'

# FUNCTION — full programming construct
myls() {
  local dir="${1:-.}"
  ls -lh "$dir"
}

Example: use a function to wrap a dangerous command:

# ~/.bashrc
rm() {
  # Ask for confirmation
  read -p "Are you sure? (y/N) " -n 1 -r
  echo
  if [[ $REPLY =~ ^[Yy]$ ]]; then
    command rm "$@"
  else
    echo "Cancelled"
    return 1
  fi
}

Environment variables: the full story#

Define and export:

# Shell variable (not inherited by subprocesses)
USERNAME=alice

# Environment variable (inherited)
export USERNAME=alice

# Both at once
export PATH=/usr/bin:/bin

# View all environment variables
printenv

# View all variables (shell + environment)
set

# View one variable
echo $USERNAME
printenv USERNAME

# Check if a variable is exported
declare -p USERNAME

$ export MYVAR=hello
$ bash -c 'echo $MYVAR'
hello

$ unset MYVAR
$ bash -c 'echo $MYVAR'
(no output — empty)

Key variables:

Unsetting a variable:

unset MYVAR
# Now $MYVAR is empty

Prompt customization: PS1 and PS2#

The shell variables PS1 and PS2 control your prompt:

# Default (often)
PS1='\u@\h:\w\$ '

# Custom
PS1='[\t] \u @ \h : \W \$ '

Escape sequences in PS1:

Example with colours:

# Red username, green hostname, blue working directory
PS1='\[\033[31m\]\u\[\033[0m\]@\[\033[32m\]\h\[\033[0m\]:\[\033[34m\]\w\[\033[0m\]\$ '

Modern approach: starship or oh-my-posh:

Instead of editing PS1, use a third-party prompt generator:

# Install starship (Rust, fast, cross-shell)
curl -sS https://starship.rs/install.sh | sh

# Add to ~/.bashrc
eval "$(starship init bash)"

Starship handles git status, language versions, duration of last command, and 200+ other features automatically.

History expansion and Ctrl+R#

History expansion:

# Re-run the last command
!!

# Re-run the last command that started with "grep"
!grep

# Re-run command number 145
!145

# Use the last argument of the previous command
echo file.txt
cat !$       # expands to: cat file.txt

# Use all arguments from the last command
!*

# Search history interactively
Ctrl+R
# Type letters; shell searches backwards
# Press Enter to run, Ctrl+G to cancel

$ grep error /var/log/syslog
... 200 matching lines ...

$ !grep      # re-runs the grep
... 200 matching lines ...

$ cat !$     # cat /var/log/syslog
... contents of syslog ...

Readline keybindings — Bash/Zsh line editing#

When you are typing at the bash prompt, these keybindings let you edit without leaving the line:

Example of editing a long command:

# You typed this long line, with a typo:
tar -xzvf /path/to/archive/my-backup-20240802.tar.gz /path/to/destnation

# Press Ctrl+A to go to the start
# Press Ctrl+F Ctrl+F Ctrl+F to move to "destnation"
# Press Ctrl+R to search for "destination" in history
# Find it, press Enter, and it replaces the word
# Or: Ctrl+U to delete from cursor to line start, retype correctly

Reloading configuration without logging out#

To apply changes to ~/.bashrc without logging out:

source ~/.bashrc

# Or use the . (dot) shorthand
. ~/.bashrc

This re-reads the file in the current shell, updating aliases, functions, environment variables, and prompt.

Verify the change:

# After adding an alias to ~/.bashrc
source ~/.bashrc
alias myls    # prints the alias definition
myls          # runs the command

7 · A Real ~/.bashrc Example#

Here is a production-quality configuration suitable for development work, condensed from the PDF example on p135–137:

# ~/.bashrc — user interactive shell setup
# This file is sourced by non-login shells (e.g., when you open a new tab in an editor)
# It is also sourced by login shells via ~/.bash_profile

# Skip if not interactive
case $- in
  *i*) ;;
  *) return ;;
esac

# ============ HISTORY ============
HISTSIZE=10000
HISTFILESIZE=20000
HISTCONTROL=ignoredups:ignorespace

# Append to history file instead of overwriting
shopt -s histappend

# ============ ALIASES ============
alias ls='ls --color=auto'
alias ll='ls -lh'
alias la='ls -la'
alias grep='grep --color=auto'
alias mkdir='mkdir -pv'
alias less='less -R'

# ============ PROMPT ============
PS1='[\u@\h \W]\$ '

# ============ FUNCTIONS ============

# extract: unarchive any compressed file format
extract() {
  if [[ $# -eq 0 ]]; then
    echo "Usage: extract <file> [destination]"
    echo "Extracts common archive formats"
    return 1
  fi

  local file="$1"
  local dest="${2:-.}"

  case "$file" in
    *.tar.bz2|*.tbz2)  tar xvjf "$file" -C "$dest" ;;
    *.tar.gz|*.tgz)    tar xvzf "$file" -C "$dest" ;;
    *.tar.xz|*.txz)    tar xvJf "$file" -C "$dest" ;;
    *.tar)             tar xvf "$file" -C "$dest" ;;
    *.zip)             unzip "$file" -d "$dest" ;;
    *.rar)             unrar x "$file" "$dest" ;;
    *.7z)              7z x "$file" -o"$dest" ;;
    *)                 echo "Unknown format: $file"; return 1 ;;
  esac
  echo "Extracted to $dest"
}

# backup: create a dated backup of a file
backup() {
  if [[ $# -eq 0 ]]; then
    echo "Usage: backup <file>"
    return 1
  fi
  local file="$1"
  if [[ ! -e "$file" ]]; then
    echo "Error: $file not found"
    return 1
  fi
  local backup_file="${file}.backup.$(date +%Y%m%d-%H%M%S)"
  cp -v "$file" "$backup_file"
  echo "Backed up to $backup_file"
}

# find_in_files: grep recursively with proper escaping
find_in_files() {
  if [[ $# -lt 1 ]]; then
    echo "Usage: find_in_files <pattern> [directory]"
    return 1
  fi
  local pattern="$1"
  local dir="${2:-.}"
  grep -r "$pattern" "$dir" --include="*.txt" --include="*.sh" --include="*.md"
}

# ============ ENVIRONMENT ============
export EDITOR=vim
export PAGER=less
export LANG=en_US.UTF-8

# Add local bin to PATH if it exists
if [[ -d ~/.local/bin ]]; then
  export PATH=~/.local/bin:$PATH
fi

# ============ SHELL OPTIONS ============
shopt -s checkwinsize  # update LINES and COLUMNS after each command
shopt -s globstar      # enable ** for recursive glob
shopt -s dotglob       # include files starting with .