systemd Fundamentals

A junior engineer asks a straightforward question: "How do I make my Python script start at boot?" The senior engineer starts typing systemctl... then pauses. Service, target, socket, timer, mount, slice — which unit type? Type=simple or forking? After=network.target or network-online.target? Install section with WantedBy? Drop-in override in /etc/systemd/system/foo.service.d/?

For something as routine as "run a program at boot," systemd offers dozens of knobs, and the internet is full of conflicting advice because different knobs matter in different situations. This lesson is the map: what every piece of systemd is, how the pieces interconnect, and which knob matters for which situation. After this, "how do I make my Python script start at boot" is three lines of config and a systemctl enable.

---

What systemd Actually Is

systemd is three things, all tightly coupled:

1. An init system — PID 1. It starts your services at boot, stops them at shutdown, and supervises them in between.
2. A dependency-based service manager — it models "service A needs service B" and figures out the correct start order and which services can run in parallel.
3. A suite of tools and daemons — journald (logging), logind (sessions), resolved (DNS), timesyncd (NTP), networkd (networking), and more. You do not have to use every piece, but they all come from the same upstream and they all use the same concepts.

Before systemd, Linux used SysV init: /etc/init.d/* shell scripts, a fixed boot order through "runlevels" (0–6), and no real idea what "service B depends on service A" meant. Boot was sequential: start service 01, wait for it, start service 02, wait for it, and so on. A slow service held up the whole boot. A badly-written init script could crash the system.

systemd replaced this with:

• Unit files: declarative .ini-style configs instead of shell scripts.
• A dependency graph: After=, Requires=, Wants= describe relationships, and systemd computes the correct start order.
• Parallel startup: units with satisfied dependencies all start at once.
• Supervision: services that die are restarted, leaks are isolated, cgroups enforce limits.
• Socket activation: a service does not have to be running to "listen" — systemd listens, and starts the service when a connection comes in.
• On-demand activation: a mount unit only activates when the path is accessed; a timer runs only when scheduled.

---

The Unit File Format

A unit file is an .ini file with named sections. Most of them have three main sections:

[Unit]
Description=My Web App
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=/etc/myapp/env
ExecStart=/opt/myapp/bin/server
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The three sections:

• [Unit] — metadata and dependencies. Same fields for every unit type.
• [<Type>] — the type-specific section: [Service] for services, [Timer] for timers, [Mount] for mounts, etc.
• [Install] — how this unit is enabled. Read only by systemctl enable.

Unit files live in three locations, in priority order (higher wins):

1. /etc/systemd/system/ — your local configuration.
2. /run/systemd/system/ — runtime-generated (do not edit).
3. /usr/lib/systemd/system/ (or /lib/systemd/system/) — shipped by packages.

If nginx.service exists in both /etc/systemd/system/ and /usr/lib/systemd/system/, the one in /etc/ wins completely. This is the "override by copying" pattern — but there is a better way (drop-ins, below).

---

The Unit Types You Will Use

Extension	What it is	Example
.service	A process to run and supervise	nginx.service, postgresql.service
.socket	A socket that activates a service on demand	ssh.socket, docker.socket
.timer	A scheduled trigger (cron replacement)	backup.timer, logrotate.timer
.mount / .automount	A filesystem mount (replaces fstab on modern distros)	var-log.mount, home.automount
.target	A grouping of units, like a milestone	multi-user.target, graphical.target
.path	Activates a service when a file/path changes	certwatch.path
.device	Kernel-detected devices (auto-created by udev)	dev-sda.device
.slice	A cgroup for resource accounting	user.slice, system.slice
.scope	A cgroup created for an existing group of processes	Containers, user sessions

You spend 90% of your time with services, targets, and occasionally timers and sockets.

# List every unit, any type
systemctl list-units --all | head -20

# Just the services
systemctl list-units --type=service
systemctl list-units --type=service --state=failed   # only failed ones

# Every unit file known to systemd (installed on disk)
systemctl list-unit-files | head -20

# See a specific unit's current status
systemctl status nginx.service
# ● nginx.service - A high performance web server and a reverse proxy server
#      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
#      Active: active (running) since Fri 2026-04-19 09:00:32 UTC; 2h 30min ago
#        Docs: man:nginx(8)
#    Process: 12300 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
#    Process: 12301 ExecStart=/usr/sbin/nginx (code=exited, status=0/SUCCESS)
#      Main PID: 12302 (nginx)
#       Tasks: 9 (limit: 38312)
#      Memory: 8.4M
#         CPU: 120ms
#      CGroup: /system.slice/nginx.service
#              ├─12302 "nginx: master process /usr/sbin/nginx"
#              ├─12303 "nginx: worker process"
#              └─...

systemctl status packs a lot in one screen: load state, active state, start time, the process tree, its cgroup, recent log lines. Learn to read it.

---

Targets: The Milestones of Boot

A target is a named set of units that should be active together. They are systemd's version of runlevels, but more flexible — a unit can be WantedBy many targets, and targets can depend on other targets.

The important ones:

Target	What it means
default.target	The "goal" of the boot. A symlink to one of the others.
multi-user.target	Non-graphical production server: network, logs, sshd, services.
graphical.target	Desktop: depends on multi-user + display manager.
basic.target	Filesystems mounted, swap on, sockets listening. Not yet running daemons.
sysinit.target	Early boot: kernel modules, basic mounts. Predecessor of basic.target.
network.target	Networking configured. Not necessarily online. Most services should not depend on this alone.
network-online.target	Networking actually up and able to reach other hosts.
rescue.target	Single-user mode: root shell, minimal services.
emergency.target	Even more minimal: root shell, root read-only.
shutdown.target / poweroff.target / reboot.target	Shutdown paths.

# What is my system booting to?
systemctl get-default
# multi-user.target

# Change it (persistent)
sudo systemctl set-default multi-user.target

# Jump to a target right now (switches modes without rebooting)
sudo systemctl isolate multi-user.target   # drop graphical, go to multi-user
sudo systemctl isolate rescue.target       # drop to rescue
sudo systemctl rescue                       # shortcut for the above

# What depends on multi-user.target?
systemctl list-dependencies multi-user.target

# The full dependency tree, expanded
systemctl list-dependencies --all

Targets are how you group services without manually adding them to a start-up list — you mark a service as WantedBy=multi-user.target, and systemd starts it whenever multi-user.target is pulled in (which is almost always, unless you drop to rescue).

---

The Dependency Vocabulary

The [Unit] section of a unit file has a handful of dependency-defining keywords. Knowing them prevents 90% of "my service does not start in the right order" problems.

Keyword	Meaning
Requires=	Hard requirement: if the required unit fails to start, this unit fails too. If the required unit stops, this unit stops.
Wants=	Soft requirement: try to start the wanted unit, but do not fail this one if it fails. Most common choice.
Requisite=	Like Requires, but the required unit must already be active — does not start it.
BindsTo=	Stronger than Requires: if the bound unit goes away for any reason, this unit follows.
PartOf=	Propagates stop/restart downward: when the parent stops or restarts, children go too.
Conflicts=	Mutual exclusion: starting this unit stops the conflicting one.
After=	Start ordering: this unit starts after the other (but only if both are starting).
Before=	Start ordering: this unit starts before the other.

The key insight: ordering (After/Before) and requirement (Requires/Wants) are independent.

• After=postgresql.service alone means "if postgres is also starting, start me after it" — but does not pull in postgres.
• Wants=postgresql.service alone means "start postgres alongside me" — but no ordering, so my service might start before postgres is ready.
• Both together are the common case: "I need postgres, and I need to start after it."

---

Drop-In Overrides: The Right Way to Customize

You want to tweak the shipped nginx.service unit — add an environment variable, change the Restart policy. The wrong way is to copy the whole file to /etc/systemd/system/ and edit it. The right way is a drop-in.

# Open an editor on a drop-in override
sudo systemctl edit nginx.service
# Opens /etc/systemd/system/nginx.service.d/override.conf in your $EDITOR

Write only the changes:

[Service]
Environment="NGINX_WORKER_PROCESSES=8"
Restart=always
RestartSec=3s

Save. systemd merges this file on top of the distro-shipped unit. When the distro updates nginx.service, your override continues to apply on top.

# See the effective, merged unit
systemctl cat nginx.service
# # /lib/systemd/system/nginx.service   <- distro ships
# [Unit]
# Description=...
#
# # /etc/systemd/system/nginx.service.d/override.conf   <- your drop-in
# [Service]
# Environment="NGINX_WORKER_PROCESSES=8"
# Restart=always

# Apply changes without restarting the service
sudo systemctl daemon-reload

# Verify the override took effect
systemctl show nginx.service -p Restart,Environment

---

Timers: The Cron Replacement

Timers are systemd units that trigger other units (usually services) on a schedule. They replace cron for most purposes and integrate with journald logging out of the box.

# /etc/systemd/system/daily-backup.timer
[Unit]
Description=Daily backup

[Timer]
OnCalendar=daily
Persistent=true                 # run at next boot if the system was off when it should have fired
RandomizedDelaySec=5min         # stagger fleets

[Install]
WantedBy=timers.target

And the matching service:

# /etc/systemd/system/daily-backup.service
[Unit]
Description=Daily backup job

[Service]
Type=oneshot
ExecStart=/usr/local/bin/do-backup.sh

Enable and start the timer (not the service — the timer triggers the service):

sudo systemctl daemon-reload
sudo systemctl enable --now daily-backup.timer

# See all timers
systemctl list-timers --all
# NEXT                         LEFT     LAST                         PASSED   UNIT
# Sat 2026-04-20 00:00:00 UTC  13h left Fri 2026-04-19 00:00:32 UTC  10h ago  daily-backup.timer
# Sat 2026-04-20 05:30:00 UTC  18h left Fri 2026-04-19 05:30:23 UTC  5h ago   logrotate.timer

Timer advantages over cron:

• Journalctl integration: every run's output goes to journald, queryable with journalctl -u daily-backup.service.
• Missed runs: Persistent=true means runs that were missed (system was off) fire on next boot.
• Randomized delay: RandomizedDelaySec= prevents the classic "100 servers all hit the backup server at exactly 00:00".
• Complex schedules: OnCalendar=Mon,Fri 03:00 and similar are much clearer than cron's 0 3 * * 1,5.
• Dependencies: you can Requires= another unit — the job will not fire if the dep is down.

---

systemctl: The Verbs You Use Every Day

# Service lifecycle
sudo systemctl start UNIT     # start now
sudo systemctl stop UNIT      # stop now
sudo systemctl restart UNIT   # stop + start
sudo systemctl reload UNIT    # SIGHUP (or whatever the unit defines as reload)
sudo systemctl reload-or-restart UNIT  # reload if supported, else restart

# Enable at boot / disable at boot
sudo systemctl enable UNIT        # create symlink so it starts at boot
sudo systemctl enable --now UNIT  # enable AND start now
sudo systemctl disable UNIT
sudo systemctl disable --now UNIT # disable AND stop now
sudo systemctl mask UNIT          # completely disable (symlink to /dev/null) — cannot even be started manually
sudo systemctl unmask UNIT        # undo mask

# Status and inspection
systemctl status UNIT
systemctl is-active UNIT           # active / inactive / failed
systemctl is-enabled UNIT          # enabled / disabled / masked / static
systemctl show UNIT                # every setting, machine-readable
systemctl show UNIT -p Restart     # just one property
systemctl cat UNIT                 # the unit file and drop-ins

# System-wide
systemctl list-units
systemctl list-units --failed
systemctl list-dependencies UNIT
systemctl --failed

# Reload systemd itself after editing unit files
sudo systemctl daemon-reload

---

Why systemd Won (and What It Costs)

The case for systemd:

• Parallel boot. Fleet nodes that used to take 90 seconds to SysV-boot boot in 15 with systemd.
• Supervision for free. Restart=on-failure replaces dozens of supervisord configs.
• Cgroup isolation by default. Every service runs in its own cgroup, and resource limits are one line in the unit file.
• Unified logging. journald collects stdout, stderr, and syslog into one queryable store.
• Socket and path activation. Services that only run when needed save memory on small systems.
• Declarative. No more 800-line init scripts.

The case against:

• Huge surface area. systemd is a lot of software. If you do not like "one binary to rule them all," this is the opposite.
• Tight coupling. journald, logind, resolved, networkd, udev — they interact, and debugging across them requires knowing all of them.
• Opinionated about PID 1. systemd pulls in specific behaviors (cgroup delegation, D-Bus interactions) that not everyone wants.

In practice, every mainstream Linux distribution ships systemd, and "I do not use systemd" is usually a choice that costs you compatibility and easy deployment automation. Learning it well is a better investment than avoiding it.

---

Key Concepts Summary

• systemd is an init system + service manager + tool suite. One process tree, one config format, one set of commands.
• Everything is a unit. Services, timers, sockets, mounts, targets, slices — same concept, different type.
• Unit files live in three directories; /etc/systemd/system/ overrides the distro. Drop-ins (systemctl edit) override cleanly without stomping on distro updates.
• Targets are groupings of units. multi-user.target is the typical server default; graphical.target adds a desktop.
• Dependencies have two dimensions: requirement (Wants/Requires) and ordering (After/Before). They are independent — use both together when you need both.
• Drop-ins are the right way to customize shipped units. systemctl edit and systemctl cat are the daily tools.
• Timers are better cron. Journalctl integration, missed-run replay, randomized delay, real dependency relationships.
• systemctl status, list-units, list-dependencies, show, cat are the inspection tools. daemon-reload after any unit file change.

Common Mistakes

• Editing shipped unit files in /usr/lib/systemd/system/ directly. Package upgrades overwrite your changes. Use drop-ins instead.
• Forgetting systemctl daemon-reload after changing a unit file. Your edits exist on disk, but systemd is still running the old config.
• Relying on After=network.target for "wait until I have internet." That target means "networking is configured," not "networking is online." Use network-online.target (and set up wait-online appropriately).
• Using Requires= everywhere. A hard requirement means a chained failure — one flap and half your services go down together. Prefer Wants= unless you genuinely want the all-or-nothing behavior.
• Enabling with systemctl start and wondering why the service is gone after reboot. start is "now"; enable is "at boot." Use enable --now for both.
• Grepping ps for your service name instead of systemctl status myapp. systemd already tracks the whole process tree via cgroups.
• Writing a timer that just ExecStart=/path/to/thing without a matching service unit. Timers trigger services; write both.
• Leaving RandomizedDelaySec= at 0 on fleet-wide timers. All your nodes running backups simultaneously at 03:00 is a self-inflicted thundering herd.
• Masking a unit with systemctl mask and forgetting about it. is-enabled returns masked — but troubleshooters who do not know to look there spend hours wondering why systemctl start refuses.

---