Working with Files and Paths

Filesystems are messier than most scripts assume. Filenames contain spaces. Sometimes newlines. Sometimes emoji. Paths contain ... Two processes try to write the same file at the same moment. The disk fills up mid-write. Any of these will break a naive Bash script.

This lesson is about the specific techniques for writing scripts that survive real filesystems — safe temporary files, atomic writes, lock files, and the patterns for iterating over filenames that might contain anything.

---

Temporary files — always use mktemp

The wrong way:

# RACE CONDITION + PREDICTABLE NAME
tmp=/tmp/myapp.tmp
echo "data" > "$tmp"
# Attacker in /tmp can pre-create /tmp/myapp.tmp as a symlink to /etc/passwd.
# Your script just overwrote /etc/passwd.

The right way:

tmp=$(mktemp)
echo "data" > "$tmp"

mktemp creates a file with a guaranteed-unique name in a safe location, with safe permissions (0600 by default). Race-free.

For a temporary directory:

tmpdir=$(mktemp -d)
# ... use $tmpdir ...

For a custom template (prefix for readability):

tmp=$(mktemp --suffix=.log myapp.XXXXXX)
# GNU syntax: --tmpdir and --suffix

# BSD / macOS syntax
tmp=$(mktemp -t myapp)

Always pair with a trap

tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

# ... use $tmp ...

Covered in the Traps lesson. Without the trap, Ctrl-C or any unexpected exit leaves the temp file behind.

---

Atomic writes — the mv pattern

You want to update a file. A naive write leaves a window where the file is half-written:

# BAD — reader can see partial content
generate_content > /etc/myapp/config.yml

The fix: write to a temp file, then rename:

tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

generate_content > "$tmp"
mv "$tmp" /etc/myapp/config.yml

mv within the same filesystem is atomic. A reader sees either the old file or the new file — never a half-written one.

mktemp in the target directory

target="/etc/myapp/config.yml"
tmp=$(mktemp "$target.XXXXXX")   # temp file in the same dir as target
trap 'rm -f "$tmp"' EXIT

generate_content > "$tmp"
mv "$tmp" "$target"

Now you're guaranteed the mv is atomic.

---

Lock files — preventing concurrent runs

Two common problems:

1. Cron fires the script again while the previous invocation is still running.
2. Two humans launch the same deploy script simultaneously.

The fix is a lock file + flock:

#!/usr/bin/env bash
set -euo pipefail

lockfile="/var/lock/myapp.lock"

exec 200>"$lockfile"
if ! flock -n 200; then
  echo "another instance is running" >&2
  exit 1
fi

# ... work ...

# Lock auto-released when script exits

Pieces:

• exec 200>"$lockfile" opens file descriptor 200 writing to the lock file. The descriptor stays open for the life of the script.
• flock -n 200 tries to acquire an exclusive lock on fd 200. -n = non-blocking (fail immediately if locked).
• When the script exits (normally or abnormally, even from SIGKILL), the OS closes the fd, which releases the lock.

Why flock and not "check if file exists"

A "check-then-create" lock:

# RACE CONDITION
if [[ -f "$lockfile" ]]; then
  exit 1
fi
touch "$lockfile"    # another process could have taken it between check and create
trap 'rm -f "$lockfile"' EXIT

Two processes can both see "file doesn't exist" at the same time; both create the lock; both proceed. flock avoids this because locking and acquiring are a single atomic OS call.

Blocking vs non-blocking

flock -n 200         # fail immediately if locked
flock 200            # wait forever
flock -w 30 200      # wait up to 30 seconds

Pick based on use case. Cron jobs typically use -n (skip this run). Interactive deploys might use -w 60 (wait a minute, then bail).

---

Handling filenames safely

Filenames can contain anything except / and null. In particular:

• Spaces (my file.txt)
• Newlines (my\nfile.txt) — yes, really
• Starting with - (-rf) — looks like a flag to most tools
• Emoji, non-ASCII characters
• Escape sequences (my\tfile)

The -- convention

Tools that might interpret a filename as a flag:

# DANGEROUS if filename starts with -
rm "$file"

# SAFE — -- tells the tool "everything after this is positional, not a flag"
rm -- "$file"

Always -- when the filename comes from user input. Not every tool supports it (some do; it's a convention). But for common commands (rm, cp, mv, grep, head, tail), yes.

Iterating over files safely (revisited)

For a known directory with safe filenames, a glob is fine:

for f in /path/to/dir/*; do
  process "$f"
done

For potentially hostile filenames (newlines, etc.), use null-delimited reads:

while IFS= read -r -d '' f; do
  process "$f"
done < <(find /path -type f -print0)

See the Loops lesson for more on this.

---

Path manipulation

basename and dirname — the Bash way

path="/home/alice/documents/report.pdf"

# Via external commands (slow)
dir=$(dirname "$path")
file=$(basename "$path")

# Pure bash (fast)
dir="${path%/*}"        # /home/alice/documents
file="${path##*/}"       # report.pdf
base="${file%.*}"        # report
ext="${file##*.}"        # pdf

Parameter expansion is orders of magnitude faster than forking basename/dirname. For one-off scripts it doesn't matter; in a loop over 10,000 paths, it matters a lot.

Canonicalizing a path

# realpath resolves symlinks and relative paths
full=$(realpath "$input")
# or (GNU readlink)
full=$(readlink -f "$input")

# To get the directory a script is in
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)

realpath is sometimes missing on minimal BusyBox systems. readlink -f is GNU-specific (not on macOS default). A portable alternative:

abs_path() {
  cd "$(dirname "$1")" && echo "$PWD/$(basename "$1")"
}

Ensuring a directory exists

# Fails if the dir doesn't exist
cp file "$target_dir/"

# Make sure it exists first
mkdir -p "$target_dir"
cp file "$target_dir/"

mkdir -p is idempotent — doesn't fail if the directory already exists, and creates parents as needed.

---

Copying and moving safely

Preserving attributes

cp file1 file2                 # basic copy
cp -p file1 file2              # preserve mode, ownership, timestamps
cp -a src/ dst/                # archive mode — recursive + preserve everything

For backups, cp -a is usually what you want. For "just make it readable," plain cp is fine.

Recursive copies with a trailing slash gotcha

cp -r src/  dst/    # copies src's CONTENTS into dst/
cp -r src   dst/    # copies src itself into dst/, becoming dst/src

Check your trailing slashes. This is a common cause of "why is there now a src directory inside dst?"

rsync — the better cp for anything non-trivial

# Mirror src into dst, deleting anything in dst that's not in src
rsync -av --delete src/ dst/

# Copy only newer files, show progress
rsync -avP src/ dst/

Rsync handles large trees, resumes, preserves permissions correctly, and is generally safer than cp for production. Learn the flags.

---

Checking disk space before writes

Disk full halfway through a write is ugly:

# Bytes free on the target filesystem
free_bytes=$(df --output=avail -B1 "$target" | tail -1)

if (( free_bytes < 1024 * 1024 * 100 )); then   # less than 100MB
  echo "error: less than 100MB free on target" >&2
  exit 1
fi

df --output=avail -B1 is GNU-specific. macOS needs different flags. For portability:

free_bytes=$(df -P "$target" | awk 'NR==2 {print $4 * 1024}')

---

Reading files — a few patterns

Whole file into variable

content=$(< file.txt)   # trailing newlines stripped
# or
content=$(cat file.txt) # slower (uses subprocess)

$(<file) is faster because it's pure Bash — no fork.

Whole file into array (lines)

mapfile -t lines < file.txt

for line in "${lines[@]}"; do
  echo "$line"
done

Line-by-line (for large files)

while IFS= read -r line; do
  process "$line"
done < file.txt

Covered in the Loops lesson. For files bigger than memory, always use while read rather than slurping.

---

Writing files atomically

Beyond the mktemp + mv pattern, a few more idioms:

Appending with locking (for log files)

If multiple processes append to the same log:

exec >> "$log" 2>&1
# Writes from multiple processes can interleave. For small writes (<PIPE_BUF),
# the kernel guarantees atomic append. For larger writes, serialize with flock:

(
  flock 200
  echo "big multiline message"
  cat another_file
) 200>>"$log"

Checksum-verified writes

For critical data:

tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT

generate_content > "$tmp"

expected=$(sha256sum "$tmp" | cut -d' ' -f1)
mv "$tmp" "$target"

actual=$(sha256sum "$target" | cut -d' ' -f1)
[[ "$expected" == "$actual" ]] || { echo "integrity check failed" >&2; exit 1; }

Paranoid but occasionally useful — e.g. over flaky NFS mounts.

---

Common file-handling mistakes

Mistake 1: not using mktemp

# BROKEN
tmp=/tmp/myapp-$$.tmp   # $$ is predictable; attacker can race you
echo "data" > "$tmp"

Mistake 2: forgetting mv atomicity requires same filesystem

# BROKEN on many systems
tmp=$(mktemp)           # probably in /tmp
mv "$tmp" /var/myapp/state.json    # if /tmp and /var differ, not atomic

Mistake 3: rm -rf "$dir" when $dir could be empty

# With -u this errors; without -u, silently rm -rf /
rm -rf "$prefix/$dir"

Always set -u. Always explicit defaults. [[ -n "$dir" ]] before any rm -rf.

Mistake 4: assuming filenames are ASCII

# BROKEN if filename contains a newline
ls dir/ | while read f; do
  process "$f"
done

Use null-delimited find instead.

Mistake 5: Ignoring symbolic links

# If $dir is a symlink, rm -rf follows it and deletes the target's contents
rm -rf "$dir"

Guard with [[ -L "$dir" ]] or realpath if the difference matters.

---

A real-world example — atomic config update

#!/usr/bin/env bash
set -euo pipefail

target="/etc/myapp/config.yml"
lockfile="/var/lock/myapp-config.lock"

# Prevent concurrent updates
exec 200>"$lockfile"
flock -n 200 || { echo "config update already in progress" >&2; exit 1; }

# Create the new file atomically
tmp=$(mktemp "$target.XXXXXX")
trap 'rm -f "$tmp"' EXIT

generate_config > "$tmp"

# Validate before committing
validate_config "$tmp" || { echo "generated config is invalid" >&2; exit 1; }

# Preserve permissions of existing file
if [[ -f "$target" ]]; then
  chmod --reference="$target" "$tmp"
  chown --reference="$target" "$tmp"
fi

# Commit
mv "$tmp" "$target"
trap - EXIT     # cancel cleanup; file is no longer temp

echo "config updated: $target"

Every pattern from this lesson: mktemp in the target's dir, atomic mv, lock file, validation before commit, preserving permissions, cleanup trap cancelled on success.

---

Quiz

---

What to take away

• Temporary files: always mktemp. Predictable names are race conditions waiting to happen.
• Atomic writes: mktemp + write + mv, with the temp file in the same directory as the target so mv is on the same filesystem.
• Lock files: use flock with an exclusive lock on a file descriptor. Don't use check-then-create.
• Filenames can contain anything. Iterate with null-delimited find | read -d '' when safety matters.
• -- separator to prevent filenames-starting-with-- from being interpreted as flags.
• Parameter expansion beats basename/dirname for speed in loops.
• rsync -a is better than cp -r for non-trivial copies.
• $(<file) is the fastest way to slurp a whole file into a variable.

Next lesson: structuring larger scripts — organization, sourcing libraries, and knowing when it's time to switch to Python or Go.