The Debugging Toolkit

When a Bash script misbehaves in production, your options for debugging are limited. You cannot attach a debugger. You cannot set a breakpoint. You can barely inspect state mid-run without printing it.

What you can do — and do well, once you know the tools — is trace exactly what commands run and what values they see. set -x is the cornerstone; PS4, BASH_XTRACEFD, DEBUG traps, and selective tracing complete the toolkit.

---

set -x — the core tool

#!/usr/bin/env bash
set -x

name="alice"
echo "hello $name"

# Output:
# + name=alice
# + echo 'hello alice'
# hello alice

Every command is printed to stderr with a + prefix AFTER all expansions (variable substitution, globbing, word splitting). The output shows you what Bash actually ran — not what was typed.

Turning it on/off locally

set -x       # enable
risky_section
set +x       # disable

Wrap just the part you're debugging. Avoids flooding output with trace lines from uninteresting sections.

Tracing a function only

problem_function() {
  set -x
  # ... suspicious code ...
  set +x
}

Or with a trap:

problem_function() {
  trap 'set +x' RETURN   # turn off x when function returns
  set -x
  # ...
}

trap '...' RETURN fires when the current function returns. Useful for "trace everything in this function and nothing else."

---

PS4 — customize the trace prefix

The default + prefix is minimal. Customize with PS4 for real context:

export PS4='+ ${BASH_SOURCE}:${LINENO}: '
set -x

# Output:
# + ./script.sh:3: name=alice
# + ./script.sh:4: echo 'hello alice'

Now each trace line tells you the file and line number it came from.

A more useful PS4:

export PS4='+${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '
set -x

# Output:
# +script.sh:3:main: name=alice
# +script.sh:8:deploy: git pull

File, line, function. Reads nicely in big scripts.

Timestamps for long-running scripts

export PS4='+[$(date +%H:%M:%S)] ${BASH_SOURCE##*/}:${LINENO}: '
set -x

# Output:
# +[14:23:01] script.sh:5: slow_command
# +[14:23:45] script.sh:6: next_command

Each line now shows when it ran. Useful for "where did the 45 seconds go?" diagnosis.

---

BASH_XTRACEFD — send trace output somewhere else

By default, set -x writes to stderr. For long scripts this pollutes logs.

BASH_XTRACEFD lets you send trace output to a specific file descriptor:

#!/usr/bin/env bash

exec 3> /tmp/trace.log      # open fd 3 to the trace log
BASH_XTRACEFD=3
set -x

# All trace output goes to /tmp/trace.log, not stderr
# Normal stdout/stderr unaffected

Now stderr stays clean — users see just error messages, not trace output. The trace is preserved for post-mortem.

Conditional tracing

Useful pattern: trace only when DEBUG env var is set:

if [[ "${DEBUG:-}" == "true" ]]; then
  exec 3>>"$HOME/.myapp/debug.log"
  BASH_XTRACEFD=3
  export PS4='+[$(date +%H:%M:%S)] ${BASH_SOURCE##*/}:${LINENO}: '
  set -x
fi

Normal users get a clean run; set DEBUG=true and you get full trace output appended to a log file.

---

The DEBUG trap

trap '...' DEBUG fires before every command. You can use it to log, inspect state, or pause interactively:

trap '(( lineno != LINENO )) && { echo "line $LINENO: $BASH_COMMAND" >&2; lineno=$LINENO; }' DEBUG

# Before every command:
# line 5: some_command
# line 6: another
# line 7: yet_another

Less commonly useful than set -x, but a good tool when you want to print only line numbers, or dump specific variables before each command.

An interactive-style debugger

_debug_step() {
  echo "next: $BASH_COMMAND (line $LINENO)" >&2
  read -p "step [enter|q|vars]? " -r ans < /dev/tty
  case "$ans" in
    q) exit 0 ;;
    vars) ( set ) >&2 ;;   # dump all vars
  esac
}
trap _debug_step DEBUG

Now you can step through a script one command at a time, with the option to dump all variables or quit. Crude but functional.

---

Inspecting state mid-script

When tracing is too noisy, print specific things at specific points:

# Dump an array
echo "args: ${args[@]}" >&2
printf 'arg: [%s]\n' "${args[@]}" >&2

# Dump an associative array
for k in "${!config[@]}"; do
  echo "$k=${config[$k]}" >&2
done

# Dump the current command
caller   # prints the line and function of the caller

# Dump the full call stack
print_stack() {
  local i
  for (( i = 1; i < ${#FUNCNAME[@]}; i++ )); do
    echo "  at ${FUNCNAME[$i]} (${BASH_SOURCE[$i]}:${BASH_LINENO[$((i-1))]})" >&2
  done
}

caller and the FUNCNAME/BASH_SOURCE/BASH_LINENO arrays let you print a call stack like a real language.

---

The -v flag

set -v prints each line as it's read (BEFORE expansion), in contrast to -x (which prints AFTER expansion):

#!/usr/bin/env bash
set -v

name="alice"
echo "hello $name"

# Output:
# name="alice"        <- printed as-written
# echo "hello $name"  <- printed as-written, before $name expands
# hello alice         <- actual execution

Occasionally useful for checking "did my here-document actually include this line?" but -x is more informative for most debugging.

Combine them: set -xv shows both.

---

set -n — check syntax without executing

bash -n myscript.sh

Parses the script without running it. Useful in CI to catch syntax errors before deployment. Does not catch runtime errors or logical bugs — only syntax.

Run it in CI or pre-commit:

find scripts/ -name '*.sh' -exec bash -n {} \;

---

Putting it together — a debug-friendly script template

#!/usr/bin/env bash
set -euo pipefail

# --- debug support ---
if [[ "${DEBUG:-}" == "true" ]]; then
  # Send trace to a file, not stderr
  exec 3>>"$HOME/.$(basename "$0").trace"
  BASH_XTRACEFD=3
  export PS4='+[$(date +%H:%M:%S)] ${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: '
  set -x
fi

# --- error reporting ---
trap 'rc=$?; echo "FAIL at ${BASH_SOURCE##*/}:$LINENO: $BASH_COMMAND (exit $rc)" >&2' ERR

# --- your script ---
main() {
  # ...
}

main "$@"

DEBUG=true ./script.sh gives you full trace with timestamps, file/line/function prefix, written to ~/.script.sh.trace. Normal users see a clean run with good error messages.

---

Handy debugging tricks

Trick 1: print a variable's type

declare -p MYVAR
# Output examples:
# declare -- MYVAR="value"       (string)
# declare -a MYVAR=([0]="a")     (indexed array)
# declare -A MYVAR=([k]="v")     (associative array)
# declare -i MYVAR="42"          (integer)
# declare -r MYVAR="value"       (readonly)

Trick 2: dump all variables

set                    # all shell variables AND function definitions
env                    # only exported variables
declare -p             # all variables with their types
compgen -v             # just names of all variables
compgen -A function    # just names of all functions

Trick 3: test a regex interactively

# Does this regex work?
[[ "test-123" =~ ^test-[0-9]+$ ]] && echo match || echo no

# What did the captures see?
[[ "v1.2.3" =~ v([0-9]+)\.([0-9]+) ]] && echo "${BASH_REMATCH[@]}"

Trick 4: time a command

time some_command
# real  0m2.345s
# user  0m0.123s
# sys   0m0.456s

Trick 5: profile a script

# Run under time, with trace
time bash -x script.sh 2>trace.log
# Or for per-line timing
PS4='+[$(date +%s.%N)] ' bash -x script.sh 2>trace.log

Analyzing trace.log shows you which commands took the longest.

---

Debugging remote scripts

Scripts that run via SSH or in containers are harder to debug. Approaches:

Capture trace into the remote log

ssh host 'DEBUG=true bash -s' < script.sh
# Local DEBUG flag, remote trace to its own file

Use bash -x directly

ssh host 'bash -xs' < script.sh 2> local-trace.log

Ship the script to the remote, run with -x, capture trace locally via stderr redirect.

For containers

docker exec -it mycontainer bash -c 'set -x; /scripts/broken.sh'

Or use an ENTRYPOINT that checks for DEBUG env:

# Dockerfile
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]

# entrypoint.sh
[[ "${DEBUG:-}" == "true" ]] && set -x
exec "$@"

Now docker run -e DEBUG=true ... gives you trace from the start.

---

Interpreting traces

A sample trace:

+script.sh:5:main: name=$(hostname)
++script.sh:5:main: hostname
+script.sh:5:main: name=myhost
+script.sh:6:main: echo 'hello' myhost
hello myhost

Reading:

• + = one level deep (main shell).
• ++ = two levels deep (subshell — the $(hostname) substitution).
• After each substitution you see the resolved value: name=myhost.
• The echo line shows the exact arguments Bash passed.

Spotting bugs: if you see a command with unexpectedly empty or split arguments, you found your quoting issue.

---

Common debugging anti-patterns

Anti-pattern 1: print statements without stderr

# Broken — mixes debug with data output
my_func() {
  echo "DEBUG: entering my_func"   # goes to stdout
  echo "result_value"               # also stdout — caller captures BOTH
}

result=$(my_func)
echo "$result"
# DEBUG: entering my_func
# result_value

Fix:

my_func() {
  echo "DEBUG: entering my_func" >&2   # goes to stderr
  echo "result_value"
}

Always send debug output to stderr (>&2). Stdout is for data.

Anti-pattern 2: leaving set -x in production

Traces bloat logs. If DEBUG=false and some branch has set -x without a matching set +x, you pollute every run.

Use the conditional DEBUG=true pattern so tracing is off by default.

Anti-pattern 3: debugging with echo without context

echo "$x"

What is this printing? In a long script, unmoored echoes are useless. Always label:

echo "x after update: $x" >&2

---

Quiz

---

What to take away

• set -x is the cornerstone — shows every command after expansion.
• Customize PS4 for useful context: +${BASH_SOURCE##*/}:${LINENO}:${FUNCNAME[0]:-main}: .
• Add timestamps to PS4 for performance debugging.
• BASH_XTRACEFD=3 + exec 3>>trace.log sends trace output to a file, keeping stderr clean.
• Gate tracing on DEBUG=true env var so normal runs are quiet.
• set -n for syntax check without execution (useful in CI).
• declare -p VAR shows variable type and value.
• Print debug info to stderr (>&2), not stdout.
• FUNCNAME, BASH_SOURCE, BASH_LINENO let you print call stacks manually.

Next lesson: common production failures — shells other than bash, argument list too long, locale issues, CRLF line endings.