Shell Scripting: Process Substitution, PIPESTATUS, and Test Expressions

Process substitution: <(cmd)

Some commands require file arguments and can't read from stdin. Process substitution creates a temporary file descriptor that looks like a file:

# diff requires two file arguments — can't diff two strings directly
diff <(echo "hello world") <(echo "hello there")
# Output:
# 1c1
# < hello world
# ---
# > hello there

# Compare two command outputs
diff <(sort file1.txt) <(sort file2.txt)

# Run comm (set operations) on two lists
comm -23 <(sort list1.txt) <(sort list2.txt)  # lines only in list1

# Verify a downloaded file's checksum against a remote manifest
diff <(sha256sum downloaded.tar.gz) <(curl -s https://example.com/SHA256SUMS)

<(cmd) expands to a path like /dev/fd/63 — a file descriptor opened to the command's stdout. The command runs in a subshell. Write substitution >(cmd) also exists for commands requiring an output file path.

PIPESTATUS: capturing pipeline exit codes

$? captures the exit code of the last command in a pipeline. For the exit code of any earlier stage, use PIPESTATUS:

# Problem: $? only shows tee's exit code
grep "ERROR" application.log | tee errors.txt
echo $?  # exit code of tee, not grep

# Solution: PIPESTATUS array
grep "ERROR" application.log | tee errors.txt
echo "${PIPESTATUS[0]}"  # grep's exit code (1 if no match, 0 if found)
echo "${PIPESTATUS[1]}"  # tee's exit code

# Capture all at once
grep "pattern" file.txt | process | save
STATUSES=("${PIPESTATUS[@]}")
echo "grep: ${STATUSES[0]}, process: ${STATUSES[1]}, save: ${STATUSES[2]}"

PIPESTATUS is overwritten by the next command, so capture it immediately after the pipeline.

Bash's set -o pipefail makes the pipeline return the first non-zero exit code:

set -o pipefail
grep "pattern" file.txt | wc -l
# If grep exits 1 (no match), the pipeline exits 1 (not wc's 0)

grep exit code 1 means 'no match' — not an error — which breaks pipefail scripts that expect false-positives to be non-fatal

GotchaShell Scripting

grep uses exit code conventions: 0=match found, 1=no match, 2=error. In a shell script with set -o pipefail, a grep that finds no lines returns 1, which terminates the script with an error. This is correct for 'did the pattern exist?' checks but wrong for filtering pipelines where 'no matches' is a valid outcome. Fix: append || true to suppress the exit code, or check PIPESTATUS explicitly and treat exit code 1 as success for your use case.

Prerequisites

• Exit codes
• set -o pipefail
• grep usage

Key Points

• grep exits 0 (match found), 1 (no match), 2 (error). exit 1 is NOT an error — it means the pattern wasn't there.
• PIPESTATUS captures each pipeline stage's exit code; must be read immediately after the pipeline.
• set -o pipefail: pipeline fails if any stage fails — use carefully with grep or commands that return 1 legitimately.
• || true after a command makes the combined expression always succeed (useful in set -e scripts).

test command reference

test (or [ ]) evaluates expressions used in if, while, and until:

# File tests
if [ -f /etc/passwd ]; then echo "file exists"; fi
if [ -d /tmp ]; then echo "is directory"; fi
if [ -e /path ]; then echo "path exists (file or dir)"; fi
if [ -L /path ]; then echo "is symlink"; fi
if [ -r /path ]; then echo "readable"; fi
if [ -w /path ]; then echo "writable"; fi
if [ -x /path ]; then echo "executable"; fi
if [ -s /path ]; then echo "non-empty"; fi

# String tests
if [ -z "$var" ]; then echo "empty string"; fi
if [ -n "$var" ]; then echo "non-empty string"; fi
if [ "$a" = "$b" ]; then echo "strings equal"; fi
if [ "$a" != "$b" ]; then echo "strings not equal"; fi

# Integer comparison
if [ "$count" -eq 0 ]; then echo "equal"; fi
if [ "$count" -gt 5 ]; then echo "greater than"; fi
if [ "$count" -lt 5 ]; then echo "less than"; fi
if [ "$count" -ge 5 ]; then echo "greater or equal"; fi

# File comparison
if [ file1 -nt file2 ]; then echo "file1 is newer"; fi
if [ file1 -ot file2 ]; then echo "file1 is older"; fi
if [ file1 -ef file2 ]; then echo "same inode (hard links)"; fi

[[ ]] (double brackets) is a bash extension that supports &&, ||, pattern matching (=~), and unquoted variables safely:

if [[ "$str" =~ ^[0-9]+$ ]]; then echo "all digits"; fi
if [[ -f /path && -r /path ]]; then echo "readable file"; fi

A CI script uses `command | grep 'SUCCESS' | wc -l > count.txt` with `set -o pipefail`. The command always runs successfully but sometimes 'SUCCESS' isn't in the output. The script fails unexpectedly. Why?

medium

grep exits 1 when it finds no matches. set -o pipefail makes the pipeline exit with the first non-zero code.

• Awc -l fails when its input is empty

  Incorrect.wc -l accepts empty input fine and outputs '0'. wc always exits 0 on success.
• Bgrep exits 1 when 'SUCCESS' is not found. With set -o pipefail, the pipeline returns 1, which terminates the set -e script.

  Correct!grep uses exit code 1 to mean 'no matches found' — not an error in grep's context, but exit code 1 is non-zero. With set -o pipefail, the pipeline exit code is the first non-zero code from any stage. grep returning 1 causes the pipeline to exit 1, which set -e treats as an error and terminates the script. Fix: append `|| true` to the pipeline — `command | grep 'SUCCESS' || true | wc -l > count.txt` — but this doesn't work because || binds to grep. Better: `command | (grep 'SUCCESS' || true) | wc -l > count.txt`. Or restructure: capture grep output and check PIPESTATUS.
• CThe pipeline redirect to count.txt fails when the file doesn't exist

  Incorrect.Shell redirection creates the file if it doesn't exist. Redirect to a new file always succeeds (assuming write permission).
• Dset -o pipefail only affects the last command in a pipeline

  Incorrect.set -o pipefail makes the pipeline exit code equal to the rightmost non-zero exit code among all pipeline stages. If grep (middle stage) exits 1, the pipeline exits 1.

Hint:What exit code does grep return when it finds no matches? How does set -o pipefail treat that exit code?