Debugging Dependencies in CircleCI: Patching node_modules, SSH Access, and Dependency Pinning — Blog

Instrumenting third-party code during CI runs

When a test fails due to behavior inside a dependency you can't modify (private library, vendored code), adding debug logging to the dependency during the CI job can surface what values are being passed. This is a diagnostic technique, not a production pattern.

Using sed to insert a debug method into an installed package:

npm install

# Insert a getter method after a specific comment in a library file
sed -i '/\/\/ Query flight data/a \
getHeader() {\
    return this.#headers\
}\
' node_modules/lib-js-orx/src/client/client.js

# Verify the modification worked
grep -A 5 "getHeader" node_modules/lib-js-orx/src/client/client.js

This modifies the installed file in the CI environment only — the change doesn't persist beyond the job and doesn't affect other runs unless it's cached. If you're using save_cache for node_modules, exclude modified packages or use a different cache key to avoid poisoning future runs.

Patching cached node_modules persists the modification to subsequent runs

GotchaCircleCI

If node_modules is cached and a job modifies a file inside it, the modification is saved into the cache on the next save_cache step. Future runs restore the modified cache and inherit the patch — which is almost never what you want for a debug instrumentation. Always invalidate the cache after a debugging session, or use a cache key that excludes the patched run.

Prerequisites

CircleCI caching
npm package structure

Key Points

Modifications to node_modules in a job are included in save_cache if the path is node_modules.
Change the cache key version (v1 → v2) after debugging to force a clean install.
npm ci always deletes and reinstalls node_modules — use it to avoid stale cache issues.
patch command is more robust than sed for complex multi-line modifications.

SSH access for interactive debugging

CircleCI allows re-running failed jobs with SSH access enabled. This opens an SSH connection to the build container, letting you inspect the environment interactively:

# In the CircleCI UI: click "Rerun" → "Rerun job with SSH"
# CircleCI shows an SSH command like:
ssh -p 64535 54.161.xxx.xxx

Inside the SSH session:

# Check what's installed
cat node_modules/some-lib/package.json | jq '.version'

# Run a specific test interactively
npx jest tests/specific.test.js --verbose

# Check environment variables (non-secret)
env | grep NODE

# Inspect the filesystem
ls -la /tmp/

The SSH session persists for 10 minutes after the final job step completes, or 2 hours from job start — whichever comes first. For longer debugging sessions, add a sleep step at the end of the job:

- run:
    name: Hold for debugging
    command: sleep 1800   # 30 minutes
    when: on_fail          # only run when a previous step failed

📝Using patch files for reproducible dependency modifications

For modifications that need to be consistently applied (not just for debugging), patch is more reliable than sed. Create a patch file from a diff:

# Generate a patch file from your local modification
cd node_modules/some-lib
git diff > /tmp/my-patch.diff

# Or create a unified diff manually
diff -u original.js modified.js > fix.patch

Apply the patch in CircleCI:

- run:
    name: Apply library patch
    command: |
      patch node_modules/some-lib/src/client.js < patches/some-lib-fix.patch

The patch file lives in the repository under patches/. This approach is version-controlled, reproducible, and visible in code review. For npm packages specifically, the patch-package tool automates this:

# After modifying node_modules/some-lib
npx patch-package some-lib
# Creates patches/some-lib+1.2.3.patch

# Add to package.json scripts:
"postinstall": "patch-package"

patch-package re-applies patches automatically after every npm install, making the modification permanent across all environments.

You use sed to add debug logging to a file in node_modules during a CircleCI job. The job caches node_modules with save_cache. On the next run, the debug logging appears in production test output even though the sed command isn't in the config anymore. Why?

medium

The debugging session was on a branch. The branch merged to main. The main branch pipeline restored the cache from the feature branch run and uses the same cache key.

Ased modifications persist on the CircleCI executor between runs
Incorrect.CircleCI spins up a fresh executor for each job. Modifications made in one job don't affect the next job's executor — they're isolated containers.
BThe modified node_modules directory was saved to the cache after the sed patch — subsequent runs restore the patched version because the cache key hasn't changed
Correct!save_cache captures the directory state at save time. The sed command ran before save_cache, so the patch was included in the cached archive. Any run that restores this cache key gets the patched node_modules. Fix: bump the cache key version (v1 → v2) to force a clean npm ci install that produces unmodified node_modules. Going forward, add the sed step after restore_cache and before save_cache only when explicitly debugging, or use npm ci which always deletes and reinstalls node_modules from scratch.
CCircleCI propagates cache from feature branches to main automatically
Incorrect.CircleCI caches are scoped to the repository, not the branch. A cache written with a specific key is accessible from any branch. The key itself doesn't encode the branch unless you include {{ .Branch }} in it.
DThe sed modification wrote to the package registry, affecting all future installs
Incorrect.sed modifies local files only — it has no effect on npm registries or package sources.

Hint:What is the state of node_modules when save_cache runs, and how long does a CircleCI cache persist?