CodeWisdom

CircleCI Caching: Keys, Invalidation, and Dependency Layer Strategy

•2 min read•Cloud Infrastructure

CircleCI caching stores build dependencies between runs to avoid repeated downloads. Cache keys control when the cache is invalidated. A bad key strategy either never invalidates (stale dependencies) or always invalidates (no speedup). The correct pattern layers file checksums and fallback keys to balance freshness and hit rate.

ci-cdcircleci

How CircleCI caching works

CircleCI caching persists a directory (e.g., node_modules, ~/.gradle) across workflow runs. On the first run, save_cache writes the directory to CircleCI's cache storage with a key. On subsequent runs, restore_cache checks for a matching key and restores the directory if found.

Cache hits skip npm install or gradle build dependency downloads. Cache misses run the full install and save again.

Cache keys determine invalidation — a key that never changes creates a permanently stale cache

GotchaCircleCI

CircleCI cache keys are strings evaluated at runtime. The key's content determines when the cache is invalidated. A static key like 'node-modules-v1' never invalidates — you'll restore a cached node_modules from six months ago even after updating package.json. A checksum of the lockfile invalidates the cache whenever dependencies change.

Prerequisites

  • CircleCI workflows and jobs
  • npm/yarn dependency management

Key Points

  • {{ checksum 'package-lock.json' }} generates a hash of the lockfile — changes when dependencies change.
  • Cache keys are immutable once written — updating a key saves a new cache, old caches are not overwritten.
  • restore_cache tries keys in order and uses the first match — fallback keys catch partial hits.
  • Caches persist for 15 days without a hit before CircleCI evicts them.

Cache key strategy

version: 2.1

jobs:
  build:
    docker:
      - image: cimg/node:20.10
    steps:
      - checkout

      - restore_cache:
          keys:
            # Primary key: exact match on lockfile
            - node-modules-v1-{{ checksum "package-lock.json" }}
            # Fallback: any cache from this branch (partial hit)
            - node-modules-v1-{{ .Branch }}-
            # Final fallback: any node-modules cache
            - node-modules-v1-

      - run:
          name: Install dependencies
          command: npm ci

      - save_cache:
          key: node-modules-v1-{{ checksum "package-lock.json" }}
          paths:
            - node_modules

      - run:
          name: Build
          command: npm run build

The fallback key pattern matters: after a cache miss on the exact key, CircleCI restores the most recent cache matching the fallback prefix. npm ci then installs only changed packages — faster than a full install from scratch.

npm ci instead of npm install is important here: npm ci installs from lockfile and deletes node_modules first if it exists. On a partial cache hit, npm ci cleans up stale entries and adds new packages, keeping the cache valid.

Invalidating the cache manually

CircleCI caches are immutable — you can't update an existing cache. To force a full re-download, change the cache key prefix:

# Bump v1 → v2 to invalidate all existing caches
- restore_cache:
    keys:
      - node-modules-v2-{{ checksum "package-lock.json" }}
      - node-modules-v2-

- save_cache:
    key: node-modules-v2-{{ checksum "package-lock.json" }}
    paths:
      - node_modules

Common reasons to bump the version:

  • A corrupted cache causing intermittent test failures
  • Changing Node.js version (cached node_modules from Node 18 may break on Node 20)
  • Adding native addons that compiled against a different glibc
📝Caching for different package managers and build tools

Yarn:

- restore_cache:
    keys:
      - yarn-v1-{{ checksum "yarn.lock" }}
      - yarn-v1-
- run: yarn install --frozen-lockfile
- save_cache:
    key: yarn-v1-{{ checksum "yarn.lock" }}
    paths:
      - ~/.cache/yarn   # yarn global cache, not node_modules

Python pip:

- restore_cache:
    keys:
      - pip-v1-{{ checksum "requirements.txt" }}
      - pip-v1-
- run: pip install -r requirements.txt
- save_cache:
    key: pip-v1-{{ checksum "requirements.txt" }}
    paths:
      - ~/.cache/pip

Go modules:

- restore_cache:
    keys:
      - go-mod-v1-{{ checksum "go.sum" }}
      - go-mod-v1-
- run: go mod download
- save_cache:
    key: go-mod-v1-{{ checksum "go.sum" }}
    paths:
      - /home/circleci/go/pkg/mod

Cache the package manager's global cache directory, not the project-local directory. This allows multiple projects using the same package to share cache entries.

Cache vs workspace vs artifacts

| | Cache | Workspace | Artifacts | |---|---|---|---| | Scope | Across workflow runs | Within one workflow | Stored after workflow | | Mutable | No (immutable per key) | Yes (jobs can append) | No | | Use case | Reuse dependencies | Pass build outputs between jobs | Download results, test reports | | Retention | 15 days per hit | Workflow lifetime (+ 15 days for rerun) | 30 days (configurable) |

Use cache for dependencies. Use workspace when job B needs the compiled output from job A. Use artifacts for human-readable outputs (test reports, binaries for download).

Your CircleCI build is caching node_modules with the key 'node-modules-v1-{{ checksum 'package-lock.json' }}'. After updating a dev dependency, the cache is restored from the old key even though package-lock.json changed. Why?

medium

The package-lock.json was modified locally and committed. The CI job restores a cache from before the update. npm ci still installs correctly but the cache restore step shows a hit.

  • ACircleCI uses a stale version of package-lock.json for the checksum because checkout runs after restore_cache
    Incorrect.If checkout correctly runs before restore_cache, this isn't the issue. But if steps are reordered, it could be — see the correct answer.
  • Brestore_cache ran before checkout — it checksummed the old package-lock.json from the previous checkout, found the old cache key, and restored it
    Correct!CircleCI evaluates {{ checksum 'package-lock.json' }} at the time restore_cache runs. If restore_cache appears before the checkout step in your config, the lockfile on disk is from the previous run (or doesn't exist), so the checksum matches the old cache key. Always place checkout before restore_cache. Once correctly ordered, the checksum of the updated lockfile won't match the old key — restore_cache will use the fallback key and npm ci will install the updated dependencies.
  • CThe checksum function only considers the file size, not content
    Incorrect.CircleCI's checksum function computes a full SHA256 of the file content. It detects any content change, including a single dependency version bump.
  • Dpackage-lock.json changes don't affect the checksum when only devDependencies change
    Incorrect.CircleCI's checksum is a hash of the entire file content. Any change to package-lock.json — including devDependencies — changes the hash and therefore the cache key.

Hint:When in the job does CircleCI evaluate the {{ checksum }} template — and where is checkout relative to restore_cache?