Tests pass locally but fail in CI after an AI agent's changes

Any AI coding tool Last checked 2026-10-26

What you saw

You run the test suite locally after the agent finishes a change and everything is green. You push. CI goes red, often on tests the agent never touched and that were passing before.

Re-running the same CI job sometimes passes on a retry with no code change — a sign the failure is non-deterministic, not a straightforward regression.

The CI log shows a different error than anything you can reproduce locally: a missing module, a different package version in a lockfile diff, a different file path separator, a different timezone offset in a date assertion, or tests failing only when run in a particular order or in parallel.

It is more common right after an agent run than after a human commit, because agents frequently add a new dependency, a new file, or a new environment read without updating the lockfile, the CI config, or test isolation at the same time.

What it actually means

'Passes locally' only proves the code works inside your current machine state: installed packages (including ones not pinned in the lockfile), cached build artifacts, environment variables, locale, timezone, and whatever test run order your test runner happened to use last. CI starts from a clean container every time, so any of those hidden dependencies that differ between your machine and the CI image will surface there first.

AI coding agents make this specific failure more likely for a few concrete reasons: they often pip install or npm install a package directly without the agent then running a full lockfile update and commit, they frequently add code that reads Date.now(), process.env, or the filesystem without the agent realizing the test environment may set those differently, and they sometimes generate a new test that passes in isolation but shares mutable global state (a database row, a temp file, a module-level cache) with an existing test — which only shows up when the suite runs in full, in CI's test order, not when you reran just the one new test file locally.

A second, separate cause is a lockfile or version-pin drift: the agent updated a dependency in package.json/requirements.txt but the lockfile (package-lock.json, poetry.lock, uv.lock) still points at the old resolved version, or vice versa. Locally you still have the old node_modules/venv on disk so nothing looks wrong; CI does a clean install from the lockfile and gets a different, sometimes incompatible, version.

A third cause, more subtle: the agent's diff looks correct in isolation but changed import order, a decorator, or a fixture's scope in a way that alters when a side effect runs. Test frameworks that rely on execution order (module-level mutable state, class-level fixtures with scope="session", global mocks that are not reset) are especially exposed.

The fix

1. First, reproduce the CI environment locally instead of trusting your machine. Run the suite in a clean container using the exact image CI uses — this alone confirms whether it is an environment difference, not a code bug.

   # Example: run the exact CI image locally
   docker run --rm -v $(pwd):/app -w /app node:20-slim bash -lc \
     "npm ci && npm test"
   
   # Python equivalent
   docker run --rm -v $(pwd):/app -w /app python:3.12-slim bash -lc \
     "pip install -r requirements.txt && pytest"

2. Check whether the agent changed a dependency without updating the lockfile. A diff here is the single most common root cause.

   # Node — confirm lockfile matches package.json exactly
   npm ci   # fails loudly if lockfile is out of sync, unlike npm install
   
   # Python (poetry)
   poetry check
   poetry lock --check
   
   # Python (pip + requirements.txt, no lock)
   pip install -r requirements.txt --dry-run

3. Run the suite with randomized or CI-matching test order locally to catch order-dependent failures before they surface only in CI.

   # pytest — install pytest-randomly, then:
   pytest -p randomly --randomly-seed=last
   
   # Jest — run with a fixed worker count of 1 to rule out parallel interference
   npx jest --runInBand
   
   # Once isolated, bisect: run just the agent's new test file alone,
   # then the full suite, to confirm it's a shared-state collision
   pytest tests/test_new_feature.py        # alone: does it pass?
   pytest                                    # full suite: does it still pass?

4. If the failure is in a date/time or locale assertion, pin the environment explicitly rather than relying on whatever the host happens to have.

   # CI job env — match timezone and locale to what tests assume
   env:
     TZ: UTC
     LANG: en_US.UTF-8
   
   # Or freeze time in the test itself instead of depending on the host clock
   # Python: freezegun
   with freeze_time("2026-01-01"):
       assert my_date_function() == expected

5. If it is a leftover global/shared state issue the agent introduced, make the new test isolate its side effects instead of relying on suite ordering to save it.

   # pytest — reset shared state per test instead of per session
   @pytest.fixture(autouse=True)
   def reset_cache():
       my_module._cache.clear()
       yield
       my_module._cache.clear()

How to stop it happening again

After any agent run that touches dependencies, always run the lockfile-strict install command (npm ci, poetry lock --check, pip-compile --dry-run) locally before you push — not just the looser install command the agent likely used.

Add a CI step, or a pre-push hook, that runs the full suite inside the same container image CI uses, not just 'whatever is on my machine'. This turns the local/CI gap into something you catch before pushing instead of after.

Tell the agent explicitly, in your rules/instructions file, that any new dependency must be followed by a lockfile update command, and that any new test must not assume a specific run order or leave mutable state behind.

Pin timezone and locale in CI config (TZ=UTC) so date-sensitive tests cannot pass only because your laptop happens to share a timezone with nobody else on the team.

Periodically run the suite with randomized test order (pytest-randomly, Jest with shuffled test files) even outside CI failures — order-dependency bugs are cheap to catch early and expensive once an agent has built more code on top of the hidden assumption.

Related questions

Is this an AI-specific bug, or would it happen with a human contributor too?

The underlying causes — lockfile drift, test order dependency, environment-specific assumptions — are not AI-specific; they are classic CI/local parity bugs. What changes is the odds: an agent is more likely than an experienced human to add a dependency without a lockfile update, or add a test without checking it is order-independent, simply because it does not see your CI logs and is not deliberately verifying cross-environment parity unless told to.

I reran the same CI job with no changes and it passed the second time. What does that mean?

That is a strong sign of either test-order/shared-state flakiness or a race condition (parallel test workers touching the same resource, a timing-sensitive assertion, or a flaky network call in a test that should be mocked). It is very unlikely to be a lockfile problem, since that would fail consistently, not intermittently.

The CI failure mentions a package version I don't recognize. What happened?

Most likely the agent added or bumped a dependency in the manifest file (package.json, requirements.txt, pyproject.toml) but did not regenerate the lockfile, or regenerated it with a different resolver/registry state than what's cached in your local node_modules or virtualenv. CI does a clean install from the lockfile and resolves differently. Run the lockfile-strict install locally to see the same version CI would pick.

Should I just tell the agent to 'fix the CI failure'?

Only after you've looked at the CI log yourself first. An agent given just 'CI is failing, fix it' with no log access will often guess, and a guess here is likely to touch the wrong test or paper over a lockfile issue with a version pin that happens to work today and breaks again on the next dependency bump. Paste the actual CI log and the lockfile diff into the conversation so the fix targets the real cause.