Dear readers,
Every week, NAVAL-SEM — my free, offline SEM (structural equation modeling) desktop app — gets a batch of Dependabot PRs. Six of them showed up this week. All patch or minor version bumps. All green checkmarks. The kind of PR most maintainers merge without a second look.
I decided to actually test them first. Here's what I found instead.
Step 1: Zero tests collected
Before touching any dependency, I ran my test suite locally to establish a baseline.
collected 0 items
no tests ran in 0.24s
My test suite has 200+ tests. Running pytest from my actual repo directory found none of them. It turned out my tests lived in a separate local validation folder — kept out of the public repo deliberately, since the test cases encode methodology I don't want competitors reverse-engineering. But that folder had never been symlinked or copied into the actual working checkout. I'd been "running my tests" against an empty directory for who knows how long.
Fix: copied the tests into the repo under a gitignored tests/ folder, so they run locally without ever being pushed publicly.
Step 2: A resolver conflict hiding behind CI's green checkmark
With tests actually discoverable, I ran uv sync --upgrade to pull in the pending bumps. The resolver refused:
× No solution found when resolving dependencies
╰─▶ Because pydantic==2.13.4 depends on pydantic-core==2.46.4
and your project depends on pydantic-core==2.47.0,
we can conclude that your project's requirements are unsatisfiable.
Two lines in pyproject.toml — a direct pin on pydantic and a separate direct pin on pydantic-core — had drifted out of sync. pydantic-core isn't meant to be pinned directly at all; it's a transitive dependency pydantic manages internally. An earlier Dependabot PR had bumped pydantic-core on its own, without anyone (bot or human) checking whether pydantic itself still agreed with that version.
CI never caught this because CI only validates the diff in a given PR — not whether the cumulative state of all your pins is internally consistent.
Fix: dropped the redundant pydantic-core pin entirely and let pydantic manage it.
Step 3: A 200-test regression run, live server included
Because my tests are integration tests that hit a running instance of the app over HTTP, "run pytest" first meant "start the app in a separate terminal." Once that was in place, the full suite ran clean — twice, once for each dependency batch (narwhals + tzdata + websockets + anyio, then setuptools separately since it's a major version bump).
201 passed, 2 skipped in 805.55s
Thirteen minutes per run. Worth it — this is exactly the run that would have caught a real regression, and it's the run a solo maintainer is most tempted to skip when the CI badge already looks green.
Step 4: Chasing a log line into a dead dependency
Somewhere in the startup logs, a line caught my eye:
No graphviz package found, visualization method is unavailable
I don't call graphviz anywhere in my code. Tracing it down, it turned out to be semopy — a hard dependency for my SEM engine — silently attempting to import graphviz at module load time, for its own optional semplot() diagram function. I've never called that function. My actual model diagrams are rendered entirely client-side: a D3/SVG canvas, screenshotted via html2canvas, and shipped to the backend as a base64 PNG for report exports.
Graphviz had been sitting in my dependency tree as dead weight — a native binary with per-OS packaging quirks that I'd have had to bundle across three release platforms (Windows, macOS, Linux) for a feature I don't use.
Fix: removed the graphviz Python package and uninstalled the system binary. Nothing broke, because nothing was using it.
Step 5: The lockfile that quietly drifted
Last thing: after merging the narwhals and setuptools PRs through GitHub's UI, my local uv.lock and the committed one on master disagreed — the PRs had only touched pyproject.toml, not the lockfile. Anyone doing a fresh uv sync --locked clone would've hit a resolution mismatch that I wouldn't have seen locally, because my own environment had already been synced ahead of the commit.
Fix: regenerated and committed the lockfile in its own PR.
The takeaway
None of these six problems showed up as a red X anywhere. Every PR CI ran was green. Every one of these issues was found by actually running the software, not by trusting the badge.
If you maintain free software — even something small — this is the tax nobody sees: an hour of archaeology behind every "trivial" merge, so that the next person who clones your repo doesn't inherit a broken environment.
Thank you for reading,
N. Singh
