What is diff coverage?

Measuring coverage only on the lines changed in a pull request, rather than across the whole codebase.

By Cláudia FernandesUpdated

Total coverage on a large legacy codebase is a number nobody can move. Diff coverage asks a tractable question instead: were the lines you just wrote tested?

This is what makes coverage enforceable on a legacy codebase. A team can require 80% coverage on new code from day one without first backfilling tests for a decade of existing code. The trade-off is that code nobody touches stays untested, and stable, rarely changed modules can be among the most critical, so the overall number only rises as fast as the codebase turns over.

What to look for

  • Whether the threshold for new code is separately configurable
  • How added versus modified lines are counted
  • Whether the gate blocks the merge or only reports
  • Behavior on pull requests that contain no testable changes

Best 5 code quality, security and code review tools with diff coverage

Every tool in the directory whose documentation covers this, 3 fully and 2 partially. Full support is shown first. Shown in a random order that changes on every visit, so no tool gets an advantage from its position.

Support is not the same as parity, since some implementations are narrower in scope, gated to a higher plan tier, or maintained only for existing customers. The note under each tool is what its own documentation describes.

FAQ

What is diff coverage?

Measuring coverage only on the lines changed in a pull request, rather than across the whole codebase.

How many tools in the directory offer diff coverage?

5 of the 20 tools in this directory offer diff coverage, 3 fully and 2 partially:

Support is not the same as parity, since some implementations are narrower in scope, gated to a higher plan tier, or maintained only for existing customers. The note under each tool is what its own documentation describes.

What should you check when comparing tools with diff coverage?

Four things:

  • Whether the threshold for new code is separately configurable
  • How added versus modified lines are counted
  • Whether the gate blocks the merge or only reports
  • Behavior on pull requests that contain no testable changes