top of page

Evaluating Open Source Before You Depend on It

S. West
Aug 16, 2025
14 min read

Updated: 7 hours ago

In March 2016 a developer unpublished 273 packages from npm during a naming dispute. One of them was left-pad, a short function that pads a string. Within minutes npm was seeing hundreds of failures a minute, because many widely used packages depended on it, and so did everything that depended on them. The disruption lasted two and a half hours.

Two years later, an attacker posing as a maintainer took over event-stream, a widely used npm package. The new maintainer added a dependency, flatmap-stream, whose version 0.1.1 contained code built to steal wallet keys from one specific application. In 2024, a developer noticed SSH logins on a Debian test system taking about 0.8 seconds instead of 0.3, and traced the slowdown to a backdoor in the release tarballs of xz, a compression library shipped with most Linux distributions.

Run any of those three through the usual evaluation checklist (stars, recent commits, issue activity, tests, documentation) and they pass. left-pad was finished, not abandoned. event-stream was widely used. xz was actively maintained. The checklist measures whether a project looks healthy. The failures came from somewhere else: how deep the code sat, who could publish a release, and what that release was allowed to touch.

This post replaces the checklist with an evaluation that starts from your system rather than the project's GitHub page.

Why a health checklist misses the dependencies that hurt

A dependency can fail you in four ways, and each one needs a different check.

Failure mode

What it looks like

Real example

The check that catches it

It disappears

The package or version is removed, renamed or yanked

left-pad, 2016

Lockfile plus a private registry or mirror for anything you can't rebuild without

It stalls

You need a fix and nobody ships one

A bug or CVE sits open for months

Time from report to fixed release, contributor concentration

It turns hostile

A release contains code its users never reviewed

event-stream 2018, xz 2024

Who can publish, how releases are built, install scripts

It changes terms

A new version ships under a licence you can't accept

HashiCorp (2023), Redis (2024)

Licence trajectory and who owns the copyright

The common checklist covers part of "it stalls" and very little of the others. Popularity in particular cuts both ways: it means more people find bugs, and it also makes the package a better target. event-stream was attacked because it was widely installed.

The test: list your ten most important dependencies and write down, for each, which of the four failure modes would hurt you most. If you can't answer for a dependency, that is the first gap. You don't know what job it does in your system.

Start with the role, not the project

The discovery question that comes before any evaluation is: what will this code be able to touch, and how hard will it be to remove? The same library can be harmless in one place and a serious exposure in another. A date-formatting helper in an internal admin script is a different decision from the same helper on the path that handles customer payments.

A running example (composite). A product team is building a customer-facing feature that imports spreadsheets: users upload CSV or XLSX files with up to 50,000 rows of order data. The team needs a parsing library for Node.js and has two candidates:

  • Library A: very popular, one maintainer who wrote nearly all of the code, last release 14 months ago, about 40 open issues with no maintainer reply.

  • Library B: about a tenth of the downloads, hosted by a foundation, four people with merge rights, releases every two to three months.

The checklist would call this close: A wins on popularity and Stack Overflow answers, B wins on maintenance. Asking about the role settles it. The parser will read untrusted files uploaded by strangers, and its output goes into the orders database. That puts it in the highest tier.

Tier

The dependency…

Evaluation depth

1. Exposed

Parses untrusted input, handles authentication, cryptography or secrets, or runs code at install or build time with access to credentials

Every check in this post, a named owner and a written decision

2. Load-bearing

Is called from many places or shapes your data model, so replacing it means weeks of work

Maintenance, licence and vulnerability checks, plus an exit plan

3. Contained

Has one or two call sites, is dev-only, or could be replaced in an afternoon

Licence check and an automated vulnerability scan

Two numbers place most dependencies in a tier quickly. The first is call sites: grep -rl "from 'library-a'" src | wc -l (or the equivalent for your language) tells you how far it has spread. The second is install-time code: in npm, a preinstall, install or postinstall entry in the package's package.json means the package runs code on every machine and CI runner that installs it.

The test: pick the last three dependencies your team added and assign each a tier. If any Tier 1 dependency was added in a pull request with no discussion, your intake process is evaluating the code review, not the dependency.

Measure maintenance with numbers, not impressions

"Actively maintained" means different things to different people. These are signals you can pull in about ten minutes, with the threshold that should worry you.

  • Contributor concentration. CHAOSS, the Linux Foundation project that defines open-source health metrics, calls this the Contributor Absence Factor (it used to be called the bus factor): the smallest number of contributors responsible for 50% of contributions. Run git shortlog -sn --since="2 years ago" on a clone and count down the list until you pass half the commits. A result of 1 means one person's life events decide your patch schedule.

  • Time to first maintainer response. Open the 20 most recent issues and note how long each waited for a reply from someone with commit rights. A median measured in weeks matters more than the total number of open issues.

  • Time from report to fixed release. Look at the project's past security advisories, or its changelog for fixes labelled as security. How many days passed between the report and a release containing the fix? That is the number you will live with next time.

  • Release regularity. A gap is fine for a finished library with a narrow scope. It is a problem when the gap coincides with open bug reports that maintainers have acknowledged.

The OpenSSF Scorecard project automates part of this. It runs checks against a repository, including Maintained (the top score needs at least one commit a week over the previous 90 days), Code-Review (whether recent changes were approved by someone other than the author), Security-Policy (a SECURITY.md with a way to report vulnerabilities) and Dangerous-Workflow (CI patterns that let untrusted pull requests run privileged code). Treat the score as a set of prompts, not a verdict: a finished library can score low on Maintained and still be the right choice for a Tier 3 job.

In the example: Library A's absence factor is 1 (one maintainer has 93% of the commits in two years), and its median first response on recent issues is "none yet". Library B's absence factor is 3, with a median first response of four days. For a Tier 1 parser, A fails on this section alone.

The test: compute the absence factor for each Tier 1 dependency. Any that come out at 1 need either a replacement, a funded relationship with the maintainer, or an entry in your risk register with a named owner.

Ask who can publish a release

This is the question the usual checklist skips, and it is the one behind event-stream and xz. In both cases the code in the repository was not what users ran. A new maintainer published a version with a malicious dependency, and in xz the key part of the backdoor existed only in the release tarballs, not in the git source.

Check three things for every Tier 1 dependency:

  1. How many accounts can publish, and has that changed recently? On npm, npm view <package> maintainers lists them. A recent change of maintainer on a widely used package isn't bad in itself, but it is exactly the moment to read the next release diff before upgrading.

  2. Can you trace the published package back to its source? npm supports provenance statements: packages published from GitHub Actions or GitLab CI/CD with npm publish --provenance carry a signed link to the source commit and build. Running npm audit signatures in your project reports which installed packages have verified signatures and attestations. PyPI's trusted publishing serves a similar purpose. A package with provenance doesn't prove the code is safe, but it does mean the release came from the repository you evaluated.

  3. Does it run code when installed? An install script runs with the permissions of whoever runs npm install, which on a CI runner often includes deployment credentials. Where you can, install with --ignore-scripts and allow scripts only for the packages that need them.

Then make sure a surprise release can't reach production unreviewed. Commit your lockfile, so that a new version enters only when someone changes it. Let your update tool (Dependabot, Renovate) open those changes as pull requests that get read, not auto-merged, for Tier 1 packages.

In the example: Library A has one publishing account and no provenance, and it has a postinstall script that downloads a native binary. Library B publishes from CI with provenance and has no install scripts.

The test: run npm audit signatures (or your ecosystem's equivalent) and list every Tier 1 dependency that has no verified attestation and runs install scripts. Each one is a place where code nobody on your team reviewed runs with your credentials.

Check the licence, the whole tree and where it's heading

Three licence mistakes recur.

Checking only the direct dependency. The licence of the library you installed says nothing about the libraries it pulls in. deps.dev, Google's Open Source Insights service, shows the full dependency graph and the licence of every package in it for npm, PyPI, Maven, NuGet, Go, Cargo and RubyGems. Locally, a licence scanner for your ecosystem (for example license-checker for npm or pip-licenses for Python) lists every package with its licence, which you can compare against an allow list.

Misreading copyleft. The rules turn on what you do with the code, not on the licence name alone. The GPL's obligations are triggered by distributing the software. The AGPL adds a network clause, and section 13 of AGPLv3 applies "if you modify the Program": a modified version must offer its source to users who interact with it over a network. Running an unmodified AGPL component behind your service is a different situation from shipping a modified one. Many companies ban AGPL entirely anyway, because the line is easy to cross by accident. Know your organisation's policy, and take real cases to whoever owns legal risk; this post is not legal advice.

Licence family

Typical obligation

Where teams get caught

Permissive (MIT, BSD, Apache 2.0)

Keep the copyright and licence notice; Apache 2.0 adds patent terms and a NOTICE file

Missing attribution in distributed apps and containers

Weak copyleft (LGPL, MPL 2.0)

Changes to the licensed files themselves must be shared

Modifying the library in place, or static linking under LGPL

Strong copyleft (GPL)

Distributed software that includes it must be offered under the GPL

Shipping it inside a mobile app, desktop app or appliance

Network copyleft (AGPL)

A modified version used over a network must offer its source

Patching an AGPL service you run for customers

Source-available (BSL, SSPL, RSAL)

Not open source by the OSI definition; usually restricts offering the software as a competing service

Discovering it at upgrade time, after a relicensing

Ignoring the trajectory. Licences change, but only for future releases. HashiCorp moved its products from MPL 2.0 to the Business Source License in August 2023. Redis moved from the three-clause BSD licence to dual RSALv2 and SSPLv1 from version 7.4 in March 2024. In both cases versions already released kept their old licence, so users weren't breached overnight. They faced a choice at the next upgrade: accept the new terms, stay on an aging version, or move to a community fork. The signal to look for in advance is a project where one company holds the copyright, typically through a contributor licence agreement, and where that company's revenue competes with hosted versions of the project. That combination is what makes relicensing possible.

The test: run a licence scan across your full dependency tree and list every package not on your allow list. Then, for each Tier 1 and Tier 2 dependency controlled by a single company, write down what you would do if the next major version were source-available.

Known vulnerabilities, and how fast they get fixed

The scanners are the easy part. npm audit, pip-audit (maintained under the Python Packaging Authority) and dotnet list package --vulnerable check your installed versions against advisory databases. OSV-Scanner checks a whole repository, across languages, against the OSV.dev database: osv-scanner scan source -r . from the project root.

What a scan can't tell you is how the project behaves when the next vulnerability arrives. That is the number from the maintenance section: days from report to fixed release. A project that took two days last time will probably take days again. A project with an advisory that sat open for five months will put you in the position of choosing between running known-vulnerable code and patching a library yourself.

Two further checks catch most of the rest. First, count transitive dependencies: npm ls --all --parseable | wc -l after installing the candidate on its own shows how many packages it brings with it. Each one is another publisher you are trusting. Second, check whether the project has a SECURITY.md with a private reporting route. Without one, researchers tend to report in public issues, so the vulnerability is public before the fix exists.

In the example: Library A has an open advisory about a malformed file that causes runaway memory use, reported seven months ago with no fix. For a feature that parses uploaded files, that is a denial-of-service bug sitting on the exposed path. Library B fixed its last two advisories in 3 and 11 days.

The test: for each Tier 1 dependency, find its most recent security fix and work out the number of days from report to release. Write it down. If you can't find any security history at all, treat that as unknown, not as good.

Make the decision, and write it down

Bring the evidence together in one record per Tier 1 decision. For the example:

Check

Library A

Library B

Role

Tier 1: parses untrusted uploads

Tier 1: parses untrusted uploads

Contributor absence factor

1

3

Median first response, last 20 issues

No reply on most

4 days

Days from report to security fix

Open for 7 months

3 and 11 days

Publishing

One account, no provenance

CI publishing with provenance

Install scripts

postinstall downloads a native binary

None

Transitive packages

31

4

Licence (whole tree)

MIT, one package unlicensed

Apache 2.0 throughout

Licence trajectory

Individual author

Foundation, no single copyright holder

Decision

Rejected

Adopted behind an adapter

The decision has three possible outcomes, not two.

  • Adopt. The evidence fits the tier.

  • Adopt with mitigation. Use it, but confine it. Put the library behind one module of your own (an adapter), so that every call site goes through code you control and a replacement touches one file instead of forty. Pin it, read its upgrade diffs, and add a test suite at the adapter boundary that a replacement would have to pass. The team adopted Library B this way, because even a well-run Tier 1 dependency deserves an exit.

  • Reject, and look for another candidate or build.

Evaluating open source by role. The tier a dependency falls into (exposed, load-bearing or contained) decides which checks to run: who can publish it, contributor concentration and fix speed, licence across the whole tree, and a vulnerability scan. Each check maps to one of four ways a dependency fails: it disappears, stalls, turns hostile or changes terms.

Record the date and the numbers, as well as the choice. Six months later, the question will be "has anything changed since we decided?", and that needs a baseline to compare against.

The test: can you produce, for your three most exposed dependencies, a record like the one above? If the decision lives only in someone's memory, it isn't a decision anyone can revisit.

When to build instead

Building your own is reasonable when three things are true: the scope is small and stable (a function you can specify in a paragraph), you can write the tests in a day, and the domain is not one where experts disagree about edge cases. A string-padding function passes all three. That was the real lesson of left-pad: a few lines that nobody needed to fetch from someone else's account.

The test fails in the other direction for anything security-sensitive or format-heavy. Cryptography, authentication protocols, and parsers for complex formats such as XLSX, PDF or XML are where a home-grown version ends up with worse bugs than the library it replaced, and nobody outside your team reviews it. For those, the answer is a better-evaluated library behind an adapter, not your own.

The same reasoning applies to automation you build internally: automation that survives team changes is automation whose maintenance someone owns, which is exactly the question to ask of a dependency.

The test: before building, write the specification and the test list. If the test list for a "simple" parser grows past a page, you are about to write a library, and you should evaluate one instead.

Re-evaluate, because dependencies change after you adopt them

Every failure in this post happened to a dependency that had already been adopted. left-pad was removed years after people started using it. event-stream changed hands after it became popular. Redis users chose it under BSD. Evaluation at intake is necessary, but the risk is created later.

Set a review cycle by tier. For Tier 1, check quarterly: has the maintainer list changed, has a release lost provenance, has the licence or the copyright holder changed, has a security fix taken longer than before? For Tier 2, check twice a year. For Tier 3, rely on automated scanning. Many of these signals can be automated: Scorecard results over time, a licence scan in CI that fails on anything outside the allow list, and update pull requests that show the maintainer and licence diff alongside the code diff.

The same discipline applies one level up, when the dependency is a vendor rather than a package. Choosing a technology partner without getting burned covers the commercial version of these questions. The attacker's side of the same problem, including dependency compromise and CI/CD pipeline attacks, is in Cybersecurity Beyond OWASP.

The test: for your Tier 1 dependencies, when was each last re-checked? If the answer is "when we added it", schedule the first review now.

What to do this week

Take one service and list its direct dependencies. Assign each a tier using the table above; most will be Tier 3, which is the point, since that is where you can stop. For each Tier 1 dependency, collect four numbers: contributor absence factor, days from report to its last security fix, number of accounts that can publish, and whether it runs install scripts. Run a licence scan across the full tree. Anything that comes out at an absence factor of 1, with install scripts, no provenance or a licence outside your allow list goes into a decision record with an owner and a date.

Final takeaway

A dependency's risk depends less on how healthy its GitHub page looks and more on what it can touch in your system, who can publish it, and what happens when it breaks. Tier your dependencies by role, give the exposed ones the deep checks (publishing, contributor concentration, fix speed, licence trajectory), and confine them behind code you own. Then check again, because most dependencies that cause trouble were fine on the day they were added.

Sources

Related reading

Keep learning. This article is part of the Automation & Process Improvement path in the ShiftQuality Learning Center. Remove the repetitive work — and keep the automation from becoming new debt.

bottom of page