How Tidelift evaluates packages

Tidelift has built a unique model of partnering with open source maintainers and paying them for the critical work they do in keeping their packages maintained inline with industry-leading secure software development practices. Together with open source maintainers, we’ve developed an informed and clear idea about what makes a package reliable and secure enough to be used by our customers.

When evaluating packages, we suggest that you use the Tidelift recommendation. The Tidelift recommendation is a holistic evaluation of the package, and whether it is developed and maintained in a way that would make it a good fit for use in an application.

The recommendation for the package is one of:

  • Not recommended: This package has flaws that make it risky to use in your environment
  • Caution advised: While the package may not have severe flaws, it either has current issues, or does not follow a number of secure development practices
  • Neutral: The package does not have obvious flaws, but also does not have additional criteria (funding, backing) that would make it a recommended long term bet
  • Recommended: The package is funded and adheres to secure development and enterprise standards, and is a good bet to build your software on top of.

The recommendation includes reasons, describing the specific criteria that caused a package to receive  its recommendation.

If you want to make further determinations, you can look at raw data that Tidelift provides for the package.

For more information on how to evaluate packages using Tidelift, see:

What makes a package not recommended?

A package is not recommended if it falls into any of these categories:

It is declared end-of-life

A package that has been declared end-of-life will no longer get updates or security fixes. End-of-life packages should not be used.

Tidelift checks the end-of-life status of packages to ensure packages are under active maintenance. Tidelift checks for end-of-life in multiple ways, such as

  • Explicit declarations in package documentation, READMEs, or mailing lists
  • Whether a package was deprecated, and then had its source repository archived

It was removed from the upstream package manager

A package that was removed from the upstream package manager was removed for a reason. At best, the maintainer decided it wasn’t worth having it published, either due to bugs, or not wanting to maintain it. At worst, it was actively hostile and/or malware, and was pulled by the package manager itself. Packages that have been removed from the package manager should not be used.

Tidelift checks to ensure that any package that has previously existed upstream remains available. 

It is deprecated

A package that has been deprecated has been flagged by its maintainer as approaching its end-of-life. While the maintainer may still choose to make updates, they may also choose not to do so. Deprecated packages should not be brought into new applications, and work should be done to replace them in applications that might be currently using them.

Tidelift tracks packages that have been marked as deprecated by their maintainers directly in their ecosystem, and researches to find packages that have been marked deprecated by other means (such as notes in the documentation.) This data is also surfaced in the “Package is not deprecated” quality check.

It does not show maintenance

Unmaintained packages will not be fixed for new bugs, features, or security vulnerabilities. Unmaintained packages are a latent risk within your organization. They should not be brought into new development, and work should be done to replace them in applications that might be currently using them.

Tidelift tracks the maintenance activity of packages, and when it reaches a risky point, does an evaluation of the package to see if it is still maintained. Tidelift considers a package unmaintained if:

  • Its source repository has been marked archived or unmaintained
  • All of the following are true:
    • It has not made a release in the past 6 months
    • It has not had code commits in the past year
    • Less than 1/3 of the issues and pull requests in the past year have been addressed

It does not publish a security policy

A security policy is a documented procedure that a software project uses when handling security issues. It involves having a designated point of contact and a process for handling issues. Having a security policy demonstrates a willingness to fix issues as they arise.

Tidelift tracks whether a package has a defined security policy.

It is not responsive to security issues

The response to security issues, and providing fixes in a timely manner, is critical when using an open source project for enterprise application development. 

Tidelift tracks whether a project makes fixed releases available.

What makes a package marked as caution advised?

A package is marked as "caution advised" if it falls into any of these categories:

It does not have a discoverable security policy

A security policy is documentation of how a package's maintainers handle security issues in their projects, including having a reporting address, a way to address them securely and confidentially, and what procedures they follow. Lack of a documented security policy could mean that security issues are not prioritized or handled properly.

Tidelift tracks where maintainers have published the security policy for their project.

use two-factor authentication when committing new code, or publishing new releases

Two-factor authentication is a key security practice that ensures that the person who is committing new code or publishing new releases is the correct, authenticated person. You've probably set up two-factor authentication for your email or your bank, as an example. It is important enough that some package managers have begun requiring it for everyone who publishes new software releases.

Tidelift tracks where maintainers have asserted that they are using two-factor authentication, or where they are required to by the systems they use.

It has no stable release, or none that is two years or more old

When choosing a package, you want a package that has some level of history. A package's history gives you confidence to know that it's not a one-off release, that the maintainer has the intention of doing updates, and that a community can be built

Similarly, you want a stable release - if a package's entire set of releases are all prereleases, it's a signal that it may not be ready yet for wider usage.

Tidelift tracks the release history of packages, and whether releases are prereleases, and uses this information to determine whether a package has a sufficient history of stable releases.

It has not reviewed who has access to push new releases

A key part of security is reviewing who has what permissions to do what, and keeping the attack surface small. Each new person with the ability to publish a release is an expansion of the attack surface for the package.

Tidelift tracks where the maintainer has validated that only the proper set of people have access.

It has at least one active vulnerability on its latest release

Having a vulnerability with no available fix is a problem - you're left without anything you can do in your environment.

Tidelift tracks whether vulnerabilities have a fixed release available.

It will bring in an issue due to its dependencies

When you add a software project to your environment, you're not just bringing in that piece of software; you're bringing in everything it depends on, and trusting the practices of the project's maintainers in managing those dependencies.

Tidelift tracks not just whether an individual package has issues, but also whether it will transitively bring an issue into your environment.

What makes a package recommended?

A package is recommended if its maintainers has made commitments to uphold secure development and maintenance standards, and has financial backing to do so.

Maintainers do this by partnering with Tidelift.

Was this article helpful?
0 out of 0 found this helpful



Article is closed for comments.

Articles in this section