What is project health and how is it calculated?

What is project health? (BETA)

Project health is a 0-100 grade given to each project that gauges how "healthy" a project is. A project is healthy when it uses 100% reviewed and approved open source (and verified internal packages).

How should you use project health scores?

Project health should be used to help determine where you want to focus your limited resources.  Having one score per project allows you to compare projects to each other and choose to focus on one project over another.

Additionally, you and your team can use project health to show how projects improve over time. Project health can show the health of a project for the last 6 months.

Ways your team may choose to use project health scores: 

• You may choose to focus on projects that have the lowest scores and work to improve them first.
• You may want to make sure all of your projects are above a certain score.
• You may choose to acknowledge that old projects have bad scores, but that it's acceptable given the age of the project.

What causes changes to your health score?

• Changing standards If you enable or disable new standards or raise standards, your health score can change. Enabling new standards is like raising the bar of your organizational open source usage. Some items may no longer meet the new standard and lower your score. On the opposite end, disabling a standard is like lowering the bar, raising your score.
• A new violation of an existing standard Security vulnerabilities, license changes, new releases, and deprecation occur all the time. If one of these affects one of your releases in use and brings it out of compliance with your standards, this will drop your project health score.
• A new alignment run Every time an alignment is run, Tidelift receives a new version of your software bill of materials. This could mean new packages are brought in as you build new features or as you fix existing issues. This would affect your project health score.
• Requesting a package Requesting a package to be reviewed and evaluated by a catalog administrator allows more traceability for when an issue occurs, and brings visibility to issues that may exist. Requesting a package that's in a project's bill of materials will improve your health score.
• Creating an exception Sometimes you need to use a release that doesn't meet your organization's standards. By creating an exception, you are indicating that the issue doesn't matter. The project health will improve when you create an exception for a release that otherwise violates your standards.

How is project health calculated? (BETA)

Project health depends a lot on what your organization feels is important, based on what standards you choose to enforce for the projects. By choosing which catalog a project aligns to, you are also choosing what standards to enforce for the project.

Each enabled standard will generate a subscore for a project. A weighted average of the subscores will make up the project health score. A weighted average allows more important issues, such as licensing or security to take a higher priority.

• A security subscore accounts for 40% of the project health score, and is based on the 'Releases have no vulnerabilities’ standard.
• A licensing subscore accounts for 40% of the project health score, and is based on the 'Releases use approved licenses' standard.
• An up-to-date subscore accounts for 10% of the project health score, and is based on the 'Releases must be up to date' standard.
• A maintained subscore accounts for 10% of the project health score, and is based on the 'Releases are actively maintained' standard.

To calculate each subscore, we score each dependency based on the standard, and take the average score for the dependencies. Each dependency subscore starts with a maximum score of 100.

Calculating subscores for up-to-date, actively maintained, and licensing

Up-to-date, maintained, and licensing subscores are the simplest to calculate by hand. For these subscores, the criteria we look for are:

1. Has the release been requested?
2. Has the catalog administrator denied or created an exception for the release?

Starting with 100, if the dependency violates the standard, we subtract 70. If a developer requests the release, bringing it officially for review, we add 10 points.

Example: If your organization does not want to be using deprecated packages and has the 'Releases are actively maintained' standard turned on, you would expect all releases to be actively maintained. If a project is using a deprecated package, the dependency would receive a score of 30. If the developer requests the dependency to be added to the catalog, the dependency would receive a score of 40.

A catalog administrator reviewing the request can choose to create an exception, at which point the dependency score increases to 100. If the catalog administrator does not approve of the release, the request will be denied and the dependency would continue to keep the score of 40.

Calculating subscores for security vulnerabilities

Security vulnerabilities require more nuance as security vulnerabilities range significantly by severity. We evaluate a few aspects:

1. What is the highest severity on this release?
2. How is this vulnerable release brought in, directly or transitively?
3. Has the catalog administrator denied or created an exception for the release?

Starting with the maximum score of 100, we subtract 10 times the CVSS score of most severe known vulnerability for the release, rounded down to the nearest integer. So if a release has a CVSS score of 9.8, we would subtract 90 points.

If the vulnerable release is brought in transitively, we add 10 points. This is because we believe direct issues are more actionable, so a potential vulnerability brought in transitively will require more effort to address than an equal potential vulnerability brought in directly. 

Finally, if a developer requests the release, we add an additional 10 points. Requesting the release brings it officially into view and reduces your organization's risk.

Note: This scoring makes it possible to have a perfect score even with releases that have documented minor security vulnerabilities. This is intentional as those are low risk and may not be worth your organization's time in addressing.

Example: Suppose your organization is using a release transitively with two known CVEs, one with a 8.3 CVSS and one with a 3.2 CVSS. Starting with 100 points, we'd subtract 80 points for using this release for a score of 20. Since it's brought in transitively, we'd add 10 points, for a score of 30. If it has been requested by a developer, we'd add another 10 points, for a score of 40.

If you review the security vulnerability with the 8.3 CVSS and deny it, the score will remain 40. If you review the security vulnerability and determine it does not apply, you would create an exception. This would normally bring your score back up to 100.

But in this case, if you still have an outstanding 3.2 CVSS, we'd have to recalculate. 100 (max) - 30 (for 3.2 CVSS) + 10 (transitive) + 10 (requested) = 90.

You could in this case choose to decide that 90 is good enough, and not review the 3.2 CVSS, and focus on addressing other releases that have worse scores.

Note: This feature is still under beta and is likely to change. We may choose to change weighting of each standard, add new standards, or even how the subscores are calculated. Please reach out if you have any feedback or thoughts at support@tidelift.com