Customers often use Tidelift to identify proactive risk management opportunities, meaning reducing the quantity and severity of vulnerabilities they have to remediate in the future rather than only reacting the vulnerabilities as they arrive.
Tidelift's unique analysis of open source packages allows you to find bad packages that are likely to cause future trouble. Tidelift's maintainer model allows you to ensure that fewer of your dependencies go bad and have to be replaced.
Metrics
The business outcomes many customers care about most are to reduce risk (fewer vulnerabilities “go live”) and to spend less time on remediation (fewer vulnerabilities and other problems need a “fire drill”).
For each metric, you will have reports and web UI views in Tidelift to help you move it.
Tidelift stores 90 days of data in our UI for a quick all-project snapshot. Reports are used to move this data to storage within your organization.
Metric 1: Violations by Standard
This metric shows you the current risk posture of your projects. This is active, open risk that you may use to prioritize time allocated for developers to fix problems.
Each point under each standard represents the number of unique violations affecting packages within your projects. A violation's details can be the same across multiple projects, so the count on this chart will be much lower than in the All-projects violations report which could be affecting a single catalog multiple times.
It is certainly possible to reactively respond to violations as they appear. However, we believe the other two metrics are better for tracking progress because they represent actionable, proactive improvements above the level of single violations.
Report: All-projects violations report corresponds to the violations-by-standard metric. This report can be used to identify vulnerabilities or other violations that are most valuable to fix, and which projects are carrying the most risk.
Metric 2: Approved Releases
This metric represents your projects’ compliance with the policies you’ve configured in Tidelift. You've implemented a process to give your developers a simple, responsive allowed or not allowed answer. How well are developers adopting this guidance?
Approved releases are those you’ve asked your projects to use; over time, you’d want to use more of them. This metric goes beyond single violations and looks at how well you're sticking to releases that avoid all violations of your standards.
Report: All-projects compliance report corresponds to the approved releases metric. This report can be used to identify projects with the most work to do to comply with an organization’s configured open source standards.
Metric 3: Package Quality (beta)
This metric represents Tidelift’s assessment of bad packages that may have future problems. Tidelift’s “not recommended” packages historically see both more vulnerabilities and have a higher percentage of vulnerabilities with no fix. This metric does not consider an organization’s own standards configuration; it's Tidelift's assessment of package risk.
Report: All projects package quality report (beta) corresponds to the package quality metric. This report can be used to find not-recommended packages, and sort/filter by how many projects use them, and whether they use them directly or transitively. It will improve your developer sprint planning practice to proactively identify packages to move away from — and avoid future fire drills.
How customers use these features
We have seen customers have success with a regular practice of improvement. It will be virtually impossible to fix everything and maintain perfection on the three quality metrics. At the same time, it’s risky and inefficient to ignore everything and allow it to get worse and worse.
Teams have good luck with a strategy of allocating a little time on a regular schedule to improve their metrics.
The all-projects reports can be used to define organizational initiatives or sprints, such as eliminating the top 10 violations, or upgrading a particular package. When doing this, one developer can create a document or guide showing how to do it, allowing other developers to quickly apply the same fix.