Overview: measuring and improving risk

Customers often use Tidelift to identify proactive risk management opportunities, meaning reducing the quantity and severity of vulnerabilities they have to remediate in the future rather than only reacting the vulnerabilities as they arrive.

Tidelift's analysis of open source packages allows you to find bad packages that are likely to cause future trouble. Customers have the option to pull Tidelift's analysis or raw data into their existing tool set, or to track their SBOMs using Tidelift's software. This article is about SBOM tracking.

Overview: the process to locate risks

  • Understand the metrics you're trying to move ... how to find them in Tidelift's UI
  • Load the dependencies of your projects into Tidelift, to power metrics and reports
  • At the management level, review all-project reports on violations and not-recommended packages and use them identify needed work.
  • At the developer level, review single-project views of violations and not-recommended packages, then take steps to remove them.

Metrics to track progress and outcomes

You can use Tidelift to track three metrics out of the box. The business outcomes many customers care about most are to reduce risk (fewer vulnerabilities “go live”) and to spend less time on remediation (fewer vulnerabilities and other problems need a “fire drill”).

For each metric, you will have reports and web UI views in Tidelift to help you move it.

Metric 1: Violations by Standard

This metric is closest to the actual goal and can be improved by improving the other two; vulnerabilities are the most important type of violation. It is certainly possible to reactively respond to violations as they appear. However, we believe the other two metrics are better for tracking progress because they represent actionable, proactive improvements above the level of single violations.

Metric 2: Approved Releases

This metric represents your projects’ compliance with the open source standards you’ve configured in Tidelift. Approved releases are those you’ve asked your projects to use; over time, you’d want to use more of them. This metric goes beyond single violations and looks at how well you're sticking to releases that avoid all violations of your standards.

Metric 3: Package Quality

This metric represents Tidelift’s assessment of bad packages that may have future problems. Tidelift’s “not recommended” packages historically see both more vulnerabilities and have a higher percentage of vulnerabilities with no fix. This metric does not consider an organization’s own standards configuration; it's Tidelift's assessment of package risk.

Screenshot 2024-05-24 at 4.41.12 AM.png

Loading the dependencies of your projects into Tidelift

For each project it’s important to upload dependencies for the branch you’re currently shipping to production. We call the list of project dependencies a “bill of materials,” and there’s an article here about the nuts and bolts of uploading it.

Some key points (see that article for more details):

  • Ideally, you’ll upload new project dependencies in a nightly batch job or a CI job, so they always reflect the latest version of the project.
  • If you use Tidelift to analyze feature branches or pull requests, for each project, you’ll want to define a default branch (such as “main”) that we use to represent the production version of the project and surface in reports and metrics.

Once projects are set up to keep their dependencies updated in Tidelift, it’s possible to pull metrics and reports from Tidelift to guide risk mitigation and tech debt remediation efforts.

Here's an example of a project's bill of materials in the Tidelift web UI, to give you a sense of what information it will include:

Screenshot - BOM.png


Managers can review all-project reports and determine initiatives to prioritize

Once project dependencies are imported into Tidelift, there are reports available (in the UI and via API) aggregating potential improvements.

These reports contain a list of potential developer work to reduce the attack surface of your applications.

Some customers choose to pull data into their own systems (such as Power BI or Kenna), while others simply view reports in Excel.

  • The all-projects compliance report corresponds to the approved releases metric. This report can be used to identify projects with the most work to do to comply with an organization’s configured open source standards.
  • The all-projects violations report corresponds to the violations-by-standard metric. This report can be used to identify vulnerabilities or other violations that are most valuable to fix. At the all-projects level, it’s also possible to find multi-project initiatives, like “fix this severe vulnerability in these 30 projects.”
  • The All projects package quality report corresponds to the package quality metric. This report can be used to find not-recommended packages, and sort/filter by how many projects use them, and whether they use them directly or transitively. Identify packages to move away from to avoid future fire drills.

These reports can also be pulled via API, which is useful to automate syncing them with other business processes.

Developers can review their own projects to identify worthwhile work

The same information from the all-projects reports can be found in the Tidelift UI on each individual project (and if you’ve configured it, on each feature branch or pull request where you upload a bill of materials).

Looking at an individual alignment,

  • The “Denied Releases” tab shows you how to move the approved releases metric, which corresponds to the all-projects compliance report. Typically, avoiding denied releases also improves your violations-by-standard metrics.
  • The “Actions to Take” tab shows you how to remediate violations by making the smallest upgrade that will solve the problem; this will also move your approved releases and violations-by-standard metrics. This tab corresponds to the all-projects violations report.
  • The “Bill of materials” tab includes a column for the Tidelift recommendation, which corresponds to the Tidelift recommendations report. Not-recommended packages pose a higher risk and should be removed if possible.

Screenshot - Actions.png

How customers use these features

We have seen customers have success with a regular practice of improvement. It will be virtually impossible to fix everything and maintain perfection on the three quality metrics. At the same time, it’s risky and inefficient to ignore everything and allow it to get worse and worse.

Teams have good luck with a strategy of allocating a little time on a regular schedule to improve their metrics.

The all-projects reports can be used to define organizational initiatives or sprints, such as eliminating the top 10 violations, or upgrading a particular package. When doing this, one developer can create a document or guide showing how to do it, allowing other developers to quickly apply the same fix.


Was this article helpful?
0 out of 0 found this helpful



Article is closed for comments.

Articles in this section