Choosing the right SAST solution

Maintaining your developers’ productivity should be paramount. Part 2 covers several factors with SAST tools that can affect productivity and the overall success of a SAST tool. Some of these are general operational requirements that have nothing to do with SAST tools; however, that doesn’t mean they are any less critical.

If you haven’t already read Part I, you can find it here.

Features

Quick Feedback

Quick feedback is the core part of the shift-left approach. If the whole point of using a static analysis tool is to find issues in your code quicker and faster, it makes sense to run these tools as early as possible. Setting up tools in your build pipeline to test everything before release is a good idea. However, if your developers end up spending a considerable amount of time writing some code only for your tools to block the release at the last minute due to some small, possibly insignificant, issue it’s going to make them unhappy.

You need to make sure you get feedback to the developers as quickly as possible, this (again) reduces the cost of fixing issues further and makes sure your developers don’t get frustrated.

The most obvious options are to use IDEs that might include extensions for your SAST tools of choice. This way, as your team is coding, they get instant feedback and can easily, if not subconsciously, fix issues during their normal working lives. There can be some issues with this; for example, it forces you to use specific IDEs and SAST tools that offer the option; also, it might have performance issues for your team’s dev environments.

Alternative options include making sure your static analysis tools of choice can be run remotely on your source code repositories as part of the development process. This way, you can try and get additional feedback hourly/daily/weekly. Whatever you do, you don’t want your developers getting feedback from static analysis tests an hour before the end of the sprint.

Update frequency

How often are updates provided? Has the vendor just released the tool and then forgotten about you? Make sure your tool does offer updates as the security climate changes and make sure you have a way to use them. If you never update the rule sets, you might catch some “poor practice”, but you won’t stop the latest vulnerabilities

Audit and Logging

You need to be able to prove that testing has occurred. If you can’t prove something’s happened, it’s as if you haven’t done anything. You need to be able to make sure you can audit your pipeline; however, you also need to make sure you check your audit results.

You can also use audit/generic logs to provide better metrics to measure your SAST tool results. Once you have done this, you try to identify and manage false positive rates with your SAST logs, as well as identify other areas you can improve.

Bonus Features

Before selecting a tool it’s good to at least ask yourself:

1. Does it explain clearly what the issue is and how to fix it effectively?

2. Does it provide code examples in each language you care about?

3. Can it automatically provide code fixes in the IDE?

4. Does the vendor also offer other additional resources/training materials to help the developers?

5. Do you have regulatory requirements? Does the SAST tool look for vulnerabilities for PCI-DSS, for example?

All of these add some sort of productivity value for your teams.

Service management

Don’t just use tools

There’s more to secure coding than using a whole load of tools. While SAST, DAST, IAST and SCA can all contribute, you need to foster a culture of security to receive the benefits of any additional tools. Some great ways to do this include:

• Actively having your security experts engage with your development teams

• Implement a fully secure development lifecycle.

• Provide beneficial and relevant security training to your developers.

User Buy-In

Developers will hate you if you just force them to use a tool. You need to make sure they agree and are engaged with the purchasing decisions and are committed to using the tools. If you don’t, you might find they get fed up with any range of problems that get blamed on your security tools and then try to circumvent the security measures designed to help them.

While their complaints could be a blend of reasonable to the downright ridiculous, you can try and force people to follow specific procedures, but it’s always much better to use a carrot than a stick. Your developers are the domain experts on what makes their lives easier; any decisions on what SAST tool to use should be developer-led.

The best-case scenario, if you don’t engage, is you miraculously manage to read their minds and pick the best tool for the job. The worst-case scenario is that productivity dies, developers refuse to fix issues, or even start looking for jobs without such stringent requirements.

Some core things to think about include:

1. Make sure the developers are involved in the selection process. Get them to: a. Receive demos. b. Arrange proof of concept testing. c. Help write the requirements for such tools. d. Lead the process.

2. When deciding what the bug bars/quality gates are for a product/team, engage with the team and make sure they agree. Have the team produce a policy that they own, rather than demanding they fix arbitrarily selected types of vulnerabilities. If what the team decides doesn’t match up with what you as a company want/need then now is the time to have a discussion about the company’s policies and what it would take to agree.

3. When implementing a new tool, make sure you have a proper change management process in place. As part of this, you need to make sure you are plumbing your tool into your environment correctly and monitoring its effectiveness. Lastly, significant changes like this need to have success and failure criteria and you need to be able to measure them if you can’t prove that you’ve made the world better then you haven’t.

Easy to Configure

By ensuring the tools you are using are easy to configure, you are making sure you can adapt to any unforeseen issues in the future. You’re also ensuring that you quickly overcome any routine changes that might affect your static analysis testing. You want to get the most out of your tools, especially if you are paying stupid amounts of money for the privilege!

The main questions you need to ask here are:

1. Is the documentation available and easily accessible? I worked in one company where the documentation for the SAST tools used in the company was blocked by the corporate firewall. You definitely need to make sure you can access the documentation for your tools.

2. Are people getting the right training? It hurts your productivity if your developers are not trained on the tools they need. Teams need to be able to update efficiently and change their pipeline without struggling to get the tools they are dependent upon into a usable state. If you expect your teams to use a SAST tool, you should give them ample opportunity to use and understand the tool. Otherwise, you might be setting the teams up for failure.

3. Can people configure the code to customise the alerts based on their requirements? Making alerts configurable makes it a lot easier for developers to prioritise the issues that are most important to them. This also ties in with reducing false positives. For example, if your SAST tool keeps flagging you don’t free memory somewhere (because it can’t identify where you free the memory) being able to customise the tool to ignore such issues can save you time in the long run.

4. Does the tool have decent false positive management? Another example is customising the tool to avoid duplication of alerts. If you have multiple teams developing libraries and collaborating, you don’t want all the alerts for one team’s code to always appear when another team just wants to use the library. Making sure your SAST tools can segregate the alerts based on who is responsible for them will help significantly improve your productivity.

5. Can you add your own rules? Adding your own rules can be one of the best ways to get rid of false negatives. Only you know what the most significant issues facing your applications are. While most products focus on generic problems that are common to most companies and industries, by understanding your critical components, and the most common problems you have, you can develop custom alerts that help target what matters the most. No tool that isn’t configurable can even consider offering the same level of accuracy. Public communities are developing to help crowdsource static analysis rules. By engaging with these communities, you can get additional rules

Easy to manage/support

It’s one thing to install a new service for your developers, another to make sure it’s managed well. The team that owns the tool should be experienced in providing service management and given the time they need to make sure they can solve issues that can come up. If you assign responsibility for the tool as an afterthought, they will just let any problems raised fall to the bottom of the todo pile. The team also need to understand the tool so they can be equipped to deal with issues that arise as well as actively engaging with the teams that are dependent upon them to make sure they can pre-empt and fix issues where possible.

If you mandate that your chosen SAST tool must analyse all released code then you must be able to make sure the tool does not go down during office hours; otherwise, code releases will grind to a halt. Guaranteeing either a pragmatic approach to code release or guaranteed SLA on SAST tool issues is needed to make sure SAST tool issues do not become a blocker.

Measurable effectiveness

Before deploying a tool, make sure you can measure its effectiveness. If you can’t measure its effectiveness, then while you may have improved the security of your codebase, you can’t prove it was cheaper than just searching for bugs manually.

Do this before you start using it as you’ll need before and after data to provide a proper comparison; otherwise, your test results will be tainted. If you are a larger company with a business analytics team, you can attempt to engage with them, thereby reducing your workload and adding a degree of independent review.

Good Performance/Low Compute cost

You might find the best tool in the world, which can meet all your requirements; however, when you first install it on all your developer’s workstations, it requires so much processing power it grinds their workstations to a halt. Alternatively, you could have it set up in the cloud/on a suite of servers and suddenly realise the cost of running the SAST tool is way higher than expected and was never accounted for initially.

For any software you decide to use, it’s always good practice to take into account the operational cost of running the software; in productivity, compute power, or money.

In conclusion...

There is more to using SAST tools than running them on your code and magically fixing problems. Technical metrics aren’t the most crucial thing to consider. Instead, reducing the cost of finding vulnerabilities is the most important thing to consider. There’s a whole wealth of different factors that can help you reduce the cost when using SAST tools; the detection rate is just one of them.