Building an AppSec Program: Part 1 of a 4-Part Series on Application Security

Matthew Keeley
June 16, 2024

Hi there! You might know me from my technical deep dives into pentesting, exploit development, and zero-day research. Today, I’m shaking things up! I’m starting a 4-part blog series all about application security and the nitty-gritty details that come with it. I’ve been in the cybersecurity world for nearly a decade, with the past couple of years focused on building an application security program from the ground up. So, let’s dive in!

Introduction to AppSec

Ever wonder what keeps apps secure from those sneaky hackers lurking on the “dark web”? It’s all about AppSec! Application security, or AppSec, is the part of an organization dedicated to preventing applications from being exploited by hackers. This might sound straightforward, but there’s a lot more to it. Achieving a high level of security involves numerous practices and tools to make your applications as secure as possible.

No application will ever be 100% secure — technology evolves too quickly, vulnerabilities from decades ago are still being discovered, and perfect code is a myth. But with a solid AppSec team acting as a first line of defense, companies can significantly reduce the risk of that dreaded word: breaches.

What is Application Security?

At its core, application security is about ensuring that the software you use and develop is protected from threats. It encompasses secure coding practices, data protection, and compliance with regulatory standards. So, what’s the ultimate goal? Simply put, it’s making sure your app doesn’t get hacked and that your users’ data remains safe and sound.

Sounds easy, right? Well, not really. It’s a continuous effort involving constant vigilance, regular updates, and staying ahead of the ever-evolving threat landscape.

Why should your company have an AppSec team?

Well, maybe it doesn’t need one — at least not right away. Here’s my take: if your company has fewer than 500 employees, you might not need a dedicated AppSec team. Instead, a small but mighty security team should suffice. The rule of thumb I use is one security engineer for every 100 software engineers. Sounds crazy, right? But this approach forces your team to focus on high-impact actions that scale effectively. Without this, you risk ending up like Airbnb, manually reviewing every Content Security Policy (CSP) that goes to production — a colossal waste of time and money.

Now, when your company grows beyond 500 employees, having an AppSec team becomes crucial. Why? It all boils down to one principle: stop vulnerabilities from reaching production. When a vulnerability slips through and hits production, the cost and effort to fix it are about five times higher than catching it beforehand. Stopping bugs early keeps your customers’ data safe and reduces the risk of costly breaches.

But let’s get one thing straight — if compliance is the only reason you’re creating an AppSec team, you’re throwing money down the drain. And if you’re big enough, maybe you don’t care about that. Compliance shouldn’t be the main driver for any security program. Teams focused solely on compliance often cut corners to get those PCI or SOC2 checkmarks. I’m not saying you shouldn’t have a compliance team — it’s important! Just don’t let your AppSec team’s primary goal be achieving SOC2 compliance. Focus on real security measures that genuinely protect your company and your customers.

Building an AppSec Team

So, now that I’ve ranted about why you should have an AppSec team, let’s talk about how you should go about building one. For this example, let’s assume we have a 1000-person company and a budget to hire 10 people for the AppSec team. How would I design it?

Key Roles and Responsibilities

Let’s play fantasy team-building and imagine we’re assembling a crew of pirates. If you had ten spots to fill, who would make the cut? Here’s a possible lineup:

  1. Leadership — (1) AppSec Manager, (1) Program Manager
    These two roles are the most important — they’re the co-captains of the ship! Your AppSec manager should know the technical ins and outs of how your team operates best (and they should, if they hired some of them). They keep the boat sailing in the right direction. The program manager has a significant impact, setting up large-scale projects and focusing hard on cross-team and cross-department collaboration. Security teams without this role often struggle to communicate with other teams, leading to unresolved bugs, missed SLAs, and a general lack of engagement with the security team. This role can also shift the company’s perspective on application security, making developers more aware and invested in implementing security into products. After all, it’s a team effort.
  2. (4) Product Security Engineers
    These are the rowers of the ship, the do-ers who get things done. They handle security reviews, pentests, and work with engineers to implement necessary security protections. I like to see these engineers have consulting experience (think BishopFox, Rapid7, Mandiant, etc.), which gives them a broad perspective on different departments’ operations and excellent communication skills — crucial for consulting.
  3. (2) Software Security Engineers
    These are the ship’s engineers. They make improvements to the ship, find advantageous currents, and build things that enhance the ship’s longevity and speed. Yeah, I’m probably taking this pirate metaphor too far, but you get the point. Software security engineers need to know how to code. Too often, I see security engineers who can “read code” but can’t write it. We don’t want those people — not on this boat. While I’ve only allocated two slots for this position, their impact is usually among the largest. These two should be writing code, developing tools, automating processes, and implementing security measures across the company, scaling the team’s effectiveness.
  4. (2) Vulnerability and Risk Management Analyst
    Last but not least, vulnerability and risk management! Admittedly, this is one of my weaker areas, but I recognize its importance within the team. These analysts are the Jira gurus. They build and maintain centralized boards where all identified vulnerabilities — from sources like Nessus, pentests, Snyk, etc. — are logged. This ensures that every vulnerability lands in one place, making it easier to assign them to the right teams for triaging and resolution. They define and track key metrics for the entire security team. This includes the number of vulnerabilities identified, time to resolution, and compliance with security policies. They generate reports that provide visibility into the team’s performance and highlight areas needing improvement. They tend do a lot more stuff, but Im going to keep it brief.

Goals, Metrics, and KPIs

No team can thrive without clear goals. Here’s how to set them:

Tracking Progress

You might think tracking progress is easy, but it’s not. Security teams often struggle to measure their processes and impact. At the end of the day, it’s really hard to quantify what preventing a critical vulnerability saved or cost the company. Especially if the company has never had a formal breach, assigning a dollar impact is tricky and often leads to prioritizing the wrong things.

So, how do we track progress effectively? One of my favorite strategies is to set a five-year vision. What should our cloud security look like in five years? Is there a standard we can align to, like AWS CIS standard? What does a five-year plan look like?

Start with a high-level vision for broad categories like securing AWS, securing CICD pipelines, and finding bugs in applications. This is the most important first step. Once you have this vision, breaking it down into smaller, manageable increments becomes much easier.

By setting clear long-term goals and mapping out a detailed roadmap, you can ensure your security efforts are focused, measurable, and aligned with your organization’s overall strategy.

What Do Team KPIs Look Like?

The right KPIs for your AppSec team will heavily depend on your organization’s current priorities and goals. For example, if your company is planning an IPO soon, how would that reprioritize your KPIs? Or, if your company has just experienced a breach and is now retroactively trying to implement security measures, how does that shift your focus?

The point is, I don’t know where your company is heading. But in general, here are some baseline KPIs I like to track:

  • Number of Vulnerabilities Detected Internally and Externally: Tracking the number of vulnerabilities discovered both internally and externally (through platforms like Bugcrowd or HackerOne) is straightforward. You can typically hook into a Jira board to gather the data you need. If your team is finding vulnerabilities faster than the development teams can prioritize and remediate them, it might be time to shift the focus to helping dev teams fix these bugs more efficiently.
  • Time to Remediate (TTR): How quickly are vulnerabilities being fixed? Are any overdue? What is the average time for each department to fix a vulnerability, and are they staying within SLAs? If not, what can the AppSec team do to help? This KPI helps you understand the efficiency of your vulnerability management process.
  • Percentage of Apps Passing Security Review Before Release: How many apps are getting reviewed and still coming out with a dozen bugs? Can we implement better threat modeling to prevent this going forward? This KPI helps ensure that applications are secure before they go live.
  • SAST Coverage: Percentage of Repos Being Scanned: Security tooling scalability is crucial. How many of your repositories are being scanned with security tools before CI/CD? Are tools like Semgrep and Snyk finding impactful bugs? How is the developer experience, and are there any improvements that can be made here? Ensuring comprehensive scanning coverage helps catch vulnerabilities early in the development process.
  • Continuous Improvement: Let’s not forget the importance of continuous improvement. Regularly reviewing and refining these KPIs based on the evolving needs and priorities of your organization ensures that your AppSec team remains effective and aligned with business goals.

Setting SMART Goals

Setting Specific, Measurable, Achievable, Relevant, and Time-bound (SMART) goals helps in keeping the team focused and motivated. Heres an example DataDog dashboard that my company uses every week to help track Goals and KPI progress:

Example DataDog Dashboard

Hot Takes and Controversial Topics

Let’s spice things up with some hot takes!

Team Positioning

Should the AppSec team be directly under the CTO/CISO, or should it be part of another department like Platform or DevOps? Each setup has its pros and cons. From my experience, the best positioning is directly under a CISO/CTO structure. Here’s why:

When the AppSec (or any security) team is under a different department, say DevOps, the team members often understand that their promotions and career growth are tied to the priorities of that department. For example, if you’re part of the DevOps org, a glowing review from marketing or any other department doesn’t hold much weight when the VP or Director signing off on promotions is focused on DevOps. This can lead to a shift in priorities, where the security team focuses more on adding security to DevOps rather than addressing the needs of the entire organization. However, when the team reports directly to a CISO or CTO, they have the authority and visibility to act at scale, impacting the entire organization’s security posture.

On the other hand, being within a department like DevOps can lead to better collaboration. Security engineers working closely with DevOps teams can build faster, and there’s often a natural exchange of knowledge — security engineers learn more about DevOps, and DevOps engineers become more security-savvy. This proximity can foster a more integrated approach to security, where it becomes a seamless part of the development process.

Budget Allocation and Prioritization

Securing adequate funding and resources can be a challenge. How do you prioritize security spending? It’s a constant balancing act between immediate needs and long-term goals. My favorite approach is an 80/10/10 split: 80% on tools, 10% on training and development, and 10% on team outings.

Security tools are crazy expensive, and rightfully so. They often do the job of 2–5 employees for the cost of one, so investing in top-notch tools with all the bells and whistles is a smart move. These tools can automate many security tasks, allowing your team to focus on more complex issues.

I like to allocate 10% of the budget for training. This allows your team to take cool courses like OffSec and stay up-to-date with the latest security practices. Continuous learning is crucial in the fast-paced world of cybersecurity.

The final 10% goes towards team outings. Not everyone on the team may care for trainings, and not all will attend the outings, but it tends to balance itself out. Plus, team outings help build camaraderie and keep morale high.

Balancing Security and Innovation

Security measures can sometimes slow down development, leading to a tug-of-war between security and innovation. Striking the right balance is crucial. Here’s my take: development comes first.

“But Matt, aren’t you a security engineer? WTF man?”

Hear me out. If you don’t have revenue, you don’t have a business. If security constantly blocks engineering teams from shipping new products and features, you’re not just hurting the engineering team — you’re hurting the entire company (and often its EBITDA). Security should complement development, not hinder it.

Security needs to integrate seamlessly into the development process, enhancing the flow and making it enjoyable for everyone with minimal interruptions. This is where security engineering comes in. By shifting security left and implementing it early in the development cycle, we can scale with the engineering teams and automate security measures. This approach ensures that security is embedded in the development process, allowing for innovation without compromising safety.


So, there you have it! Building an AppSec program isn’t just about checking boxes — it’s about creating a security culture that scales with your company and supports innovation. Whether you’re just starting out or refining an existing team, AppSec is an ongoing journey, not a destination. Keep evaluating, adapting, and growing your strategies to stay ahead of threats. Good luck!

Subscribe to our blog

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Matthew Keeley
February 14, 2023

Check out our other posts