Episode #297 - True/False Positives, Phishing Package Maintainers

This week on Absolute AppSec, Seth and Ken return for an in-depth conversation on true and false positives, the determination of which depends upon matters of context and business impact. Both factors must be taken into account in order to avoid rabbit holes. This discussion was spurred by a recent article from signalblur of magonia.io discussing alerts in a security operations center. In short, only considering the existence of a flaw is not enough by itself; true impact comes by understanding context. After the discussion of true and false positives, Seth and Ken turn to the recent successful phishing of an npm package maintainer that resulted in the exposure of millions of projects depending on popular npm packages. It happens, folks, protect yourselves. To see this episode, go to https://www.youtube.com/@AbsoluteAppSec/streams, find us on Spotify, or look up Absolute AppSec wherever you get your podcasts.

The heart of this episode revolves around a recent article titled “What Framing Security Alerts as a Binary True or False Positive Is Costing You.” It introduces Signal Detection Theory, developed in the 1950s, which frames detections into four categories: true positive, false positive, true negative, and false negative. While this starting point is helpful, the article argues that modern AppSec requires a more nuanced approach that incorporates intention, behavior, and business impact. For example, within a true positive classification, it could be detecting a malicious attack requiring response, or suspicious activity worth investigating, or a real finding that is irrelevant in context with the business. For Ken, this article resonated with him because his company focuses on contextual security analysis, which often reveals the limits of simple binary labels. He gives the example of detecting MD5 usage—a signal that might be technically correct but meaningless if used in a non-security context (e.g., a demo). Classification requires business context, not just technical detection.

Seth is constantly saying that context is everything. He points out that consultants often report vulnerabilities at their theoretical worst-case severity, while development teams view them through the lens of their business threat model. This disconnect frequently leads to friction about severity ratings. For example, Seth describes self-XSS with CSP protections. Technically, it’s a vulnerability, but the real-world impact may be negligible. He stresses that findings only become actionable when framed with business context; otherwise, developers and SOC teams waste time on irrelevant alerts. While the industry has long known that this was an issue, what’s new is the attempt to build frameworks and language that help both security practitioners and business stakeholders reason about alerts in more meaningful ways. Like everything in the security world, this connects back to current trends in AI-driven tooling, where the LLMs are trained through human feedback to better understand how to triage issues.

The discussion moves to the recent npm debug package breach, which Ken is familiar with as part of his work. The maintainer’s account for a popular npm package was compromised, allowing attackers to inject malicious code. In this case, the malicious code was designed to drain cryptocurrency and transfer it to a specific Bitcoin wallet. Despite millions of projects being affected, the attackers reportedly only made about $500. The breach was a result of a phishing attack. The maintainer received an email that appeared legitimate and clicked a link to update their two-factor authentication, leading to a social engineering bypass of their account’s security. While this is unfortunate, it is a common occurrence, as humans are prone to mistakes and often take the path of least resistance. Ken praised npm for a quick response, as the malicious package was only active for about three hours. He also mentions that his company’s build system, which runs an audit before installing, successfully blocked the malicious library from being introduced.

The hosts advocate for an “assumed breach” mindset, which treats a security incident not as a matter of “if” but “when”. Instead of wasting time and resources on training employees to avoid every possible phishing email, organizations should focus on building robust defense mechanisms to mitigate the damage once an attack inevitably succeeds, including strengthening controls like logging, monitoring, and alerting. Organizations must regularly test their incident response plans and practice flexing those muscles, because without proper preparation, an incident will likely be harder than you expected and expose you for longer than you’d like. Also worth remembering that, with the development of AI, phishing attempts are more sophisticated and realistic, making a strong security posture even more critical.

Are you looking to bulk up your clothing collection for fall weather? Well, the Absolute AppSec merch store might just have what you’re looking for. Pick out a hoodie or beanie to keep your core temperatures nice and toasty.

And, if you have thoughts you’d like to share with Seth and Ken and the Absolute AppSec audience, join us in Slack. Show topics often originate as discussion points with the Slack audience, and the newsletter incorporates that discussion as well. So, join in there to offer ideas for what you’d like the podcast to cover as well as pick the brains of a number of industry experts.

Stay Secure,

Seth & Ken

Related Episodes

https://www.youtube.com/live/DEYR7pZXJyk - Episode #230 - False Positives vs. Negatives, Scaling Vuln Management - An earlier discussion of False Positives and False Negatives, pitting them head-to-head which is worse for organizations.

https://www.youtube.com/watch?v=3AcL_37gnhY - Episode #170 - Security Basics, Social Engineering, Plan for Failure - In this discussion, Ken and Seth point out that if we as a security team or organization design for failure, when the inevitable happens, we’ll know that our organizations can recover. More support for how an org like npm responds so well to something like the supply-chain breach in the news this week.

https://www.youtube.com/watch?v=-yXMLsqmMpw - Episode #45 - Sean Poris - product security, bug bounties, assurance, and more - Sean, now PwC, formerly Yahoo and Verizon (where he was at the time of this episode), is a long-time industry stalwart who’s thought long and hard about getting good results out of AppSec and ProdSec programs.

Absolute AppSec Happenings

https://www.dryrun.security/blog/meet-code-insights-mcp-your-secure-code-concierge - A shoutout to Ken and DryRun, they have released a new MCP server that allows for easy query of findings and code changes across all of the pull requests monitored by their tool. Super easy way to get ask about significant (or not) changes to code even when it’s not documented well.

https://dayvster.com/blog/in-defense-of-cpp/- “In Defense of C++” by Dayvi Schuster, in which we hear the arguments for why C++ remains a useful, viable language. Key graf: “With the right approach and mindset, C++ can be a joy to work with and can yield high-performance and efficient applications.”

https://anycrap.shop/ - In the age of AI, we have a site that will create any product you can imagine. Take a look at this thing we generated: a cake plate the cake slides off of. https://anycrap.shop/product/cake-plate-the-cake-slides-off-of.

Upcoming Events

Where in the world are Seth and Ken?

September 23-25, 2025 - Enterprise Tech Leadership Summit - Las Vegas, NV - Ken will be attending so check in with us in the slack if you’d like to see about catching up with him for a pseudo-extension of Vegas Summer Camp.

December 8-11, 2025 - Next-Gen Secure Code Review: Black Hat Edition - Seth and Ken are bringing a four-day exclusive course to Black Hat Europe in London, UK. This is a great opportunity to get a truly in-depth understanding of Secure-Code Review and how it can be empowered through LLM-tooling. Seth and Ken have innovated industry-leading trainings in both of these topics, so this four-day course promises to provide a lot of valuable insight.