Episode #266

This week on Episode #266 of Absolute AppSec, Seth (@sethlaw) and Ken (@cktricky) tackle the unrealistic expectations of penetration testing, inspired by recent posts from Andrew Wilson in our slack channel. Clients often expect that a single penetration test finds every vulnerability possible, whether or not those expectations are appropriate. The duo shares their experience with those expectations as AppSec professionals, reflecting on how it has affected their respective careers. To watch this episode, go to https://www.youtube.com/watch?v=WDlElw15xHY, or find Absolute AppSec wherever you get your podcasts.

Ken and Seth are excited to announce a new virtual training course on harnessing LLMs for Application Security. Set for December 2-3, this course will guide participants through threat modeling, code analysis, and efficient AppSec practices using LLMs. For those curious about tools like LangChain and Retrieval-Augmented Generation (RAG) in AppSec, register at https://training.absoluteappsec.com.

Penetration tests should be viewed as a risk-based approach, focusing on identifying significant vulnerabilities within a set timeframe rather than exhaustively examining every potential flaw. Our hosts emphasize the importance of setting clear client expectations— penetration testing is a snapshot in time, and new vulnerabilities can emerge as an application evolves. In reality, continuous testing and a realistic understanding of what testing can achieve are more effective for good security posture. This, in combination with the lack of standardization in the penetration testing industry, results in a disconnect between client expectations and service provider practices. Ken and Seth discuss how terms like “pen-testing,” “vulnerability assessment,” and “red teaming” are often used interchangeably, creating confusion over what each service actually entails. The lack of precise terminology is indicative of a broader need for the cybersecurity industry to formalize its practice and better communicate what should be expected from a penetration test. They suggest that, while different vendors may use varying methodologies, clients should focus on the quality of the results over the specific technical methods used. Greater transparency and formalization in the industry could help clients make more informed decisions and ensure that they are getting the services they need. Security is a continuous journey, not a destination that can be reached with a one-time test. While penetration testing is valuable, security assessments should be scheduled regularly to adapt to the ever-evolving landscape of threats and vulnerabilities.

In discussing the particular security needs of product-focused teams, Ken and Seth explore the challenges that arise when teams face a flood of low-priority issues. Automated tools often generate extensive reports, leading to a backlog of bug tickets for minor or informal findings that distract from higher-impact vulnerabilities. Ken shares how their product takes a layered approach to prioritize findings by looking at the developer’s familiarity with the code, the frequency of changes, and the importance of the affected areas within the application. This context-driven analysis allows security teams to focus on the most significant risks and avoid becoming bogged down by low-impact issues. Educating clients on this quality-over-quantity approach enables development and security teams to stay agile and focused on meaningful security improvements.

“Attack modeling” is a new approach to application security that’s gaining traction as an alternative to traditional threat modeling. While threat modeling focuses on profiling types of attackers, such as insider or external threats, attack modeling zeroes in on the specific types of attacks that align with an application’s vulnerabilities. Ken and Seth explain how attack modeling allows security teams to prioritize defenses against realistic and impactful threats. For example, instead of considering all potential threat actors, attack modeling emphasizes the probable attack vectors and exploits that could be used against an application’s specific architecture. This shift in focus can help teams better align their security controls with the actual risks their applications face, making security strategies more effective and less theoretical. By concentrating on the most relevant attack scenarios, attack modeling helps streamline security efforts and ensure that the most critical protections are in place to counter genuine risks.

If you haven’t yet, get festive and show your support of Absolute AppSec by visiting the merch store and picking up our limited-edition holiday items. As always, join Seth and Ken in Slack.

Stay Secure,

Seth & Ken

Related Episodes

https://youtu.be/wNODQSmGUTQ - Episode #182 - Twitter, LastPass, Testing Edge Cases - Seth and Ken express opinions about disclosures and building out security programs. Further discussion on password managers and LastPass breach. Finally, a bug bounty report shows the importance of testing edge cases and using a bounty program to supplement integration testing.

https://youtu.be/4xYJPyBo9XE - Episode #161 - Language Semantics, Blockchain Validations, Pentest Stories - A blast from the past as Ken and Seth reminisce about past penetration testing and security stories. A discussion of language semantics and how programming language basics are similar to spoken language basics.

https://youtu.be/HlGcJRhgNG0 - Episode #92 - Working from Home, Skreen, Evolution of AppSec - Thoughts on the evolution of application security and penetration testing since the beginning of our careers.

Absolute AppSec Happenings

https://techcrunch.com/2024/11/05/mozilla-foundation-lays-off-30-staff-drops-advocacy-division/ - Mozilla lays off a significant portion of people. Overall, whether this is a positive or negative to browser security is up in the air. Being limited to a single browser engine across every browsing device could have unforeseen consequences.

https://www.darkrelay.com/post/http-security-headers - Dark Relay has created a resource for understanding the various security headers. Good background for anyone in security or development to brush up on what various headers are intended to do and whether they have a security impact.

https://github.com/ghostsecurity/reaper - New tool from Ghost security that integrates AI into application security testing. These sorts of integrations will become more and more common as it proves effective and we learn to manage expectations and make AI efficient for our use cases.

Upcoming Events

Can you spot the individuals wearing crocs and socks?

November 21-22, 2024 - DeepSec, Vienna, Austria - Seth will be presenting “Modern vs. 0ld 5k00l” where everything old is new again.

December 2-3, 2024 - Harnessing LLMs for Application Security - Virtual Training, dates may change - A newly developed course that focuses on strategies for using more than just the chat interface of an LLM to secure software.

February 14-15, 2024 - CactusCon 13, Mesa, AZ - Both Seth & Ken have submitted here and will most likely attend even if the talks and panels don’t get picked up. A great regional conference that has been running for over a decade!