How to Manage Open Source Security Vulnerabilities in the Age of AI Scanning (Without Shutting Down Your Repos)

Introduction

Recent news from the UK's National Health Service (NHS) highlights a growing tension: open-source repositories are being shut down because advanced LLM tools like Anthropic's Mythos are getting better at finding security vulnerabilities. But is closing everything down the right move? As Terence Eden, a former NHSX engineer, points out, most NHS repos contain data sets, internal tools, guidance, and front-end designs that pose zero security risk. The COVID contact tracing app was open-sourced without any security incidents. And this contradicts the UK's Tech Code of Practice point 3: "Be open and use open source."

This guide will show you how to protect your organization's open source code from AI-powered security scanning without resorting to full closure. You'll learn to assess risk, implement smart security measures, and maintain transparency—all while staying aligned with open source best practices.

What You Need

• An inventory of all your open source repositories – including datasets, internal tools, guidance, research tools, and front-end design repos.
• Security scanning tools – both traditional and AI-powered (like Mythos or similar).
• A risk assessment framework – e.g., CVSS scoring or custom criteria.
• Access to the UK Tech Code of Practice (or similar government guidelines) for reference.
• Documentation of your code's security history – especially if you've open-sourced sensitive projects before.
• A cross-functional team – including developers, security engineers, legal/privacy experts, and a decision-maker.

Step-by-Step Guide

Step 1: Audit Every Open Source Repository

Start by listing all repos your organization maintains. Categorize each as:

• Low risk: datasets, guidance documents, internal tooling, front-end design files, research tools.
• Medium risk: libraries or APIs that process non-sensitive data.
• High risk: code handling personal data, authentication, or national security.

Review each repo's contents and past security incidents. As Eden notes, the majority of NHS repos are low risk—"nothing in them which could realistically lead to a security incident." Don't assume all repos are equally dangerous.

Step 2: Run a Controlled AI Security Scan

Instead of reacting to the threat of LLM scanners with a blanket takedown, run the same AI tools on your repos in a controlled environment. Use a staging copy if possible. Record the findings. For low-risk repos, the results will likely show no meaningful vulnerabilities. For high-risk repos, you'll get actionable alerts.

Step 3: Assess Risk vs. Benefit

For every vulnerability found, ask:

• Is the vulnerability exploitable in a real-world context?
• Does it affect sensitive data or infrastructure?
• How critical is the open-source release to public trust, collaboration, or innovation?

Remember: the NHS's COVID app was open-sourced on day one despite extreme scrutiny, and it caused zero incidents. Transparency builds trust and improves code quality.

Step 4: Implement Targeted Protections, Not Blanket Closure

For high-risk repos, consider:

• Delayed release – publish code after a security review, not proactively.
• Sanitized versions – remove sensitive strings, API keys, or internal notes.
• Private forks with public summaries – share high-level architecture without exposing exploitable code.
• Bug bounty programs – encourage responsible disclosure before LLMs find flaws.

For low- and medium-risk repos, keep them fully open. The benefit far outweighs the risk.

Step 5: Align with Existing Open Source Policies

Check your country's digital service guidelines. In the UK, the Tech Code of Practice explicitly states: "Be open and use open source." A decision to close all repos contradicts this. Document how your security measures still honor the spirit of openness.

Step 6: Communicate Your Strategy Externally

Publish a brief public statement explaining your approach. Highlight that you are not abandoning open source but are adopting smart, risk-based protections. This maintains community trust and prevents backlash. Use the original NHS situation as a counterexample to show you learned from it.

Step 7: Monitor and Iterate

Set a schedule for re-scanning repos (e.g., quarterly or after major LLM updates). As AI scanning evolves, adjust your protection levels. Keep a log of decisions and revisit them annually.

Tips for Success

• Don't panic. The vast majority of open-source repos are not critical security risks. As Eden says, "the open source code caused zero security incidents" for the COVID app.
• Involve the community. Let developers and users know your reasoning. They can help spot false positives.
• Use your own data. If you have historical evidence of safe open-sourcing (like NHSX did), rely on it.
• Consider a vulnerability disclosure policy – make it easy for researchers (and AI scanners) to report issues responsibly.
• Remember the principles. Open source isn't just about code; it's about transparency, collaboration, and public good. Closing repos should be the last resort, not the first.