The Dangers of Coding With AI

My co-founder Jake and I have created a service for touring bands called BandAlert. The point of BandAlert is to make it easier for bands to get fans showing up at their gigs. The process is super simple: a fan can scan a QR code at a show, that QR code auto-populates a text message to subscribe to the band, and afterwards the fan gets a text message about new gigs the day before.

Jake just got back from NAMM where he talked with a bunch of bands, and I thought we should make it easy to promote the service to bands who want to try it out. So I’ve been working on reworking the registration path and building out a new flow where we can generate promotional codes on the admin backend and share those codes with bands so they can sign up and get three months of free service.

I’ve been doing most of my coding with Claude Code and have very detailed CLAUDE.md files in every directory that explain the coding standards, the infrastructure, and how to get new code written, including testing, coverage, linting, and all those things.

Security-First Abstractions

As I’m quite worried about LLMs introducing security vulnerabilities, I spend quite a bit of time figuring out the interfaces for new API routes. I’m using Node and Express, and I wrote an authenticated wrapper around the Express app that makes it obvious what kind of authentication and authorization is required. The route setup code doesn’t even get to see the Express app directly. This makes it impossible to accidentally create unauthenticated routes.

One of the new routes was the ability to validate a promotional code during band registration. A band can input the promo code and then see whether the code worked and what tier of service they got. This needed to happen triggered by the frontend for the UI flow and the user experience.

Of course, with my security background, one of the things that immediately occurred to me was that an adversary might want to enumerate our promo codes to find valid ones and use them. However, with the authenticated Express wrapper, I had also provided request handlers that apply various kinds of rate limits on a particular route.

Three Rounds to Get It Right

I was quite surprised that Claude Code initially created a route that wasn’t using the authenticated Express wrapper at all and wasn’t using any of the rate limiters:

1

app.post('/api/promo/validate', async (req: Request, res: Response) => {

No authentication. No rate limiting. Just a raw Express route wide open to the internet. When I noticed this I told Claude, “Look, you’re not supposed to use the Express app directly. We never use the Express app directly. Look at the other code. You need to use the authenticated Express wrapper to make sure that everything works properly.”

A few minutes later Claude wrote new code, and now it used the authenticated Express wrapper, but it was using the unauthenticated_post method and didn’t apply any rate limiting to it.

So I went back and said, “Look, this needs to be aggressively rate limited because we don’t want this to be a route that ends up being abused.” Claude then inserted the rate limiter, but it was still using an unauthenticated route:

1
2
3
4

authApp.unauthenticated_post(
    '/api/promo/validate',
    (req, res, next) => aggressiveUnauthenticatedRateLimiter(req, res, next!),
    async (req: Request, res: Response) => {

The context here is that band registration is only possible if the user already authenticated to the service. So the proper answer would have been an authenticated route with an authenticated rate limiter. I had to go back and forth multiple times before we ended up with what the code should have been from the start:

1
2
3
4
5

authApp.authenticated_post(
    '/api/promo/validate',
    AuthType.AUTHENTICATED_USER,
    (req, res, next) => aggressiveAuthenticatedRateLimiter(req, res, next!),
    async (req: Request, res: Response) => {

The whole point of the authenticated Express wrapper is that it makes the security posture of each route obvious at a glance: authenticated_post instead of unauthenticated_post, AuthType.AUTHENTICATED_USER spelling out who can access it, and aggressiveAuthenticatedRateLimiter tied to the authenticated user to prevent enumeration. All of this existed in the codebase already. Claude Code just didn’t use any of it.

The Bigger Picture

This is just one example, and lately I’ve been noticing that I need to make changes to almost every single file created by Claude Code. Overall progress has been much slower than it used to be.

For anybody who ends up using these AI coding tools, they probably make a lot of assumptions about the underlying security of the code that is being written. I have many more examples where Claude Code or other coding AI tools ended up writing code that was not secure. This is probably completely obvious for anybody who has been in this space, but it’s definitely something that needs to be top of mind.

My previous approach had been: I will work around these limitations by making sure that all of my interfaces and abstractions are security-first. That’s why I was so surprised that even though all of the other code in the BandAlert project uses these abstractions, the first version that Claude Code produced wasn’t using any of them.

Be safe out there. Write safe code. Don’t rely too much on these tools to keep you safe.