Tokens Are Liabilities: Surviving the Age of Credential Leaks

I once pushed an AWS access key to a public repository at two in the morning. By five past the hour, an automated bot had spun up thirty crypto-mining instances in us-east-1. That minor lapse in judgment cost a few thousand dollars and kept me awake for three days straight. When I read that Grafana recently suffered a GitHub token leak leading to a source code breach and a subsequent extortion attempt, my initial reaction was pure, unadulterated empathy.

Grafana handled the situation exactly as they should have. A threat group claiming to be the CoinbaseCartel demanded a ransom to prevent the release of downloaded source code. Following FBI guidance, Grafana refused to pay. They invalidated the credentials and audited their systems to confirm no customer data was accessed.

We all know the standard advice. Do not commit secrets. Use environment variables. Rotate your keys. But as a solopreneur who manages everything from database migrations to frontend deployments, I know that advice falls apart at scale. Systems grow complex. Tokens get passed around CI/CD pipelines and local development environments. They also end up in third-party integrations. Eventually, a token slips through.

Static tokens are liabilities. You cannot secure a system by asking developers to simply be more careful. You have to design an architecture where secrets are either ephemeral or entirely absent. Here is how we build that system.

The Problem with Personal Access Tokens

The core issue with GitHub Personal Access Tokens, and really any static API key, is their lifespan. A developer generates a token with read/write access to repositories. To avoid fixing a broken pipeline in six months, they set the expiration to "Never" and drop it into a CI environment.

If that environment is compromised, or if a rogue script logs the environment variables to a build console, the token is gone. The attacker now has the exact same privileges as the developer.

We need to shift from static credentials to identity-based access. If a system needs to read code, it should prove its identity at runtime and request a temporary lease. It must then discard the credentials immediately after the job finishes.

Solution 1: Ephemeral Access via OIDC

OpenID Connect is the grown-up way to handle CI/CD authentication. Instead of storing an AWS secret inside your GitHub repository settings, you configure your cloud provider to trust GitHub's identity provider.

When a GitHub Action runs, it requests a short-lived JSON Web Token from GitHub. It presents this token to AWS. AWS validates the signature and checks the repository name before granting temporary access.

Here is how you set up a GitHub Action to deploy a Node.js application to AWS without ever storing a static secret.

name: Deploy Application
 
on:
  push:
    branches:
      - main
 
permissions:
  id-token: write
  contents: read
 
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4
 
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsDeployRole
          aws-region: us-east-1
 
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
 
      - name: Install and Build
        run: |
          npm ci
          npm run build
 
      - name: Deploy to S3
        run: aws s3 sync ./dist s3://my-production-bucket

The critical piece here is id-token: write. This tells GitHub to mint an OIDC token for this specific workflow run. If an attacker breaches the repository and tries to extract secrets, they will find an empty vault. There are no static AWS keys to steal.

Solution 2: Automated Secret Scanning

OIDC solves infrastructure deployments, but we still have third-party API keys. Services like Stripe and SendGrid, along with OpenAI, do not universally support OIDC for server-to-server communication. You are forced to use static keys.

I rely on pre-commit hooks to catch these keys before they enter the git history. Relying on remote scanners is a mistake. Once the key hits the remote server, you have to treat it as compromised.

Here is a lightweight, dependency-free TypeScript script I use to scan staged files for common token patterns. You can hook this into Husky or your preferred git hook manager.

import { execSync } from 'child_process';
import { readFileSync } from 'fs';
import { exit } from 'process';
 
// Define regex patterns for high-risk tokens
const SECRET_PATTERNS = [
  { name: 'GitHub PAT', regex: /(ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9_]{36}/ },
  { name: 'AWS Access Key', regex: /AKIA[0-9A-Z]{16}/ },
  { name: 'Stripe Secret Key', regex: /sk_(test|live)_[0-9a-zA-Z]{24}/ },
  { name: 'Slack Bot Token', regex: /xoxb-[0-9]{11}-[0-9]{11}-[a-zA-Z0-9]{24}/ }
];
 
function getStagedFiles(): string[] {
  try {
    const output = execSync('git diff --cached --name-only', { encoding: 'utf-8' });
    return output.split('\n').filter(file => file.trim().length > 0);
  } catch (error) {
    console.error('Failed to retrieve staged files.');
    return [];
  }
}
 
function scanFiles() {
  const files = getStagedFiles();
  let foundSecrets = false;
 
  for (const file of files) {
    try {
      const content = readFileSync(file, 'utf-8');
      const lines = content.split('\n');
 
      lines.forEach((line, index) => {
        for (const pattern of SECRET_PATTERNS) {
          if (pattern.regex.test(line)) {
            console.error(`\n[!] SEVERE: Possible ${pattern.name} found in ${file} at line ${index + 1}`);
            console.error(`    > ${line.trim()}`);
            foundSecrets = true;
          }
        }
      });
    } catch (err) {
      // Skip binary files or unreadable directories
      continue;
    }
  }
 
  if (foundSecrets) {
    console.error('\nCommit blocked. Please remove the secrets from your code.');
    exit(1);
  }
}
 
scanFiles();

I keep this script in an .internal directory and wire it to run on pre-commit. While imperfect and unable to catch obscure custom database passwords, it stops the catastrophic ones. The goal is to build a defense-in-depth system that saves you from your own late-night fatigue.

Solution 3: Runtime Secret Fetching

Many developers load all API keys into a .env file during local development and deploy those same variables into their cloud provider's environment settings. This creates sprawling copies of the same secret.

I prefer fetching configuration at runtime from a centralized vault. This allows you to rotate a key in one place without redeploying your entire fleet of services.

Here is a TypeScript utility using the AWS SDK v3 to fetch configuration dynamically when a Node.js server starts.

import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';
 
interface AppConfig {
  databaseUrl: string;
  paymentApiKey: string;
  emailProviderKey: string;
}
 
const secretsClient = new SecretsManagerClient({ region: 'us-east-1' });
 
let cachedConfig: AppConfig | null = null;
 
export async function loadAppConfig(): Promise<AppConfig> {
  if (cachedConfig) {
    return cachedConfig;
  }
 
  try {
    const command = new GetSecretValueCommand({ SecretId: 'production/app-config' });
    const response = await secretsClient.send(command);
 
    if (!response.SecretString) {
      throw new Error('Secret string is empty');
    }
 
    const parsedConfig = JSON.parse(response.SecretString) as AppConfig;
    
    // Cache the config to prevent repeated API calls
    cachedConfig = parsedConfig;
    return cachedConfig;
 
  } catch (error) {
    console.error('Failed to load secrets from vault. Application cannot start.');
    process.exit(1);
  }
}

You call this function once during your server bootstrap process, right before you start listening for incoming requests.

import express from 'express';
import { loadAppConfig } from './config';
 
async function bootstrap() {
  const config = await loadAppConfig();
  const app = express();
 
  app.post('/charge', async (req, res) => {
    // Use config.paymentApiKey directly in memory
    // No environment variables required
  });
 
  app.listen(3000, () => {
    console.log('Server is running with secured configuration');
  });
}
 
bootstrap();

By keeping the secrets in memory and out of the environment variables, you eliminate the risk of a misconfigured logger dumping process.env to Datadog or CloudWatch.

Wrapping Up

We love to judge major companies when they end up in the news for a breach. But building software is chaotic. We leave a trail of tokens and passwords, along with service accounts, in our wake. Grafana caught and contained their leak before refusing to play the extortion game. They did the hard things right.

As engineers, our job is to assume the breach will happen. Design your systems so that when a token inevitably leaks, it is either already expired or scoped down to absolute uselessness. Kill static keys and enforce runtime identity so you can sleep a little better at night.