Skip to Content
LearnCode Review5 WhysInvalid Paymasters

Invalid Paymasters

This learning is based on a real production incident that affected customers.

Let’s examine how to ensure code and design reviews properly consider feature-flag isolation and downstream product impact, using a real-world example where newly onboarded v0.7 paymasters were inadvertently assigned to test-net policies, breaking smart-contract-account (SCA) transactions for new entities.

The Scenario

While the Paymaster Gas Station (PGS) team was preparing future support for ERC-4337 v0.7, here’s a breakdown of what happened:

  1. The team was preparing for ERC-4337 v0.7 support by adding new paymasters
  2. They added unfunded v0.7 paymasters to production for pre-registering signers
  3. The policy creation code assumed only one paymaster per chain existed
  4. The code simply took the first paymaster record from the database
  5. Due to database ordering, ~1 in 16 new entities got assigned the unfunded v0.7 paymaster
  6. Transactions using these unfunded paymasters failed with “AA31 paymaster deposit too low” error
  7. Unit tests only covered the happy path with mocked data
  8. Integration tests for multiple paymaster scenarios were missing
  9. The error was classified as a client-side issue, bypassing monitoring
  10. The issue went undetected for 26 hours
  11. Mitigation took 2.5 hours (removing v0.7 paymasters and fixing policies)
  12. Two customers were affected with four failed transactions on testnet
paymaster.go
func (r *repoImpl) GetCirclePaymasterByBlockchainEP(
  ctx context.Context,
  blockchain common.Blockchain,
  entryPoint *string,
) (*model.Paymaster, error) {
  res, err := r.getPaymastersByBlockchainsAndProviderEP(
    ctx,
    []common.Blockchain{blockchain},
    model.PaymasterProviderCircle,
    entryPoint,
  )
  if err != nil || len(res) == 0 {
    return nil, err
  }
 
  // No error handling or fallback
  return &res[0], nil
}

PR Comment

Choose the comment that you think is the most constructive and helpful.

Click here to learn more

Key Lessons

1. Understanding Product Impact

  • New infrastructure (v0.7 paymasters) must remain isolated until end-to-end ready
  • Random selection of an unfunded paymaster directly blocks user onboarding flows
  • Client-classified errors can mask server defects; classify and alert on symptoms too

2. Testing Strategy

  • Don’t rely solely on mocked unit tests for infra changes
  • Add integration tests that create new entities and exercise default-policy paths
  • Include scenarios with multiple paymasters per chain and verify funded-status checks

3. Code/Design Review Best Practices

  • Challenge hidden assumptions (e.g., “exactly one paymaster”)
  • Require evidence that feature-flag boundaries are enforced
  • Ensure reviewers look for observability gaps and alert coverage

Tips for Reviewers

1. Ask Product-Focused Questions

  • How does this change affect first-time wallet creation UX?
  • What fallback exists if the chosen paymaster is unfunded?
  • Could the error rate increase silently?

2. Verify Testing Strategy

  • Do tests cover multiple-paymaster scenarios?
  • Is there an automated flow that funds or validates new paymasters?
  • Are alerts routed to on-call within minutes?

3. Document Dependencies

  • List paymaster contracts and their funded status per env
  • Note feature flags gating v0.7 traffic
  • Link associated cleanup or rollback tools

Common Pitfalls to Avoid

1. Assuming One Resource Instance

  • ❌ “There’s always one paymaster per chain.”
  • ✅ “Selection logic handles >1 paymaster deterministically.”

2. Treating Client Errors as Non-Critical

  • ❌ “AA31 is a client problem.”
  • ✅ “AA31 could signal an unfunded paymaster—alert and investigate.”

3. Skipping Integration Tests for “Simple” DB Changes

  • ❌ “Unit tests mock the DB; we’re good.”
  • ✅ “Integration tests verify real DB ordering and defaults.”

Remember: robust reviews weigh technical correctness, feature-flag isolation, and product impact. Catching hidden assumptions early saves customers from production outages!

Last updated on
Input ValidationLaunchDarkly