Cross-Service Impact

Let’s examine how to properly review changes that affect critical user permissions and access controls, using a real-world example where a data corruption incident blocked developers from upgrading to mainnet and customers from completing onboarding.

The Scenario

A data corruption incident in the identity service affected the authorized representative (AR) flags, which led to:

1. Mint users are unable to continue onboarding after login.
2. Developers unable to upgrade to mainnet.
3. Diameter users not seeing AR flags.
4. Periodic review submission data potentially impacted.
5. Extended resolution time due to complex data dependencies.

While the immediate trigger was an untracked PR that accidentally removed an SQL WHERE clause, the broader root causes were:

1. Lack of proper error handling and fallbacks in dependent services.
2. No validation of affected row counts in SQL updates.
3. Missing monitoring for permission state changes.
4. Insufficient safeguards against unintended data modifications.
5. No automated checks for critical permission changes.

The incident started with a seemingly simple PR to fix a button in Diameter for re-designating ARs, but cascaded into a system-wide issue due to these underlying gaps.

The Code Change

-- Original query with proper WHERE clause
UPDATE user_records
SET is_authorized_representative = false
WHERE entity_id = :entityId
  AND email != :newArEmail;

-- PR change that caused the incident
-- TODO: Simplify the query to fix button not working
UPDATE user_records
SET is_authorized_representative = false;  -- WHERE clause accidentally removed!

Impact on Services

The lack of proper error handling and fallbacks meant that when AR flags were corrupted:

const MainnetButton = () => {
  const { data } = useQuery(USER_PERMISSIONS_QUERY);
 
  if (!data?.canUpgradeToMainnet) {
    return (
      <Tooltip content="Only the owner can upgrade this account to a production environment">
        <Button disabled>Upgrade to Mainnet</Button>
      </Tooltip>
    )
  }
 
  return <Button>Upgrade to Mainnet</Button>
}

const MainnetButton = () => {
  const { data, error } = useQuery(USER_PERMISSIONS_QUERY);
 
  if (error) {
    return (
      <ErrorBoundary fallback={
        <ServiceErrorMessage
          message="Unable to verify permissions"
          action="Please contact support if this persists"
        />
      }>
        <Button disabled>Upgrade to Mainnet</Button>
      </ErrorBoundary>
    )
  }
 
  if (!data?.canUpgradeToMainnet) {
    return (
      <Tooltip content="Contact support if you believe you should have access. Reference ID: AR-ACCESS">
        <Button disabled>Upgrade to Mainnet</Button>
      </Tooltip>
    )
  }
 
  return <Button>Upgrade to Mainnet</Button>
}

PR Comment

Choose the comment that you think is the most constructive and helpful.

Nicky Coffeecommented on Apr 2, 2025

The access control changes look good, we can merge this.

🚫

Less Helpful Approach

• Doesn’t verify data integrity
• Ignores monitoring needs
• No mention of testing strategy
• Misses cross-service impact

Kelcey Aquacommented on Apr 2, 2025

We should add some error handling here.

🚫

Less Helpful Approach

• Too vague about error handling needs
• Doesn’t address data validation
• No mention of monitoring
• Doesn’t help prevent similar issues

Dinah Aquacommented on Apr 2, 2025

The code looks good, but we should add monitoring and data validation.

⚠️

Better Approach

• Considers data integrity
• Suggests monitoring
• Still misses fallback handling
• Doesn’t address cross-service impact

Kamilah Amethystcommented on Apr 2, 2025

Thanks for improving the mainnet upgrade access! Before we merge, let’s ensure we have everything covered:

1. Can we add validation to ensure exactly one AR per entity?
2. Should we implement monitoring for AR flag inconsistencies?
3. Can we improve error messages to help support triage?
4. Have we tested the impact on related services (Mint, Diameter)?
5. Should we add automated checks for data integrity?

Let’s also document these changes in our service dependency diagram.

Most Helpful Approach

• Addresses data validation
• Suggests comprehensive monitoring
• Considers cross-service impact
• Promotes better documentation
• Helps prevent similar issues

Why Wasn’t This Caught?

Several factors contributed to this issue making it to production:

1. Untracked Change: The PR was created outside the normal process, bypassing initial review gates.
2. Missing Safety Checks: No automated validation of SQL changes that affect all records.
3. Limited Error Handling: Services assumed the AR flags would always be valid.
4. No Monitoring: No alerts for unusual permission changes or AR flag modifications.
5. Testing Gaps: Test data didn’t represent production scale.
6. Review Focus: Code reviews focused on the UI change, not the underlying data modification.