Feature Flag Variations

This learning is based on a real production incident that affected customers.

Let’s examine how to properly test feature flags and their variations, using a real-world example where incorrect flag configuration led to disabled blockchain withdrawals.

The Scenario

A code change introduced a bug that affected Circle Mint UI, preventing customers from withdrawing more than 1MM on-chain.

The 1MM withdrawal limit was incorrectly applied to all chains instead of just ACE chains.
The feature flag did not prevent the issue due to incorrect assumptions about the query results when the flag was off.
Manual testing did not cover scenarios with the feature flag turned off.
Lack of unit tests for scenarios with the feature flag turned off.
No standardized practices for verifying feature flag scenarios in different environments.

Before

Blockchain.ts


  const aceFiltered = currencyFiltered.filter((chain) =>
    isAceSupported != null ? chain.isAceSupported === isAceSupported : true,
  )
 
  return sortBy(aceFiltered, 'name')

PR Comment

Choose the comment that you think is the most constructive and helpful.

Layne Bronzecommented on Apr 2, 2025

The withdrawal limit logic looks good. Have we verified that this limit is applied to ACE chains?

Janaya Greencommented on Apr 2, 2025

The withdrawal limit logic looks good. Can we add a unit test to ensure this limit is only applied to ACE chains and not to others?

🚫

Less Helpful Approach

Suggests adding a unit test but does not specify what the test should cover.
Does not check if there are existing tests that could be modified or extended.
Does not provide details on how to implement the unit test or what specific conditions it should verify.
Does not consider other potential issues related to the withdrawal limit, such as performance impacts or integration with other systems.

Lexis Tancommented on Apr 2, 2025

The withdrawal limit should only apply to ACE chains. Have we considered the scenario where the feature flag is turned off? It would be good to add a test case for this scenario.

Edyth Peachcommented on Apr 2, 2025

Thanks for updating the withdrawal limit for ACE chains! Before we merge, let’s ensure we understand the full impact:

Since it’s relying on a modification to an existing function, does upstream usage need an update? We’ll want to ensure it doesn’t introduce breaking logic changes.
Can we make sure getBlockchains is the correct function to use so that there’s no unintended changes in the future?
Do we have tests to cover the success and error cases? We’ll want to at least cover the over/under cases for ACE and non-ACE.
Since there’s a feature flag involved, we’ll need to ensure tests are run against the flag being on and off.
We’ll want to make sure the feature flag logic is clearly separated so avoid problems if/when the flag is removed.
We’ll want to follow up with the team to discuss when and how feature flags should be applied.

This will help us catch any unintended consequences before deployment.

Click here to learn more

Key Lessons

1. Feature Flag Handling

Thoroughly consider all scenarios, especially when feature flags are turned off.
Feature flags being turned off should be considered the default case.
Document the intended behavior of feature flags.
Verify feature flag interactions in different environments.
Test all affected features with feature flags both on and off.

2. Review Process

Ensure functions works as documented
Explicit functions are better than implicit functions
Changes should include corresponding tests
Ensure tests are run against a combination of inputs and scenarios

3. Best Practices

Standardize feature flag management processes.
Create clear documentation for feature flag usage.
Avoid confusion by implementing feature flag logic in a single location

Tips for Reviewers

1. Ask Rule-Focused Questions

How should this logic combine?
What’s the intended behavior?
How do we verify the logic?
Example: “Does using this flag produce the correct result?“

2. Verify Testing Strategy

Are all affected features tested?
Do we have proper test coverage?
Did we evaluate the possible scenarios?
Example: “Let’s try with the flag on and off”

3. Document functions

Shared functions should work as documented
Ask questions if there is logic that cannot be explicitly configured, but can affect the results
Ensure functions can maintain backwards compatibility
Example: “We’ll need to add the feature flag logic in, but make sure the default case produces the same result”

Common Pitfalls to Avoid

1. Focusing on the feature flag being enabled

❌ “The flag changes look good, let’s merge.”
✅ “The flag changes look good, but we need to make sure the default case is still valid”

2. Implicit implementations of a feature flag

❌ “x() will produce the new result format without any new arguments”
✅ “x() needs a new argument to produce the new result format else it will produce the original”

3. Scattered implementations of a feature flag

❌ “The feature flag is added to the shared utility functions and can be used without changes in our top level UI function”
✅ “This UI function has a feature flag, so we’ll need to call the utility functions differently”

Remember: Feature flags cause a split in potential scenarios. Make sure to verify each one.