Fallback Mechanism Between Anthropic and OpenAI API

⚠ Important: Before implementing, consult with your legal team to ensure compliance with the providers' Terms of Service. Lunar.dev provides a platform for building API middleware and tools to work within API provider boundaries, but it is the user's responsibility to enforce compliance correctly.

Use Cases

• High Availability and Redundancy – Ensures uninterrupted AI services for real-time applications like chatbots or automation tools. If Anthropic’s API experiences downtime or throttling (429 errors), OpenAI’s API is automatically used.
• Cost Optimization – Businesses can balance AI model usage dynamically, using the provider that is most cost-effective for specific tasks.‍
• Performance Optimization – OpenAI and Anthropic models have different latency and computational strengths. This mechanism switches providers dynamically based on the fastest response time.

Solution Structure

Step-by-Step Breakdown

1. Triggering Fallback Based on API Throttling – If Anthropic returns a 429 Too Many Requests response, the request is filtered and rerouted to OpenAI.
2. Transforming Requests Between Anthropic and OpenAI – A custom Translator processor reformats Anthropic API calls to OpenAI’s API structure, preserving request intent.
3. Executing API Calls to the Selected Provider – Once transformed, the request is forwarded to OpenAI. The response is then returned to the user.‍
4. Switchback Timer Mechanism – When Anthropic is available again, requests gradually shift back from OpenAI to Anthropic using a custom Timer processor.

Concrete Example

Problem:
An AI-powered legal research assistant uses Anthropic’s Claude API for document analysis. During peak usage, Anthropic enforces strict rate limits (429 errors), temporarily rejecting requests. The assistant cannot afford service interruptions, and latency needs to remain low.

Solution:
By implementing a fallback mechanism, when Anthropic throttles requests, the assistant automatically switches to OpenAI’s GPT API. Once Anthropic is available again, the system gradually redirects traffic back to Anthropic instead of abruptly switching providers.

Caching Flow YAML Configuration

endpoints:
  - url: "https://api.anthropic.com/v1/complete"
    method: POST
    remedies:
      - name: Filter Processor  # Detect Anthropic API throttling
        enabled: true
        config:
          match_condition:
            status_code: 429  # Trigger fallback when Anthropic returns 429
          action: reroute
          target: "translator"

      - name: Translator Processor  # Custom processor to convert Anthropic requests into OpenAI requests
        enabled: true
        config:
          source_format: anthropic
          target_format: openai
          mappings:
            model: "model"  # Directly map model names
            input: "prompt"  # Anthropic uses 'input', OpenAI uses 'prompt'
            temperature: "temperature"
            max_tokens: "max_tokens"
          additional_parameters:
            openai_version: "gpt-4"

      - name: API Request Processor  # Executes the transformed API call
        enabled: true
        config:
          provider: "openai"
          authentication: "api_key"
          timeout_seconds: 10

      - name: Timer Processor  # Custom processor to gradually shift traffic back to Anthropic
        enabled: true
        config:
          wait_time_seconds: 300  # Wait 5 minutes before switching back
          retry_policy: gradual
          retry_interval_seconds: 60  # Check availability every 60 seconds

Explanation:

• Requests to Google Maps API’s geocoding endpoint are intercepted.
• Cached responses are indexed using URL, query parameters, and headers, ensuring that different queries don’t overwrite each other.
• Cached entries expire after 24 hours to prevent stale data.
• The middleware stores full responses to serve repeated requests instantly.
• Optional size limits help manage cache memory usage.

This approach reduces API costs and latency, ensuring that frequently requested geolocation data is quickly available while staying within API limits.

Final Notes

This fallback flow ensures uninterrupted AI model availability, cost savings, and performance optimization by dynamically switching between Anthropic and OpenAI. The Filter processor detects throttling, the Translator processor reformats API requests, and the Timer processor ensures a smooth transition back to Anthropic.