How Lunar.dev Helped BitDam Manage 100K of Weekly API Calls

‍

Blocking malicious email-driven attacks is a challenging endeavor. For BitDam, a cyber threat detection company, it required making hundreds of millions of weekly API calls to Microsoft 365. Through these calls, they can scan all emails and prevent malicious phishing and malware attacks.

Tell Us About BitDam

The 40-people engineering organization with DevOps, backend developers and additional engineering roles - had developers with a specialized focus on managing Microsoft 365 APIs. This is no small feat, given the complexities involved.

Their responsibilities included maintaining API changes to ensure compatibility, ensuring the throttling threshold is met to prevent penalties, error handling due to invalid requests or server issues, managing token lifecycles, handling issues when API calls fail, and more.

What were BitDam’s API usage challenges?

The engineering team was making hundreds of millions of weekly API calls to Microsoft 365. This was essential for real-time scanning of emails. However, as expected, such a high volume of API requests caused throttling issues. Every time we reached the rate limit we got a 429 (Too Many Requests) error. This required us to wait before we were able to make a new request. This is not only annoying, it also put our business model at risk. We needed to be able to scan emails in real-time to prevent malicious attacks. 

‍

We used two main mechanisms to solve this and reduce the number of calls we made. First, on user signup, a webhook was set to trigger alerts for new emails. However, it wasn't completely reliable. Second, we implemented a polling mechanism. A scheduled task polled for new emails every two minutes.

Both approaches had complexities and weren’t able to completely avoid meeting rate limits  and the subsequent errors from Microsoft 365. There were also rate limit penalties.

To mitigate this, logic was added to control the polling frequency. These requests were scoped per user and per tenant, adding another layer of complexity. 

This issue was a constant headache for developers. They were constantly getting alerts about issues. For example, scaling up led to a surge in error alerts. The problem was never fully resolved, requiring ongoing attention.

How did Lunar.dev help manage these errors in production?

Before Lunar.dev, we tried rolling our own API middleware to tackle Production issues. However, one of the challenges was unexpected management issues that kept popping up.

Lunar.dev handles throttling, token management, and multi-tenancy out of the box. The solution is based on industry knowledge and best practices, saving us the time and effort of trying to deal with all the third-party API complexities.

What is game changing about using Lunar.dev for API usage management?

Lunar.dev is an out-of-the-box solution that cuts down time, effort and resources spent on API consumption and management. The tool is developed with knowledge on edge cases and best practices that takes developers time to acquire. With Lunar, they don’t have to.

In addition, Lunar.dev developers can implement advanced logic to tackle issues, saving even more time.

Why should organizations care about their 3rd-party API usage?

When scaling, API consumption and management is especially important. By optimizing API calls, organizations can save time and cut down on traffic and compute costs. For example, if an organization makes hundreds of millions of API requests a week, like we do, even a 1% improvement is significant.

Lunar.dev provides a service that does that for organizations, letting them focus on their core competency.

Which types of companies should use a solution like Lunar.dev?

Any company should be interested in Lunar.dev. Early-stage companies will benefit the most from Lunar. It saves the hassle of building an API management solution from scratch. Enterprises that are already developing integrations with cloud services and other third party tools can also greatly benefit from Lunar. They are already experiencing the challenges of API consumption and management and Lunar can help them.

Within these companies, Lunar.dev can significantly help engineering managers, since the platform is key to cleaning up that backlog and reducing technical debt.

Do you have a tip to share on 3rd-party API management?

There will be a lot more APIs in your future of businesses as the API economy propels forward. Prepare for a rainy day by investing in API management solutions like Lunar.dev. It's better to be proactive than reactive when the API issues pile up.

‍

BitDam has been acquired by Datto, the leading global provider of cloud-based software and technology solutions purpose-built for managed service providers.