Optimizing Lunar.dev: Why We Prototyped in Python But Embraced Go for the MVP

Into the Void of Prototyping

In 2022, our journey at Lunar.dev began with a simple idea and a vision for a Minimum Viable Product (MVP). Geared with just the concept of optimizing 3rd party API usage with an OSS agnostic tool, we embarked on a mission fueled by interviews and extensive user research. We had hired our first engineers - shout out to Jonathan Lande (today our R&D Team Lead!) and Eitan Goldberg - and set sail with prototyping.

“Now what”? As the development phase loomed ahead, we faced a pivotal fork down the road - Should we favor development velocity or product performance?

At this juncture, before we even proved a problem-solution fit, we decided to dedicate 6 months working with design partners building a prototype that we will later discard.

So we started prototyping in Python trying to solve non-production issues. We were running in a low-scale API-usage environment. Back then, our main value was the new layer of visibility we provided our design partners, showing them a deeper understanding of how API consumption issues affect the product they are building.

We quickly realized that if we want to solve the issues we help our partners discover, we need to build a production-ready API-proxy. And we would need a language that could deliver solid performance without compromising on other essential factors to achieve that mission. Quicky recap: The purpose of an API proxy is to act as an intermediary between one or more applications and the APIs they use. It handles the request and response process, providing a layer of abstraction and control between the applications and the APIs. [We wrote quite a lot about what's an API-proxy or egress proxy here].

Trading Python for Go

‍

While our core founding team had tons of knowledge in Python. Their background comes from Cybersecurity where low-level and opcode level optimization can have a huge impact on application performance. However, we then realized that Python was way too high-level for our mission.

On the other hand, C/C++ would slow down our development velocity. Go may not be the absolute fastest language available, it certainly outpaces contenders high-level scripting languages like Node.js and Python, while providing a more robust development experience than low-level languages such as C/C++.

One of the compelling aspects of Go is that it resides in a sweet spot within the performance-velocity-accuracy triangle. It offers commendable speed while maintaining a balance between accuracy and development speed.

This equilibrium suited our project requirements perfectly.

‍

Memory management was another critical concern. In languages like C, memory is not managed for you, leaving significantly more room for errors and potential security breaches. However, Go's memory management system is a boon. It promotes writing code with fewer bugs, unlike languages with garbage collectors that may not guarantee uninterrupted execution, especially during memory peaks.

One of Go's standout features is its handling of pointers. Developers can work efficiently with pointers without the need to send entire objects across different parts of the codebase. This not only enhances efficiency but also allows us to combine high-level language features with the precision of low-level languages.

Finally, Go excels in concurrency, offering built-in support for threads and parallel execution. This feature has been instrumental in Lunar.dev's success. We can efficiently handle a multitude of requests concurrently, ensuring our proxy deliver exceptional performance.

The seamless integration of concurrency into Go makes it an ideal choice for developing Lunar.dev, eliminating the need for additional, complex concurrency libraries and enhancing the overall development experience.

‍

‍

Minded to the Community

Another compelling reason for choosing Go was its thriving open-source community.

The popularity of Go has been steadily growing, attracting more developers to its ecosystem.

However, performance was just one piece of the puzzle.

We had to consider developer onboarding time, an aspect where Go truly shines.

The ease with which developers can jump into Go and start contributing was a significant factor in our decision-making process.

Since we knew we wanted Lunar.dev to be open-source, encouraging contributions from the developer community.

Leveraging Go as our language of choice enabled us to tap into this expanding pool of Go enthusiasts who could enhance our project and help us build stronger connections within the open-source community.

‍

En Route: Still on the GO

‍

If a developer were to create their own type of home-grown Lunar.dev, they would likely embark on a bumpy and untraveled road filled with hurdles and challenges in achieving the same level of performance in general, and especially what we've achieved with Go.

While other languages may offer their strengths, Go is proving to be a great choice for us in terms of low-latency proxy performance. In conclusion, our decision to build Lunar.dev using Go was driven by a combination of factors that align with our project's requirements.

Go's balanced approach to performance, ease of developer onboarding, memory management capabilities, and thriving open-source community made it the perfect fit for our endeavor.

We look forward to continuing our journey with Go and the Gophers community, and further contributing to the growing success of Lunar.dev.

‍

We're aiming for the stars! If you're experiencing any API failed requests, check out lunar.dev's open-source solution * star * us! Or get in touch at info@lunar.dev.