Table of Contents
- This Project Felt Different
- Learning While Building
- Production
- Observing
- How AI Changed My Relationship with Implementation
- Conclusion
- Disclaimer
This Project Felt Different
Over the past year or so, AI has gradually become part of my daily development workflow.
This time, I had to implement a distributed rate limiter for a few API endpoints after we noticed that some clients were occasionally generating traffic spikes that put pressure on our OpenSearch cluster.
Until then, rate limiting was one of those concepts I understood in theory. I knew why it existed and was familiar with the common algorithms, but I had never actually built one myself.
Whenever I start a feature, I like to map out the problem first. I try to understand what I need to learn before I start thinking about the implementation. Using AI had already become part of how I approached new features, and this time was no different.
At the time, I thought I was just building another feature with AI. Looking back, I see that it changed my relationship with implementation.
Learning While Building
Before writing code, I first had to gather information about the API clients to define proper thresholds. I analyzed traffic patterns and how traffic behaved during previous incidents. Then I could define the tiers we wanted: warning, alert, and critical. These tiers would also drive the metrics and alerts, helping us identify unusual traffic patterns before they became an issue. In the long term, once a request reached the critical tier, the API could reject it with an HTTP 429 response.
Throughout this process, AI was my thinking partner. We discussed traffic patterns, thresholds, and the trade-offs behind the implementation. I validated its suggestions and adapted them to our environment. I explored the code to find the right integration point and configuration, and to introduce the distributed cache that would handle the thresholds. The implementation was in Java, and AI made it simpler to work through parts of the language that I still didn't know deeply.
While AI accelerated my implementation, my pull requests were reviewed by another senior engineer with strong Java experience. His feedback helped me align with the team’s patterns and better understand some of the design decisions behind the implementation.
Production
Once the implementation was finished, I deployed everything with rate limiting disabled. I knew production would require attention, so rather than hardcoding the thresholds and the enablement of the feature, I exposed them as configuration properties. This made it easier to tune the values as we learned from production.
I first tested everything in our sandbox environment to validate the setup. As expected, I had to fix a few things here and there before everything worked as intended.
Testing in sandbox also helped me understand the metrics well enough to build dashboards for better observability. Both enabling the rate limiter and enforcing request rejections were controlled through configuration. That allowed me to enable rate limiting without enforcing rejections, observe when clients reached the thresholds, and validate whether the values made sense before actually returning HTTP 429 responses.
Once everything was in place, it was finally time to enable the rate limiter in production and see how it behaved with real traffic, while still not rejecting requests.
Observing
Over the following days, I spent more time observing than changing the implementation. The dashboards were my main tool for understanding how everything was behaving. While analyzing the production metrics, I noticed repeated failures during cache evaluation.
After some investigation and discussion with my lead, we found that the rate limiter depended on a synchronous cache evaluation. During periods of repeated failures, every request still had to wait for the cache evaluation to fail, adding unecessary latency. To avoid that, I introduced a cooldown mechanism that temporarily bypassed cache evaluations after consecutive failures.
Looking back, my lesson was not the cooldown itself. It was realizing that observability was not just something I added after the implementation; it was an essential part of the process. Without the dashboards and production metrics, I would not have understood how the system behaved in practice or had the confidence to prepare it for the next rollout step.
How AI Changed My Relationship with Implementation
AI has become my implementation partner. It helps me explore ideas, understand concepts that are not yet clear to me, and navigate through a codebase that, although I have worked on it for two years, is still a complex system. I also have only three years of Java experience, so there is still a lot I do not know deeply.
With this feature, and others I have implemented before, I found myself spending more time understanding the problem, evaluating trade-offs, and validating whether the proposed solution actually made sense.
At the same time, this experience reinforced that AI did not replace the people around me. Having my pull requests reviewed by another senior engineer with much more Java experience helped me align with the team’s conventions and improve parts of the implementation that AI alone would not have caught. Discussions with my lead helped me think through the production behavior of the system and make different engineering decisions.
I also wish I had had more discussions during the pull request reviews. That is entirely on me; I could have asked for them. For many times, I was exhausted by all the generated code. We still need to review it, understand what AI is suggesting, compare it with feedback from experienced engineers, and build our own understanding before making a decision.
It can feel like this: “Okay, AI suggested this. Fine. But then another engineer suggested another way. Now I need to understand both solutions, compare them, and build my own understanding before I do something.”
Conclusion
Looking back, I do not think this project was memorable because I implemented a distributed rate limiter, even though I really wanted that to be the case when I started the task. It was memorable because it changed how I think about implementation.
It shifted where I invested my effort. The code came together faster, but understanding the problem, making trade-offs, collaborating with experienced teammates, and taking responsibility for the final solution remained entirely human.
I think that is why this ended up not really being a post about a rate limiter after all.
Disclaimer
I finished this implementation end of June and it took me weeks until I finished this post. There's some help of AI of course to polish stuff here and there, but I try to keep my soul on it and some freestyle.
One last note: I am no longer part of the team that owns this service.
Before leaving, I documented the implementation, the rollout strategy, the dashboards, and the main engineering decisions behind the feature. When I left, the rate limiter was still disabled, but everything was in place for a gradual rollout. From that point on, someone else would take over its maintenance and decide when, or if, it should be enabled.
I honestly hope everything went fine... and that nothing exploded after I left.
This image is for the Search Team ❤️
If you're still here, thanks for reading :)