At Momento, we’ve built a blazing-fast serverless cache. Making our cache simple and delightful is a crucial part of our mission. We strive to be shockingly simple.

What does that mean for you, as a developer? Well, for the server side of our cache service, it means that you shouldn’t have to worry about managing clusters, scaling, replication—any of those types of things that cost you tons of engineering time and prevent you from focusing on the differentiators that actually matter to your unique business. You should just be able to create a cache and go. Let us worry about the rest.

But that’s only half of the story! If you need a cache, then you obviously have some code somewhere that needs to read and write values to the cache. That means that you need a client library for your favorite programming language that knows how to talk to the cache service.

Cache clients as a first-class concern

Most database and cache services provide excellent server-side capabilities, but the client side of the equation is left as an exercise for the reader. Often you’re stuck sifting through multiple choices of client libraries on GitHub, loosely basing your assessment of the quality of the library on something like “which one has more stars”? And even after you choose a library, there’s a good chance that you’ll spend a chunk of your precious time tuning it—playing with all the knobs for things like connection pools—to make it work best in your environment. Or worse: debugging inappropriate configuration values after an outage!

Momento Cache is taking a different approach. We want your client-side experience to be as delightful as your server-side experience. So, we provide official Momento client libraries for most programming languages. We think of these as our “front door”; they’re the first place you’re really going to start forming an opinion about Momento, so it’s on us to make sure they live up to the same high standards of quality that we have for our server-side product. Our cache clients are a first-class concern.

This blog is the first in a series where we will dig into some technical detail about how we think about and tune our clients. The second installment details how we’ve tuned the JavaScript cache client, and an upcoming entry will dive into Python. The irony of writing these is that we hope that you will never need to think about any of the nerdy things that we will cover. The whole point is that we are doing this work so that you don’t have to. So, if your interest in reading this article is to learn what you need to know about ramping up with one of our cache clients: you’re done! We’ve got you. You can stop reading now and just go try them out!

But, if you’re curious to see how the sausage is made—what we think about as we polish the SDKs to make sure they shine for you—then come along, dear reader!

Getting into the weeds

Familiar faces: timeouts, retries, and backoffs

Timeouts, retries, and backoffs are some commonly overlooked configurations in client libraries. These parameters seem inconsequential—until a rainy day, when they cause a catastrophic outage. For example, if your cache client has a five-second default timeout for connecting to the cache server, you may not notice any issues with that setting for weeks or months while your cache server is healthy. Then one day, your cache server becomes unavailable for some reason. Now all of your requests are blocked for five whole seconds before your code falls back to the underlying data source! This kind of sudden, unexpected bottleneck can wreak havoc on your entire system.

On the other hand, a poorly configured retry strategy can result in false cache misses and put unnecessary load on the database server. And a poorly configured backoff strategy can cause a storm of retries, overloading your servers and potentially making a bad situation much worse.

A tailored approach to dev and prod

When it comes to tuning things like connection pools, timeouts, and retries, there’s usually not a one-size-fits-all solution that works well for both development and production environments. 

For example, in a development environment, you’ll often have higher network latencies and possibly less network stability, so you’ll want to choose somewhat relaxed values for timeouts and retry strategies. Otherwise your development work might be interrupted by request timeouts that you don’t care about at dev time.

But in a production environment where your cache requests are issued from your application servers, with low latencies to your cloud provider of choice, you’re likely to want more aggressive timeouts and retry strategies. This is how you can ensure that your application’s latencies meet your own service-level agreements.

So how can you get the best of both worlds?

Manual Tuning

Some client libraries expose configuration as a set of keys and values that you need to fiddle with. “I want a timeout of 500 milliseconds, retry up to 5 times, with a backoff factor of 2 and jitter of up to 5 milliseconds.” Then you’ll need to go through the same exercise again for each different environment you run in.

We will support the ability to tune the Momento clients that way if you like, or if you have a unique environment that requires an unusual configuration. But in most cases, we don’t want you to have to do all that work—and we are working hard to make sure that you don’t.

Pre-built static configs

We are currently finishing up a round of performance tuning work to define a set of pre-built, static configuration bundles for each of our supported programming languages. Each client library will soon include ready-to-use configurations such as a ^{DevEnvironmentConfig} and a ^{ProdEnvironmentConfig}, which should give you a great starting point that will cover the 90% case for each type of environment.

We’re doing all of the chaos-testing to make sure that the pre-built configs are a good fit for a given environment. You only need to read the brief descriptions of the pre-built configs, decide which one most closely represents your environment, and hit the ground running!

Touch-free adaptive configs

Choosing between a small handful of pre-built static configurations is a lot less work than tuning all of your config settings by hand, but even that simplification leaves room for improvement. You still have to identify which of the pre-built configurations is the best match for your environment. Imagine, instead, an adaptive configuration that keeps a sparse histogram of your client-side latencies, so that it can make dynamic adjustments to the timeout and retry values based on the real-time latencies it’s observing in your environment!

We don’t have an ETA for any adaptive configs at this point, but it’s another example of how we are thinking about ways to make our client library user experience shockingly simple. This is hard to get right and we’re keen to make it production ready. Stay tuned for updates on that front.

Really getting way into the weeds

Check out our deep dive into tuning the JavaScript SDK where we share details about gRPC configuration, code optimization, maximizing single-core CPU performance, and other things you can simply forget about with Momento—and stay tuned for upcoming Python, C#, and Java installments!

In the meantime, go ahead and get started with Momento Cache for free.

Chris Price

Author

Chris was previously a software engineer at Momento who hails from Portland, OR. When not gardening or riding bikes, he enjoys analyzing and improving software performance and geeking out over metrics. He has technical leadership experience at companies of various shapes and sizes, from tiny startups to AWS. He passionately worked to improve the productivity and happiness of all engineers who have ever experienced the joys of building and operating a cache cluster, by contributing to Momento’s platform.