Imagine you’re sitting on your couch streaming a football game. This isn’t any old football game. This is your team and they’re going up against their rivals. Your team is down by five points, there are two seconds left in the game, it’s fourth down, and they have the ball in the red zone. This is the last play of the game. The ball is snapped, the quarterback drops back… and your internet goes out.

The internet is only out for a few seconds, maybe a smart device in your house turned on and kicked your TV off the wifi momentarily. But when it comes back, the game is over. You see your team won – hooray! But HOW? What happened in those final two seconds? Your internet dropped only for a few seconds and now you have no idea how your team won because when it came back, the stream picked up at the current point in the game, not where you left off. Cue the self-pity music.

What you experienced was poor handling of real-time data processing. 

But what is real-time data processing? Technically speaking, real-time data processing is the consistent, uninterrupted delivery and handling of messages or events. So when your internet blipped for a few seconds, you had an interruption in messages. In this example, messages are audio and video frames. When your internet came back, you completely missed the frames of the touchdown pass, which made you angry (understandably so). 

But if the stream provider handled their real-time data processing a little differently with message buffering or replay, when your internet came back the football game would have resumed where it left off and you could have seen that touchdown pass for yourself and lived in the moment. 💚

Why is real-time data processing mission-critical for enterprises?

It’s one thing to say real-time data processing is a critical part to the success of enterprises, but it’s another thing entirely to truly understand what that means. Many of us (myself included) think of real-time notifications as non-critical functions. If a WebSocket gets into a bad state, what’s the big deal? Your users have to refresh a page to see an update?

The truth is that notifications are just a tiny part of real-time data processing. Real-time data is the heart of soul of major industries like media and entertainment, gaming, and fintech. Let’s look at what’s on the line based on how companies handle their real-time data processing.

Customer Retention

The example we just talked about is an easily relatable scenario. But if that happened to you, who would you get mad at? Your internet service provider? The streaming provider? The network you were watching? Maybe yourself for having too many devices on your network? No, that last one can’t be right.

As it turns out, it is the streaming service provider who needs to up their game. They’re lucky it’s difficult for people to figure out who to point the finger of blame at because if word got out that the provider had poor live streaming quality, people would move over to a different provider in a heartbeat. In markets that are saturated with providers, like media and entertainment, there’s little keeping consumers loyal to your service. Off the top of my head, I could call out Netflix, Peacock, Disney+, Prime Video, and Discovery+ as services my kids watch. If Prime Video starts providing a subpar experience, we’re out of there.

Other examples where real-time data processing is mission-critical

FinTech

Let’s take a more serious example. Let’s talk about money. Specifically, let’s talk about high-frequency trading (HFT). HFT is a type of stock trading used by financial firms to automatically buy and sell stocks as prices fluctuate in real-time. This allows the firms to make constant, small profits from rapid price changes. This isn’t something you can do with your personal portfolio, it’s what the big guys do.

Price fluctuations occur every few milliseconds. If we estimate a price change every 5ms, that means they change 720,000 times per hour! This type of high-speed data change requires data processing to not only be real-time, but also always up to date. A dropped or delayed message could mean a competitor beats you to a sale or even worse—executing a transaction with an invalid price. 

Let’s take an example. Say the stock price for Apple (AAPL) is sitting at $222.94. As I’m writing this, that’s the actual current value. If an HFT firm is watching Apple and is looking to buy $1M worth of shares, that’s 4,485 shares. After writing that sentence, I checked again and AAPL is now worth $222.66 (8 seconds later). That would be 4,491 shares. So a difference of 6 shares or $1,335.96. If the software had dropped the price change messages or didn’t deliver the messages in real-time, that’s a costly mistake.

Except it gets worse.

High-liquidity stocks like Apple are traded tens of thousands of times a day. If we assume a worst-case scenario with these opportunity costs, that could result in a loss of over $10M a day. Yikes.

Something like this happened in real life back in 2012 with the Knight Capital Group. They shipped a code change to their fleet of servers, failed to update one of them, and went off the rails with their processing of real-time data. Ultimately it cost them $440M in 45 minutes. It’s a fascinating story, I encourage you to read the article.

ゲーミング

Imagine you’re playing Fortnite, or any massive multiplayer online (MMO) game. Every movement, action, bullet, and structure from every player must be reflected across all player’s screens instantly. Pro players average 4-8 inputs per second when they play, and each game starts with 100 players. This means each game is syncing ~600 inputs to 100 players every second. Doing the math, that’s approximately 18 million events in the first 5 minutes 🤯

But what happens when these events are delayed, dropped, or delivered out of order? Any gamer could tell you they result in one of the most frustrating gameplay experiences out there: lag. If one player is lagging, their experience changes completely. They could experience enemy movement appear like they’re teleporting, structures could be invisible or missing completely, you could take phantom damage (meaning you took damage but didn’t see the attack because the video frames were dropped), or players could be rubber banding. Not exactly what you’d call an enjoyable experience.

I need to pause for a moment to talk about eSports. eSports is a multi-billion dollar industry with massive audiences, major sponsors, and opportunities for hockey stick growth for many gaming companies. To drastically oversimplify, it’s competitive video game playing.

If your game has even a tiny delay in message delivery or routinely drops messages, you could give players (who are playing for hundreds of thousands of dollars) an unfair advantage. This could lead to both player and spectator unhappiness, which results in damage to your reputation and competitive viability as a company, which ultimately leads to significant loss of revenue. Sounds like a mission-critical aspect of that service to me.

How to process real-time data right

You might be wondering “That sounds hard. How do these major enterprises do it?” Well first of all, you’re right. It is hard. But it doesn’t have to be. There are several crucial considerations when building a fault-tolerant, real-time, massively scalable event delivery system.

Use low-latency infrastructure

Don’t laugh. I know that sounds ridiculously obvious. But saying is a lot easier than doing. Using high-performance servers located geographically close to your users reduces the time it takes to get data to and from them.

Implement reliable message delivery protocols

You want to send messages as quickly as possible with the lowest risk of failed delivery. Using a protocol like gRPC over WebSockets provides faster serialization and data transfer, built-in mechanisms for errors, retries, and timeouts, and allows for multiple streams per connection. To get the maximum throughput, you have to go low level with your designs.

Monitor proactively with real-time data analytics

You don’t want to get 45 minutes into a catastrophic event before you realize something is wrong. You need to know instantly and you need to respond fast – ideally with an automation. Getting things back to normal as fast as you can is crucial.

Use distributed systems and take advantage of cloud scalability

Divide the workload up across multiple compute resources. Keep their utilization healthy without pushing them too hard or you’ll start to get dips in performance. Use mechanisms that automatically scale when traffic surges occur. 

How Momento partners with you on mission-critical data processing

Momento is purpose-built to be ultra-low latency with the services provided on the platform. From caching to real-time notifications with Topics to high-throughput storage (coming soon!), your mission-critical workloads are safe with us.

Our SDKs are built on top of gRPC, giving you peak performance through HTTP/2, protobuf serialization, and bi-directional streaming. Monitor your hit rates, errors, data transfer, and more in our web console or have them forwarded to your favorite observability vendor. Finally, breathe easy with instant availability and virtually limitless capacity of our serverless services built on top of AWS.

We’ve left no stone unturned when it comes to optimizing performance and providing features that let you fix problems before your customers even know they exist. When failure is not an option, choose Momento.

Happy coding!