Executive Summary:

The Media Collection and Viewership Beaconing Service (MCVBS) at FOX–a system designed to gather and analyze viewer data–powered by Momento, Amazon Athena, and Amazon Simple Storage (Amazon S3), delivered uninterrupted real-time metrics and accurate post game insights during record-breaking Super Bowl LIX. MCVBS easily handled the load of 15.5 million concurrent viewers, offered real-time metrics up to 1 minute faster than alternative dashboards, and had 100% availability during the event. On game day, MCVBS seamlessly ingested over 25 billion beacons from the Tubi app at almost 2 million beacons per second. 

MCVBS consists of rich player-side beaconing–device type, stream information, delivery network–from each Tubi device with server-side support powered by Momento to process, store, and report insights from the individual beacons. Real-time insights enabled operators at FOX to make data-driven decisions to optimize content delivery, enhance viewer experience, and diagnose any device or CDN–Content Delivery Network–issues happening during the event. 

Momento enabled deep post-event analytics, powered by Amazon Athena, and Amazon S3 to identify peak viewership moments, device counts, viewership numbers, and device types.  Athena made it easy to express analytics questions as SQL queries. Athena was able to process terabytes of beacon data within seconds thanks to Momento Gateway’s transformation of individual beacons into compressed Parquet batches. 

Motivation:

Tent-pole streaming events are consistently breaking viewership records, increasing their visibility and presenting technical challenges, as any issues during the event can impact millions of viewers. Delivering the best possible viewer experience requires real-time visibility into actionable metrics. Existing tools like Mux and Conviva offer broad visibility but are optimized for generalized queries making them less effective at delivering timely insights needed for in-game operations. Furthermore, having telemetry from all beacons during the event enables free-form analytics like breaking down UHD viewership by device, identifying peak viewership magnitude and time, etc… 

The MCVBS team at FOX aimed to build a system that leveraged the most up to date data possible, providing near real-time insights derived from client-side beaconing–delivered once every 10 seconds per device. Initially, the solution was built on Amazon API Gateway and worked for smaller events, but faced scalability challenges with dropped requests and throttling during high viewership events like the Super Bowl. 

Solution:
FOX selected the Momento Gateway, Momento Cache, and Amazon Athena backed by Amazon S3 to build a robust architecture that can seamlessly absorb peak viewership of tens of millions and deliver timely insights.

Each Tubi device sent a beacon directly to Momento Gateway every 10 seconds, peaking at 15.5 million devices during the half time show at Super Bowl. The devices connected to the Momento Gateway, transforming each beacon and coalescing them into larger blobs–binary data–in S3. The Gateway also stored precomputed metrics directly in Momento Cache to power a live viewership dashboard. The dashboard offered unique real-time insights into the stream, empowering the FOX operators and executives to maintain situational awareness of the viewer experience.

Denormalizing the beacons–transforming the complex beacon data into simpler formats for storage–using previously determined Momento counters allowed rapid lookups of key metrics from materialized views–pre-calculated results stored to increase the speed of data retrieval. These metrics powered a dashboard where each browser could simply look up values for specific metrics from Momento Cache, facilitating a fast and interactive experience delivering real-time metrics.

As Momento Gateway processed each beacon, it simultaneously batched them up into larger files, transformed into Parquet–a format that stores data by columns for faster analytics–and compressed using ZSTD, to deliver them to Amazon S3. This enabled FOX to use Athena to analyze beacons with rich queries expressed simply in SQL.

“It took us less than a day to add support for streaming data to Amazon S3, and it was worth it because it gave us access to the full power of Athena! Best of all, even during the game, Athena was less than 20 seconds behind real-time metrics–giving us the ability to run complex analytics in near real time to diagnose anything unexpected” said Ellery Addington-White, founding engineer at Momento.

The Athena queries were instrumental in debugging client side beaconing and making MCVBS more robust for game-day. It is incredibly powerful and simple to use SQL queries to find any discrepancies.

Momento Cache, a high-performance in-memory datastore, delivered low-latency access to pre-computed metrics loaded from the Momento Gateway. It also cached the results of user queries to Athena to improve performance for common analytics requests. This data is read from Momento Cache via the Momento Gateway, powering an internal dashboard for FOX engineers and executives.

To ensure game-day readiness, the team rigorously load-tested the system, simulating twice the expected peak audience. The results confirmed the architecture was capable of handling the immense traffic demands with ease.

Results:
MCVBS, powered by Momento and AWS, flawlessly delivered 100% analytics uptime for FOX engineers and executives during the game. The fully-serverless architecture seamlessly scaled to handle the surging viewer demand, ensuring uninterrupted observability even with higher-than-anticipated viewership. In fact, the Momento Gateway processed a peak of 1.8 million messages per second from 15.5 million devices with zero delivery errors!

Momento Gateway and Momento Cache seamlessly handled the Super Bowl traffic, managing 15.5 million concurrent connections and nearly 2 million beacons per second—all with zero errors or missed beacons. 

“Seamlessly absorbing this kind of load took meaningful preparation, load testing, and operational excellence. It’s what we train for every day at Momento,” said Daniela Miao, CTO and Co-Founder at Momento. 

The combined work of FOX, AWS, and Momento produced a solution that confidently satisfies the most extreme demands of live sports broadcasting. Powerful, high-performance components from Momento and AWS easily handled millions of events per second. With the MCVBS, FOX provides a blueprint for media companies–as well as gaming, finance, adtech, and more–to leverage real-time data processing at massive scale.

Khawaja Shams

Author

Khawaja is the CEO and Co-Founder of Momento. He is passionate about investing in people, setting a bold vision, and team execution. Khawaja has experience at AWS where he owned DynamoDB, and subsequently owned product and engineering for all 7 of the AWS Media Services. He was awarded the prestigious NASA Early Career Medal for his contributions to the Mars Rovers.