How caching can help reduce latency

A fast response time is essential for a good user experience when streaming videos online. Therefore, latency should ideally be at one second (1000ms) or below.

When a client asked us to develop some new features for their video streaming service, we found their platform was outgrowing its capacity due to increasing content and subscribers. Therefore, we needed to update the system while ensuring latency and response times remained within 1000ms.

Updating the platform

The Client’s subscriber home page shows video content in multiple carousels. Each row of content shows videos from one or more queries, for example, specific tags or events. To provide a more personalized experience for the users, we developed features such as adding tags to categorize videos by factors such as events or status (e.g. live or video on demand). Videos were also filtered by the user's subscription and country.

After developing the new enhanced features, we tested the system and discovered that latency would have been higher than before we made the changes. The increased number of filter parameters meant we needed to store more data into the cache layer. With the current execution flow, we would need to make more requests on the cache layer to retrieve the information we needed; therefore, we needed to improve the caching model by changing its position in the workflow.

How does caching increase performance?

It’s quicker to access cached data, so optimizing the cache will help with latency and performance. Cached data also reduces the number of requests on the database, reducing the load.

Database hotspots (frequently accessed data) can cause the system to slow down or crash. Caching this data will mitigate the hotspots as there will be fewer requests to the underlying database.

If done correctly, caching decreases latency and improves performance, which means fewer delays, improving user experience.

How changing the caching model improved performance

Caching is a critical element in speeding up response times as it reduces the need for unnecessary queries to the database. However, it needs to be in the right part of the workflow. In this case, we sped up response times by changing the caching architecture.

In the original workflow, we cached raw data, retrieving the data for each user and video.

To avoid thousands of calls to the cache layer, which would slow the platform down, we changed the process to cache the already partially processed list. So that the platform could still react to changes in video lists, we set a reasonable low time to live (TTL) of one minute.

Changing where caching happened in the workflow allowed us to reduce the number of calls to the cache layer. FORGE, our in-house publishing platform, allowed us to move the cache to a different layer in the workflow, moving caching from inside the video processing method (its input) to the outside (its output).

The FORGE engine decouples data gathering and data rendering, so we could use it in this case to render the API output.

Previously, we only cached the retrieved video metadata in FORGE, while the user data was collected during the creation of the final list. This process required many internal calls between the web server and Redis. Therefore, we moved the list processing logic to a bespoke data retriever service so that the FORGE caching layer caches the final list.

With the new model, we can have more permutations and more users because the single user’s lists will be cached, reducing the need of retrieving data from Redis for every user request. Therefore, the organization can now grow its capacity for content and subscribers while keeping performance high.

Results

After refactoring the cache, we tested the system again. The response time was now stable at 1000ms, so we launched the new features.

The next step to optimizing the cache will be to move from a TTL based invalidation mechanism to a proactive cache invalidation one. However, as this is a more complex logic, we would only add it when the business needs it.

By changing the caching model, our client can now sell more subscription packages and continue to grow their subscriber base without limits on the number of permutations of subscriptions and countries.

An increase in users and content is exciting for any business; however, it is essential to have the infrastructure to handle it in place. Caching plays a critical part in latency and response times and must happen in the correct place in the workflow.