<aside> 💡 This was a challenge I observed at work. I proposed this solution as an internal RFC. This blog discussed redacted any details of internal systems.

</aside>

Overview

My company captures customers' invaluable data and promises to preserve it in forever. The worth of the historical data will only increase over time. The expansive data set will offer customers unparalleled insights into analyzing their website performance over an extensive time span.

However, As data accumulates, storage costs will also rise. While data's value grows over time, the infrastructure and costs to store it securely and efficiently at scale must be considered.

Also, as the volume of data grows over time, the computational resources required to process, analyze, and query the entire dataset will also increase. This can lead to longer processing times, increased latency, and higher operational costs. Furthermore, reprocessing the entire dataset each time when a new event is defined can be both time-consuming and expensive.

In this post, I want to discuss a possible data reduction strategy using data sampling to significantly reduce the amount of data to store to reduce storage and processing costs while ensuring that the user experience remains seamless and unaffected.

Problem Analysis

Let’s first analyze how user would use the extended data:

• stats over data point: When customers analyze the data over a long history. They are caring about the stats like events count, conversion rate etc. instead of individual session/users
• trends matters: when analyzing over an extended time range. The user usually care about a stats values trend over time.

A straightforward attempt might be aggregate all the stats possible and then just throw the raw data away after a period, for example, 6mo. However, this won’t work for the following reasons:

• Curse of Dimensionality
• losing the capability to compute on the historical data. For example, creating a newly defined event.

Solution Proposal

To address the above issues, we propose implementing a data sampling strategy that preserves the statistical properties of the data while significantly reducing the amount of data stored.

Our proposed sampling strategy consists of two parts:

Random Sampling

We will perform a random sampling of the data when they reach the retention period we configured. We will only keep a small portion of the data selected, and purge the rest.

The sampling rate for this phase will be configurable, allowing customers to control the tradeoff between data reduction and accuracy.