Case Study: Olive Branch Helps Client Reduce Data Costs 99%

About our case studies. To guard from disclosing information that our clients may not want shared, we do not use company per individual names in our articles.

Data storage is has become complicated. Your options for for database technology are far greater than in the past, each solution offering different strengths and weaknesses. I remember a time when if you wanted to store any meaningful volume of data, your options were Microsoft SQL or Oracle ... and then eventually MySQL. But with the increase in available computing power and the drop in the cost of high performance drive space the number of options for database solutions has skyrocketed. Choosing the right technology to match your requirements can poivde a high performance, low cost solution, but using the wrong solution can kill your budget. Read on to learn how Olive Branch Technology slashed one company's data storage costs by more than 99%.

One of our clients was faced with mounting database costs resulting from a system that was built on top of Microsoft SQL. We're not going to knock MSSQL here, it is an absolutely fantastic relational database. It is easy to maintain, powerful and fast; an impressive technology. But with that power comes cost. Operating large scale relational databases is not cheap. As you add up licensing costs, server costs, drive costs, etc. you need to allocate a good chunk of your budget to operating the database. Even if you are using a hosted solution such as those offered by Amazon Web Services or Microsoft Azure as your costs rise significantly when you are working to genuinely large datasets.

Databases are not one-size-fits-all.
​
Knowing how to choose and use the right tool for the job can save your budget.

The wrong technology, or not knowing how to use the technology you have chosen, can kill your budget.

When our client first came to us, their database costs were only a few thousand dollars per year. However, forecasts of business growth put their database costs in up into the six figures. Upon careful review we characterized their data needs as follows:

• Data needed to be retained indefinitely.
• It was unusual to actively use old older data, but in the event it was needed, the data needed to be available immediately.
• Most of the data storage volume was time-series data.
• Almost all of the actively used data was 30 days old or less, with the majority of the day-to-day operations using data that was less than 24 hours old.
• Most time series data was accessed in either 1 day chunks or 30 day chunks, it was extremely rare to have to operate on more than one month of data at once.
• Combining or aggregating data across multiple sources was extremely rare.

A few things jumped out at us:

1. Using a relational database like MSSQL for time series data is complicated. A solution that works for a relatively small volume of data won't scale well. Scaling time series data is complicated and gets expensive with relational databases.
2. The hot-data, the data being accessed frequently, was really only 24 hours old at the most.

Here's what we did.

Scrapped the relational database. As wonderful as MSSQL is, it just wasn't the right solution for this system. It was expensive, and as  the business grew, it would be difficult to maintain performance of the time-series-data analysis.

Introduced NoSQL databases, For basic entities like account data, customer information, etc. that needed to be stored, we created a solution that leveraged Google Datastore (now Firestore). This involved denormalizing the database and setting up structured entities to save in DataStore. Fast and inexpensive, we could store all of the data entities we needed with minimal cost

Our bottom line as improved greatly after working with Jim and his team at Olive Branch Technology"
​ - Vice President, Undisclosed Company

Used Big Query for time series data. Big Query is a data warehouse. A data warehouse is designed for storing and batch processing huge volumes of data but are not necessarily know for having zippy access times. Given that the majority of the information was going to sit and wait for if, or when, it needed to be accessed, we decided to put all of the data into a Big Query data warehouse. However, we did leverage a high degree of partitioning (breaking the data storage into chunks) that allowed us to very quickly find and retrieve the data that we needed, when we needed it.

Caching strategies. For user interfaces that needed fast response times, we implemented a caching strategy that functioned much like a real-time database. Keeping commonly accessed information readily available while performing longer data interactions asynchronously in the background, we were able to match, and even exceed response time when compared to leveraging a SQL server for active storage.

As a result of these changes,  for the first year of operation data storage costs were reduced by 99.7%. We expected the costs savings to continue to be strong, but not that strong. After a few years in production and having reached anticipated data volumes (many orders of magnitude beyond year one volumes) we followed up by pricing out hosted SQL server instances necessary to meet the current demands and found that depending on the solution provider, our chosen technology stack still represented a 99.7% to 99.9% savings over the original architecture.

This kind of savings is atypical. This system happened to have just the right combination of requirements to allow us to leverage high performance, low cost solutions. It is also important to note that there is not anything inherently inexpensive about Google's data storage solutions. In fact, this same combination of storage technologies, when used without careful attention to implementation could cost thousands of times more per month.

The key, is knowing what your needs are, knowing what technology solutions can best meet those needs, and knowing how to best implement those solutions. It's all about choosing the right tool for the job.
​