Keys to benchmarking your scale-out storage

How to benchmark your scale-out storageWhy even bother benchmarking your storage?

The first step in determining what kind of benchmark software that should be used has more to do with understanding Why you are doing a benchmark and what you want out of it. There are many reasons to run benchmarks but the results probably won’t be useful unless you understand why you are doing the benchmark.

Here is an example of benchmarking gone wrong:

Engineering is executing a scale-out test on their final build of the product, the number looks great and you decide to take the results and use it for marketing the product.realizing that the synthetic workload that is great for testing scale but isn’t applicable to real-life application workloads.

This is even more common with customers and proof-of-concepts (POC) where a customer thinks they are benchmarking the storage but unless they use the right tool (workload generator & statistics gathering), they are probably benchmarking some other component of the system. This happens often, even with the best intentions.

The best way to get results in a benchmark is to spend more time on the test plan, particularly outlining the results you expect to see, before jumping in and focusing on how you can automate the testing and how to get the biggest number. Some of the tools these days are so simple to run that we often overlook this part the process and often end up with inaccurate  results.

Key benchmarks tools to consider

I have run into primarily three categories of benchmarks. These categories are very different, serve different purposes however one interesting aspect is that the same tool can be used in different categories, just configured differently.

For example, running a marketing benchmark for 5 minutes may generate a high number, running that same benchmark for 24 hours would test something completely different, endurance and stability.

I should also mention that the list of tools is not comprehensive (there are literally 100s of benchmarking tools) and I have focused on the ones I have the most experience with as this post will be followed up with more details on actual benchmarking results.

Engineering benchmarks

These are benchmarks used to categorize the performance of a particular product, or sub-product or even a single function call. They are focused on squashing bugs and ensuring that the design is allowing the product to scale as planned.

Stability and I/O-torture-style testing falls into this category as well.

Typical tools used

  • Home-grown tools – often adopted from Open Source then modified to fit the purpose of the test
  • fio (flexible i/o) – designed to test performance and stability of the system
  • iozone – file system benchmark tool. Focused on measuring performance of file operations
  • SpecSFS – NFS only workloads, typically used to test scalability

 

Marketing benchmarks

These are the most visible and you have been exposed to them many times. They are designed for a single purpose and that is to show really great performance in order to get your attention. Often the details behind the results are obscured and very difficult to reproduce without help.

Typical tools used

  • SpecSFS (NFS) – Configured to optimally scale to a “good” outcome. This benchmark can be configured in such a way that a brag-number can be generated. One really good aspect of SpecSFS is that a full disclosure report has to be provided if the results are used publically which prevents a savvy user from hiding the details
  • iometer – block and file level benchmark
  • fio (flexible i/o)
  • TPC-C, TPC-H and so forth. Industry standard benchmark tools for database workloads
  • Application-specific. There are industry verticals with such a well-defined application set that benchmarks are available to measure a very narrow performance profile
  • LoginVSI or similar VDI benchmark tools
  • VMmark – Virtual Machine workload simulator

 

Customer benchmarks

Here is where the rubber meets the road. These benchmarks vary widely, you can often argue (with yourself in the mirror or on mute, we are talking to a customer!!) their validity but most importantly, the customer is running them and they have a big impact on the purchasing decision.

Typical tools used

  • Simple, home-grown “scripts” – can be as simple as creating lots of files and directories and then doing operations on those files
  • iometer – block and file level benchmark
  • iozone – file system benchmark tool. Focused on measuring performance of file operations
  • SLOB – (Silly Little Oracle Benchmark) – utilizes Oracle to generate an IO workload for the storage layer. This could be used if the end goal is to understand how a database application, specifically Oracle, would run on the storage
  • Oracle CalibrateIO or Orion – very specific to Oracle workloads
  • VDI benchmarks such as LoginVSI or simply using VMware View and a bunch of scripting to get the job done
  • SQL IO simulator for Microsoft SQL environments

This article series will focus on taking you through a series of benchmarks for a scale-out storage system. As I dive deeper in a couple of the benchmarks we will learn more how the benchmarks behaves in a Coho scale-out storage environment.

Stay tuned!

References

SpecSFS – http://www.spec.org/sfs2008/

iometer – http://sourceforge.net/projects/iometer/

iozone – http://www.iozone.org/

SLOB – http://kevinclosson.net/2013/05/02/slob-2-a-significant-update-links-are-here/

Oracle CalibrateIO – http://www.oracle-base.com/articles/misc/measuring-storage-performance-for-oracle-systems.php

Orion – http://www.oracle-base.com/articles/misc/measuring-storage-performance-for-oracle-systems.php

LoginVSI – http://www.loginvsi.com/

SQL IO –  http://www.microsoft.com/en-us/download/details.aspx?id=20163

VMmark – http://www.vmware.com/products/vmmark/overview

fio – http://freecode.com/projects/fio


Interested in Coho Data? You should download our datasheet or  get the independent analyst review by ESG here.

6,879 total views, 1 views today