Big Data Performance Testing: A Look at the Bigger Picture

Every growing business looks forward to the time when the performance of processing, testing, and pulling important insights out of mass quantities of data catches up with the huge amount of data collected on the web. It might seem that the challenge that big data imposes is far too complicated and overwhelming to meet, however, with the right strategy and tools at your disposal, you can manage this process relatively smoothly.

Strategies Necessary for Performance Testing

1. Data ingestion – The process in which data is ‘absorbed’ or ingested into the larger system. This includes both data that is meant for immediate use as well as data meant to be archived or ‘warehoused’.    
   The focus here should be on routing the files or entries to their right destination in a timely manner. The process of ingesting and sorting data should be able to exceed the rate at which data is collected, with consideration for peak times. This part of the process is about ingestion and sorting, not validation.
2. Data processing – This data that is gathered from many sources will need to be deduplicated, aggregated, and often de-anonymized depending on the use of said data. In Big Data driven targeted marketing, even without uniquely identifying information, one can take sets of data from different sources and connect information about individual prospects to a high degree of certainty with just a few complementary data points, allowing a profile to be built based on all the various sources in which the data was originally from.
   This process of data mapping is heavily varied depending on the framework and overall methodology of the operation. In a way, this is a large part of the ‘secret sauce’ of any Big Data operation, and strategies for this step are the part in which businesses wish the beat the competition on.
   The data is often processed in batches, and the system needs to be able to offer reliability and scalability- after all, the amount of data we collect is ever increasing, and it’s not like you’re going to dump most old data. The systems here may require ‘unique’ infrastructure to run complex operations on later, as there’s often a lot of parallel processing involved. Types of GPU based servers that were originally a niche area for things like video rendering and scientific research are now used for Machine Learning and AI as well (which is a driving point of big data). Infrastructure for this type of computing is often more expensive as while the rate of improvement for CPUs has dropped drastically, GPU technology is much closer to the rate of improvement of Moore’s law. This means the service lifetime of a GPU cluster is considerably lower before it becomes not worth the space and power cost of running it.
3. Data persistence – The next area to focus on is the way the data is structured and archived. There are many options for data storage- ‘data marts’, management systems for relational database storage, data warehousing, and more. The key here is consistency and adaptability.
   You’ll essentially just be throwing data on a pile, and it will keep growing. Finding a scalable solution that’s cost effective while maintaining whatever speed of access you may need is important. What if you need something in the middle of that pile and you need it right now? It better not be on tape storage or Amazon Glacier! Often, Big Data solutions involve a tiered system, ranging from ‘hot’ to ‘cold’ depending on how often and how quickly the data needs to be accessed. After all, solutions for storage of data that can be retrieved at high speed or with high amounts of random operations is much more expensive than slower, denser, and less flexible data archiving solutions.
4. Reporting and analytics – This is the process of pulling insights from the processed data, the other part of the ‘secret sauce’ of Big Data. After data has been processed, methods will be applied to attempt to uncover hidden patterns, correlations, emerging market trends, or really any type of information that is useful to the business undertaking the Big Data operation. The focus will be on applying the right algorithms, often involving the latest in two other buzzwords- Machine Learning and AI. This is where the parallel computing infrastructure mentioned before is important- the reporting and analytics section is more about the methods applied to the data, and the Data processing infrastructure must be adequate, and ready for the rapid changes in the methodology of the reporting and analytics.

How to Approach Performance Testing

1. Data insertion rate – The rate at which data can be ingesting into the system, and the rate at which is it moved- mostly between different parts of the overall system
2. Query Performance – The processing speeds of queries and data retrieval from systems

1. Yahoo Cloud Serving Benchmark, or YCSB – A client for cloud service testing that can read, write, and update depending on specific workloads
2. Sandstorm – a tool for automated performance testing in order to support testing for big data performance
3. Apache Jmeter – A tool that provides the necessary plug-ins for test out databases