Four Vs drive big data solutions: volume, velocity, variety and veracity. Volume and velocity are technical considerations usually receiving a healthy dose of attention among architects and coders. However, variety and veracity often determine success or failure, and they tend to sneak up on teams who haven’t fully considered them. Key performance indicator, or KPI, analysis can head off variety and veracity problems before they escalate.
When some folks say “KPI analysis,” they mean KPI monitoring and evaluation—the business intelligence perspective. I’m talking about a DevOps analysis discovering which KPIs should be monitored and what data is needed. One step in this analysis traces each desired KPI back through necessary transforms to its required data sources, often termed as facts. Such an analysis clarifies variety and quickly establishes a path to veracity.
From a strategy perspective, KPI analysis can also mitigate the symptoms of “big bad data” that devour an organization, consuming people and resources rapidly. Here are five ways KPI analysis helps DevOps teams get a handle on big data.
- It prevents over-measuring. If you don’t know what you’re looking for, more data probably won’t help you find it. Teams experiencing FOMO (fear of missing out) expand their data gathering, hoping more variety uncovers something interesting. Once your KPIs have been discovered and fully analyzed, stop exploring and start exploiting. Stick to monitoring KPIs directly driving the most impact for your business.
- It minimizes bloating. In any computational system, the most expensive operation is moving data. Simply moving stuff around eats processor, network and storage bandwidth. Even with cost-effective storage available, unnecessary data means a bloated infrastructure and less efficiency. KPI analysis identifies a minimum spanning set of data sources. A workload-optimized solution then concentrates on efficient algorithm implementations, yielding results aligned with your KPIs.
- It stops wallowing. Allowing people too close to raw big data is extremely dangerous. Off-the-shelf analytics tools often carry a lot of non-KPI variables with them. Cloud-based martech tools are some of the biggest offenders; they are essentially showing off how many ways they can expose their underlying data. Avoiding the temptation to wallow through intriguing but ultimately irrelevant data saves precious brain cells. Customize reports and optimize dashboards around your KPIs.
- It de-obfuscates conversations. Nope, not talking about JavaScript here. We want everyone in the organization talking about the same KPIs and only those KPIs. You’ve seen those presentations where someone tries to dazzle an audience with mostly meaningless charts and graphs. Obfuscating your KPIs underneath a pile of other observations wastes time and risks defeating the purpose. Trust your KPIs and focus all organizational conversation on those.
- It thwarts non-causality. This is one place where data scientists and analysts really earn their money. Part of a solid KPI analysis determines causality—if your data sources change by some amount, the KPI needle responds by a quantified amount. Understanding those relationships is critical in avoiding false conclusions from data. If your KPI isn’t moving despite changes at the data sources, or your KPI is changing and the obvious data sources aren’t, step back and look at causality and transforms again.
If you’re seeing these symptoms developing in your organization, it might be a sign automation was rushed in before KPIs were completely analyzed. At that point the toughest symptom to deal with is bloating. You may be stuck with a sized and funded infrastructure, and you really shouldn’t throw out good data already captured. Ongoing KPI analysis can lead your team to optimizations moving forward.
Ideally, KPI analysis is performed upfront and these symptoms never develop. Realistically, complex systems have cycles of learning before their behavior is fully understood. Even a good upfront analysis may benefit from a revisit as trends start becoming apparent. When KPIs are rolled up periodically, you’ll need a minimum of six periods of results before expected causality is confirmed, maybe more. If that period is monthly, you’ll need ample patience.
You’re spending too much on a big data solution to afford a big bad data outcome. KPI analysis is an important step, one DevOps teams with both technical and business expertise can excel at with some practice.