We all get it. Creating a better customer experience comes from intelligent use of customer data. But our customers have been busy creating an enormous trail of cookie crumbs for us to collect; it can be tough to find a pocket big enough to stuff all of the crumbs into, let alone piece the trail together.
In other words, storing and processing all of this data in real-time into a usable format to inform each customer interaction is a challenge.
Or is it?
Costly and Cumbersome
Traditionally, we have used large, expensive servers to meet our clients’ data and analytics requirements, costing hundreds of thousands of dollars. When our clients wanted to scale up to more computing power, it was a sizeable investment—one that was hard to reverse if the purchase didn’t work out. What’s more, the way in which these machines process data often meant long wait times to complete reports and analytics. So much for real-time decision making!
This is all on the verge of changing.
Big power in little servers
There is immense power in using a hundred small servers instead of one big server, and it’s possible to adopt this computing infrastructure with Apache™ Hadoop®. If you haven’t heard of Hadoop before, it is an open source software that allows for the distributed processing of large data sets across clusters of computers—drastically improving processing speed and allowing clients to easily (and inexpensively, at less than $10K per server) scale and up and down to meet their data computing needs.
The challenge with migrating to Hadoop was that it would require completely overhauling everything that we do; from training our analysts and our clients’ analysts to creating new campaign solutions; from rewriting all of our reports to changing how we cleanse, match and enrich data. Huge benefits, but HUGE change.
Enter Splice Machine
Back in Feb 2013, I was listening to a podcast about big data and discovered Splice Machine. Their technology basically creates a relational database, but it sits on top of Hadoop®. In a nut shell, bringing Splice Machine on board would allow us to get the massive performance benefits out of the small server clusters while using our existing tools, all of our concepts for reporting, campaign management, etc. It could all function as it does today with very little change.
It was a no-brainer to try this one out.
What it all means for you: faster matters
As marketers, we all need to connect to customers on more channels than ever before. More channels mean more data. The amount of data we need is only going to keep increasing, and we’re only going to need to do more things with our data, incrementally over time. This innovation gives us all a leg-up on this ever-expanding need.
Working with Splice Machine, we have improved performance and enabled easier scaling, without requiring clients to retrain staff, build new skills or buy new tools.
The result is that our clients have the ability to collect and crunch more data in a shorter amount of time to enable more relevant, one-to-one customer experiences. It also means clients have flexibility to scale their computing resources up and down to improve performance without big investment or risk. Go ahead, experiment a little. Try out something new. You’re not stuck with $300K of unused infrastructure if it doesn’t work out.
Making it real
While we’re still in the midst of a phased approach to implementing a Splice Machine-enabled Hadoop infrastructure, we have proven that this approach would improve query performance speed by five to 10 times (with even higher speeds possible with more servers). We are now testing real applications with real data and intensive batch, analytics and real-time workloads, to make sure this will work smoothly for us in production.
The next step is to actually bring this new infrastructure into production mode, run it in parallel to the old system for a period of time and then cut over completely.
Although not yet fully-functional, we’ve already been recognized with the Ventana Research 2014 Technology Leadership Award in the category of Big Data. I’d like to think we’re off to a pretty good start…and that we’re only going to see increased value from here on out.