cloud_computingWhat is the technology stack that makes big data go? And how does it work with cloud computing? Here’s how three successful companies – SoftLayer, Cloudant and Rosetta Stone – work at different layers of the big data technology stack.

Bare metal is the foundation of the big data technology stack

The foundation of a big data processing cluster is made of machines. Like relational data clusters, these machines usually have plenty of memory, CPU and storage. However, big data machines don’t have to be scaled up – they can be scaled out by adding more machines. The ability to scale out makes them a good match for cloud computing.

SoftLayer is a hardware IaaS provider – it does not deal with NoSQL directly but does deliver the clusters required to run them. Nathan Day, chief scientist at Softlayer, said they can “deliver a cluster of servers for things like the NoSQL solutions, so with things like Riak and Mongo, a customer can come and say ‘I want my own cluster of NoSQL servers. I want three of them in Amsterdam. I want three of them in Singapore.'”

These machines are often physical rather than virtual because bare metal makes performance less painful – with the unfortunate side effect of making the bill more painful. Day commented on bare metal versus virtual machines. “We did a comparison test between deploying Mongo on bare metal and doing a deployment on a cloud…It’s very consistent on bare metal, as you’d expect, because you’re single tenant running on your hardware – it behaved very predictably. In a public virtual machine cloud, where you can’t control aspects of storage and even CPU and RAM access, the results varied wildly”.

The database service in the middle

Building this layer requires database expertise. The designers must answer tricky technical questions such as when to shard, how much memory is enough, and what the difference is between Hadoop and Cassandra.

Cloudant, the managed NoSQL provider, provides DBaaS to its customers, placing this DBaaS layer on platforms from IaaS providers like RackSpace, Azure and Joyent. If customers want cheap DBaaS, Cloudant can supply them with an AWS virtual cluster. If customers prefer hardware, Cloudant can supply a SoftLayer cluster like Day described. Cloudant CEO Derek Schoettle said, “Whereas they provide IaaS and we provide DBaaS, our joint customers benefit from a tight coupling of our respective services. As such, some of Cloudant’s biggest and most important accounts are on SoftLayer infrastructure.” By Nick Hardiman read more