nosqlAll applications have inherent limits in what they can support. As organizations grow, so do their expectations on the apps they use. Development and operations teams need to plan for large scale from day one — not react to it when it finally happens.

When you hit the limitations of your system, you’ll need to determine how best to scale up. This can be done by changing the cluster hardware configuration, patching or replacing the database software, re-designing schemas, migrating data to other services, or a host of other tasks that often exacerbate issues in the short term or prolong downtime. The process can be costly and error-prone as organizations learn what pitfalls their database management systems (DBMS) vendor hasn’t told them about or hasn’t yet discovered.

The limitations of the traditional, relational approach to the DBMS are becoming more exposed as enterprise cloud services become more ingrained in daily business. NoSQL databases offer a solution to large scale but can be complex to implement. So what do you need to know?

Design tradeoffs

Traditional relational databases are designed for consistency. There is a raft of NoSQL databases that target different use cases, and often emphasise availability of the data. The differences lie in either using expensive hardware that is powerful enough to provide database consistency at large scale (‘vertical scaling’), or using new database software that supports availability across groups of commodity servers (‘horizontal scaling’).

Popular relational databases like MySQL scale well vertically but are complex to scale horizontally. As demand grows for enterprise cloud applications, there are good reasons for using NoSQL databases that are designed to scale horizontally:

  •      Size of data — if your data set doesn’t fit on one machine, you need two
  •      Concurrency — if a server can handle 10,000 req/sec and you get 20,000, you need two machines

When a database needs more than one machine to handle demand, you hit CAP theorem. Posed by Eric Brewer, CAP theorem states that a distributed database system can prioritize two of three properties: consistency, high availability, and partition tolerance.

While horizontally scaled databases can effectively and easily scale with cloud applications, multi-machine system introduces the problem of partitioning events; nodes will fail, networks will get cut off. Partitioning leads us to a choice between high availability partitions (AP) and consistency partitions (CP). With CP, all nodes must agree before data is written. This means that availability is hampered during a network partition as no one is able to write. For AP, consistency is sacrificed to allow the storage layer to remain available and usable to the application. The drawback is that different clients may see different data during periods when nodes haven’t all received the latest data. by Simon Metson read more