big data servicesToday, the majority of application developers are embracing the agile approach. Agile has thus become standard practice for many organizations. However, database developers, particularly those working on relational databases, have not readily adopted agile development methods, partly because they need to be conscious of the state of a database when implementing changes.

Typically, database professionals have used manual processes that are not able to scale up to the faster development cycles that are at the core of agile. Database development processes, specifically the development of relational databases, need to be capable of scaling up to a comparable level and until this happens they will continue to create a bottleneck in agile organizations. 

Organizations need to automate their pipeline, to free up that bottleneck. Database development teams also need to address risk, maintain quality and shorten the development cycle for the entire organization. 

Agile Database Development

What is at the core of agile database development? In essence, agile database development is akin to agile application software. Agile database development is thus a set of principles and values which lets teams react quickly and flexibly to changing conditions. 

Over the last decade, we have seen a dramatic shift in the quality, speed, and agility of application code development. Nevertheless, a vast number of organizations have not realized the macro benefits of a better quality of service and quicker time to market.  

This is because agile practices and tools were generally only applied to application code, and many other crucial areas, including database code and database development, remained unaddressed. Database testing process is similar to other types of software testing that need a thorough examination. Database unit testing is a vital part of the software development process. To bridge this gap, teams have begun adopting a DevOps approach, which extends many of the agile best practices further than the application development team.

As teams start to adopt DevOps, however, many are continuing to overlook crucial components of the process. Organizations need to incorporate agile and DevOps in the whole software stack, from development right through to operations. If this is not done teams will continue to experience slow returns in quality, time to market and effectivity of the engineering teams.  

5 Tips On How To Do It Right

1. Iterative Development

Organizations use agile iterative development to break down large projects into smaller pieces and continuously test in repetitive cycles. Agile teams can use this practice to see, from a different angle, new features that they need to add to the end service or product. Agile iterative development creates a flexible environment for product development. 

2. Daily Meetings

Organizations should plan for regular meetings, as part of their implementation of the agile approach. These meetings should be concise, and each member of the team should say how their tasks are progressing and what steps need to be taken to meet their goals. Organizations can use this practice to keep track of and monitor the team’s performance and to see if there are any hindrances that may affect product development. 

3. Automate Where Possible

Organizations should implement strong version controls on all components and code. Developers should also monitor their changes, throughout the development phase, and should take stock at the end of each day. If his approach is used all team members will have access to the latest version of the application and should know what was modified, who modified it, and when it was modified. 

This also applies to the database code, as this code can also be version-controlled. Team members should ideally integrate it and plug it into the same version control system used for applications. 

Once an organization has put version control into practice, processes that occur later along the development pipeline can be automated. Team members can introduce continuous integration to automatically test changes, and to ensure they are not affecting existing changes. 

4. Remove Vulnerabilities

Where data has been recognized as at risk and sensitive, organizations should limit access and make sure that the data is not widely shared. Developers, however, may need a copy of the production database in their test or development environments to make sure that changes will be effective once they are deployed. Nevertheless, those same production databases typically contain sensitive data that needs to be safeguarded.  

One way to address this problem is to develop a version of the production database with a restricted dataset of anonymous data. However, this means the database will not be of a size where the impact on performance can be evaluated, so this solution is generally not realistic. Another approach is to create a copy of the production database and to mask the data manually. However, this will quickly become outdated as new changes are deployed to the production database.   

A reasonable solution is for organizations to apply data masking measures such as encryption, pseudonymization, anonymization, and aggregation. They can also use third-party tools to mask database copies. This data will still behave and look the same as the real data. 

5. A Few Points to Help You Get Started  

The transition process may take time, so organizations need to be patient and build their strategy one step at a time. 

  • Introduce standards—enforce code documentation templates, naming conventions, and points where the code is reviewed.
  • Use an agile approach—develop a backlog of business requests, monitor hours against tasks, ensure business stakeholders are involved and prioritize tasks, and introduce sprints. 
  • Collaborate—introduce messaging tools. Demand peer reviews before you introduce code into the test environment and make sure you test the code before you transition it into production.   
  • Use a cloud database—as cloud migration increases in popularity, automation has never been easier. While organizations can apply automation to all databases, automation tools for cloud databases often offer quicker and easier solutions.

Wrap Up

There are technical obstacles that make agile database development challenging for many organizations. Databases have a state, whereas applications don’t. Consequently, iterative development is more complicated as each change endures. 

This means that teams need to approach changes with care and either restore changes to a prior state or move forward. Databases form the basis of many applications, in some instances also supporting different versions of an application, thus it can be difficult to apply quick, iterative development operations to the database. 

While there may be challenges to overcome when dealing with database development, many teams have been able to overcome the obstacles. To do so they have had the backing of strong business sponsorship. Teams also need appropriate tools and processes to bring automation and visibility to realize tasks and database development. Once teams have established these processes they can make the move to an agile database development process.