What’s the problem with agile?

Agile development is becoming more and more popular. However, agile development on its own is not enough if the data isn’t agile too. This is nothing new – I’ve written and said this before – and test data management is becoming increasingly popular for the provisioning of data for development environments. I’ve also written about the need for service virtualisation for the provisioning of test data that isn’t otherwise accessible. This is still an emerging market but this too is growing. However, there is a third element to agile data – which, I admit, I had not previously considered – and that is database schema changes.

We live in a world where database schema changes are done via scripting and, at least in a great many companies, requests for schema changes are presented via request tickets, which are implemented at the DBA’s convenience and in no particularly controlled manner. While there are certainly products (often limited to a single database) for the change management of scripts this is hardly an automated function and, regardless of whether these are in use, the process is not intrinsically agile.

A popular resolution to this issue is available through the Apache Liquibase Open Source project, which uses a database-independent library for tracking, managing and applying database changes. However, it doesn’t really provide the sort of automation one would like, it uses a command line interface and, frankly, its feature set is limited.

Just recently, Datical announced Datical DB. This is a product that extends Liquibase for enterprise use and adds functionality required by larger, distributed enterprise development and DevOps teams. To begin with Datical DB provides a graphical user interface. More to the point it is model-based. Datical DB reverse engineers your database schema (you can also use modelling environments such as Embarcadero or CA ERwin to update your database and then Datical DB can interrogate the DB for those changes) and the product then provides change management at the schema level rather than at the script level. It also allows you to simulate, in-memory, the effect of database changes in production environments (or any other environment) before changes are actually made to the database. There are facilities for automatically detecting logical errors as well as changes that impact on data integrity such as truncating a datatype. Change management means that you can tie schema changes to the author of that request and to the underlying requirement and the software can also provide a full audit trail of changes to each instance of whatever database you are monitoring. And it can tell you which version of the schema each database instance is using so that you can easily check which instances are up-to-date. Like Liquibase, Datical DB also allows you to rollback any changes that have resulted in failures – but Datical DB adds more control and flexibility in the way rollbacks can be executed.

Liquibase supports some 15 databases plus JDBC but Datical is initially focusing on Oracle, DB2, SQL Server, MySQL and Postgres though it could easily add support for others, thanks to Liquibase’s capabilities. Given SAP’s recent announcement about the success of Sybase ASE it would make sense for Datical to add this sooner rather than later, as well as Sybase Anywhere. Datical supports any schema types for its supported databases, so will support things like star schemas. However, Liquibase does not support any specialised data warehousing products so, if Datical wants to move into this area it would have to build something appropriate based on JDBC.

For its supported databases I think Datical should be a real boon. It fixes a DevOps problem that has not been widely recognised and it rounds out agile data requirements for agile development.