Data Migration

Data migration is a subject that has traditionally been ignored by both analysts and the market. However, that is changing, and with this article we are announcing the availability of two papers on the topic that are available for free download from the Bloor Research web site.

The first paper is a data migration survey, which can be found here. We wanted to establish how large the market for data migration is, how effectively companies were implementing data migration projects and, where they were not (most of the time), what were the major issues that caused time and budget overruns.

While I don’t want to steal the thunder from these papers a couple of points stand out from the survey. The first is the sheer size of the annual spend on data migration: for the projects we were investigating we estimate this at $5bn this year. However, this is only a lower bound, because we targeted only Global 2000 companies where the whole project (data migration is always a part of a larger application focused project) was budgeted at $1m or more. There must be many companies outside of this group that undertake major data migration projects and there must be many smaller projects. So what the total size of the market is anyone’s guess: it could be $7bn, $8bn, $10bn? This is pretty huge for something that has not, hitherto, been considered as a market in its own right.

The second thing that I want to highlight from this survey is that almost 20% of projects had no separate budget or timescale associated with the data migration element of the project. This is dreadful and appalling. For obvious reasons we could not measure the overruns on these projects but I expect them to have been significantly greater than those who made at least some attempt to treat data migration as a subject worthy of consideration in its own right.

The second paper we have published, which can be found here is a white paper discussing the issues involved in data migration. It has been sponsored by, though not written for, a number of companies that are active in this space.

This second paper starts by defining exactly what data migration is and contrasts it with data movement, ETL and other data integration requirements. We then go on to discuss the various points that you need to consider during a data migration project. In particular, we discuss what you need to do before you even start the project, as well as issues that may arise during the course of such a project.

Again, I do not want to pre-empt the discussion in this paper but the one point I will highlight is that we believe that a pre-cursor to delivering data migration projects on time, on budget and to meet user requirements is that data migration should be regarded as a discipline in its own right, with its own methodology, best practices and tools. Today, data migration is treated in many IT departments, that should know better, as a dead end job with no prospects. This is unlikely to encourage expert practitioners and high calibre personnel to get involved: which means that you get a vicious circle with low expectations, low prestige and even lower enthusiasm. In order to break this cycle of defeat, expertise in data migration needs to be recognised as a valid, and valuable, career path. We believe that recognising data migration as a market in its own right is a first step in this direction.