6 Classic Data Migration Mistakes Well Worth Avoiding
We have seen in the Data Migration Professionals forum on LinkedIn that data migration failures, overruns or budget blowouts are sadly not a distant memory for many organisations. So why are they still a common occurrence for many organisations?
Why do so many companies still fall into the trap of ignoring the need for essential resources in areas like data quality management, skilled leadership, appropriate technology and a solid methodology?
In this post I want to discuss some of the data migration failure causes that I have seen in the past in the hope that some companies will spot them and react before it’s too late.
1: Lack of Appropriate Expertise
This is surely the biggest failure and an all too common one.
In a past interview for a data migration leadership role I was told the project had no methodology, no technology, limited budget and a deadline of 6-8 months to complete. They offered me the job and the next day they retracted it stating that the parent company didn’t feel they needed a subject matter expert on the project for what was a straightforward migration to a known target system.
This is not uncommon. There is a gross underestimation of just how complex and challenging data migrations can become. It is often perceived as a data ‘shunt and grunt’ exercise, tagged on to the end of the much higher visibility target implementation.
The fact is that your CRM or ERP system may look nice and exciting on the salesman’s laptop but that is all it will ever be if you’re not assigning the correct people to the key roles within the team.
2: ‘Beermat’ Forecasting and Budgeting
It sometimes feels like a data migration project has been scoped, forecasted and budgeted for using a combination of a wet finger in the air and a child’s abacus.
Stop cutting corners when it comes to analysing the complexity and cost involved in your data migration. If you’re a business leader or sponsor ask some tough questions of your suppliers or bidders.
How has your solution provider or in-house migration team arrived at those staffing figures if they haven’t even examined the data?
If you cut corners in forecasting you either end up paying suppliers endless wads of cash that you hadn’t planned (not good) or the contractors walk off the job when the coffers run dry (disastrous).
Working on the client side I’ve seen some bid responses that look fantastic from a price point but there is one catch – they haven’t got a hope in hell of succeeding because you can’t deliver a complex, specialist migration that requires skilled staff and great technology on a rock bottom budget.
3: Non-Existent Specialist Data Quality Software and Skills
A data migration project is not like a long-term data integration project where people can learn ‘on-the-job’ over time to perfect their skills. You need skilled data quality services providers or in-house teams who understand the data quality software assigned to the project and can get results from day one.
Having been assigned unskilled workers on projects in the past I’ve witnessed my productivity drop to nothing as you spend countless hours coaching and cajoling your staff to become effective.
Data quality is a specialist skill that encompasses discovery, profiling, root-cause analysis, measurement, monitoring, rule creation, reporting, mitigation – and so much more.
The same goes for data quality software. You need a product that is up to the job. If you’re doing a small data migration with a few thousand records then fine, use SQL. But trust me, when you’re up into the millions of records, hundreds of tables and your data analysis guys are becoming a bottleneck for the whole project you’ll be glad you shelved out on some decent data quality software.
The good news is that all data quality vendors should lease you the software for the project so you don’t get hit for a full license.
If they refuse to do this let me know and I’ll connect you to known vendors who offer leasing of data quality software on a per-project basis. Long-term licensing for data migration is archaic so shop around if you don’t get any joy.
4: Waiting (Endlessly) for the Target to be Ready
The reality is that it doesn’t matter if the target is firmed up yet. It will matter later in the project if the implementation team can’t keep their lid on the design changes but at the outset don’t sit around waiting for a technical specification for the target schema, functions and business processes.
Get started early doors. There is so much work to do on the data quality and data analysis front it will make your eyes water. If you have done the upfront data migration impact assessment activity which I have advised in the past you should have a handle on the nasties but please don’t wait for some anticipated target readiness state; it may be too late by then to hit the deadline.
5: Data Migration Methodology – So What Is That Then?
Philip Howard at Bloor Research has pointed to the introduction of methodologies (like Johny Morris’ PDMv2) as a clear indicator of the falling failure rate of data migration projects in their latest data migration study.
This is great news but there are still many organisations trying to tackle a migration without a proven blueprint of all the appropriate components required. I think a common problem here is that they often rely on the target manufacturer’s methodology. These can be suspect in my experience so if you’re in any doubt get a second opinion.
6: Suppliers Need To Be Managed – Who is going to do it?
Managing suppliers on a data migration project can be challenging. It often helps to assign a specialist who can help mediate on the client’s behalf to keep all the relationships intact.
The challenge with suppliers often stems from the creation of a poorly specified terms of reference at the outset. This can be compounded even further by a lack of upfront investigation and analysis into ‘what lies beneath’ all those legacy systems that look in good shape data-wise until you profile the data.
Supplier management requires a dedicated role. It is not something you can stick on a project plan and assign 1 hour a week to. It needs someone to make quick, often compromising decisions so ideally someone with a data migration background who can understand both sides of the equation (business and supplier) is useful.