Data Migration Best Practices for your Next Project

Data Migration Best Practices

In this guide we present 8 activities that have been proven to improve the likelihood of success in data migration projects.

Expand, share, critique but above all learn from them and spot opportunities to implement them when conditions are right.

There is a huge variation in the quality and experience of consultancies and systems integrators offering data migration services.

We are still seeing major, multi-million pound mission critical data migration initiatives awarded to suppliers who, to be frank, have no hope of successfully delivering the project on time and in budget due to a lack of available expertise and skill.

In at least one example this year I've witnessed an organisation securing a proposal and then instantly scrabbling around in search of the appropriate skills in the market.

So, what's the solution?

Well, we've printed enough articles, surveys and reports to hopefully enable organisations to ask the right questions of their suppliers.

I also recommend that organisations should utilise the services of an independent consultancy to help vet the service providers and ensure they have the necessary credentials.

I also believe the vetting and bidding processes undertaken by most organisations are simply not intensive enough.

Far too many inexperienced suppliers are winning bids based on price, not capability so ensure that your selection criteria is focused on capability, not cost.

I'm still seeing the "we think our data is good enough" scenario, even with large corporates who really should know better.

The fact is that without an adequate data quality rules, data discovery and data quality management process your project has little or no chance of coming in on time and on budget.

I recently spoke with one member, a project manager, who delivered his first ever data migration project on time, under budget and to widespread acclaim within the business.

The secret for success, in his own words, was this:

“We spent about 10 months understanding the data intimately. The business community was 100% involved in that process. The final migration was actually quite simple because we knew exactly what to expect” Project Manager, UK Based Utilities Organisation

What was even more interesting about his case was that the team used low-cost technology and basic data analysis techniques that are freely available.

Data quality management and thorough data discovery are not optional, they should form the backbone of your migration.

If you adopt a thorough approach to data quality your integration and implementation phases will slot into place with ease, ignore data quality and you are heading for a world of pain.

Okay, I'm going to fire a shot across the bow of a few systems integrators here.

In recent years I have witnessed far too many well established systems integrators making a total hash of data migration scoping and costing.

For example, I really have no idea how an integrator can arrive at a budget and timeline for a complex data migration without having carried out at least some form of a pre-migration assessment beforehand.

How can you possibly estimate a migration without understanding the complexity of the legacy landscape, the relationships involved, the volumes and data quality levels required to deliver the migration?

Yet, we repeatedly see systems integrators pulling figures out of thin air based on previous migrations or completely inappropriate techniques drawn from software development paradigms.

The end-customer organisations are also to blame here.

They must take far more responsibility for selecting the right partner and ensuring that an adequate pre- migration assessment is undertaken.

Yes, this will cost but at what cost is a failed migration?

We will be exploring the techniques of data migration scoping early in the New Year so please add your comments if you wish to see anything specific in this area.

Data migration projects imply risk. For example, how do you know the project is actually viable? What is the risk if the project overruns or eats up all the budget?

There are countless risks in every data migration but few organisations seem to manage these risks effectively.

Risk-management starts with a pre-migration assessment or audit to fully understand the challenges ahead.

It should then continue throughout the migration, reacting to issues found at each phase and iteration.

Too many projects stick blindly "to the plan" without adapting to the conditions and issues discovered.

Don't make the same mistake, create a risk management strategy that guides the direction and deliverables of your project.

This point is closely connected with the issues surrounding poor choice of supplier.

Data migration is a specialist subject and requires specialist skills.

I'm still witnessing too many projects going forward without a clear data migration subject matter expert available on the project.

Sure, these skills are rare and often difficult to hire but proceeding without the guidance and regular support of an experienced data migration specialist is a major risk to success.

Data Migration Pro now has over 16000 members worldwide, many of these are migration specialists so if you desperately need a specialist resource just ask us and we'll connect you.

Alternatively, our Data Migration Pro LinkedIn Group accepts job posts from members plus there are other data migration communities on LinkedIn which are well worth a look for specialists.

Data migration projects are first and foremost about people, not technology.

The way in which you motivate and weave the business and user community into your project will dictate your ultimate success.

Forget offshore data migration farms and cleansing centres.

Instead develop closer relationships with suppliers and business representatives, the dividends will prove far higher than any outsourcing savings.

Motivate them to share their knowledge and expertise by explaining the benefits of a successful data migration to their personal and corporate goals.

Make the business understand it is THEIR data being migrated and that it is THEIR responsibility to get involved to ensure that their knowledge is shared for the common good.

 Still sticking to the waterfall approach for your data migration projects? Perhaps it's time to change.

Why not design your data migration projects around results-focused iterations.

Go agile and build rapid discovery elements into each iteration and plan your project based on what you discover, not what you thought you knew.

Don't make the mistake of planning your entire project at the outset, you simply don't know enough at day 1.

Convince the business that an agile, transparent approach is worthwhile for managing risk and delivering data that the business can actually benefit from.

 Whilst I don't want to get bogged down in vendor debates I do feel it is important to reiterate the rather obvious point that the quality of your migration toolset will dictate the quality of your overall migration.

Coding your migration using a combination of shell scripts, SQL and home-grown routines may appear cost-effective but there are a host of reasons why it may be time to change.

Tracing data lineage, loss of key technical resources, version control management, limited data quality features, speed, complexity - there are a myriad of different reasons why it's time to explore tools and technology that could accelerate and simplify your next migration.

If your business dictates that your migration must be seamless and implemented in a limited window then you must identify the appropriate technology that can support those aims. Your decision should be based on business needs initially.

Remember that data migration projects are quite different to data integration style projects so ensure that this is reflected in your licensing strategy.

Negotiate hard and trade vendors off against each other in the selection process. There are some excellent deals to be had in these trying times and no vendor likes to lose out to their nearest competitor.

Evaluate service providers based on the technology which they plan to use, will it really deliver what you need?

Summary

Too many companies are still making too many mistakes when it comes to data migration projects.

This list provides some pointers for helping your organisation chart a successful data migration.

Dylan Jones

Co-Founder/Contributor - Data Migration Pro
Principal/Coach - myDataBrand

Dylan is the former editor and co-founder of Data Migration Pro. A former Data Migration Consultant, he has 20+ years experience of helping organisations deliver complex data migration, data quality and other data-driven initiatives.

He is now the founder and principal at myDataBrand, a specialist coaching, training and advisory firm that helps specialist data management consultancies and software vendors attract more clients.