Six Misconceptions about Data Migration

It is fair to say that data migration is one of the less well understood disciplines in IT and business, confusion often abounds.

In this guide, Norma L.Davis of Connective Edge Resources highlights some of the main misconceptions she has witnessed.

Norma goes on to provide some invaluable lessons for overcoming these common, but costly, data migration misconceptions.

As organizations embark on a system implementation, one of the most important and riskiest activities is data migration—the movement of data from the legacy system to the new system.

While the selection, implementation, and operation of an enterprise application for enterprise resource planning (ERP), enterprise asset management (EAM), or any other system certainly has technical elements, such systems require much more than a technical skill set to implement. All of the activities involved with implementing and operating an enterprise application require business management skill sets as well, and individuals with those skill sets must monitor closely and communicate effectively with the technical staff that is configuring and implementing the application. This often means that senior management must be closely involved with the implementation, right down to the process of migrating data from a legacy system to the new application.

But why does data migration require management oversight? Isn’t migration simply the function of transferring tabular data from one system to another?

A Case in Point

Some time ago, I was involved in the implementation of IFS Applications in a mid-market manufacturing company. It was the day after the implementation team had completed data migration, and we were at first elated that we had eclipsed our goal of an 85 percent success rate. In fact, 98 percent of the data was successfully migrated from the legacy system to IFS Applications.

However, our elation quickly turned to concern when we met to do our quality audit of the data, and found that the data definitions had been changed by the technician who handled the migrations. It turns out that, in an attempt to get the data to migrate successfully from a technological perspective, changes had been made to the data’s format. In some cases, data was put into different fields than was originally planned, once again, to get data to flow more easily into the new application.

While these changes allowed the data to migrate, it also meant that the altered format conflicted with the underlying business logic and workflows we had designed within IFS Applications.

It is really hard to fault the technician, who naturally wants to do what is necessary to maximize data transfer success. But because the criteria that constitute a successful migration are not technical- but business-oriented, migration cannot be left entirely to information technology experts.

We were able to recover from this gaffe during our second migration, but this initial lack of supervision set us back a number of weeks. Subsequently, we transferred responsibility for any changes to data from technical staff to a management team representative.

This experience gave me pause, and inspired me to develop the following list of six common misconceptions about data migration.

While IT personnel certainly have the skill set to extract the old data and to import it into a new database, other functions in the migration process belong to an implementation team consisting of senior representatives of each functional area touched by the new application environment. These individuals are the ones who truly know how the data was used in the old system, and how it will be used in the new system.

The example above illustrates why this is the case. Technically savvy people may not recognize at what point operational and management insight are required in the migration process. Or perhaps a newer IT person, anxious to prove his or her abilities by turning around a migration process with a very high success rate, changes the format of the data to make its migration a smoother process.

Lesson learned: Assign owners by data type or function to make decisions, and advise IT personnel during the migration process.

Many small to medium businesses have a one-to-two person IT department, and even a larger department might not have the correct skill set to undertake the technological aspects of a migration process.

Depending on the age of the legacy system, extraction of the data may require software coding and the use of other software packages to aid in the process. Not all IT staff are skilled and resourceful in applying such methods to get the data from the legacy system.

During one recent project, the legacy system at a client organization required the development of software code in Cobol in order to get some of the data loaded into the migration tables in the correct format. Only one of the company’s IT staff knew Cobol, and working alone, he could not develop all the coding within the time frame required to meet our implementation schedule.

Situations like this require either the training of other existing in-house staff, or that the process be outsourced. Each of these options had its own unique costs and benefits.

Management faced with this situation must keep its own counsel in determining if existing staff can handle the migration, since few diligent technology experts want to admit they do not possess the necessary skills; they may actually long for the challenge of taking on a new and different process.

Lesson learned: Don’t assume that your IT staff possesses all the technical skills or that it has the workload capacity to handle data migration. Taking into consideration the skill set and workload of existing IT staff—keeping in mind that the team likely still needs to support the legacy system during implementation—may prevent bottlenecks that could delay project completion.

Often it is assumed that because a company needs the most current and up-to-the-minute information in its new system, data migration is performed right before going live. Not so. Rather, data migration is done early, and done often.

An initial migration should be one of the first activities following the implementation team’s training. This early migration will help you understand where your data is coming from, the quality and condition of the data, and which tables and fields in a new system that the data should be migrated to.

The results of early migration passes will also help the implementation team understand how the new software works, allowing it to make better decisions. It might take multiple migration passes before a company is in a position to test business processes in a new system.

Lesson learned: Migration is a process, and not a big-bang event; it should be seen as a central element of the implementation. Planning the number of migration passes will help you determine the length of the implementation timeline.

When problems are encountered with data during migration, making changes to correct the problem is not necessarily the proper course of action. Changes in the first couple of migrations are to be expected. But the primary reason multiple data migration passes are performed is to continually change and improve the data, the tables, and the coding, until the legacy data properly populates the new system.

Later in the process, however, you must control the number of variables, and that means reducing the number of changes. The problem with change is that much of the data have interdependencies with other data.

Changes to data and data definition are to be avoided whenever possible, because any change has the potential of impacting, to an exponential degree, other data. Making just one “minor” change on a data element, a field, a code, or a table may impact many, many data and processes. For example, a simple change to a unit of measure from “each” to “pounds” can affect the quantity on hand, the unit cost, the cost of goods sold, and the total inventory value.

Lesson learned: Any changes made to data, tables, or coding should be closely monitored, especially as an organization approaches its go-live date. The project manager must be fully aware of requested changes, the benefits of each requested change, and more importantly, the risks associated with each change. Because of the impact a single change can make, formalizing a review and approval process for data changes is necessary.

If you assume your data has migrated successfully and that you can proceed directly to testing your business processes, you are fooling yourself.

A data migration never loads exactly as planned. Surprises and glitches occur with every migration—even during the last one. The data must be audited in the new system, on every screen and on every field, to ensure the data populates appropriately and in the correct format. Even simple data can present problems that, if not discovered early, can cause you to stumble later.

A case in point is a company with a legacy system that had customer and supplier addresses populated together. The street, city, state, and zip code all resided in one field for each address. In the new system, there was a different field for each of these elements of the address. When the data was migrated to the new system, the whole address was populated into the street address field, leaving the city, state, and zip code fields blank. A great deal of formatting was necessary to correct this situation.

Lesson learned: You know the old adage about assumptions. Never assume anything about data. Check and audit every piece of data and every field after migration to the new system. Only then will it be time to test your processes.

Many times, companies are so focused on meeting their projected go-live date for their new application that they want to cut corners, delaying necessary changes to data until after the fact.
Putting changes off until after going live can be a mistake for the simple reason that it is then easy to put making changes off even longer. As busy as an implementation team is before going live, it is at that point that it can focus on making the new system operate as it should.

After going live, team members will likely be more focused on their usual pre-implementation duties, and will be too busy to even address these changes. Moreover, specialized outside resources that might be part of a consultant or vendor team will not be at your immediate disposal after going live.

Limiting use of consulting time after going live is one of the best ways to reduce total cost of ownership for your enterprise application.

Lesson learned: Get it done right before going live. After going live, making changes will be much more difficult, and much more expensive.

The challenges of a system implementation can be somewhat daunting, but approaching the data migration process with some best practices will help to avoid potential problems that can delay a successful cutover to the new system.

A good, solid approach to data migration that allows for adequate time and resources will be instrumental in supporting the new business process with the new information system, and will allow everyone to realize the benefits earlier.

© Copyright: Connective Edge Resources 2008
Author: Norma L.Davis
Published: 4th February 2008