What is Landscape Analysis and why is it important to your data migration?
Landscape Analysis is one of the most vital tasks you should take on your data migration
Does your data migration resemble a leap of faith?
Are you hoping for the best and planning with a healthy dose of optimistic guesswork?
If so you might just benefit from an activity termed “Landscape Analysis”.
Look Before You Leap
In past surveys, we find that most data migration projects overrun.
The reasons for this vary but one of the biggest problems is inaccurate scoping.
I have lost count of how many data migration timelines I’ve seen which are conceived with a combination of optimism and guesswork.
Many project managers will simply look at the date the business requires the target platform then roll back their migration project from that point.
Thus ignoring issues like target bottleneck syndrome and whether the migration is actually feasible.
For example, imagine you are going on a mountaineering expedition.
You have been given a mountain to climb, would you base your estimates on how long it took to climb your last mountain?
It could be a similar height, similar terrain and you could be going with the same climbing buddies but striding off without suitable planning would present a considerable risk.
So most experienced climbers spend some considerable time and effort understanding exactly what is required to climb this new peak. Every mountain is different and no two migrations are alike.
So, the aim of landscape analysis is to perform a “data migration simulation” far enough in advance such that we don’t get caught with our trousers down midway through the project.
In terms of the “when” to run landscape analysis I would say execute it before you even sign off the project. Signing off a project assumes you know the resources, likely timescales and risks involved. How can you if you’ve not simulated the likely end-result?
So, obtain some minimal funding and launch your landscape analysis process to help you measure the viability of the project and get a head-start, you will probably need it.
Key Landscape Analysis Activities
We can break down this phase into 4 main activities:
Business Process Analysis
Data Modelling Analysis
Data Discovery Analysis
Mapping Analysis
The key to executing these activities is to run them almost in parallel, you don’t have the luxury of time to run a simple waterfall process in most data migration projects.
You need to set aggressive time-boxed cycles and then rapidly learn, share, plan and adapt as governed by the terrain you discover.
In short, your project plan will be dictated by what the landscape analysis process uncovers, not by what is written down in an optimistic project plan based on previous projects.
Business Process Analysis
What services does the target platform need to deliver that is different to what we do now?
That is business process analysis in a nutshell, we need to identify precisely what the business expects of the data in a language we can all understand.
Keep it simple
You don’t need to create endless process modelling charts which no-one really understands.
Instead, focus on getting the right people together and just hammer out what the business expects the new target platform to support.
Broad brush-strokes, not fine detail
We’re looking for the big requirements at this stage.
We will delve a lot deeper into the specifics later in the migration, landscape analysis is always time-constrained so pick out as many key requirements, get them agreed and then identify their impact.
What are the pressures on data?
What we need to uncover here are change of use pressures on the data.
For example, in a recent migration we found that legacy systems had a reasonably good level of quality for the type of business they currently run.
However, to support the new business, complete with one-touch automation, ERP and workflow processes, the data simply was not fit enough to deliver.
So, simply looking at the legacy and target schemas with a view to their correlation will not give you these answers.
Don’t wait until for needless detail
In addition, the target model may not even exist yet so we need something to move forward with.
Waiting until the target schema is firmed down is way too late to begin landscape analysis.
Remember that most organisations know why they are moving to the new environment, you just need to extract that knowledge and ensure that everyone understands AND AGREES what the future will look like.
Document it, share it and then validate it against the data.
In every situation I’ve done this there have been differences in opinion on what the future business processes should be so it pays to get this phase implemented as early as possible so we can articulate and resolve these issues.
Contentious but vital
Business process analysis is a contentious subject, many believe it should not be included in a data migration, we’re just shunting data right?
True but that data must support the new business services so it pays to understand what they will be.
Data Modelling Analysis
Once again we don’t typically have weeks and months to carry out landscape analysis so make it short, snappy and super focused..
80/20 rule is the most critical tool in your data migration
Use the pareto (80/20) rule across the board.
Which 20% of data items are critical to our migration – keys.
So, rather than spend endless hours identifying whether a timestamp in your legacy platform will map to a different format in your target platform, focus instead on the key structural relationships of your legacy environment then identify the major gaps with the target.
Discover from above and below
Work at a conceptual level first, then drill down to logical and physical.
If those terms are alien then get an experienced data modeller on the team.
You need to attack data modelling from above and below, workshop the high-level models first and use data discovery and model re-engineering using appropriate tools from below.
Get the business experts to explain the high-level conceptual models with their respective subject area relationships.
How do customers relate to equipment, services and contracts?
Drill down into the logical layer and then finally bring in your physical modelling analysis from the data profiling results to help identify gaps in any pre-conceived notions of how the business models are actually instantiated.
Do the models have the right relationships?
Look at what correlation failures there are between the legacy systems that need to be migrated.
Are there any consolidation challenges?
Are they major, can you find a solution?
You don’t need to engineer a solution just identify the risks and the key inter-system relationships.
This phase should produce a detailed set of legacy and target models that, combined with the data discovery results, can be passed to the mapping phase for a more detailed assessment.
Data Discovery Analysis
This acts as a hub of services feeding intelligence into the other activities focusing on fact not documentation.
This cannot be achieved in isolation, all business and project personnel must come together during data discovery, it is not solely a tools or technical phase.
Challenge, learn and share
For example, if we need a logical model of our billing system then don’t rely on a 10 year old entity relationship diagram, re-engineer the models from the data profiling results or your model re-engineering tool.
Get the data to give you the answers, don’t rely on documentation that will be almost certainly out of date or lack recent data quality analysis.
In one recent migration impact assessment we discovered that a foreign key expected to drive a link to a 3rd party data set critical to the migration was actually 20% complete.
This caused hysteria in the project because the planning team had viewed the entity relationship diagram created several years before and just assumed the link would be valid.
Identify and eliminate assumptions
Remember, always use the data to drive your project planning not preconceived wisdom. Building your project on assumptions is the fastest road to migration failure.
When you are creating your planning activities, ask yourself – “How many of these activities and timelines are pure assumption and guesswork?”
There are no paint-by-numbers data migration project plans, every migration is different so discover, measure and adapt based on what the data tells you.
The right technology is critical
I prefer to use a feature-rich data quality product for this phase utilising data profiling, data cleansing, matching, de-duplication and advanced parsing.
Why do we need all these features?
Data quality tools will enable you to carry out mini-migrations and relationship validations which will tell you whether the migration is likely to be viable.
In order for us to gauge the success of the migration we need to know where the pain points are and our data quality analysis will uncover this far more quickly than standard scripts or ETL tools alone.
However, the killer tool for this phase has to be a data visualisation product.
You need some method of rapidly interpreting the huge amount of data this phase will product and in my experience data quality tools alone don’t cut it.
There are products now which cost a fraction of the overall project cost, can be leased for specific time-frames, are a breeze to use and will deliver exactly what you need for this phase – accurate intelligence.
Don’t forget that the business needs to be involved during every activity so ensure you give them information in a format they can understand and utilise.
Mapping Analysis
This is where we put all our knowledge together to enable us to identify the likely success of the migration based on our intelligence gathering exercise.
Find, articulate and measure the big issues
We need to uncover issues like:
Will the legacy and target environments fit.
Are the models compatible?
Will the new business services be supported by the legacy data?
What effort will be required to resolve the issues found?
Mapping analysis is all about identifying how our legacy and target objects will link together to deliver our migration.
Once we understand the routes and challenges of this we will be in a much better shape to deliver our migration strategy and project plan.
Stay focused, don’t get too detailed
You do not need to map endless columns, we will do that later in the migration.
For now we are being agile, performing small iterations.
If we find an issue, drill down deeper and quantify it.
If there are no issues, move on, don’t focus on non-critical attribute mappings, just the most vital relationships.
In Summary
Landscape Analysis is a data migration simulation activity that will help you de-risk and accelerate your project.
It should be performed rapidly, iteratively and with the best skills and tools at your disposal.
It is an agile process, performing it in a waterfall fashion will lead to failure on projects with tight deadlines.
We never have enough time to perform landscape analysis adequately so be ruthless with your pareto (80/20) analysis.
Focus on the critical business processes and objects to be migrated and supported.
Customer involvement is paramount as you simply cannot understand the legacy environment without significant support from the in-house business and technical teams.
Record your results and findings, feed these into further planning, design and build phases.
Create a shared repository and wiki-like environment so information is freely available.
Above all, communicate, challenge and verify. If there is an assumption, validate it and share the results.