Data cleaning – should this be done before or after a system migration?

Comedienne Roseanne Barr once stated that she wasn’t going to clean up until ride-on vacuum cleaners were invented. While Roseanne circumvents this philosophical conundrum by employing housekeepers, most retail database managers operate on leaner resources, with data quality an ongoing challenge. However the spectre of data quality typically remains in the background – until a major project such as ERP migration and Fulfilment EDI comes along.

The decision to migrate a key system is not taken lightly. ERP projects, for example, typically cause business disruption, loss of revenue, and reduced productivity – and that is when they go according to plan! Examples such as Hershey Foods’ botched SAP deployment, which lost $100 million in Halloween sales and 8% of their share price in 1 day, and Nike’s $400 million worldwide i2 deployment that lost $100 million in sales, reflect that even well-managed companies are not immune to project management failures. Key findings are that planning and system suitability are key to robust outcomes.

Once the decision to migrate to a new ERP is made, various questions around scope and timelines must be addressed – including when and how data cleaning will be undertaken, which is the focus of this blog post (many other critical elements need to be addressed). Traditional project management tenets require data to be cleansed prior to deployment of the new system, whereas current lean project management standards advocate an ongoing process to eliminate waste of resources.

Supporters of the traditional model believe that data cleaning should be conducted as a discrete project, generally asking employees to collaborate with suppliers over Excel lists of products, descriptions, and attributes, with the goal to cull and correct SKU lists. Once this is completed, updated data is imported to the new ERP. This method avoids import of dead SKUs and incorrect data, for a pristine commencement of the new ERP – at least in theory. The broad criticism of this approach involves the amount of time the data cleanse process takes – which, in our experience, is 6+ months – and the practical reality that even after this huge investment of time and resources, quality problems persist. In addition to data entry errors, which are virtually impossible to eliminate, a significant proportion of ‘corrected’ data will become invalid during the cleanse project cycle. ‘Churn’ is impossible to eliminate.

Proponents of modern ‘lean’ project management, with which SPS Commerce agrees, find that any advantage of cleaning data prior to ERP migration is outweighed by the negatives.

TRADITIONAL ‘CLEAN FIRST’ APPROACHTime: a discrete cleanse project delays the actual ERP migration by 6+ monthsProductivity: team members in buyer, logistics & warehouse, and accounts teams see an overwhelming proportion of their work hours absorbed by the cleanse projectInefficiency: every SKU must be checked individually for validity and complianceQuality: team members focus on the cleanse task, to the neglect of core dutiesError: data entry errors occur as manual updates are transcribed to Excel spreadsheets, including data for new fields that don’t exist in the legacy ERPComplacency: companies believe that the project will effectively eliminate database error, and plans for ongoing maintenance are neglectedFuture needs: this method does not inherently lead to an automated system of attribute management

The alternative, correcting data after the system migration, leverages modern technologies and methods to avoid pitfalls associated with the traditional approach. With SPS Commerce, for example, automated Fulfilment EDI solutions be used as trigger-points for data review, on an ad-hoc basis.

CONTEMPORARY ‘CLEAN AFTER’ APPROACHTime: the ERP migration is not delayed by data cleanseProductivity: time allocated to data cleaning is spread across a longer periodEfficiency: team members review data which has been found problematic, rather than reviewing every SKUQuality: data maintenance is considered an ongoing core responsibility and conducted in parallel with other dutiesError: error rates are minimised through direct in-system ERP entryProcedural rigour: companies implement a continuous process of data review and managementFuture proof: retailers and consumer product companies are readily able to connect assortment (attribute automation) to their information system architecture

Modern lean project management practices support a clean-after approach, for better resource management, a higher quality outcome, and future-proofing. However, if you are considering an ERP deployment, please talk to our consultants or partners to recommend a solution that fits your precise needs.