Data transformation with Python for a global insurance company.

The Client

Core Consultants helped a global insurance company to develop an automated data ingestion pipeline.

The Challenge

The client's data solution consisted of code, which was hard to configure and lacked flexibility, taking the operational team up to a week at a time to perform the necessary validation and transformation tasks.

The Solution

Using our knowledge of Spark and Python, we had to automate and increase the efficiency of these vital processes by supporting the operations department to automate their lengthy data validation procedure, which usually required several iterations of reviews across various teams. 

After transforming the data using SQL, we then developed a Python code to produce load-ready files within seconds.

The Outcome

Increased Efficiency: Reduced the time taken for ingesting complex data to just under a few seconds.

Improved Customer Experience: The Python code skills addressed a skills gap and helped transform the customer's user experience.

Modernised Processes: The client's legacy processes were transferred to an environment using Spark and Python, creating an efficient ingestion process across the whole organisation.