ETL Migration for IT Company
A large-sized IT company the development of digital solutions contained vast amounts of data for the company structure, projects, customers, employees, and their education certificates of 3,000+ employees.
The existing system provided them with more functionalities than needed, leading to overpricing. The decision to change was financially based, and pricing was the reason.
Try Visual Flow
Challanges
Solutions
-
ETL jobs MigrationMigrate all ETL jobs and data pipelines from IBM DataStage to the Visual Flow low code solution. -
ConfigurationInstall and configure the new target database. Develop corresponding jobs and data pipelines in the Visual Flow solutions. -
TrainingEducate the client’s data and non-data analysts on how to use their new software. Create extensive technical documentation for the delivered Visual Flow solution. -
Simple UICreate a simple drag-and-drop interface/UI for the panel for creating ETL job pipelines. -
Deadline-drivenEfficient planning, setting up clear goals to achieve & approved work scope allowed to meet deadlines before the imminent shutdown of servers hosting the old target database and IBM DataStage.
Results
-
5 times lower operational costs
The migration decreased the cost of ownership of the ETL system by at least five times compared to the original DataStage usage.
-
3 times Improved performance
The IBM DataStage logic has been optimized and successfully migrated to Visual Flow, resulting in 3 times improved performance.
-
Usability: 8-10 times high-speed task performance
The Data Analysis team is now able to build new jobs and pipelines 8 to 10 times quicker, using the simple drag-and-drop interface/UI for the panel and technical documentation for the Visual Flow solution.
Sooner or later, many companies meet data transformation challenges during their business operation course. It was the case with one of our clients who approached the Visual Flow professional service team with the request to migrate ETL jobs from a current application to something modern and open source.
The Story of the Project and the Business Behind It
Let us start with some general information about our client. It’s a large-sized tech-driven organization focusing on the development of custom digital solutions. They invest heavily in the education of software engineers to improve their hard and soft skills.
The customer system was developed around the IBM Datastage software. It contained ETL data for the company structure, projects, customers, employees, and their education certificates. Just for reference: there was data of 3,000+ employees. It includes information for completed and passed courses, internal promotion and compensation increase lists, and data that allows for monitoring the availability of engineers for potential projects.
Our client faced a complex issue. First, the IBM Datastage provided them with more functionalities than they needed, meaning they were paying for services they did not use . Pricing was the reason for the changes.
The fast-approaching expiration date of the hosting license created a sense of urgency for the data migration.
So, they approached the Visual Flow team to help migrate away from DataStage and old servers. We were to find a solution that would meet our client’s requirements regarding the tech stack. It should be modern, cloud-native, scalable, and open-source. It also needed to be intuitive enough and come with low code and job visualization options. The latter functionalities were due because they greatly assist non-tech data analysts, and HR use them in their day-to-day ETL routines.
Goals and Objectives for the Visual Flow team
After a negotiating and planning phase; the Visual Flow team came up with clear goals to achieve. The team created an approved work scope.
- Migrate all ETL jobs and pipelines from IBM DataStage to the Visual Flow software solution.
- Provide HR analysts and resource managers with the possibility to build simple ETL job pipelines should they need new data for their evaluation and appraisal initiatives.
- Create a simple drag-and-drop interface/UI for creating ETL job pipelines.
- Install and configure the new target database.
- Create extensive technical documentation for the delivered Visual Flow solution.
- Develop corresponding jobs and pipelines in the Visual Flow solutions.
- Educate the client’s data and non-data analysts on how to use their new software.
The project was complicated by the requirement to conduct all the processes within one month without compromising the quality, reliability, or performance of ETL processes. As mentioned, deadlines are explained with the oncoming shutdown of the servers.
How Visual Flow Responded to the Challenges
-
Install a new target database. -
Create schemas and tables. -
Establish a relationship between tables in the new target database. -
Document the old DataStage logic. -
Make changes to the old logic, keeping in mind the features of the original Visual Flow solution. -
Install and configure the Visual Flow solution. -
Create a project inside the Visual Flow solution. -
Create and validate connections to source and target databases. -
Create Visual Flow jobs in the designer, so they are parallel with jobs in DataStage. -
Create pipelines — analog of sequence jobs in DataStage. -
Check the performance of jobs and pipelines and fix errors. -
Perform data validation between old and new target databases. -
Optimize the performance of the Visual Flow solution. -
Disable scheduling for DataStage sequence jobs. -
Add a schedule to the Visual Flow cron pipelines.
The Business Outcomes of Our Work
Business-level projects are not about meeting technical requirements, but rather about outcomes obtained. Here is a summary of the achieved results through this project implementation:
- The IBM DataStage logic has been optimized and migrated to Visual Flow, resulting in 3 times improved performance.
- Engineers have transferred all data from the old target database to the new one. Now it is Cognos-compatible and suitable for building reports without changing the logic.
- The Data Analyst team can now build new jobs and pipelines 8 to 10 times quicker.
- The cost of ownership of the ETL system was decreased by at least 5 times compared to the original DataStage usage.
Do you have a data migration or a similar ETL project in mind?
Are you inspired by the results Visual Flow provides the clients? If so, contact us today! Our professional services team will always be happy to provide you with extra details or schedule a free strategy session.
Try Visual Flow
AWS Cloud Migration for a Global Company
A large, international business with a strong emphasis on technology hired Visual Flow to help them move their clients’ infrastructure to cloud services.
When our ETL migration experts first started working together, our client’s end-user was heavily storing data for the following internal service tasks:
- tracking production costs;
- dealer’s data;
- the date that the client shared with the dealers;
- client/employee details.
Along with many additional tools, like Cron, Shell Script, Java, and JavaScript, the infrastructure mostly rested on the Oracle database and Visual Flow ETL tool.
As part of its modernization efforts, the corporation standardized its manufacturing processes, strengthened its organizational framework, and introduced new practices to manage development quality.
Business Challenges
-
Primary Goal #1Move business operations to cloud-based new infrastructure. -
Primary Goal #2Match Visual Flow's data migration service to the company's business operations.
Solution Highlights
The client established a separate team in charge of the creativity and integration of many systems into one infrastructure. With the help of architects and engineers of Visual Flow team the client adapt solutions for infrastructure modernization. Our teams worked on the overall ecosystem’s top-level structure, general techniques, and regulations for each system.
The Visual Flow experts immediately set this migration into motion after making suggestions for the process’s development and auditing. The involvement of a Visual Flow vendor improved the quality of judgments made throughout the architectural design stage.
Because of the business’s unique characteristics, each department had its architecture, and different teams were formed to oversee its adaptation and development. They also participated in the migration of current local services to cloud technologies: Visaul Flow, PySpark, ECS, Kubernetes, EMR + PySpark.
A “seamless” transfer was one of the key requirements; teams throughout the migration process made sure that access to current services remained continuous.
Results
-
Enhanced Fault Tolerance
Enhanced fault tolerance via bug fixes and the use of pre-built cloud-platform services to facilitate interaction between various client services.
-
Operational Continuity
Existing services remained operational during the move.
-
Cost Savings
Cost savings via the rejection of certain server capacity as a result of the cloud service transfer and using open source AWS compilable ETL tools as Visual Flow Flexible Financial Planning.
Flexible financial expenditure planning for growth because of the cloud solution’s pay-as-you-go model. -
Accelerated IT Hiring
A shorter time to fill IT expert positions was achieved via standardizing standards, specifying explicit guidelines, and outlining the technological stack.
-
Optimized Workforce
Reduction in the number of professionals with specialized knowledge who are unemployed full-time as a result of their retraining in the company’s chosen technological stack.
Do you have a data migration or a similar ETL project in mind?
Are you inspired by the results Visual Flow provides the clients? If so, contact us today! Our professional services team will always be happy to provide you with extra details or schedule a free strategy session.
Try Visual Flow
Visual Flow's Compatibility for a Commercial Bank
Description: part of an international financial consortium, a sizable Eastern European commercial bank confronted the difficulty and expenses of growing, as well as the impossibility of adding new capabilities resulting from the present data storage and infrastructure configuration.
To serve its customers and keep data safe throughout internal data loading and processing, the bank built and extended its on-premise infrastructure using bespoke solutions.
Challenges
Using on-site infrastructure was the toughest obstacle because it meant always supporting the hardware against rising data volumes. The bank demanded fresh methods for handling data that the present infrastructure could not handle. Given this, it turned to Visual Flow, as business intelligence services provider, in search of a solution for the various conflicting problems it encountered.
-
Challenge #1The customer was actively working on a mobile application for Internet banking and adding new features, including online card processing from a website or application and online payment capabilities. Due to major user interface flaws and incompatibility with certain smartphone models, existing solutions failed to meet the client's expectations. Furthermore, it was not feasible to add intended features using the current program, such as fresh forms of cost estimates. The bank was in the midst of an aggressive expansion period and was acquiring new clients. The user experience was crucial. Consequently, revenue and clients were lost due to the sluggish rollout of new features and the cumbersome user interface. Adding new features at a time of fast expansion and addressing user environment problems rapidly were the main difficulties.
-
Challenge #2In order to scale the bank's servers, the necessary equipment was bought and set up in preparation. A portion of the capacity remained unused while waiting for loading, needing maintenance and taking up space when consumption was optimized. Concurrently, there was a dramatic rise in the overall cost of data backup, server room upkeep, cooling, repairs, and maintenance workers. The most important thing to do with server capacity was to design a system for storing and organizing data that could handle projected expansion in the next 2–5 years efficiently. -
Challenge #3As a result of the frequent state audits, staff from many departments had to compile detailed reports to meet the needs of each auditor. Data from the databases still had to be retrieved by hand, even when the queries were repeated. The company had significant issues clearing the previous audit due to its inability to provide two reports about transactions. This was caused by a lack of an adequate technological foundation. The lack of technological competence and the high level of manual work required to compile this data was the biggest obstacle to submitting audit reports. Our ETL migration experts had to eliminate flaws as soon as possible and creating a robust data pipeline. At the end, their intervention allowed the company to meet audit requirements efficiently.
Goals
-
Goal #1
Keep the quality of the bank’s services and products high or raise the bar.
-
Goal #2
Lessen the growing workload associated with managing their server and infrastructure.
-
Goal #3
Free up development time for new features by removing technological constraints.
-
Goal #4
Make sure the company is technically prepared to meet the demands of auditors.
Solution Highlights
Data Management Solutions to Cloud Migration Service
Our architects set out to create a solution once they had recognized the problems and the objectives. The bank’s representatives emphasized scalable, fault-tolerant systems that are dependable and easy for the bank’s IT experts to manage.
Several potential answers were found:
- Customer migration to the cloud using our comprehensive data migration service and no code ETL tool;
- Installing OpenShift from RedHat;
- Converting client services to work on AWS Cloud.
The most suitable AWS services were selected to replace the client’s existing solutions, ensuring compatibility and scalability. Additional services were incorporated to address specific tasks while providing growth potential. A separate focus was placed on implementing a robust visualization and reporting system, utilizing Tableau for advanced analytics and interactive dashboards.
Our Client’s Gains from Outsourcing Data Management to the Cloud:
- Once development and testing were complete, the client was given the chance to use the new capability right away. ETL migration experts ensured a seamless integration of data pipelines, minimizing downtime and streamlining data operations.
- The customer received a backup system with a prior functional version in the case of the main system failure, and no difficulties were detected. This served as a temporary solution until the underlying issue was resolved.
- The customer’s primary focus shifted away from increasing hardware capacity due to the fact that monitoring systems now automatically propose required improvements. The engineers at the bank now can easily scale up or down with the click of a mouse.
- Better expansion of client businesses.
- Enhanced happiness for clients as a result of first-rate service and lightning-fast new features.
Key Results
-
#1
With the help of the Visual Flow team, the bank was successful in moving 80 percent of its server infrastructure to the cloud.
-
#2
Within two months of the Visual Flow team’s modifications, there were no more infrastructure failures than the thirteen that had occurred before.
-
#3
Methodology for building cloud-native applications using RedHat, OpenShift, and other suitable cloud alternatives for current services.
-
#4
Data aggregation across many clusters using Visual Flow as ETL tool. Providing the necessary 99.99% uptime with fault-tolerant data storage in two data centers.
Do you have a data migration or a similar ETL project in mind?
Are you inspired by the results Visual Flow provides the clients? If so, contact us today! Our professional services team will always be happy to provide you with extra details or schedule a free strategy session.
Try Visual Flow
Moving an IT Company's Human Resources Database to the Cloud
Description: human resources data was dispersed throughout our client’s many systems and divisions, which caused a lot of problems. Efficient personnel management and thorough HR analytics were impeded by this decentralization. The customer migrated HR data to the cloud in order to achieve better decision-making and unified data management.
Challenges
-
Bringing together information from several HR systems;
-
Getting to the bottom of data duplication and discrepancies caused by using many HR systems made it difficult to rely on a single source of truth;
-
Building data flow diagrams;
-
Implementing the data quality management process;
-
Masking personally identifiable information according to access permissions;
-
Develop ETL workflows to efficiently integrate data into the global data warehouse solution;
-
Ensuring that the HR software in the cloud correctly works with other company systems and applications.
Outcomes
-
Saving human resourses
All parties involved with human resources — teams, supervisors, and employees — have safe, anywhere access to critical data, as well as real-time analytics achieved.
-
Successful migration
Successful migration for 39 locations is achieved by carefully following all regulations and taking advantage of our data migration service. Our ETL experts played a critical role in designing and implementing efficient data pipelines, ensuring a smooth transition and maintaining data integrity.
-
Protection
Protected sensitive HR data was achieved by the use of advanced encryption and security mechanisms.
-
Encryption and security
Sensitive HR data was secured using state-of-the-art encryption and security methods.
-
Easy integration
Self-service features and collaboration increased staff members’ team work engagement, boosting satisfaction by 50%. With the help of the Visual Flow team, data silos were broken down via the easy integration of various HR tools and apps.
Do you have a data migration or a similar ETL project in mind?
Are you inspired by the results Visual Flow provides the clients? If so, contact us today! Our professional services team will always be happy to provide you with extra details or schedule a free strategy session.
Try Visual Flow
Contact us
