Implementing EDM and Real-Time Analytics with AWS
A leading US eCommerce API and end-to-end software services provider aimed to enhance their sales, facing both internal and external pressures to boost overall performance. Their strategy involved utilizing a Cloud Data Warehouse (AWS) and Data Analytics.
Challenges
- Consolidate data from industry leaders.
- Improve cutting-edge analytics for industry data
- Establish as an authoritative data provider in the industry
- Enhance slicing and dicing of complex risk management KPIs
- Implement self-service BI across the organization
- Develop a client-centric reporting framework
Results
- Enhanced Data Integration: Streamlined merging of varied data sources for improved quality and accessibility.
- Advanced Analytics: Deeper insights and better decision-making through cutting-edge analytics tools.
- Risk Management Improvement: More detailed analysis of complex KPIs for effective risk handling.
- Self-Service BI Empowerment: User-friendly data exploration and reporting across the organization.
- Client-Centric Reporting: Tailored frameworks for improved client engagement and satisfaction.
Solution
- Migrated various data types from multiple sources (Salesforce, JSON, XML, Java APIs, semi-structured data) using Talend ETL tool to AWS-hosted warehouse.
- Employed Spark with Python for high-performance data loading.
- Utilized KMP Algorithm and POSIX operators for pattern matching and data standardization.
- Implemented real-time data push from sources to target server for Power BI visualization.
Benefits
Cost Savings:
40%
Conversion
+20%
Tools:
- Talend Studio
- Java Code for API Calls
- Python Code for AWS Redshift UDF (Analytics)
- Spark with Python (for Data Loading)
- Amazon Redshift and S3
- Power BI