- Development, maintenance, and optimization of ETL processes in Informatica PowerCenter and Apache Airflow.
- Experience with Oracle PL/SQL: optimizing complex queries, developing stored procedures, and automating data processing workflows.
- Supporting real-time microservices with Apache Kafka to ensure seamless data streaming and processing.
Projects:
1. Customer Selection Process
Migration from Oracle to Informatica
- Led the full migration of the customer segmentation process for the #kartakarta product, transitioning from PL/SQL-based Oracle procedures to Informatica PowerCenter.
- Optimized 10+ complex ETL procedures, improving data processing speed and reducing execution time.
2. Migration of 50+ ETL Pipelines from Informatica to Apache Airflow
- Successfully redesigned and migrated over 50 ETL workflows from Informatica PowerCenter to Apache Airflow, improving orchestration and monitoring capabilities.
- Enhanced scalability and automation by integrating Airflow DAGs with cloud-based storage and processing systems
3. Fraud Detection Microservice using Apache Kafka & AWS
- Designed and deployed a real-time fraud detection system for financial transactions using Apache Kafka and AWS services (S3, Lambda, and DynamoDB).
- Implemented streaming data processing pipelines that analyze incoming transaction data and detect anomalies and reduced fraudulent transaction processing time by 30%
4. Automated JSON File Parsing System
- Built a scalable ETL pipeline to ingest and process large volumes of JSON files from various sources, improving data integration efficiency.
- Developed custom Python scripts for parsing and transforming JSON data into structured formats for database storage.