SaaS ETL Platform
An ETL-as-a-service platform. Users plug in their own data sources and database design, and scheduled Airflow jobs load the data into MinIO and MySQL on AWS EC2.
Problem
The idea was to let users schedule data ingestion from their own sources, into a database they design themselves, without having to set up Airflow or servers.
What I did
- Built the ETL pipeline in Apache Airflow. Raw files land in MinIO (S3-compatible) and processed data goes into MySQL.
- Let users define their own data sources and database structure, so they decide what gets extracted and how their tables look.
- Packaged Airflow, MySQL, MinIO and the Flask web app as Docker containers and ran them on several AWS EC2 instances. I designed the VPC and subnet layout.
- Tested the whole flow, from ingestion to storage to the web app.
user-defined source ──► Airflow DAG ──► raw files ──► MinIO (S3-compatible)
└──► processed ──► MySQL (OLTP)
│
Flask web app ◄──┘
Result
Final course project, graded 10/10.