Amazon: Re-architecture of the ML big-data data pipelines
advertising.amazon.comOne of the main contributors out of the two, to re-design and implement a new version of the data pipeline that simplifies the architecture and allows faster evolutions. This pipeline is handling petabytes of data daily and used in one of the strategic products of Amazon Advertisement organization. My team leveraged Iceberg and other technologies and refactored the existing Spark jobs to a cleaner and simpler design. This project reduced the number of job applications from 15 to 1 for a module.