Google AdWords - Change History, Logs Processing
Change History is a feature of AdWords where advertisers can view all changes made to their ad campaigns (changed via API and manually). I made full-stack changes to migrate all Ads change history data to a new database and serve this new data to user.
- Wrote mapreduce pipeline for pre-rendering all historical change data and storing into new database. This allows deprecation of old database.
- Wrote mapreduce pipeline for verifying all migrated data is present for confidence before turning down the old database.
- Changes to frontend for serving from new database and storage format.
- Assisted with design decision of how to store migrated dta.
EvenFlow is the Google Ads real-time processing pipeline that converts raw event logs for views and clicks into stats and metrics used for analysis and billing. I worked on the integration test suite for the end-to-end logs processing pipeline.
Visit Google AdWords
- Improved pipeline instrumentation to record duration of each pipeline stage for each run.
- Created tool to generate graph showing runtime broken down by pipeline stage, showing parallel and serial sections.
- Created visualization to show total pipeline latency changes over the last few months. Lead to identifying increased latency over time.
- Improved regression test speed by running more stages in parallel, cutting down overhead time for a section.
- Reduced regression test resource utilization by creating system to store golden run data instead of recomputing each run.