New Customer Onboarding — AWS Data Platform
2022 – 2023
Sole engineer standing up a new client's data pipeline architecture from scratch, supporting a 10-person analyst team.
Impact
- ›Designed and built the S3 data lake to Redshift pipeline architecture, orchestrated via Prefect on Docker/Lambda
- ›Cut duplicate-record rates from roughly 8% to under 1% via business-key hashing and idempotent upsert logic
- ›Negotiated technical scope directly with the client on-site, without a dedicated account lead
Being the only engineer on a client onboarding, with a 10-person analyst team depending on the pipelines you build, means the architecture has to be right the first time — there’s no second engineer to catch what you miss.
The problem
A new client needed a full data pipeline stood up from nothing: ingestion, storage, deduplication, and a reliable foundation for a 10-person analyst team to build reporting on top of. I was the sole engineer on the project, and also the primary technical point of contact for the client directly.
What I built
I designed the pipeline architecture around an S3 data lake feeding into Redshift, orchestrated with Prefect running on Docker and Lambda. A significant part of the work was deduplication logic across tables with up to 5 million rows — business-key hashing combined with idempotent MERGE/upsert patterns tailored per data source, since a naive approach doesn’t hold up at that scale or across heterogeneous sources.
I also worked directly with the client, including on-site scoping and negotiation in Barcelona, to define the technical implementation plan without a dedicated account lead in between.