We are transforming our current infrastructure (Data Lake on S3 + Redshift) into a modern Lakehouse architecture based on Apache Iceberg. You will join a compact, high-performing team of two engineers to scale this new architecture and migrate data from Redshift. We actively use Claude Enterprise for code generation, so your role involves high-level prompting, validation, and architectural oversight.
Key Requirements:
- technical stack:
- Apache Spark
- Apache Flink
- Apache Iceberg
- Hadoop (AWS EMR)
- AWS Glue (Jobs, Catalog)
- Python
- Deep understanding of Data Lake and Lakehouse concepts;
- Building and scaling production-grade data pipelines and performing complex migrations;
- Nice-to-have: Scala (for legacy code support);
- English: upper-Intermediate (speaking) with exceptional listening skills to navigate diverse UK and Northern Irish accents.
What You Will Do:
- Scale and evolve the new Lakehouse architecture using Apache Iceberg;
- Lead the transition of data and logic from Redshift to the new platform;
- Leverage Claude Enterprise to generate code, focusing on its validation and quality control;
- Collaborate with a supportive mentor and a critical Solution Architect to ensure top-tier delivery.
- Ability to create and defend Proof of Concepts (PoC) before technical experts