Principal/Lead Data Engineer required to architect and design scalable data platforms, driving the data architecture vision across Data Lake, Pipelines, and/or Mesh.

You will need to have a background as a Data Engineer with hands-on experience on several large enterprise data lake projects, preferably with strong Python. You must also have experience as a technical lead/principal with multiple cross functional teams (both onshore and offshore), building data platforms, customer facing data products and/or machine learning systems, with experience of product analytics tools (Mixpanel, Power BI, Athena). Experience working with LLMs in Data engineering and using AI as an accelerator is also key to this role.

Technology Requirements:

Data Architecture & Design: Data Lakes (eg, AWS S3, Azure Data Lake, Google Cloud Storage), Data Mesh principles, domain-oriented data ownership and federated governance, data modelling (OLAP/OLTP, dimensional modelling, schema evolution)

Data Engineering & Pipelines: ETL pipelines (using tools like AWS Glue, Apache Spark), Map-Reduce, streaming data platforms (eg, Kafka, SQS), Real Time and batch processing paradigms

Cloud & Infrastructure: cloud-native data services (AWS Glue, Azure Synapse, GCP BigQuery, Databricks), Infrastructure-as-Code (IaC) (using Terraform, CloudFormation, Lakeformation)

Programming & Scripting: Python and SQL, C#, CI/CD pipelines and DevOps practices for data workflows

Data Governance & Security: Data cataloging and lineage tools (eg, Collibra, Apache Atlas, OpenMetaData), data privacy, encryption, access control (eg, IAM, RBAC, ABAC), and compliance frameworks (GDPR)

Observability & Reliability: Monitoring and alerting for data systems, data quality frameworks (eg, Great Expectations, Monte Carlo), designing for resilience, fault tolerance, and disaster recovery

This role will be mostly remote but does require attending the office in Richmond upon Thames 1-2 days a week.