Role: Databricks Data Engineer (Insurance)
Location: London, United Kingdom (hybrid, 3-4 days in office)
Type: Permanent, ASAP start
My client, a well-established global data and analytics consultancy, is looking for a highly skilled Azure Databricks Data Engineer to design, develop and deliver cloud-based data integration and analytics solutions within their Insurance portfolio.
The Role:
You'll work across data engineering, modelling, and transformation initiatives - building scalable pipelines, optimising performance, and ensuring best practices in data quality and governance. This is a hands-on technical role requiring strong Azure and Databricks experience, coupled with solid insurance domain knowledge. You'll work on high-impact insurance data transformation programmes, collaborating with experienced data professionals in a forward-thinking, innovative environment, with exposure to cutting-edge Azure and Databricks technologies.
Key Responsibilities:
- Design, build, and deploy scalable data pipelines using Azure Databricks, Azure Data Factory, and Azure SQL Database
- Optimise Databricks environments for performance, cost, and security
- Develop and maintain data integration, ETL/ELT processes, and data quality frameworks
- Translate business requirements into data models, ensuring consistency and accuracy across data layers
- Work with stakeholders to deliver effective data solutions and analytics
- Create technical documentation including data flow diagrams, architecture design, and test plans
- Stay current with emerging data engineering and Azure ecosystem technologies to drive continuous improvement
Essential Experience:
- Insurance industry experience is essential, with a strong understanding of policy, claims, and regulatory data
- Minimum 5 years' management experience - leading technical delivery or overseeing project teams within data engineering or integration environments
- 8-10 years' experience in data engineering and integration, ideally in enterprise-scale environments
- Deep expertise in Azure Databricks, Azure Data Factory, Azure Synapse, and Azure Data Lake
- Hands-on experience with Spark (PySpark/SparkSQL), ETL/ELT design, and data pipeline orchestration
- Strong data modelling skills (Dimensional, ODS, Data Vault) and experience with data warehousing concepts
- Experience working with large, complex datasets and distributed computing environments
- Proficiency in SQL, version control (e.g. Git), and CI/CD pipelines
- Familiarity with Agile delivery and DevOps practices (Azure DevOps, JIRA, etc.)
Preferred:
- Knowledge of Enterprise Data Management (EDM) practices, data governance, metadata management, and lineage tools
- Exposure to multi-cloud environments (AWS or GCP)
- Understanding of modern data architecture philosophies (Dimensional, ODS, Data Vault) and real-time cloud data capabilities
What's on Offer:
- Competitive salary plus bonus
- Private healthcare, life assurance, income protection, and pension scheme
- Additional financial wellbeing benefits
- Strong investment in professional development and training
- Share purchase scheme
- Flexible hybrid working model
- A genuinely inclusive, equal-opportunity employer
Please note: candidates must already hold the right to work in the United Kingdom.
