Senior Data Engineer – Data Products & AI

Beacon Intelligence | London (Hybrid)| Salary: £90,000 + Bonus

At Beacon Intelligence, we are a SaaS data and analytics business focused on delivering high-value insights through our proprietary platform. Data sits at the core of everything we do — powering our products, driving customer value, and enabling scalable growth.

We have built a modern data platform over the last 24 months, consisting of Airbyte as our ETL tool and Databricks as our scalable data processing layer, all hosted on Azure Cloud Services. This platform underpins our product offering and is central to our continued evolution as a data-driven SaaS business.

The Role

As a Senior Data Engineer (Data Products & AI), you will play a key role in building and scaling Beacon’s data products.

This is not a traditional data engineering role focused purely on pipelines. Instead, you’ll work at the intersection of data, product, and commercial teams, turning datasets into customer-facing, revenue-generating solutions.

You’ll combine strong engineering expertise with a data product mindset, structuring, enriching, and operationalising data through modern platforms and AI systems.

Key Responsibilities

Data Product Development

• Design and build scalable data products that underpin Beacon’s commercial offerings

• Transform raw and third-party data into structured, enriched, product-ready datasets

• Partner with product and commercial teams to define how data is packaged, accessed, and monetised

• Deliver data via APIs, internal tools, and customer-facing platforms

AI & Data Enrichment

• Apply AI/LLM capabilities to enhance datasets (e.g. classification, tagging, summarisation, insight generation)

• Build and integrate LLM-powered product features

• Implement RAG frameworks to improve data quality and unlock new functionality

• Support development of AI-driven products (search, recommendations, insight tools)

• Ensure AI systems are scalable, governed, and production-ready

Platform & Engineering

• Build and maintain robust data pipelines (batch and streaming)

• Own and evolve the data platform (Databricks, Azure, Airbyte)

• Ensure high standards of data quality, testing, and observability

• Improve efficiency through automation, CI/CD, and engineering best practices

• Support deployment and lifecycle management of data and AI systems

Commercial & Stakeholder Impact

• Collaborate with senior stakeholders to identify high-value data opportunities

• Translate business requirements into scalable data and AI solutions

• Act as a bridge between technical and non-technical teams

• Contribute to prioritisation based on commercial impact

Leadership & Growth

• Influence the evolution of Beacon’s data product and AI strategy

• Mentor junior engineers and promote strong engineering standards

• Support the development of a high-performing, commercially focused data team

What We’re Looking For

Core Engineering Experience

• Strong experience with modern data platforms (e.g. Databricks) and cloud (Azure preferred)

• Advanced Python and SQL, with a focus on scalable, production-quality solutions

• Experience with Apache Spark (PySpark) for large-scale data processing

• Proven experience building data pipelines and data models

• Experience working with APIs and data delivery mechanisms

• Familiarity with CI/CD, Git, and modern development practices

Data Product Mindset

• Experience building or supporting data products or customer-facing data solutions

• Strong understanding of how to structure and expose data to end users

• Ability to think beyond pipelines and focus on commercial value

• Experience working closely with product or commercial teams

AI / LLM Experience

• Hands-on experience or strong familiarity with LLMs (e.g. Azure OpenAI, LangChain)

• Experience applying AI to real-world use cases

• Experience working with unstructured data (text, documents, web data)

• Understanding of embeddings, vector databases, or semantic search

• Awareness of AI governance and data ethics

Desirable Experience

• Experience with Airbyte or modern data ingestion tools

• Infrastructure-as-code (e.g. Terraform)

• Exposure to LLM frameworks (LangChain, LlamaIndex)

• Experience with experimentation or A/B testing

• Background building AI-enabled or data-driven products

• Life sciences or healthcare domain experience

How You Work

• Product-focused – you think in terms of outcomes, not just pipelines

• Commercially aware and impact-driven

• Pragmatic and delivery-oriented

• Collaborative, with strong stakeholder engagement skills

• Curious about AI and emerging technologies, with a focus on scalable implementation

Why Join Beacon Intelligence

• Build products, not just infrastructure, with real-world impact on patients

• Help shape how AI is embedded into a growing SaaS/data platform

• High-impact role with influence across product and commercial strategy

• Work with a modern, evolving tech stack

• Strong ownership and growth opportunities

• Collaborative environment with teams that value data and insight