A leading Data & Analytics consultancy is looking for an AI Engineer (Databricks / Lakehouse AI) for their practice in Sydney

A brief job description is below with a more detailed one after that:

Required

• Strong proficiency in Python
• Hands-on experience with Databricks
• Solid understanding of machine learning concepts
• Experience with Apache Spark
• Familiarity with SQL
• Experience building and deploying ML models in production
• Exposure to LLMs, embeddings, vector search, and RAG architectures

Preferred

• Experience with Azure
• Experience with Mosaic AI, Databricks Feature Store, and Unity Catalog
• Background in data engineering or analytics engineering

DETAILED SPECIFICATION

Role Overview

The AI Engineer is responsible for designing, building, and deploying AI-powered solutions on the Databricks Lakehouse platform. This role bridges data engineering, machine learning engineering, and applied AI, enabling scalable analytics, predictive models, and generative AI use cases across the enterprise.

The ideal candidate has hands-on experience with Databricks, strong Python and Spark skills, and practical exposure to machine learning and GenAI workloads in production environments.

Key Responsibilities

AI & Machine Learning Development

• Design, build, and deploy machine learning and AI models using Databricks Machine Learning and Mosaic AI
• Develop end-to-end ML pipelines including data preparation, feature engineering, training, evaluation, and inference
• Implement LLM-based solutions (e.g. RAG, fine-tuning, prompt engineering) using Databricks and open-source models
• Integrate ML models into downstream applications via batch and real-time inference

Data Engineering & Lakehouse Enablement

• Build scalable data pipelines using Apache Spark, Delta Lake, and Databricks Workflows
• Collaborate with data engineers to ensure high-quality, governed feature and training datasets
• Leverage Unity Catalog for secure, governed access to data and AI assets

MLOps & Productionisation

• Implement MLOps best practices using MLflow for experiment tracking, model registry, and lifecycle management
• Automate model deployment, monitoring, and retraining pipelines
• Monitor model performance, data drift, and operational metrics in production

Collaboration & Stakeholder Engagement

• Work closely with data scientists, engineers, architects, and business stakeholders
• Translate business problems into AI-driven solutions with measurable outcomes
• Contribute to AI standards, reusable frameworks, and best practices within the organisation

Required Skills & Experience

Core Technical Skills

• Strong proficiency in Python for data science and AI workloads
• Hands-on experience with Databricks (notebooks, jobs, MLflow, Delta Lake)
• Solid understanding of machine learning concepts (supervised/unsupervised learning, model evaluation)
• Experience with Apache Spark for large-scale data processing
• Familiarity with SQL for data exploration and transformation

AI & GenAI Experience

• Experience building and deploying ML models in production
• Exposure to LLMs, embeddings, vector search, and RAG architectures
• Experience using frameworks such as Hugging Face, LangChain, or similar (preferred)

Cloud & Platform

• Experience on Azure (preferred), AWS, or GCP
• Understanding of cloud security, identity, and cost management considerations for AI workloads

Preferred Qualifications

• Experience with Mosaic AI, Databricks Feature Store, and Unity Catalog
• Knowledge of CI/CD for ML pipelines
• Background in data engineering or analytics engineering
• Experience working in regulated or enterprise environments (e.g. Energy, Financial Services, Property)

What Success Looks Like

• AI solutions are production-ready, scalable, and governed
• Models deliver clear business value and are trusted by stakeholders
• AI workloads are efficiently integrated into the Databricks Lakehouse
• Best practices for MLOps, security, and cost optimisation are consistently applied

Why Join

• Work on cutting-edge AI and GenAI solutions using the Databricks Lakehouse
• Influence how AI is operationalised across the organisation
• Collaborate with experienced data and analytics professionals
• Continuous learning and exposure to emerging AI capabilities

Working Place

Sydney, NSW, Australia