Schedule a call →

<p>Interview fractional leaders this week for <strong>free</strong>!</p>

We partner with Microsoft to bring their vast suite of enterprise solutions to clients, with Microsoft consulting services & technologies. Learn more here.

Spyglass MTG Icon

Spyglass Solutions is a leading women-owned consulting firm specializing in Data, AI, and Security solutions on the Microsoft clouds. With over 15 years of experience, we offer expertise in AI Transformation, Data Platform Integration, Business Analytics, and more. Our services include M365 Migrations, Azure Platform solutions, Managed Services, and much more. As a Microsoft Solutions Partner with 5 Solutions Badges and FastTrack Ready status, we pride ourselves on delivering excellence to our clients. Our team of experts, including our experienced leadership team, is dedicated to your success. Whether you are looking to embark on an AI transformation journey, modernize on Azure, or enhance your security model, Spyglass Solutions is here to help. Contact us today to learn how we can pave the way for your business success in the digital age. Partner with us and experience the difference with Spyglass Solutions.

Spyglass MTG

Spyglass MTG Logo

# Lead Data Engineer - Databricks & Microsoft Fabric

**Location:** Remote (Preference for candidates located in New England)

**Engagement:** Contract or Contract-to-Hire

## Overview

Spyglass MTG is seeking a highly skilled Lead Data Engineer with strong expertise in Databricks and Microsoft Fabric to help design, build, and scale modern data platforms in Azure.

This role is both strategic and hands-on. The ideal candidate will lead the development of enterprise data solutions while actively building scalable pipelines, lakehouse architectures, and data platforms that support analytics, reporting, and AI initiatives.

The Lead Data Engineer will work closely with data architects, analysts, and business stakeholders to deliver high-performance data solutions that enable organizations to leverage their data more effectively.

## Responsibilities

• Design and implement modern data platforms using Databricks and Microsoft Fabric

• Build and maintain scalable data pipelines and ingestion frameworks

• Develop and optimize Spark and PySpark workloads within Databricks

• Implement lakehouse and data warehouse architectures

• Design and manage data models supporting analytics and reporting

• Collaborate with data scientists and analytics teams to support AI and advanced analytics initiatives

• Implement data quality, governance, and security best practices

• Provide technical leadership and mentorship to data engineering team members

• Optimize performance, reliability, and scalability of data platforms

## Requirements

• 7+ years of experience in data engineering or data platform development

• Strong hands-on experience with Databricks and Apache Spark

• Experience implementing Microsoft Fabric data solutions

• Proficiency in Python and PySpark

• Experience working with Azure data services and cloud-based data platforms

• Experience designing modern lakehouse or data warehouse architectures

• Strong understanding of data pipeline design, ETL/ELT processes, and data modeling

## Preferred

• Experience supporting AI or machine learning workloads

• Experience working in enterprise data environments

• Familiarity with CI/CD and data platform automation

Spyglass - Lead Data Engineer - Databricks & Microsoft Fabric

A unified analytics platform, powered by Apache Spark

Databricks

Data Engineering

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Apache Spark

Spyglass MTG is seeking a Lead Data Engineer with expertise in Databricks and Microsoft Fabric for a contract or contract-to-hire role. The position involves designing and building data platforms in Azure, leading data solutions development, and collaborating with stakeholders. The role is remote with a preference for candidates in New England.

Lead Data Engineer

Data

## Overview

Spyglass MTG is seeking a highly skilled Lead Data Engineer with strong expertise in Databricks and Microsoft Fabric to help design, build, and scale modern data platforms in Azure.

This role is both strategic and hands-on. The ideal candidate will lead the development of enterprise data solutions while actively building scalable pipelines, lakehouse architectures, and data platforms that support analytics, reporting, and AI initiatives.

The Lead Data Engineer will work closely with data architects, analysts, and business stakeholders to deliver high-performance data solutions that enable organizations to leverage their data more effectively.

## Responsibilities

• Design and implement modern data platforms using Databricks and Microsoft Fabric

• Build and maintain scalable data pipelines and ingestion frameworks

• Develop and optimize Spark and PySpark workloads within Databricks

• Implement lakehouse and data warehouse architectures

• Design and manage data models supporting analytics and reporting

• Collaborate with data scientists and analytics teams to support AI and advanced analytics initiatives

• Implement data quality, governance, and security best practices

• Provide technical leadership and mentorship to data engineering team members

• Optimize performance, reliability, and scalability of data platforms

## Requirements

• 7+ years of experience in data engineering or data platform development

• Strong hands-on experience with Databricks and Apache Spark

• Experience implementing Microsoft Fabric data solutions

• Proficiency in Python and PySpark

• Experience working with Azure data services and cloud-based data platforms

• Experience designing modern lakehouse or data warehouse architectures

• Strong understanding of data pipeline design, ETL/ELT processes, and data modeling

## Preferred

• Experience supporting AI or machine learning workloads

• Experience working in enterprise data environments

• Familiarity with CI/CD and data platform automation

Lead Data Engineer - Databricks & Microsoft Fabric - Paylocity

Fast, easy, and collaborative Apache Spark–based analytics service

Azure Databricks

PySpark

Integrated cloud services and infrastructure to support computing, database, analytics, mobile, and we...

Microsoft Azure

Senior Data Engineer

Data Architect

Technical Lead

Contract or Contract-to-Hire Lead Data Engineer to build data platforms for tech company

Spyglass MTG

About Spyglass MTG

Jobs

Past roles

About this page