Software Dev Engineer III

Role overview

We’re looking for a Data Engineer who thrives on building robust, real-time and batch data products on Microsoft Fabric. You’ll design and operate ingestion from streaming sources (Event Hubs/Service Bus/Confluent Kafka), model curated Silver/Gold layers in Lakehouse, optimize KQL/Spark pipelines, and enable trustworthy, fast Power BI dashboards (including Direct Lake and semantic models).

What you’ll do

• Design and implement scalable data pipelines (batch + streaming) from diverse sources (REST, SFTP, RDBMS, Kafka/Event Hubs/Service Bus) into a lakehouse and OneLake.
• Model and curate datasets using medallion architecture; build reusable frameworks for ingestion, schema evolution, and incremental processing.
• Write efficient transformations in Spark (PySpark/SQL) and/or KQL; create materialized views, update policies, and optimization strategies for cost/perf.
• Implement CDC, watermarking, late-arrivals handling, and idempotent writes for append/merge scenarios.
• Enforce data quality, observability, and lineage (DQ rules, expectations, SLAs, alerts, metadata catalogs).
• Apply security & governance best practices (PII hashing/tokenization, access controls, secrets management).
• Productionize workloads with orchestration (Airflow/ADF/Azure Synapse/Step Functions/Glue), CI/CD, testing, and rollout strategies.
• Partner with product/analytics teams to define SLAs, table contracts, and consumption patterns; create reliable semantic layers.
• Troubleshoot performance, skew, and reliability issues; tune storage (Delta/Parquet/Iceberg) and compute configurations.

What you’ll bring

• 6+ years of data engineering experience (title flexible: Data Engineer / Senior Data Engineer).
• Strong SQL and one of Python/Scala. Deep familiarity with Spark (PySpark/SQL) and distributed data patterns.
• Hands-on with one or more clouds (Azure/AWS/GCP) and a lakehouse stack (e.g., Databricks, Delta Lake, Fabric Lakehouse/Eventhouse, Synapse, BigQuery/Snowflake a plus).
• Streaming experience: Kafka/Confluent, Azure Event Hubs, or Service Bus; schema registry, exactly-once/at-least-once semantics.
• Solid understanding of medallion architecture, CDC, SCD, upserts/merge, partitioning, Z-ordering, compaction, and vacuum.
• Orchestration & DevOps: Airflow/ADF/Glue/Step Functions; Git-based workflows, unit/integration tests, environments, and IaC (Terraform/ARM/CDK) preferred.
• Data quality & governance: expectations/testing, lineage/metadata, RBAC/ABAC, PII protection (hashing/salting/tokenization).
• Comfortable owning services in production: monitoring, alerting, SLIs/SLOs, on-call rotation.

What You’ve Done (Must-Haves)

• 5+ years in data engineering with cloud data platforms (Azure preferred).
• Hands-on with Microsoft Fabric components: Eventhouse (KQL), Lakehouse (Delta on OneLake), Spark notebooks, Data Factory (Fabric pipelines), Power BI (including Direct Lake).
• Solid SQL/KQL/PySpark; comfort with nested JSON, mv-expand, update policies, materialized views, partitioning.
• Built production-grade streaming + batch pipelines; handled late/duplicate events, watermarking, and idempotency.
• Strong grasp of data modeling, performance tuning, and data quality (unit tests, anomaly checks, SLAs).

Nice to Have

• Confluent Kafka private networking patterns; CDC from operational stores.
• Azure ecosystem: ADLS/OneLake, Key Vault, AAD, Purview, Event Hubs, Service Bus.
• MLOps/feature store basics; Python packaging & testing (pytest).
• Governance & compliance (GDPR/CCPA), PII handling, and secrets management.

Tech Stack You’ll Touch

• Microsoft Fabric: Eventhouse/KQL, Lakehouse/Delta, Spark notebooks, Data Factory, Power BI (Direct Lake)
• Azure: Event Hubs, Service Bus, AAD, Key Vault, Purview
• Langs/Tools: SQL, KQL, PySpark, Python, Git, CI/CD (ADO/GitHub)