Job description
Project Role : DevOps Engineer
Project Role Description : Responsible for building and setting up new development tools and infrastructure utilizing knowledge in continuous integration, delivery, and deployment (CI/CD), Cloud technologies, Container Orchestration and Security. Build and test end-to-end CI/CD pipelines, ensuring that systems are safe against security threats.
Must have skills : Microsoft Azure DevOps
Good to have skills : NA
Minimum 5 year(s) of experience is required
Educational Qualification : 15 years full time education
Summary: You’ll be the engineer who makes data move reliably. You’ll build Azure DevOps CI/CD that orchestrates ADF and Databricks Workflows to deliver data into a Medallion (Bronze/Silver/Gold) architecture, and you’ll own Terraform-based IaC that provisions secure, scalable Azure and Databricks resources across environments. Roles & Responsibilities: -• Build CI/CD for data ingestion
o Author and maintain Azure DevOps (ADO) YAML pipelines to version, validate, and deploy ADF pipelines and Databricks Workflows/Jobs.
o Implement parameterized, reusable pipeline templates (multi-stage, approvals, gates) for dev/test/prod with artifact promotion.
o Orchestrate batch and streaming patterns (e.g., ADF triggers + Databricks Jobs/Auto Loader) into Bronze ? Silver ? Gold layers.
o Integrate code quality, unit tests, and data checks (pytest, dbx/tests, Great Expectations or equivalent) in CI.
• Provision cloud & lakehouse with IaC
o Design Terraform modules and workspaces to provision Azure (RGs, VNets/PE, Key Vault, Storage, Event Hubs, Azure SQL, Monitor/Log Analytics) and Databricks (workspaces, UC metastore/catalogs/schemas, grants, clusters/cluster policies, Jobs, Repos/Asset Bundles).
o Manage Terraform state (remote backends), environment promotion, and drift detection; enforce Policy-as-Code (Azure Policy, Sentinel/OPA).
o Implement secure networking (VNet injection, Private Link, firewall rules), identity (AAD groups, service principals, workload identities), and secrets management (Key Vault ? Databricks secret scopes).
• Operate and improve the platform
o Set up observability for pipelines (Azure Monitor, Log Analytics, Databricks metrics & audit logs), SLOs, alerting, and incident runbooks.
o Optimize spend (autoscaling, spot/On-Demand mix, job cluster vs warehouse usage, caching, cluster policies).
o Champion Git strategies (trunk/PR), code reviews, and release tagging; document standards and how-to guides. Professional & Technical Skills: • 5+ years in DevOps/Platform Engineering supporting data platforms on Azure.
• Strong hands-on with:
o Azure DevOps: Repos, multi-stage YAML pipelines, variable groups, service connections.
o Azure Data Factory (ADF): authoring, parameterization, linked services, triggers, integration runtimes.
o Databricks: Workflows/Jobs, Repos/Asset Bundles, Delta Lake (Bronze/Silver/Gold), notebooks, job clusters & cluster policies; familiarity with Unity Catalog (catalogs/schemas/grants).
o Terraform (AzAPI/AzureRM + Databricks providers): modules, workspaces, remote state, pipelines.
o Azure networking & security: VNet/Private Endpoints, RBAC, Key Vault, MSI/service principals.
• Proficiency in Python and Bash/PowerShell for automation and CI tasks.
• Solid understanding of data ingestion patterns (batch/streaming, CDC/Auto Loader, file- and API-based sources).
This job post has been translated by AI and may contain minor differences or errors.
