Azure Databricks Engineer (PySpark) Job at Sira Consulting, an Inc 5000 company, Branchville, NJ

NE4vQnpEWGhwRmhoRmV6aEN1azdTR21CRGc9PQ==
  • Sira Consulting, an Inc 5000 company
  • Branchville, NJ

Job Description

Job Role: Azure Databricks Engineer (PySpark)

Location: Branchville, NJ

Job Type: Full-time

Job Summary:

We are looking for an experienced Azure Databricks Engineer with expertise in PySpark to design, develop, and optimize big data solutions on Azure cloud. The ideal candidate will work with large-scale data processing, ETL pipelines, and performance optimization using Databricks, Spark, and Azure services.

Key Responsibilities:

  • Design, develop, and implement data pipelines and ETL processes using Azure Databricks and PySpark .
  • Optimize Spark jobs for better performance and cost efficiency.
  • Work with structured, semi-structured, and unstructured data to create scalable data solutions.
  • Collaborate with Data Engineers, Data Scientists, and Business Analysts to meet business requirements.
  • Implement best practices for data security, governance, and compliance on Azure Cloud .
  • Work with Azure services such as Azure Data Factory (ADF), Azure Data Lake (ADLS), Azure Synapse, and Azure SQL .
  • Troubleshoot performance bottlenecks and optimize query execution plans .
  • Develop and maintain CI/CD pipelines for data pipeline automation.
  • Write clean, reusable, and efficient PySpark code following industry best practices.

Required Skills & Qualifications:

  • 10+ years of experience in Azure Databricks and PySpark .
  • Strong expertise in Apache Spark (RDDs, DataFrames, Datasets).
  • Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), and Azure Synapse Analytics .
  • Strong understanding of SQL and data modeling techniques.
  • Experience with version control tools (Git, GitHub, or Bitbucket) .
  • Hands-on experience with CI/CD pipelines using Azure DevOps .
  • Knowledge of Scala or Python for big data processing.
  • Strong analytical and problem-solving skills.
  • Experience working in Agile/Scrum environments .

Education:

  • Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field .

Job Tags

Full time,

Similar Jobs

PacifiCorp

Engineer (I,II) Klamath Falls, OR #113101 Job at PacifiCorp

 ...engineering staff on more complex projects. Develops engineering studies, plans,...  ...results. Guide and direct associate level engineers, technicians and drafters. ...  ...Intern, Design Engineer, Energy, Engineering, Entry Level PI7d776e6cecfb-30492-3657858... 

The H&K Group

Shop Mechanic 2nd Shift Job at The H&K Group

 ...environments Hours regularly exceed 40 in a week and 8 in a day H&K Group, Inc.s (H&Ks) Hazleton Division (formerly Hazleton Site Contractors) has been a proud member of our heavy civil construction team since 2002. Strategically located in Humboldt Industrial Park just... 

American Red Cross

Blood Component Manufacturing Technician Job at American Red Cross

 ...mission-based organization, we believe our team needs great support to do great work. Our comprehensive benefits help you in balancing home and work. With our resources and perks, you have amazing possibilities at the American Red Cross to advance your learning. ~... 

IN4IT

Cloud & DevOps Engineer Job at IN4IT

 ...IN4IT is growing and looking for a Cloud and DevOps engineer with a strong focus on containerization using Public Cloud Technologies and Kubernetes. The ideal candidate is a fast learner, a self-starter, and fearless in taking the initiative. Your profile: You... 

Career Systems Development Corporation

Heavy Truck Driving Instructor - Sign - On Bonus $1,000 Job at Career Systems Development Corporation

 ...instructions in a designated subject area. Duties/Responsibilities: Candidate must be able to be Certified to teach Commercial Truck Driving in the state of Maine (Company Sponsored) . Candidate must be able to be part of a cohesive team working together with two other...