Job Description

Job Title - DataBricks Data Engineer

Location - Pittsburg, PA

**Must have 8+ years of real hands on experience**

We are specifically seeking a Data Engineer-Lead with strong expertise in Databricks development.

The role involves:

Building and testing data pipelines using Python/Scala on Databricks
Hands on experience to develop and lead the offshore team to perform development/testing work in Azure data bricks
Architect data platforms using Azure services such as Azure Data Factory (ADF), Azure Databricks (ADB), Azure SQL Database, and PySpark.
Collaborate with stakeholders to understand business needs and translate them into technical solutions.
Provide technical leadership and guidance to the data engineering team and need to perform development.
Familiar with Safe Agile concepts and good to have working experience in agile model.
Develop and maintain data pipelines for efficient data movement and transformation.
Onsite and offshore team communication and co-ordination.
Create and update the documentation to facilitate cross-training and troubleshooting
Hands on experience in scheduling tools like BMC control-M and setup jobs and test the schedules.
Understand the data models and schemas to support the development work and help in creation of tables in databricks
Proficiency in Azure Data Factory (ADF), Azure Databricks (ADB), SQL, NoSQL, PySpark, Power BI and other Azure data tools.
Implementing automated data validation frameworks such as Great Expectations or Deequ
Reconciling large-scale datasets
Ensuring data reliability across both batch and streaming processes

The ideal candidate will have hands-on experience with:

PySpark, Scala, Delta Lake, and Unity Catalog
Devops CI/CD automation
Cloud-native data services
Azure databricks/Oracle
BMC Control-M

Location: Pittsburgh, PA

Job Tags

Part time, Work experience placement,

Similar Jobs

Cyntergy

Program Coordinator Job at Cyntergy

...Coordinator Were looking for self-starters and eager learners. Beyond that, youll need Working knowledge of computers and MS Office. Working knowledge of Deltek Vision, Newforma, Smartsheet, and SharePoint is a plus. Ability to handle multiple assignments...

Phoenix Loss control

Field Claims Investigator Job at Phoenix Loss control

Job Description Job Description Job Type: Contract Workplace Type: Hybrid (50% remote, 50% fieldwork) Compensation: $20/hrplus$.50/mi COMPANY OVERVIEW Phoenix Loss Control (PLC) is aUS-based business servicesproviderin the cable, telecom, and utilities...

Loram Maintenance of Way

Technical Instructional Designer Job at Loram Maintenance of Way

...Job Description Job Title: Technical Instructional Designer \n FLSA Status: Exempt \n Department: Human Resources \n Reports... ...development, and learning program production while providing expert consultation on program measurement, visualization, and multi-modal...

SAVI EHS

Environment, Health and Safety Manager Job at SAVI EHS

...Job Description Environmental, Health and Safety Manager SAVI EHS Charleston, South Carolina (On-site) Compensation $45.00 $50.00 per hour $800 weekly per diem Employment Type Full-Time Job Description EHS Manager SAVI EHS is seeking an...

Multipurpose Arena Fort Worth

Roving Event Security (Part-Time) Job at Multipurpose Arena Fort Worth

Security: Roving Event Security (Part-Time)Trail Drive Management Corporation Dickies Arena of Fort Worth, TX Dickies Arena is a 14,000 seat, spectacular multipurpose venue located adjacent to the Will Rogers Memorial Center campus in Fort Worth, Texas. The arena...

Azure data engineer (Pittsburgh) Job at Cognizant, Pittsburgh, PA

bjhia3I0akpVT1NQVDhmdWVLd2JZVmpUeUE9PQ==