At Duke Health, we're driven by a commitment to compassionate care that changes the lives of patients, their loved ones, and the greater community. No matter where your talents lie, join us and discover how we can advance health together.
Primary Role
The Data Engineer will be responsible for designing, developing, and optimizing data pipelines, manage servers, and integrate APIs. They will collaborate closely with data scientists and analysts to meet data requirements and ensure robust data management solutions.
Work Performed
Level 1
Individual contributor works under the direct supervision of level 3 or above and collaborates as a member of a team
Basic understanding of local databases (Clarity; Caboodle) and underlying tables pertaining to role/domain
Basic understanding of ETL technologies and scripting languages and terminology related to role
Basic understanding of cloud data engineering technologies, including storage accounts, data pipelines, data lakes, structured vs. unstructured data, etc.
Ability to design, construct and test data management systems under guidance of senior team members
Assists in server administration tasks, ensuring system performance and availability
Basic knowledge of SQL and relational databases
Follows all policies and procedures set forth by DUHS/DHTS/ACE as pertains to role
Responds to customer requests for assistance as assigned and prioritized by manager
Participates in code review and learning best practices in data engineering
Maintains ongoing communications (timely, clear, concise, and accurate) with manager keeping them apprised of request status
Maintains appropriate project documentation as assigned such as GITLAB or data dictionary
Level 2
Meets all Competencies in Level 1 AND:
Works with new hires/Level I team members to train on use of tools and technologies to accomplish job duties?
Demonstrates advanced knowledge of the assigned domain/role. Understands data, workflows, and tools related to role.
Demonstrates advanced knowledge of, and experience with, cloud data engineering concepts, including (but not limited to) Microsoft Azure Lakehouses and Delta Tables, Azure Data Factory, Azure Synapse, and medallion-style data architecture.
Demonstrates ability to work with stakeholders to understand business needs and perform business analysis independently >75% of the time.
Develop technical requirements from business requirements independently >75% of the time.
Can write/modify accurate SQL queries independently > 90% of the time
Ability to optimize SQL queries, stored procedures and scripts for performance reliability
Can write complex SQL inclusion/exclusion criteria for cohort development
Manages and optimizes server infrastructure for performance and reliability
Troubleshoot and resolve complex data pipeline, server and API issues
Ability to locate data elements independently > 75% of the time
Knowledge/utilization of basic data design and infrastructure
Participates in team code review
Actively participates/contributes to database QA/submit requests for new database tables/attributes
Actively seeks out?participation?in Special Projects and is able to effectively accomplish project assignments with minimal supervision
Ability to manage multiple projects to completion in a timely manor
Ability to develop and present topics as assigned/needed by customer and/or ACE team
Active participation in Duke Analytic Community or other networking forums applicable to job functions
Level 3
Meets all Competencies in Level 2 AND:
Demonstrates?expertise in?systems and software related to project area to accomplish job duties (e.g. SSIS, Azure Data Factory, etc..) ?
Resolves escalated issues/incidents to ensure high availability and performance of ETL processes
Architects and designs robust data solutions
Expert knowledge and experience with Microsoft Azure Data Engineering tools, concepts, and industry best practices.
Expert locating data elements as required. Assists team in locating data elements as needed.
Works with team members to train on advanced skills in using tools and technologies to accomplish job duties
Leads the development of complex data pipelines and API integration, ensuring scalability and efficiency
Defines standards and best practices for data pipeline design, development and maintenance
Demonstrates ability to work independently to solve problems and complete assigned work >95% of the time
Demonstrates ability to work with customers to understand business need or problem to address independently >95% of the time
Leads Projects with minimal supervision
Change agent with regard to team processes, workflows when issues/gaps are found
Collaborates with data architect on design and implementation of data solutions (lakehouse, data warehouse, etc.)
Serves as moderator, leader or channel owner for a forum such as Duke Analytics Community or other professional networking organization
Required Qualifications at this Level
Education/Training
Bachelor's degree in a related field, or four years of equivalent technical experience required.
LICENSURE/CERTIFICATION:
None required. Preferred certification in development technologies, as appropriate for the position.
Experience
Level 1 - No experience required beyond the minimum education (or equivalency) requirement.
Level 2 - Two years of directly related experience is required.
Level 3 - Four years of directly related experience is required.
Duke is an Affirmative Action/Equal Opportunity Employer committed to providing employment opportunity without regard to an individual's age, color, disability, gender, gender expression, gender identity, genetic information, national origin, race, religion, sex, sexual orientation, or veteran status.
Duke aspires to create a community built on collaboration, innovation, creativity, and belonging. Our collective success depends on the robust exchange of ideas-an exchange that is best when the rich diversity of our perspectives, backgrounds, and experiences flourishes. To achieve this exchange, it is essential that all members of the community feel secure and welcome, that the contributions of all individuals are respected, and that all voices are heard. All members of our community have a responsibility to uphold these values.
Essential Physical Job Functions: Certain jobs at Duke University and Duke University Health System may include essentialjob functions that require specific physical and/or mental abilities. Additional information and provision for requests for reasonable accommodation will be provided by each hiring department.
As a world-class academic and health care system, Duke Health strives to transform medicine and health locally and globally through innovative scientific research, rapid translation of breakthrough discoveries, educating future clinical and scientific leaders, advocating and practicing evidence-based medicine to improve community health, and leading efforts to eliminate health inequalities.
