Data Engineer - 60009025

Job Responsibilities

• Design, build, and test ELT\ETL pipelines responsible for data ingress, curating, and warehousing of data based on functional and non-functional requirements.
• Develop and maintain enterprise data cataloging processes to ensure datasets are properly documented, discoverable, and accessible, including managing metadata, data lineage, classifications, and integration with data governance tools.
• Develop and manage the deployment of data pipelines and applications through CI\CD pipelines and\or GitOps based automation.
• Work with architecture, product\service owners, and the development team to clarify work item functional and non-functional requirements including tasks needed to complete the work item so accurate effort estimates can be provided in backlog refinement meetings.
• Evaluate new and emerging technologies to determine which ones can help innovate and\or implement new feature requests.
• Participate in peer code reviews to ensure code quality and consistent use of patterns and styles.

Minimum and Additional Requirements

• A bachelor's degree in computer science, information technology, or a related technical field and 2 years of experience in cloud-based data platforms (e.g. Azure Synapse, AWS Redshift, Google BigQuery, Databricks, Snowflake, etc.

• Proven experience with data lakehouse architectures.
• Expertise in designing and optimizing ELT\ETL workflows in distributed computing environments.
• Proficient in writing modular, reusable, and testable code for data ingestion and transformation.
• Python and SQL. Familiarity with Java is a plus.
• Hands-on experience with Apache Spark.
• Understanding of batch and real-time data processing paradigms.
• Strong grasp of data modeling concepts (dimensional, normalized/denormalized, star/snowflake schemas) and data warehousing tools and concepts.
• Experience working in agile development environments.

Additional Comments

The Department of Administration is committed to providing equal employment opportunities to all applicants and does not discriminate on the basis of race, color, religion, sex (including pregnancy, childbirth, or related medical conditions including, but not limited, to lactation), national origin, age (40 or older), disability or genetic information.

Supplemental questions are considered part of your official application. Any misrepresentation will result in your disqualification from employment. Please complete the state application to include all current and previous work history and education. A resume will not be accepted nor reviewed to determine if an applicant has met the qualifications for the position.

• Health, dental, vision, long-term disability, and life insurance for employees, spouse, and children. Click here for additional information.
• 15 days annual (vacation) leave per year
• 15 days sick leave per year
• 13 paid holidays
• Paid Parental Leave
• S.C. Deferred Compensation Program available (S.C. Deferred Compensation)
• Retirement benefit choices *
  • State Retirement Plan (SCRS)
  • State Optional Retirement Program (State ORP)

• Enrollment in one of the listed plans is required for all FTE employees; please refer to the contribution section of hyperlinked retirement sites for the current contribution rate of gross pay.