Python Data Engineer

Requirements

• Strong proficiency in Python (pandas, PySpark, or similar frameworks).
• Solid knowledge of SQL (query optimization, joins, indexing, performance tuning).
• Experience with ETL frameworks/workflow orchestration (Airflow, or custom pipelines).
• Hands-on with cloud platforms (AWS S3, Redshift, Glue, GCP BigQuery, Azure Data Factory).
• Knowledge of data warehousing concepts and schema design (Star/Snowflake).
• Familiarity with file formats such as JSON, Parquet, Avro.
• Exposure to big data ecosystems (Hadoop, Spark, Hive, Kafka) is a plus.
• Experience with containerization and orchestration (Docker, Kubernetes).
• Familiarity with CI/CD practicesfor data pipelines (GitHub Actions, Jenkins, etc.).
• Exposure to real-time/streaming platforms (Kafka, Kinesis, Pub/Sub).

Roles and Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL workflows using Python and SQL.
• Work with structured, semi-structured, and unstructured data to ensure efficient ingestion, transformation, and storage.
• Optimize data processes for performance, scalability, and cost-efficiency.
• Manage and monitor data workflows to ensure accuracy, consistency, and reliability.
• Implement data quality checks, validation, and error-handling mechanisms in pipelines.
• Integrate data from multiple sources (databases, APIs, cloud storage, streaming platforms).
• Work with cloud environments (AWS/Azure/GCP) for data storage, processing, and orchestration.
• Maintain clear documentation of data workflows, schemas, and business logic.
• Support and troubleshoot production issues in data pipelines and recommend improvements.

Nice to Have

• Awareness of data security and governance (IAM roles, encryption).
• Knowledge of monitoring/logging tools (CloudWatch, Grafana, ELK stack).
• Awareness of data security and governance (IAM roles, encryption).