Skills highlighted in blue are preferred key skills
1. Design and develop scalable batch, streaming, and near-real-time data pipelines for enterprise analytics workloads.
2. Build robust ingestion and transformation workflows using the required data engineering platforms, programming languages, and cloud services.
3. Implement data validation, reconciliation, error handling, monitoring, and recovery mechanisms across pipelines.
4. Optimize processing jobs, storage patterns, SQL queries, and distributed workloads for performance and cost efficiency.
5. Work with data models, source systems, warehouses, lakehouses, and downstream analytical consumers to ensure reliable data delivery.
6. Collaborate with analysts, data scientists, architects, and business teams to understand requirements and define technical solutions.
7. Write technical documentation covering mappings, pipeline logic, dependencies, operational procedures, and data quality controls.
8. Participate in code reviews, testing, production support, incident resolution, and continuous improvement of the data platform.
Full-time opportunity for a Python PySpark Data Engineer with 7–10 years of experience, focused on PySpark, ETL, ELT.
Freelancer