Details

Description

Job summary

• Participate in the design of data ingestion and processing workflows in PySpark, based on defined business rules.
• Develop and encapsulate scripts using PySpark to implement data transformations and processing logic.
• Implement and manage Oozie pipelines to schedule and orchestrate processing steps efficiently.
• Use Git for version control and collaborative code management.
• Apply knowledge of CI/CD principles to support continuous delivery and integration within Big Data environments.
• Demonstrate understanding of Big Data concepts such as component encapsulation, queue management, Spark session handling, and CI/CD pipelines tailored for Big Data workflows.
• Prepare and execute unit tests (UT), including the creation of necessary data sets, and utilize ALM tools when applicable.
• Document work thoroughly through deliverables in Git, Confluence, and/or standard document formats like .doc and .xls.


Required competencies/experience:
•
• Python – Spark - PySpark
• Experience with development using ASP.NET 8, Blazor, and Syncfusion components.
• Familiarity with CI/CD platforms such as Azure DevOps and SonarQube.
• Working knowledge of containerization technologies, specifically Docker.
• Experience with testing frameworks like xUnit and Moq for unit and integration testing.
• Proficient in SQL data manipulation and querying.

Seniority of practice : experience at least 3,5 years

Requested competencies / experience:
• Big Data clusters secured with Kerberos authentication, based on HortonWorks distributions HDP 3.1.5 and HDF 3.5.1, currently migrating to Cloudera.
• Core technologies and tools include:
• Apache Spark with Python
• HDFS (Hadoop Distributed File System)
• Hive
• HBase
• Oozie workflow scheduler
• YARN resource management
• Apache Atlas for metadata management
• Jenkins for CI/CD
• Kerberos for security and authentication