226 Big Data Hadoop jobs in South Africa

Contract

4 to 25 years

Negotiable

03 September 2025

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform. The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Key Responsibilities:

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage, and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

Technical Skills & Experience:

• 3+ years of experience in data engineering or ETL development within a big data environment.
• Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop.
• Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language).
• Experience working with unstructured JSON data and transforming it into structured formats.
• Solid understanding of data lake architectures: Raw, Published, and Functional layers.
• Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi.
• Experience with schema design, data modeling, and partitioning strategies.
• Comfortable with version control tools (e.g., Git) and CI/CD processes.

Nice to Have:

• Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation).
• Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc.
• Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment.
• Familiarity with data quality frameworks (e.g., Deequ, Great Expectations).

Qualifications:

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.

• Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus.

• In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

• If you have not received any feedback after 2 weeks, please consider you application as unsuccessful.

Big DataApache HadoopApache HivePySparkSQLJSONData Engineering

BankingFinancial Services

• Collaborate on strategy: Work with HR and management teams to develop strategies for workforce planning.
• Evaluate HR tools and systems: Assess the performance of HR software and suggest optimizations

The Responsibilities Of The Role Include

• MIS & Strategic Reporting: Deliver actionable HR insights through data analysis and strategic reporting.
• Power BI Dashboard Creation: Design and maintain dynamic Power BI dashboards to visualize key HR metrics.
• Management Reporting: Design, compile and present regular HR reports to support leadership decision-making.
• Collect and analyze HR data: Evaluate metrics like turnover rates, employee satisfaction, and absenteeism.
• Generate reports and dashboards: Present insights to HR teams and leadership to inform decision-making.
• Monitor workforce trends: Identify patterns in hiring, productivity, or retention to recommend improvements.
• Support compliance efforts: Ensure HR practices align with employment laws and company policies.

Job Description: The successful candidates will be responsible for handling payroll queries, data collation, HR data collation, and report building. They will also be involved in creating and automating reports on company performance and key performance indicators (KPIs). Proficiency in SharePoint, Excel, Power Query, Power Pivot, Power BI, Word, PowerPoint, and Office 365 is essential.

Required Skills:

• Technical Skills:
• Proficiency in Microsoft Excel, including advanced functions, Power Query, and Power Pivot.
• Experience with Power BI for data visualization and reporting.
• Knowledge of SharePoint for data management and collaboration.
• Familiarity with Office 365 applications (Word, PowerPoint, etc.).
• Analytical Skills:
• Strong analytical and problem-solving abilities.
• Ability to interpret complex data and provide actionable insights.
• Communication Skills:
• Excellent written and verbal communication skills.
• Ability to present data and reports clearly and effectively.
• Organizational Skills:
• Strong attention to detail and accuracy.
• Ability to manage multiple tasks and meet deadlines.
• Teamwork:
• Ability to work collaboratively in a team environment.
• Strong interpersonal skills and the ability to build relationships with stakeholders.

Responsibilities: Key Responsibilities:

• Payroll Queries and Data Collation:
• Manage and resolve payroll-related queries.
• Collect, organize, and analyze payroll data.
• HR Data Collation and Reporting:
• Gather and compile HR data from various sources.
• Generate comprehensive HR reports for management review.
• KPI Reporting and Automation:
• Develop and maintain reports on company performance and KPIs.
• Automate reporting processes to improve efficiency and accuracy.
• Data Management and Analysis:
• Ensure data integrity and accuracy.
• Perform data analysis to support decision-making processes.
• Collaboration and Communication:
• Work closely with other departments to gather necessary data.
• Present findings and reports to management and stakeholders.

Qualifications: Preferred Qualifications:

• Previous experience in a similar role within an MIS 12 months plus.
• Certification in data analysis or related fields.
• Degree or certification in statistical analysis field.

In alignment with the Employment Equity Act, preference will be given to applicants from historically underrepresented groups/ aligned with our EE targets

• Job Description:
  • Collaborate on strategy: Work with HR and management teams to develop strategies for workforce planning.
• Evaluate HR tools and systems: Assess the performance of HR software and suggest optimizations

The responsibilities of the role include:

• MIS & Strategic Reporting: Deliver actionable HR insights through data analysis and strategic reporting.
• Power BI Dashboard Creation: Design and maintain dynamic Power BI dashboards to visualize key HR metrics.
• Management Reporting: Design, compile and present regular HR reports to support leadership decision-making.
• Collect and analyze HR data: Evaluate metrics like turnover rates, employee satisfaction, and absenteeism.
• Generate reports and dashboards: Present insights to HR teams and leadership to inform decision-making.
• Monitor workforce trends: Identify patterns in hiring, productivity, or retention to recommend improvements.
• Support compliance efforts: Ensure HR practices align with employment laws and company policies.

Responsibilities: MIS / Strategic Reporting:

• Develop and manage accurate and timely MIS reports for the Human Resources Department, providing insights into key metrics such as attrition, Performance Management, early Warning System insights etc.
• Design, develop, and maintain HR dashboards and reports to support strategic decision-making.
• Analyze HR data to identify trends, patterns, and insights that inform workforce planning and organizational development.
• Collaborate with HR leadership to define KPIs and reporting requirements. Collaborate with senior leadership to prepare monthly, quarterly and annual strategic reports.
• Collaborate with colleagues in the Global HRIS team to align efforts and approach in reporting.

Power BI Dashboard Creation:

• Build and manage interactive Power BI dashboards and visual reports for various HR metrics (e.g., headcount, attrition, diversity, recruitment funnel) to provide stakeholders with easy access to key HR metrics and trends.
• Ensure data accuracy, consistency, and real-time reporting capabilities.
• Train Leaders and HR team members on dashboard usage and interpretations.
• Customize dashboards based on department and leadership needs, ensuring actionable insights are at hand.
• Continuously improve dashboard usability, accessibility, and functionality, providing training as necessary to end-users.

Management Reporting:

• Prepare monthly and quarterly HR performance reports for management, ensuring all KPIs and targets are tracked.
• Work with various teams to gather and compile data, ensuring consistency and accuracy in all reporting.
• Highlight key issues and provide recommendations for operational improvements in HR performance.
• Consolidate data from multiple sources (HRIS, ATS, payroll systems) into comprehensive reports.
• Support audit and compliance reporting requirements

Qualifications: Certifications : Power BI or other data-related certifications are a plus.

Bachelors Degree with 8+ years of Experience

o Minimum of 3 years of experience in an MIS, data analysis, or HR reporting role.

o Strong experience in creating and managing Power BI dashboards and reports.

o Experience in preparing management reports and strategic insights for senior leadership.

o Strong understanding of HR processes and metrics.

• Technical Skills:

o Advanced proficiency in Microsoft Excel (pivot tables, VLOOKUP, macros, etc.) and Power BI.

o Strong understanding of database management, data analysis, and reporting methodologies.

o Experience with HR platforms and managing workflows.

The successful candidates will be responsible for handling payroll queries, data collation, HR data collation, and report building. They will also be involved in creating and automating reports on company performance and key performance indicators (KPIs). Proficiency in SharePoint, Excel, Power Query, Power Pivot, Power BI, Word, PowerPoint, and Office 365 is essential.

Required Skills

• Technical Skills:- Proficiency in Microsoft Excel, including advanced functions, Power Query, and Power Pivot.
• Experience with Power BI for data visualization and reporting.
• Knowledge of SharePoint for data management and collaboration.
• Familiarity with Office 365 applications (Word, PowerPoint, etc.).
• Analytical Skills:- Strong analytical and problem-solving abilities.
• Ability to interpret complex data and provide actionable insights.
• Communication Skills:- Excellent written and verbal communication skills.
• Ability to present data and reports clearly and effectively.
• Organizational Skills:- Strong attention to detail and accuracy.
• Ability to manage multiple tasks and meet deadlines.
• Teamwork:- Ability to work collaboratively in a team environment.
• Strong interpersonal skills and the ability to build relationships with stakeholders.

Exact location – Rosebank, Firestation, walking distance to Gautrain Station

Responsibilities will include, but not be limited to the following:

• Develop and optimize data pipelines for collecting, processing, and storing large volumes of production and operational data.
• Play a pivotal role in documenting business processes, procedures, and policies for transparency and consistency.
• Design and implement scalable data architecture solutions tailored to production/operation environments.
• Collaborate with clients, project managers, engineers, and operators to understand data requirements.
• Ensure data integrity, security, and compliance with project requirements and outcomes.
• Monitor and troubleshoot data flow issues and optimize data processing workflows.
• Support the integration of IoT devices, sensors, and industrial systems with data platforms.
• Generate reports and dashboards to visualize real-time and historical data insights.
• Stay updated with the latest industrial data engineering tools and technologies.
• Travel to operational sites when required to ensure hardware and data capturing is functional.

Qualifications and experience:

• Bachelor's or master's degree in Computer Science, Data Engineering, Industrial Engineering, or related field.
• Proven experience in data engineering, especially in industrial or equipment operations settings.
• Strong proficiency in programming languages.
• Experience with cloud platforms and data tools (AVEVA, Power-BI)
• An understanding of the power platform, its functionalities, and its integration potential will be an advantage.
• Knowledge of IoT protocols and industrial automation systems.
• Familiarity with data modelling, ETL processes, and database management.
• Excellent problem-solving and communication skills.
• Attention to detail and a commitment to accuracy.

Ready to architect the future of data on Google Cloud Platform?

Join Lumina Africa (PTY) LTD and lead innovative data solutions using cutting-edge GCP technologies. We're seeking a creative Data Engineer who thinks beyond traditional approaches and brings fresh perspectives to cloud-native data architectures.

What Makes This Role Exceptional:

GCP Innovation Hub
- Work exclusively with Google Cloud's latest data and AI services

Global Tech Group
- Part of Lumina Tech Group with operations across Dubai, London, and South Africa

Cloud-First Culture
- Build scalable, serverless data solutions from day one

Rapid Growth Environment
- Shape our expanding South African data practice

Core GCP Technologies You'll Master:

• BigQuery
• Design and optimize large-scale data warehouses
• Cloud Dataflow
• Build real-time and batch processing pipelines
• Pub/Sub
• Implement event-driven data architectures
• Cloud Composer (Airflow)
• Orchestrate complex data workflows
• Dataproc
• Manage Spark and Hadoop workloads
• Cloud Storage
• Architect data lake solutions
• Vertex AI
• Integrate ML pipelines with data engineering workflows

What You'll Architect:

• Design cloud-native data pipelines using GCP services
• Build real-time streaming solutions with Pub/Sub and Dataflow
• Optimize BigQuery performance for petabyte-scale analytics
• Implement Infrastructure as Code using Terraform and Cloud Deployment Manager
• Create innovative data solutions that challenge conventional approaches
• Mentor teams on GCP best practices and modern data patterns

Essential GCP Expertise:

• 3+ years
  hands-on experience with Google Cloud Platform
• BigQuery mastery
• complex SQL, partitioning, clustering, optimization
• Cloud Dataflow
• Apache Beam, streaming and batch processing
• Python/Java
• Strong programming skills for data pipeline development
• Terraform/Cloud Deployment Manager
• Infrastructure as Code
• Pub/Sub & Cloud Functions
• Event-driven architectures
• GCP Certifications
  preferred (Professional Data Engineer, Cloud Architect)

Bonus Skills:

• Experience with
  dbt
  for data transformation
• Kubernetes
  and
  Cloud Run
  for containerized workloads
• Looker
  or
  Data Studio
  for visualization
• Apache Spark
  on Dataproc
• Cloud Security
  and
  IAM
  best practices

Why Choose Lumina Africa:

GCP-Focused Career Path
- Specialize in Google Cloud's data ecosystem

Competitive Package
- Market-leading salary + GCP certification support

Hybrid Working
- Modern offices with flexible arrangements

Innovation Budget
- Resources for experimenting with new GCP services

International Projects
- Collaborate across UAE, UK, and African markets

Fast-Track Growth
- Lead data initiatives in our expanding practice

Ready to Build the Future on GCP?

If you're passionate about Google Cloud Platform and ready to architect innovative data solutions that drive South Africa's digital transformation, we want to hear from you.

We are seeking an experienced
Data Engineer
with strong expertise in
Google Cloud Platform
to join a fast-growing, innovative organisation. This role offers the chance to design, build, and optimise scalable data pipelines and architectures that support impactful decision-making across the business.

If you are analytically sharp, self-motivated, and enjoy working in dynamic environments, this could be the perfect opportunity. A passion for African business, curiosity, and a sense of humour will help you thrive in our energetic and forward-thinking culture.

Key Responsibilities

• Design and develop scalable data pipelines and architectures using
  Google Cloud Platform technologies
  (BigQuery, Dataflow, Pub/Sub, Cloud Storage).
• Build and manage ETL processes to transform diverse data sources into structured, reliable formats.
• Collaborate with data scientists and analysts to deliver solutions that enable insights and smarter decisions.
• Maintain documentation for pipelines, data models, and architecture to ensure clarity and consistency.
• Troubleshoot and resolve data issues while safeguarding quality and integrity.
• Optimise data workflows for performance, scalability, and cost efficiency.
• Automate data-related processes to streamline operations.
• Stay ahead of industry trends and adopt best practices in Google Cloud Platform and data engineering.

Requirements

• Bachelor's or Master's degree in Computer Science, Data Science, or related field.
• 5+ years of experience
  as a Data Engineer or in a similar role.
• Strong programming skills in
  BigQuery, Python, SQL, and Google Cloud Platform
  .
• Proven experience with
  ETL development and data modeling
  .
• Familiarity with
  data lakehouse concepts
  and techniques.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.
• Hands-on experience with
  Google Cloud Platform technologies
  (BigQuery, Dataflow, Pub/Sub, Cloud Storage).
• Experience in financial services would be an advantage.

Apply

Danielle Paxton

Senior Specialist Recruitment Consultant

Do you like building data systems and pipelines? Do you enjoy interpreting trends and patterns? Are you able to recognize the
deeper meaning of data
?

Join Elixirr Digital as a
Data Engineer
and help us analyze and organize raw data to provide valuable business insights to our clients and stakeholders

As
Data Engineer
, you will be responsible to ensure the availability and quality of the data so that it becomes usable by target data users. You will be working on a set of operations aimed at creating processes and mechanisms for the flow and access of data in accordance with the project scope and deadlines

Discover the opportunity to join our
Data & Analytics
department and work closely with a group of like-minded individuals with cutting-edge technologies

What you will be doing as a Data Engineer at Elixirr Digital?

• Working closely with Data Architects on AWS, Azure, or IBM architecture designs.
• Maintaining and building data ecosystems by working on the implementation of data ingestions, often in collaboration with other data engineers, analysts, DevOps, and data scientists.
• Ensuring the security of cloud infrastructure and processes by implementing best practices.
• Applying modern principles and methodologies to advance business initiatives and capabilities.
• Identifying and consulting on ways to improve data processing, reliability, efficiency, and quality, as well as solution cost and performance.
• Preparing test cases and strategies for unit testing, system, and integration testing.

Competencies and skillset we expect you to have to successfully perform your job:

• Proficient in Python with extensive experience in data processing and analysis.
• Strong SQL expertise, adept at writing efficient queries and optimizing database performance
• Previous working experience with the Azure/AWS data stack.
• Experienced in software development lifecycle methodologies, with a focus on Agile practices.

We Could Be a Perfect Fit If You Are:

• Passionate about technology. You anticipate, recognize, and resolve technical problems using a variety of specialized tools for application development and support.
• Independent. You are a self-motivated and ambitious individual, capable of managing multiple responsibilities effectively.
• Problem-solver. You think creatively and find solutions to complex challenges.
• Creative and outside-the-box thinker. You look beyond blog posts and whitepapers, competitions, and even state-of-the-art benchmarks to solve real-world problems.
• Communicator. Strong verbal and written communication skills are essential to ensure effective collaboration and timely delivery of results within the team.
• Proficient in English. We work across continents in a global environment, so fluent English, both written and spoken is a must.

Why is Elixirr Digital the right next step for you?

From working with cutting-edge technologies to solving complex challenges for global clients, we make sure your work matters. And while you're building great things, we're here to support you.

Compensation & Equity:

• Performance bonus
• Employee Stock Options Grant
• Employee Share Purchase Plan (ESPP)
• Competitive compensation

Health & Wellbeing:

• Health benefits plan
• Flexible working hours
• Pension plan

Projects & Tools:

• Modern equipment
• Big clients and interesting projects
• Cutting-edge technologies

Learning & Growth:

• Growth and development opportunities
• Internal LMS & knowledge hubs

We don't just offer a job - we create space for you to grow, thrive, and be recognized.

Intrigued? Apply now