1,039 Data Scientists jobs in South Africa

Big Data Data Engineer job vacancy in Johannesburg.

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform.

The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Big Data Data Engineer job vacancy in Johannesburg.

Key Responsibilities:

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage, and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

• 3+ years of experience in data engineering or ETL development within a big data environment.
• Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop.
• Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language).
• Experience working with unstructured JSON data and transforming it into structured formats.
• Solid understanding of data lake architectures: Raw, Published, and Functional layers.
• Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi.
• Experience with schema design, data modeling, and partitioning strategies.
• Comfortable with version control tools (e.g., Git) and CI/CD processes.

• Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation).
• Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc.
• Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment.
• Familiarity with data quality frameworks (e.g., Deequ, Great Expectations).

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.
• Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus.

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform. The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Key Responsibilities:

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage, and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

Technical Skills & Experience:

• 3+ years of experience in data engineering or ETL development within a big data environment.
• Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop.
• Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language).
• Experience working with unstructured JSON data and transforming it into structured formats.
• Solid understanding of data lake architectures: Raw, Published, and Functional layers.
• Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi.
• Experience with schema design, data modeling, and partitioning strategies.
• Comfortable with version control tools (e.g., Git) and CI/CD processes.

Nice to Have:

• Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation).
• Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc.
• Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment.
• Familiarity with data quality frameworks (e.g., Deequ, Great Expectations).

Qualifications:

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.
• Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus.

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

* If you have not received any feedback after 2 weeks, please consider you application as unsuccessful.

Contract

4 to 25 years

Negotiable

03 September 2025

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform. The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Key Responsibilities:

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage, and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

Technical Skills & Experience:

• 3+ years of experience in data engineering or ETL development within a big data environment.
• Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop.
• Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language).
• Experience working with unstructured JSON data and transforming it into structured formats.
• Solid understanding of data lake architectures: Raw, Published, and Functional layers.
• Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi.
• Experience with schema design, data modeling, and partitioning strategies.
• Comfortable with version control tools (e.g., Git) and CI/CD processes.

Nice to Have:

• Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation).
• Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc.
• Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment.
• Familiarity with data quality frameworks (e.g., Deequ, Great Expectations).

Qualifications:

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.

• Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus.

• In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

• If you have not received any feedback after 2 weeks, please consider you application as unsuccessful.

Big DataApache HadoopApache HivePySparkSQLJSONData Engineering

BankingFinancial Services

Job Purpose

We require a Big Data Specialist to assist with harmonising data from diverse sources, to analyse, problem solve, reconcile, develop solutions and build reporting in various tool sets, to enable Business understanding and decision making.

• Data Integration, harmonisation and reporting:
  • Collaborate with cross-functional teams to understand data requirements.
  • Design and implement efficient data pipelines using Abinitio and Denodo.
  • Extract, transform, analyze and or load data from various sources.
  • Leverage SAP connectors to seamlessly integrate SAP data.
• Data Consolidation and Harmonization:
  • Pull data from multiple sources (including SAP, legacy systems, APls, and external databases).
  • Develop strategies to ensure data consistency, accuracy, and reliability.
  • Create unified views of data for reporting reconciliation and analytics purposes.
• Performance Optimization:
  • Identify bottlenecks and optimize data processing workflows.
  • Monitor and fine-tune production jobs to ensure optimal performance and reconciliation of various data sources.
• Data Modeling and Architecture
  • Design and maintain data models that facilitate efficient querying and reporting.
  • Optimize data structures for scalability and responsiveness.
  • Enhance data flows and provide specifications for IT architecture builds.
• Governance
  • Develop governance frameworks for data flows.
  • Design appropriate controls to monitor master data and financial reconciliation.
• Collaboration and Documentation
  • Work closely with data engineers, data scientists, and business.
  • Document data integration processes, best practices, and troubleshooting guidelines for Business.

• Bachelor's degree in Computer Science, Information Systems, or a related
• Minimum of 3 years of experience in big data technologies, including Abinitio and
• Proficiency in SAP connectors and hands-on experience integrating SAP data.
• Strong understanding of data modeling, ETL processes, and data warehousing
• Familiarity with cloud-based big data platforms (e.g.,Azure, GCP) is
• Excellent problem-solving skills and ability to work independently
• Deep understanding of SAP on premise, S4Hana, BW4Hana, Denodo

• Governance, Risk and Controls
• Organisational behaviour theory
• Principles of project management
• Relevant regulatory knowledge
• Stakeholder management
• Strategic planning
• Talent management
• Business writing skills
• Management information and reporting principles, tools and mechanisms
• Client Service Management

• Building Partnerships
• Customer Focus
• Decision Making
• Facilitating Change
• Inspiring others
• Business Acumen
• Building Organizational Talent
• Compelling Communication

What you'll do:

• Platform Administration: Install, configure, upgrade, and monitor Cloudera/CDP clusters, manage HDFS/Ozone storage, and ensure security (Kerberos, Ranger, Sentry).
• Data Pipelines: Build and optimize ingestion and processing pipelines using NiFi and Informatica BDM/DEI, supporting both real-time and batch flows.
• ETL Integration: Develop Informatica mappings and workflows, leveraging pushdown execution to Hive/Impala/Spark; integrate diverse on-prem and cloud data sources.
• Performance Governance: Optimize queries, orchestrate jobs (Airflow, Oozie, Control-M), and ensure compliance with governance/security standards.

Your Expertise:

• Strong hands-on expertise in Cloudera tools: Hive, Impala, HDFS, Ozone, Hue, NiFi.
• Proficiency with Informatica BDM/DEI (ETL/ELT, pushdown optimization, data quality).
• Solid SQL, Linux administration, and scripting (Bash, Python).
• Familiarity with cloud data platforms (AWS, Azure, GCP) and orchestration tools.
• 4+ years in big data administration/engineering, including 2+ years in Informatica BDM/DEI.

Qualifications:

• Bachelorâ€s degree in Computer Science, Engineering, or related field.
• Experience in hybrid or cloud-based big data environments.

Soft Skills:

• Strong troubleshooting and problem-solving mindset.
• Ability to work independently and within cross-functional teams.
• Clear communication and documentation skills.

Other information applicable to the opportunity:

• Contract position
• Location: Johannesburg

Why work for us?

Want to work for an organization that solves complex real-world problems with innovative software solutions? At iOCO, we believe anything is possible with modern technology, software, and development expertise. We are continuously pushing the boundaries of innovative solutions across multiple industries using an array of technologies.†/p>

You will be part of a consultancy, working with some of the most knowledgeable minds in the industry on interesting solutions across different business domains.†/p>

Our culture of continuous learning will ensure that you will have all the opportunities, tools, and support to hone and grow your craft.†/p>

By joining IOCO you will have an open invitation to developer inspiring forums. A place where you will be able to connect and learn from and with your peers by sharing ideas, experiences, practices, and solutions.†/p>

iOCO is an equal opportunity employer with an obligation to achieve its own unique EE objectives in the context of Employment Equity targets. Therefore, our employment strategy gives primary preference to previously disadvantaged individuals or groups.

UNU-WIDER is seeking exceptional candidates for the position of Research Assistant, based in Pretoria, South Africa, to support the SA-TIED programme. This role involves managing and enhancing tax datasets, assisting researchers, and ensuring high standards of data confidentiality.

For the full job description and application details, please click here.

UNU offers three types of contracts: fixed-term staff positions (General Service, National Officer and Professional), Personnel Service Agreement positions (PSA), and consultant positions (CTC). For more information, see the Contract Types page.