2,585 Data Engineers jobs in South Africa

Data Engineer

The Data Engineer's role entails building and supporting data pipelines that must be scalable, repeatable, and secure. This role functions as a core member of an agile team, whereby these professionals are responsible for the infrastructure that provides insights from raw data, handling and integrating diverse sources of data seamlessly. They enable solutions by handling large volumes of data in batch and real-time by leveraging emerging technologies from both the big data and cloud spaces. Additional responsibilities include developing proof of concepts and implementing complex big data solutions with a focus on collecting, parsing, managing, analysing, and visualising large datasets. They know how to apply technologies to solve the problems of working with large volumes of data in diverse formats to deliver innovative solutions. Data Engineering is a technical job that requires substantial expertise in a broad range of software development and programming fields. These professionals have a knowledge of data analysis, end-user requirements, and business requirements analysis to develop a clear understanding of the business need and to incorporate these needs into a technical solution. They have a solid understanding of physical database design and the systems development lifecycle.

Responsibilities

• Architects' Data Analytics Framework
• Translates complex functional and technical requirements into detailed architecture, design, and high-performing software
• Leads Data and batch/real-time analytical solutions leveraging transformational technologies
• Works on multiple projects as a technical lead, driving user story analysis and elaboration, design and development of software applications, testing, and building automation tools
• Development and Operations
• Database Development and Operations
• Policies, Standards, and Procedures
• Business Continuity & Disaster Recovery
• Research and Evaluation
• Creating data feeds from on-premise to AWS Cloud
• Support data feeds in production on a break-fix basis
• Creating data marts using Talend or a similar ETL development tool
• Manipulating data using Python
• Processing data using the Hadoop paradigm, particularly using EMR, AWS's distribution of Hadoop
• Develop for Big Data and Business Intelligence, including automated testing and deployment

Requisite Experience, Education, Knowledge, and/ or Skills

• Bachelor's Degree in Computer Science, Computer Engineering, or equivalent
• AWS Certification
• Extensive knowledge in different programming or scripting languages
• Expert knowledge of data modelling and understanding of different data structures and their benefits and limitations under particular use cases
• Capability to architect highly scalable distributed systems, using different open source tools
• 5+ years of Data engineering or software engineering experience
• 2+ years of Big Data experience
• 2+ years' experience with Extract, Transform, and Load (ETL) processes
• 2+ years of AWS experience
• 5 years of demonstrated experience with object-oriented design, coding, and testing patterns, as well as experience in engineering (commercial or open source) software platforms and large-scale data infrastructures
• Big Data batch and streaming tools
• Talend
• AWS: EMR, EC2, S3
• Python
• PySpark or Spark

We are looking for an experienced Big Data Developer to join an international banking technology team in Málaga, Spain. In this role, you will contribute to the development of business applications within the Regulatory & Compliance domain, covering the full software lifecycle from problem analysis to deployment.

You'll work with modern big data technologies, collaborate with users to understand business needs, and provide innovative solutions that meet regulatory and compliance requirements. This is a fantastic opportunity to advance your career in a global environment while enjoying the lifestyle benefits of living in Spain.

• Participate in the end-to-end software lifecycle, including analysis, design, development, testing, and deployment.
• Collaborate with business users to identify requirements and deliver strategic technology solutions.
• Optimise and analyse code, applying best practices such as threat modelling and SAST.
• Manage tools and processes for documentation and Application Lifecycle Management (ALM).
• Plan and deliver projects using Agile methodology.
• Support incident resolution, including planned interventions.
• Execute unit, integration, and regression testing.
• Manage release processes and deployment tools.

Required:

• 3+ years of experience as a Big Data Developer.
• Bachelor's degree in Computer Science, Telecommunications, Mathematics, or a related field.
• Proficiency with GitHub.
• Strong knowledge of databases (Oracle PL/SQL, PostgreSQL).
• Experience with Java and JavaScript.
• Hands-on ETL experience.
• Fluency in English (Spanish is advantageous).

Preferred:

• Familiarity with microservices frameworks (Spring Boot), OpenShift.
• Knowledge of Flink, Drools, Kafka, DevOps tools.
• Agile methodology experience with tools such as Jira and Confluence.
• Exposure to S3, Elastic, and Angular.
• Experience in Transactional Regulatory Reporting.
• Innovative mindset and ability to generate strategic ideas.

Other Requirements:

• Availability to travel.
• Willingness to relocate to Málaga, Spain.

Contract

4 to 25 years

Negotiable

03 September 2025

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform. The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Key Responsibilities:

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage, and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

Technical Skills & Experience:

• 3+ years of experience in data engineering or ETL development within a big data environment.
• Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop.
• Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language).
• Experience working with unstructured JSON data and transforming it into structured formats.
• Solid understanding of data lake architectures: Raw, Published, and Functional layers.
• Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi.
• Experience with schema design, data modeling, and partitioning strategies.
• Comfortable with version control tools (e.g., Git) and CI/CD processes.

Nice to Have:

• Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation).
• Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc.
• Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment.
• Familiarity with data quality frameworks (e.g., Deequ, Great Expectations).

Qualifications:

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.

• Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus.

• In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

• If you have not received any feedback after 2 weeks, please consider you application as unsuccessful.

Big DataApache HadoopApache HivePySparkSQLJSONData Engineering

BankingFinancial Services

iOCO is seeking a skilled Big Data Administrator / Engineer with strong hands‑on experience in Cloudera’s ecosystem (Hive, Impala, HDFS, Ozone, Hue, NiFi) and proven expertise in Informatica BDM / DEI . The role involves administering and configuring big data platforms, deploying / supporting clusters, and building optimized pipelines to move and transform large‑scale datasets. Experience with alternate platforms such as Hortonworks, MapR, AWS EMR, Azure HDInsight, or Google Dataproc will be advantageous.

• Platform Administration : Install, configure, upgrade, and monitor Cloudera / CDP clusters, manage HDFS / Ozone storage, and ensure security (Kerberos, Ranger, Sentry).
• Data Pipelines : Build and optimize ingestion and processing pipelines using NiFi and Informatica BDM / DEI, supporting both real‑time and batch flows.
• ETL Integration : Develop Informatica mappings and workflows, leveraging pushdown execution to Hive / Impala / Spark; integrate diverse on‑prem and cloud data sources.
• Performance Governance : Optimize queries, orchestrate jobs (Airflow, Oozie, Control‑M), and ensure compliance with governance / security standards.

• Strong hands‑on expertise in Cloudera tools : Hive, Impala, HDFS, Ozone, Hue, NiFi.
• Proficiency with Informatica BDM / DEI (ETL / ELT, pushdown optimization, data quality).
• Solid SQL, Linux administration, and scripting (Bash, Python).
• Familiarity with cloud data platforms (AWS, Azure, GCP) and orchestration tools.
• 4+ years in big data administration / engineering, including 2+ years in Informatica BDM / DEI.

• Bachelor’s degree in Computer Science, Engineering, or related field.
• Experience in hybrid or cloud‑based big data environments.

• Strong troubleshooting and problem‑solving mindset.
• Ability to work independently and within cross‑functional teams.
• Clear communication and documentation skills.

• Contract position
• Location : Johannesburg

Want to work for an organization that solves complex real‑world problems with innovative software solutions? At iOCO, we believe anything is possible with modern technology, software, and development expertise. We are continuously pushing the boundaries of innovative solutions across multiple industries using an array of technologies. 

You will be part of a consultancy, working with some of the most knowledgeable minds in the industry on interesting solutions across different business domains. 

Our culture of continuous learning will ensure that you will have all the opportunities, tools, and support to hone and grow your craft. 

By joining IOCO you will have an open invitation to developer inspiring forums. A place where you will be able to connect and learn from and with your peers by sharing ideas, experiences, practices, and solutions. 

iOCO is an equal opportunity employer with an obligation to achieve its own unique EE objectives in the context of Employment Equity targets. Therefore, our employment strategy gives primary preference to previously disadvantaged individuals or groups.

What you'll do:

• Platform Administration: Install, configure, upgrade, and monitor Cloudera/CDP clusters, manage HDFS/Ozone storage, and ensure security (Kerberos, Ranger, Sentry).
• Data Pipelines: Build and optimize ingestion and processing pipelines using NiFi and Informatica BDM/DEI, supporting both real-time and batch flows.
• ETL Integration: Develop Informatica mappings and workflows, leveraging pushdown execution to Hive/Impala/Spark; integrate diverse on-prem and cloud data sources.
• Performance Governance: Optimize queries, orchestrate jobs (Airflow, Oozie, Control-M), and ensure compliance with governance/security standards.

Your Expertise:

• Strong hands-on expertise in Cloudera tools: Hive, Impala, HDFS, Ozone, Hue, NiFi.
• Proficiency with Informatica BDM/DEI (ETL/ELT, pushdown optimization, data quality).
• Solid SQL, Linux administration, and scripting (Bash, Python).
• Familiarity with cloud data platforms (AWS, Azure, GCP) and orchestration tools.
• 4+ years in big data administration/engineering, including 2+ years in Informatica BDM/DEI.

Qualifications:

• Bachelorâ€s degree in Computer Science, Engineering, or related field.
• Experience in hybrid or cloud-based big data environments.

Soft Skills:

• Strong troubleshooting and problem-solving mindset.
• Ability to work independently and within cross-functional teams.
• Clear communication and documentation skills.

Other information applicable to the opportunity:

• Contract position
• Location: Johannesburg

Why work for us?

Want to work for an organization that solves complex real-world problems with innovative software solutions? At iOCO, we believe anything is possible with modern technology, software, and development expertise. We are continuously pushing the boundaries of innovative solutions across multiple industries using an array of technologies.†/p>

You will be part of a consultancy, working with some of the most knowledgeable minds in the industry on interesting solutions across different business domains.†/p>

Our culture of continuous learning will ensure that you will have all the opportunities, tools, and support to hone and grow your craft.†/p>

By joining IOCO you will have an open invitation to developer inspiring forums. A place where you will be able to connect and learn from and with your peers by sharing ideas, experiences, practices, and solutions.†/p>

iOCO is an equal opportunity employer with an obligation to achieve its own unique EE objectives in the context of Employment Equity targets. Therefore, our employment strategy gives primary preference to previously disadvantaged individuals or groups.

UNU-WIDER is seeking exceptional candidates for the position of Research Assistant, based in Pretoria, South Africa, to support the SA-TIED programme. This role involves managing and enhancing tax datasets, assisting researchers, and ensuring high standards of data confidentiality.

For the full job description and application details, please click here.

UNU offers three types of contracts: fixed-term staff positions (General Service, National Officer and Professional), Personnel Service Agreement positions (PSA), and consultant positions (CTC). For more information, see the Contract Types page.

Global Influences

• Global push for data-driven decision-making in telecom, public services, and financial services
• Convergence of data transformation with AI, cloud, ESG-linked tech governance, and platform models
• Growing emphasis on deep, experience-led strategic thinking that replaces theoretical guidance with structured, practical steer from actual industry doers
• Rise of blueprint-based transformation in support of cost, agility, and monetisation mandates

At MTN we believe that understanding our people’s needs and aspirations is key to creating experiences that delight you at work, every day. We are committed to fostering an environment where every member of our Y’ello Family is heard, understood, and empowered to live an inspired life. Our values keep us grounded and moving in the right direction. Most importantly, they keep us honest. It is not something we claim to be. It is in our DNA.

As an organisation, we consider it our mission to create an exciting and rewarding place to work, where our people can be themselves, thrive in positivity and ignite their full potential. A workplace that boosts creativity and innovation, improves productivity, and ultimately drives meaningful results. A workplace that is built on relationships and achieving a purpose that is bigger than us. This is what we want you to experience with us!

Our commitments go beyond an organisational promise. It is in our leadership and managerial ethos to meaningfully partner with our employees, customers, and stakeholders with a vision to realise our shared goals.

Our values dubbed, LIVE Y’ello, are the cornerstone of our culture.

• Lead with Care
• Can-do with Integrity
• Collaborate with Agility
• Serve with Respect
• Act with Inclusion

• Lead the rollout of the Group DDO blueprint across all MTN markets, supported by localized implementation plans tailored to each Opco
• Operationalize blueprint principles, including data maturity advancement, monetization readiness, and AI adoption
• Monitor adherence to blueprint KPIs and proactively flag deviations in architecture or operating models
• Collaborate with Group DataCo, AI, and Platform teams to update the blueprint on a quarterly basis, incorporating feedback and lessons learned

• Drive all customer DDO engagements including maturity assessments, transformation strategy design, and control implementation and operationalization, and co-create with customer executives the near and long-term transformation roadmaps
• Facilitate roadmap signoffs at the ExCo level and embed KPIs into business planning and performance management cycles
• Advise on capability sequencing, monetization readiness (including data quality & governance, target operating model, capability readiness, systems, and platforms), and AI integration maturity to ensure strategic alignment and execution feasibility
• Support risk-adjusted prioritization across technology, data, governance, and product workstreams
• Collaborate with IT, Cloud, and Network leadership to provide an end-to-end diagnostic, solution design and execution plan for external and internal clients
• Build the DDO Transformation book of business and deliver on the financial targets for the service line

• Define target-state data and analytics capabilities across key domains including data science, engineering, architecture, and governance
• Support HR, L&D, and functional teams to drive competency mapping and organizational design aligned with transformation goals
• Build capacity through agile, squad-based delivery, federated CoEs, and cross-OpCo communities of practice

• Deliver structured coaching engagements for OpCo CEOs, CTIOs, CFOs, and CDOs on strategic data transformation
• Align local sponsorship models and steerco routines with Group-level transformation governance
• Support leadership in articulating AI and data strategy messaging for investors, regulators, and market stakeholders

• Establish and monitor data transformation value metrics aligned with ROIC, time-to-market acceleration, and EBITDA uplift
• Embed data transformation KPIs into Group and OpCo performance scorecards to ensure accountability and impact
• Track benefit realization and operational impact of blueprint execution through structured quarterly reviews

• Maintain a centralized repository of cross-OpCo case studies, implementation toolkits, and transformation playbooks
• Build and empower a world-class centralised internal consulting team with a mix of full-time and contractor resources to deliver and scale the DDO consulting business line
• Represent MTN in internal and external data transformation forums and executive sessions

• Master’s degree in business strategy, Information Systems, or Digital Transformation related field
• Executive certification in Operating Model Transformation, Agile Scaling, or AI Strategy preferred

• Atleast 10+ years in top-tier management consulting, technical leadership, or enterprise-wide transformation roles, with strong focus on digital transformation, digital business building, and data-driven decision making
• Proven success leading complex operating model change across multi-country organisations and engaging successfully at senior leadership levels to drive clear action and serving as a leadership advisor
• Demonstrated ability to build and leverage delivery partnerships to scale customer impact. P&L ownership and deal-making experience is highly preferred
• Very Strong knowledge of digital operating models, data structures and governance end to end, data quality, analytics platforms and methodologies, capability maturity assessment methods, and a strong sense of cross-industry data monetisation strategy, opportunities, and best practices