2,222 Kafka Developer jobs in South Africa

We are looking for an experienced Big Data Developer to join an international banking technology team in Málaga, Spain. In this role, you will contribute to the development of business applications within the Regulatory & Compliance domain, covering the full software lifecycle from problem analysis to deployment.

You'll work with modern big data technologies, collaborate with users to understand business needs, and provide innovative solutions that meet regulatory and compliance requirements. This is a fantastic opportunity to advance your career in a global environment while enjoying the lifestyle benefits of living in Spain.

• Participate in the end-to-end software lifecycle, including analysis, design, development, testing, and deployment.
• Collaborate with business users to identify requirements and deliver strategic technology solutions.
• Optimise and analyse code, applying best practices such as threat modelling and SAST.
• Manage tools and processes for documentation and Application Lifecycle Management (ALM).
• Plan and deliver projects using Agile methodology.
• Support incident resolution, including planned interventions.
• Execute unit, integration, and regression testing.
• Manage release processes and deployment tools.

Required:

• 3+ years of experience as a Big Data Developer.
• Bachelor's degree in Computer Science, Telecommunications, Mathematics, or a related field.
• Proficiency with GitHub.
• Strong knowledge of databases (Oracle PL/SQL, PostgreSQL).
• Experience with Java and JavaScript.
• Hands-on ETL experience.
• Fluency in English (Spanish is advantageous).

Preferred:

• Familiarity with microservices frameworks (Spring Boot), OpenShift.
• Knowledge of Flink, Drools, Kafka, DevOps tools.
• Agile methodology experience with tools such as Jira and Confluence.
• Exposure to S3, Elastic, and Angular.
• Experience in Transactional Regulatory Reporting.
• Innovative mindset and ability to generate strategic ideas.

Other Requirements:

• Availability to travel.
• Willingness to relocate to Málaga, Spain.

Big Data Data Engineer job vacancy in Johannesburg.

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform.

The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Big Data Data Engineer job vacancy in Johannesburg.

Key Responsibilities:

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage, and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

• 3+ years of experience in data engineering or ETL development within a big data environment.
• Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop.
• Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language).
• Experience working with unstructured JSON data and transforming it into structured formats.
• Solid understanding of data lake architectures: Raw, Published, and Functional layers.
• Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi.
• Experience with schema design, data modeling, and partitioning strategies.
• Comfortable with version control tools (e.g., Git) and CI/CD processes.

• Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation).
• Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc.
• Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment.
• Familiarity with data quality frameworks (e.g., Deequ, Great Expectations).

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field.
• Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus.

Contract

4 to 25 years

Negotiable

03 September 2025

We are seeking a skilled Data Engineer to design and develop scalable data pipelines that ingest raw, unstructured JSON data from source systems and transform it into clean, structured datasets within our Hadoop-based data platform. The ideal candidate will play a critical role in enabling data availability, quality, and usability by engineering the movement of data from the Raw Layer to the Published and Functional Layers.

Key Responsibilities:

• Design, build, and maintain robust data pipelines to ingest raw JSON data from source systems into the Hadoop Distributed File System (HDFS).
• Transform and enrich unstructured data into structured formats (e.g., Parquet, ORC) for the Published Layer using tools like PySpark, Hive, or Spark SQL.
• Develop workflows to further process and organize data into Functional Layers optimized for business reporting and analytics.
• Implement data validation, cleansing, schema enforcement, and deduplication as part of the transformation process.
• Collaborate with Data Analysts, BI Developers, and Business Users to understand data requirements and ensure datasets are production-ready.
• Optimize ETL/ELT processes for performance and reliability in a large-scale distributed environment.
• Maintain metadata, lineage, and documentation for transparency and governance.
• Monitor pipeline performance and implement error handling and alerting mechanisms.

Technical Skills & Experience:

• 3+ years of experience in data engineering or ETL development within a big data environment.
• Strong experience with Hadoop ecosystem tools: HDFS, Hive, Spark, YARN, and Sqoop.
• Proficiency in PySpark, Spark SQL, and HQL (Hive Query Language).
• Experience working with unstructured JSON data and transforming it into structured formats.
• Solid understanding of data lake architectures: Raw, Published, and Functional layers.
• Familiarity with workflow orchestration tools like Airflow, Oozie, or NiFi.
• Experience with schema design, data modeling, and partitioning strategies.
• Comfortable with version control tools (e.g., Git) and CI/CD processes.

Nice to Have:

• Experience with data cataloging and governance tools (e.g., Apache Atlas, Alation).
• Exposure to cloud-based Hadoop platforms like AWS EMR, Azure HDInsight, or GCP Dataproc.
• Experience with containerization (e.g., Docker) and/or Kubernetes for pipeline deployment.
• Familiarity with data quality frameworks (e.g., Deequ, Great Expectations).

Qualifications:

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field.

• Relevant certifications (e.g., Cloudera, Databricks, AWS Big Data) are a plus.

• In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent

• If you have not received any feedback after 2 weeks, please consider you application as unsuccessful.

Big DataApache HadoopApache HivePySparkSQLJSONData Engineering

BankingFinancial Services

What you'll do:

• Platform Administration: Install, configure, upgrade, and monitor Cloudera/CDP clusters, manage HDFS/Ozone storage, and ensure security (Kerberos, Ranger, Sentry).
• Data Pipelines: Build and optimize ingestion and processing pipelines using NiFi and Informatica BDM/DEI, supporting both real-time and batch flows.
• ETL Integration: Develop Informatica mappings and workflows, leveraging pushdown execution to Hive/Impala/Spark; integrate diverse on-prem and cloud data sources.
• Performance Governance: Optimize queries, orchestrate jobs (Airflow, Oozie, Control-M), and ensure compliance with governance/security standards.

Your Expertise:

• Strong hands-on expertise in Cloudera tools: Hive, Impala, HDFS, Ozone, Hue, NiFi.
• Proficiency with Informatica BDM/DEI (ETL/ELT, pushdown optimization, data quality).
• Solid SQL, Linux administration, and scripting (Bash, Python).
• Familiarity with cloud data platforms (AWS, Azure, GCP) and orchestration tools.
• 4+ years in big data administration/engineering, including 2+ years in Informatica BDM/DEI.

Qualifications:

• Bachelorâ€s degree in Computer Science, Engineering, or related field.
• Experience in hybrid or cloud-based big data environments.

Soft Skills:

• Strong troubleshooting and problem-solving mindset.
• Ability to work independently and within cross-functional teams.
• Clear communication and documentation skills.

Other information applicable to the opportunity:

• Contract position
• Location: Johannesburg

Why work for us?

Want to work for an organization that solves complex real-world problems with innovative software solutions? At iOCO, we believe anything is possible with modern technology, software, and development expertise. We are continuously pushing the boundaries of innovative solutions across multiple industries using an array of technologies.†/p>

You will be part of a consultancy, working with some of the most knowledgeable minds in the industry on interesting solutions across different business domains.†/p>

Our culture of continuous learning will ensure that you will have all the opportunities, tools, and support to hone and grow your craft.†/p>

By joining IOCO you will have an open invitation to developer inspiring forums. A place where you will be able to connect and learn from and with your peers by sharing ideas, experiences, practices, and solutions.†/p>

iOCO is an equal opportunity employer with an obligation to achieve its own unique EE objectives in the context of Employment Equity targets. Therefore, our employment strategy gives primary preference to previously disadvantaged individuals or groups.

UNU-WIDER is seeking exceptional candidates for the position of Research Assistant, based in Pretoria, South Africa, to support the SA-TIED programme. This role involves managing and enhancing tax datasets, assisting researchers, and ensuring high standards of data confidentiality.

For the full job description and application details, please click here.

UNU offers three types of contracts: fixed-term staff positions (General Service, National Officer and Professional), Personnel Service Agreement positions (PSA), and consultant positions (CTC). For more information, see the Contract Types page.

Big Data Developer - Regulatory & Compliance (Relocation to Spain)

We are looking for an experienced Big Data Developer to join an international banking technology team in Málaga, Spain . In this role, you will contribute to the development of business applications within the Regulatory & Compliance domain, covering the full software lifecycle from problem analysis to deployment.

You’ll work with modern big data technologies, collaborate with users to understand business needs, and provide innovative solutions that meet regulatory and compliance requirements.

• Participate in the end-to-end software lifecycle , including analysis, design, development, testing, and deployment.
• Collaborate with business users to identify requirements and deliver strategic technology solutions.
• Optimise and analyse code, applying best practices such as threat modelling and SAST.
• Manage tools and processes for documentation and Application Lifecycle Management (ALM).
• Plan and deliver projects using Agile methodology.
• Support incident resolution, including planned interventions.
• Execute unit, integration, and regression testing.
• Manage release processes and deployment tools.

Required:

• 3+ years of experience as a Big Data Developer .
• Bachelor’s degree in Computer Science, Telecommunications, Mathematics, or a related field.
• Proficiency with GitHub .
• Strong knowledge of databases (Oracle PL/SQL, PostgreSQL).
• Hands-on ETL experience.
• Fluency in English (Spanish is advantageous).

Preferred:

• Familiarity with microservices frameworks (Spring Boot), OpenShift.
• Knowledge of Flink, Drools, Kafka, DevOps tools .
• Agile methodology experience with tools such as Jira and Confluence .
• Exposure to S3, Elastic, and Angular .
• Experience in Transactional Regulatory Reporting .
• Innovative mindset and ability to generate strategic ideas.

• Availability to travel.

• Mid-Senior level

• Full-time

• Information Technology

• Staffing and Recruiting

• 35 years C#/.NET Core development experience in production systems.
• Strong understanding of microservices principles (domain-driven design, bounded contexts, service-to-service communication).
• Proficiency in containerisation (Best practices, image optimisation, debugging containerised apps).
• Hands-on deployment experience with AKS or Kubernetes (RBAC, ConfigMaps, Secrets, Ingress, scaling strategies).
• Apache Kafka (production experience):
• Administering Kafka clusters
• Designing event-driven applications and event schemas
• Monitoring (Prometheus/Grafana, Confluent Control Center, or similar)
• Handling data consistency and exactly-once/at-least-once semantics
• Experience with Azure cloud services :
• Azure DevOps (pipelines, repos, artifacts)
• Azure Monitor / Application Insights
• Networking basics (VNETs, load balancers, firewalls)
• Azure Storage and Messaging (Event Hubs, Service Bus a plus)
• PostgreSQL experience (schema design, queries, performance tuning).