Senior Data Engineer

Senior Data Engineer

Location: Woodstock, Cape Town
Salary: R960k – R1.2m per annum negotiable depending on experience

About the Role:
Opportunity available for a Senior Data Engineer reporting to the Head of Search and Data

2+ years’ experience working with server-less services such as GCP or AWS
4+ years’ experience building and optimizing ‘big data’ data pipelines, architectures and data sets
A successful history of manipulating, processing and extracting value from large disconnected datasets
Build processes supporting data transformation, data structures, metadata, dependency and workload management
Strong analytic skills related to working with unstructured datasets
Working knowledge of message queuing, stream processing and highly scalable ‘big data’ data stores
Technologies used:
AWS: S3, EC2, RDS, Redshift, Lambdas, CloudWatch etc

Create and maintain optimal data pipeline architecture
Designs data integrations and data quality framework
Assemble large, complex data sets that meet functional/ non-functional business requirements
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Keep our data separated and secure across national boundaries through AWS security services
Writes unit/integration tests, contributes to engineering wiki, and documents work
As a team member:
Continuously keeping the team up to date with possible changes
Report progress in daily stand-ups
Weekly sprint meetings
Weekly meeting with the Head of Search and Data to align, share learnings and suggest improvements
Participate in the scrum process and daily stand-ups
Seasoned communicator as regards to technical and business/ functional aspects of assigned work
Be comfortable working in an agile and fast-paced environment