LLMOps Engineer

ABOUT THE COMPANY

The future of Africa will be defined by its leaders and their ability to unleash the incredible potential of the continent. ALX sees unprecedented opportunities for our fellow Africans to innovate, re-imagine and be the catalysts for transformation on the continent. ALX is a Pan-African leadership movement for high potential and high impact leaders that enables them to take on Africa’s greatest challenges and most exciting opportunities. By collaborating, growing and networking together in the ALX community space, members will maximize their impact as they shape the future of the continent. The first ALX Leadership Lab will launch in Nairobi to bring the ALX community to life. We offer unique Leadership Accelerators for Young Leaders and for Young Professionals in leadership roles, as well as high impact workshops and events.

JOB SUMMARY

Role SummaryProject A is ALX’s AI learning platform — a set of LLM products used by learners. Every one generates a stream of LLM data, and every one has hypotheses baked into it about what “working” means. The LLMOps Engineer owns the analyzer function: turning that stream into an honest answer about whether the products work. Take RAG as one example — documents must be stored accurately, fetched accurately, and fetched in the right mixture: three separate failure modes, each needing its own eval. Every product decomposes like that. This is a junior-to-mid role with a deliberate growth path: you start close to the technical lead’s designs and grow into full ownership of the function.You will work in collaboration with Anthropic Engineers, a cross functional  team of AI engineers, product managers and data scientists to design world class learning experiences.Specific ResponsibilitiesEvaluation SuitesBuild and run eval suites per product, decomposed by failure mode, running on schedule and on every release — regression testing so nothing ships if it broke what worked.Keep evals cost-effective as the product line grows.Reporting, Data & CollaborationOwn the reporting loop — findings from evals and platform data in front of the team and stakeholders, including surfacing unintended or problematic model behaviour before learners do.Steward the core datasets the team depends on, including classified customer-support data.Partner with the AI Product Manager on instrumentation — they instrument the product, you build the evals over what is captured. This is a measurement role, not infrastructure — no model hosting or serving.Skill Requirements - EssentialPython & data: solid Python and a data inclination,  comfortable shaping and analysing messy LLM-generated data.Decomposition: the ability to look at an AI product and decompose it into success and failure metrics.Eval landscape: familiarity with Langfuse, RAGAS, DSPy, or similar — depth in one, awareness of the rest. These tools are learnable; we hire the fundamentals underneath them.Desirable (not required): experience keeping evals cheap at scale; dashboarding and reporting; classical statistics.Essential Traits for SuccessYou want to own a function, not execute tickets.You communicate well and like collaborating,  you will support every builder on the team.You can point to any project, even a small one, where you measured an AI system honestly.

REQUIRED SKILLS

Programming, IT management, System administration, Python, Server networking

REQUIRED EDUCATION

Diploma, Associate's degree