Senior ML Engineer (Evaluation)
kaiko.ai
Amsterdam
3 weeks ago
Senior ML Engineer (Evaluation)
Senior ML Engineer (Evaluation) needed for kaiko.ai's clinical AI assistant team. Responsible for designing and operating large-scale evaluation pipelines for multimodal large language models in healthcare. Requires excellent Python skills, ML infrastructure experience, and strong software engineering fundamentals.
Hybrid
Full-time
Senior
Python
Workflow Orchestration
Salary
Not specified
Core Qualifications
Technical (Must-have)
Pythonworkflow orchestrationautomated pipelinesML infrastructurelarge language modelsmultimodal modelsdistributed compute systemsGPU workloadscluster schedulingresource management
Soft Skills
collaborationownershipambitioncuriosityproblem-solving
Tools (Must-have)
CI/CDmonorepo toolingcontainerisationconfig managementobservability
Preferred Qualifications
Technical (Nice-to-have)
DagsterRayvLLMlm-eval-harnessHF Evaluatered-teamingload testing
Key Responsibilities
- Design, operate, and mature automated pipelines and workflows for large-scale evaluation jobs
- Maintain and mature inference and eval services ensuring correctness, reproducibility, and throughput
- Ensure functional integrity of eval stack through rigorous testing and validation
- Own Eval/MLOps end-to-end: service deployments, model and artifact versioning, eval data organization, and post-deployment observability
- Develop towards a technical lead: set engineering direction, make architectural decisions, and support other engineers in execution
ML EngineerEvaluationClinical AIHealthcareMultimodal LLMPythonML InfrastructureSeniorAmsterdamZurich