1 day ago

Senior LLMOps Engineer

Heidi

Sydney

1 day ago

Apply

Heidi is seeking a Senior LLMOps Engineer to join the model team in Sydney, Australia. The role involves building the operational layer around production LLMs, including observability, tracing, evaluation, and feedback systems. The ideal candidate has 2-3 years of hands-on LLMOps experience at a mature AI company and can own systems end-to-end.

Core AIOn-siteFull-timeSeniorLLMOpsObservability

Senior LLMOps Engineer

Apply

Core AIOn-siteFull-timeSeniorLLMOps

Salary

Not specified

Work Location

Sydney, New South Wales, Australia, AU

Work Model

On-site

Experience Required

3 years

Employment Type

Full-time

Experience Level

Senior

Core Qualifications

Technical (Must-have)

LLMOpsObservabilityMonitoringAlertingDatadogDistributed TracingData ModelingLLMsAI AgentsUnit EconomicsIncident ResponseEvaluationFeedback SystemsIntercomCSAT

Soft Skills

OwnershipLeadershipProblem SolvingCommunicationCollaboration

Preferred Qualifications

Technical (Nice-to-have)

Backend EngineeringData PlatformML InfrastructureHealthcareRegulated Environments

Key Responsibilities

•Build the deployment health dashboard with live visibility into every model in production.
•Make every incident traceable with complete session-to-model lineage.
•Stand up the improvement flywheel using Intercom tickets and qualitative CSAT feedback.
•Surface the full story by retrieving complete execution traces for flagged sessions.
•Close the loop by filtering high-value feedback into training data and shipping improved models.
•Own per-model P&L by measuring revenue against inference cost.
•Raise the LLMOps bar by bringing proven practices from mature AI companies.
•Partner across the model team to ensure observability is built in.

LLMOpsAIHealthcareEngineeringSeniorSydneyOn-siteFull-timeObservabilityLLMs

Key Responsibilities

•Build the deployment health dashboard with live visibility into every model in production.

•Make every incident traceable with complete session-to-model lineage.

•Stand up the improvement flywheel using Intercom tickets and qualitative CSAT feedback.

•Surface the full story by retrieving complete execution traces for flagged sessions.

•Close the loop by filtering high-value feedback into training data and shipping improved models.

•Own per-model P&L by measuring revenue against inference cost.

•Raise the LLMOps bar by bringing proven practices from mature AI companies.

•Partner across the model team to ensure observability is built in.