12 Jun 2026

Senior Research Engineer (Agentic Behavior)

JetBrains

Amsterdam

12 Jun 2026

Apply

JetBrains seeks a Senior Research Engineer to build evaluation infrastructure and improve AI coding agents for Kotlin. The role involves error analysis, benchmark creation, and post-training techniques. Requires strong Python and LLM eval experience.

AIHybridFull-timeSeniorPythonSQL

Senior Research Engineer (Agentic Behavior)

Apply

AIHybridFull-timeSeniorPython

Salary

Not specified

Work Location

Amsterdam, North Holland, Netherlands, NL

Work Model

On-site with remote flexibility

Employment Type

Full-time

Experience Level

Senior

Core Qualifications

Technical (Must-have)

PythonSQLKotlinLLMsAI coding agentsData analysisEvaluation pipelinesPost-trainingSFTDPOGRPOPyTorchAWS AthenaGitCI/CD

Soft Skills

Problem solvingProduct-aware mindsetOwnershipCollaboration

Preferred Qualifications

Technical (Nice-to-have)

RLHFTRLverlMegatronInspect AIPromptfooLM-evaluation-harnessWeights & BiasesMLflowLangfuseAndroidGradleKMPSpringKtor

Key Responsibilities

•Design and implement tooling to capture, classify, and analyze errors from AI coding agents generating Kotlin code.
•Build observability pipelines over agentic traces from JetBrains IDEs, Junie, Claude Code, Cursor, etc.
•Design, implement, and maintain evaluation pipelines for Kotlin code generation quality.
•Build simulation environments for realistic Kotlin developer tasks.
•Own evaluation infrastructure: metrics, experiment tracking, regression checks, benchmarking.
•Experiment with post-training techniques (SFT, DPO, GRPO) to improve Kotlin-specific model behavior.
•Investigate context engineering approaches (CLAUDE.md, compiler-as-verifier, Kotlin LSP, MCP).
•Run A/B comparisons and before/after analyses on real codebases.
•Collaborate with model providers (Anthropic, OpenAI, Google) to drive improvements.
•Design and build open-source benchmarks for AI coding agent performance on Kotlin tasks.
•Create task datasets covering server-side, multiplatform, build systems, Android, etc.
•Maintain and evolve benchmarks to remain challenging and contamination-resistant.

Research EngineerKotlinAI agentsLLMsEvaluation infrastructurePost-trainingBenchmarksJetBrainsAmsterdam

Key Responsibilities

•Design and implement tooling to capture, classify, and analyze errors from AI coding agents generating Kotlin code.

•Build observability pipelines over agentic traces from JetBrains IDEs, Junie, Claude Code, Cursor, etc.

•Design, implement, and maintain evaluation pipelines for Kotlin code generation quality.

•Build simulation environments for realistic Kotlin developer tasks.

•Own evaluation infrastructure: metrics, experiment tracking, regression checks, benchmarking.

•Experiment with post-training techniques (SFT, DPO, GRPO) to improve Kotlin-specific model behavior.

•Investigate context engineering approaches (CLAUDE.md, compiler-as-verifier, Kotlin LSP, MCP).

•Run A/B comparisons and before/after analyses on real codebases.

•Collaborate with model providers (Anthropic, OpenAI, Google) to drive improvements.

•Design and build open-source benchmarks for AI coding agent performance on Kotlin tasks.

•Create task datasets covering server-side, multiplatform, build systems, Android, etc.

•Maintain and evolve benchmarks to remain challenging and contamination-resistant.