Senior Software Engineer em Portugal

Responsibilities

• Design and ship features spanning data models, APIs, background workers, and UI components.
• Build and improve evaluation workflows for coding agents and language models, including sandboxed trials, scoring, and result aggregation.
• Develop benchmark quality checks using verifiers, rubric-based grading, and LLM-as-a-judge evaluation.
• Integrate new models, agents, and benchmarks through generalizable systems rather than special-case implementations.
• Improve system performance and reliability through efficient queries, safe migrations, and concurrency controls.
• Review pull requests and maintain consistent engineering standards across the codebase.

Requirements

• Typically 5+ years of experience building and operating production backend systems, including ownership of a service from design through on-call.
• Strong Python experience with Django or a similar framework, PostgreSQL, and a task queue such as Celery.
• Hands-on experience with LLM evaluation, including benchmark construction, agent-trial scoring, and result aggregation and comparison.
• Ability to assess benchmark task quality, verifier weaknesses, leaked answers, ambiguous instructions, and misleading checks.
• Experience owning loosely defined work through design, migrations, tests, edge cases, and production delivery.
• Comfort working with TypeScript and React when frontend changes are required.
• Clear technical writing and ability to make pragmatic architecture and implementation tradeoffs.
• Preferred experience includes Harbor, Terminal-Bench, SWE-bench, Inspect, sandboxed or containerized agents, Docker, Kubernetes, Modal, Daytona, distributed job scheduling, concurrency controls, database locking, coding agents, MCP, and open-source evaluation or infrastructure projects.

Benefits

• Work at the frontier of AI and help leading AI labs improve models through datasets, reinforcement learning environments, and benchmarks.
• Contribute to AI research and potentially showcase work at conferences such as ICLR, ICML, and NeurIPS.
• Apply frontier AI research to enterprise challenges and collaborate with experienced colleagues from major technology companies.
• Work in a startup-paced environment with substantial ownership and impact.
• The posting does not state a specific work arrangement, contract duration, or compensation details.