Descrição da vaga
EPAM Systems
Portugal, Portugal
Senior ML / Evaluation Engineer em Portugal, Portugal is listed on Jobeax. Browse 30,000+ vacancies available.
We're looking for a Senior ML / Evaluation Engineer to join our team in Portugal in a fully remote working mode. In this role, you will own the design and implementation of advanced evaluation frameworks for an Enterprise Agent Development Platform—a production-grade, cloud-native ecosystem enabling scalable, secure AI agent deployment. You will create evaluation strategies that combine LLM-as-judge grading with deterministic checks, define enterprise evaluation standards, and implement CI/CD deployment gates to enforce quality metrics prior to release. This position requires strong expertise in ML system testing, evaluation design, and integration into automated pipelines for agentic environments.
Design and implement multi-layer evaluation frameworks for agentic workflows and AI-driven applications Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks Develop deterministic evaluators as AWS Lambda functions for rule-based validation Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines Enable online evaluation in production by integrating sampling-based evaluation strategies and PII detection guardrails Incorporate observability signals (OpenTelemetry spans) from AWS AgentCore into grading frameworks for trace-level assessment Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms Collaborate with platform, orchestration, and DevOps teams to maintain evaluation reliability and scalability 5+ years of experience in ML engineering, AI evaluation frameworks, or AI platform development Hands-on expertise designing LLM evaluation frameworks (LLM-as-judge and deterministic graders) Practical experience implementing CI/CD deployment gates for ML model or AI agent quality assurance Proficiency in Python for building evaluation logic (deterministic Lambda-based evaluators) Strong understanding of advanced validation dimensions, including multi-turn context integrity and workflow-level scoring Familiarity with AWS AgentCore Evaluations API (CreateEvaluation, GetEvaluationResult) Exposure to AWS Bedrock Guardrails for compliance and sensitive data validation Experience integrating evaluation metrics into AWS CloudWatch for monitoring and alerting Knowledge of OTel instrumentation and trace ingestion for quality scoring inputsVagas semelhantes
Thought Machine
Responsibilities - Support the engineering team in building reliable and scalable applications. - Design and build tools that improve scalability and high availability while automating operational processes. - Develop observability and orchestration tooling for monitoring and managing customer deployments. - Contribute ...
Portugal, Lisboa
€66,000 por año
... partner in solving tough problems to keep the Internet moving forward, you’ll fit right in. Available Locations: Lisbon About the Department Cloudflare’s Network Engineering Team builds and runs the infrastructure that runs our software. The Engineering Team is split into two groups: one handles product development and the other ...
Portugal, Lisboa
act digital
We are looking for a Data Engineer with strong experience in Teradata and ETL tools to join a team working on data platforms in a structured and collaborative environment. About the Role You will be responsible for designing, developing, and maintaining data pipelines and data processing solutions , ensuring data quality, ...
Portugal, Lisboa
Siemens Energy
Seeking a Senior AI Engineer to design, build, and deploy enterprise-grade AI applications using Generative AI, LLMs, and RAG on Azure and AWS. Responsibilities include mentoring junior engineers and ensuring scalable, secure AI solutions.
Portugal, Amadora
SAP is looking for a Senior DevOps Engineer in Portugal to manage cloud infrastructure, monitor build landscapes, and automate CI/CD processes. Requires experience with Kubernetes, cloud, networks, and 5+ years in DevOps.
Portugal, Oeiras
Continental in Portugal, Braga is seeking a qualified engineer to lead wear-testing initiatives within the Evaluation Center Wear. You will design structured testing methods, collect and analyze sensor and wear data, and translate results into practical guidance for design and engineering teams. You will build tester profiles, ...
Portugal, Lousado
Jobgether is seeking a Machine Learning Task Auditor on a freelance basis to audit ML tasks used to train and evaluate AI models. Based in Portugal, you will work remotely on challenging technical assignments, reviewing data pipelines, algorithms, and testing approaches to ensure accuracy, realism, solvability, and reproducibility. ...
Portugal, Portugal
90k
... architecture decisions independently yet, but you should be a confident, self-directed practitioner. The role sits at the intersection of machine learning engineering, MLOps, and Generative AI. You will build robust ML pipelines, deploy and monitor AI services, and develop scalable APIs and retrieval systems that enable ...
Portugal, Setúbal
... balance, beginner's mind and community involvement are our identity, our foundations, and our culture. What you need to join us: 5+ years of experience in data engineering or a closely related role, with hands-on production experience in Databricks (minimum 2 years). Significant experience building data workloads in Databricks, ...
Portugal, Santarém
emagine
... you passionate about building scalable backend systems, modern cloud-native applications, and high-performance distributed architectures? We're looking for a Senior Backend Engineer to join an international technology team working on the modernization of large-scale enterprise platforms. You'll play a key role in transforming ...
Portugal, Bragança
Decskill
... allows us to offer an international context, diverse challenges, and long-term career opportunities, while staying close to our teams. We would like to meet a Senior OT Cybersecurity Engineer for a hybrid project based in Coimbra, Porto, Lisboa or Viseu. Requirements: - Knowledge of the full product security lifecycle (requirements ...
Portugal, Coimbra
Decskill
... allows us to offer an international context, diverse challenges, and long-term career opportunities, while staying close to our teams. We would like to meet a Senior OT Cybersecurity Engineer for a hybrid project based in Coimbra, Porto, Lisboa or Viseu. Requirements: - Knowledge of the full product security lifecycle (requirements ...
Portugal, Coimbra
Decskill
... allows us to offer an international context, diverse challenges, and long-term career opportunities, while staying close to our teams. We would like to meet a Senior OT Cybersecurity Engineer for a hybrid project based in Coimbra, Porto, Lisboa or Viseu. Requirements: - Knowledge of the full product security lifecycle (requirements ...
Portugal, Coimbra
Decskill
... allows us to offer an international context, diverse challenges, and long-term career opportunities, while staying close to our teams. We would like to meet a Senior OT Cybersecurity Engineer for a hybrid project based in Porto, Viseu, Coimbra or Lisboa. Requirements: - Knowledge of the full product security lifecycle (requirements ...
Portugal, Porto
... where there’s always room to discuss the quirks of living on this planet, then we might just be a perfect fit. About the role We’re looking for an experienced Senior AI Engineer to join our team working at the frontier of applied AI in Requirements Management and Hardware Engineering more in general. This is a hands‑on role ...
Portugal, Lisboa
emagine
... you passionate about building scalable backend systems, modern cloud-native applications, and high-performance distributed architectures? We're looking for a Senior Backend Engineer to join an international technology team working on the modernization of large-scale enterprise platforms. You'll play a key role in transforming ...
Portugal, Setúbal
We are seeking a Senior Automation & Platform Engineer to automate repetitive operational activities, improve infrastructure efficiency and standardization, and integrate operational workflows across platforms and tools. The engineer will develop proactive monitoring, validation, and remediation capabilities to reduce recurring ...
Portugal, Lisboa
We are looking for a Senior Cloud & Systems Engineer to join an international environment and take an active role in the management, evolution, and security of cloud and systems infrastructure. The position combines Microsoft Azure, Microsoft 365, Windows environments, identity management, networking, security, and automation ...
Portugal, Portugal
18808 - 70787.1
... inspiration. This is your opportunity to explore the future of healthcare through technology and digital innovation, supporting our mission To Serve Patients. Senior Cloud Security Engineer At Amgen, technology is more than an enabler, it is a catalyst for discovery, transformation, and better patient outcomes. Every innovation ...
Portugal, Lisboa
emagine
Are you an experienced Data Engineer passionate about building scalable data platforms, automating complex pipelines, and working at the intersection of Data Engineering, DevOps, and MLOps ? We’re looking for a technically strong and proactive Senior Data Engineer to join a growing data engineering environment. You’ll play ...
Portugal, Lisboa