Descrição da vaga
EPAM Systems
Portugal, Portugal
Senior ML / Evaluation Engineer em Portugal, Portugal is listed on Jobeax. Browse 30,000+ vacancies available.
We're looking for a Senior ML / Evaluation Engineer to join our team in Portugal in a fully remote working mode. In this role, you will own the design and implementation of advanced evaluation frameworks for an Enterprise Agent Development Platform—a production-grade, cloud-native ecosystem enabling scalable, secure AI agent deployment. You will create evaluation strategies that combine LLM-as-judge grading with deterministic checks, define enterprise evaluation standards, and implement CI/CD deployment gates to enforce quality metrics prior to release. This position requires strong expertise in ML system testing, evaluation design, and integration into automated pipelines for agentic environments.
Design and implement multi-layer evaluation frameworks for agentic workflows and AI-driven applications Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks Develop deterministic evaluators as AWS Lambda functions for rule-based validation Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines Enable online evaluation in production by integrating sampling-based evaluation strategies and PII detection guardrails Incorporate observability signals (OpenTelemetry spans) from AWS AgentCore into grading frameworks for trace-level assessment Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms Collaborate with platform, orchestration, and DevOps teams to maintain evaluation reliability and scalability 5+ years of experience in ML engineering, AI evaluation frameworks, or AI platform development Hands-on expertise designing LLM evaluation frameworks (LLM-as-judge and deterministic graders) Practical experience implementing CI/CD deployment gates for ML model or AI agent quality assurance Proficiency in Python for building evaluation logic (deterministic Lambda-based evaluators) Strong understanding of advanced validation dimensions, including multi-turn context integrity and workflow-level scoring Familiarity with AWS AgentCore Evaluations API (CreateEvaluation, GetEvaluationResult) Exposure to AWS Bedrock Guardrails for compliance and sensitive data validation Experience integrating evaluation metrics into AWS CloudWatch for monitoring and alerting Knowledge of OTel instrumentation and trace ingestion for quality scoring inputsVagas semelhantes
Iterable
€66,500 por año
... Implement abstractions, tooling, and reusable patterns that enable other teams to build ML- and LLM-powered experiences efficiently. - Partner with backend engineers to productionize ML features with strong reliability, observability, and performance characteristics. - Prototype applied ML solutions to validate feasibility ...
Portugal, Lisboa
Iterable
€66,500 por año
... Implement abstractions, tooling, and reusable patterns that enable other teams to build ML- and LLM-powered experiences efficiently. - Partner with backend engineers to productionize ML features with strong reliability, observability, and performance characteristics. - Prototype applied ML solutions to validate feasibility ...
Portugal, Lisboa
€3,025 por mês
... T-Shaped AI / ML Engineer to join a high-ownership, agile team tackling one of our most critical scaling challenges: risk and fraud on the Tiko platform. In our engineering ecosystem, T-shaped isn’t a buzzword‑it means you bring a broad, rock-solid foundation in software engineering and data infrastructure (the horizontal bar), ...
Portugal, Lisboa
Stealth AI Startup
... is to help companies make better decisions by understanding the hidden relationships that shape the global economy. We’re looking for a Founding Forecasting & ML Engineer to help build the models behind that vision. Role We’re looking for an ML Engineer who enjoys understanding how the real world works and turning that ...
Portugal, Porto
... ownership of at least one production software, data, ML, GenAI or automation component; - Strong hands-on proficiency in Python and SQL with sound software-engineering and testing practices; - Strong capability in at least one of classical ML, GenAI/RAG/agents or MLOps/platform engineering, with working knowledge of adjacent ...
Portugal, Lisboa
DataOnline Corp. in Porto seeks a senior ML Engineer to own end-to-end ML solutions for industrial IoT telemetry. You’ll frame problems, build pipelines, deploy models in production, and monitor drift with an emphasis on reliability and scalability. You will collaborate with data engineers and developers, use agentic coding ...
Portugal, Porto
Designs and scales AI agent infrastructure for industrial decisions, focusing on reasoning quality, performance, and reliability. Requires advanced Python, LangChain/LangGraph, and scalable systems expertise.
Portugal, Porto
Tiko is seeking a T‑Shaped AI / ML Engineer to join a high-ownership, agile team tackling scale challenges: risk and fraud on the Tiko platform. You will deploy modern AI and LLM-based systems in production and own outcomes from day one. You will build, monitor, and optimize intelligent systems, ensuring architectures are ...
Portugal, Lisboa
Docker is seeking a Senior Security Engineer in Offensive Security to drive proactive security across its products and cloud infrastructure. You will perform penetration testing, threat modeling, and exploit development to surface risks and collaborate with engineering to implement durable fixes. The role spans AWS, GCP, ...
Portugal, Portugal
Intermedia Intelligent Communications is seeking a Senior Frontend Engineer in Portugal to join our Dev & Engineering team. This full-time role blends technical leadership with hands-on development in a primarily remote environment, with occasional visits to Coimbra or Aveiro. You will drive frontend architecture, build ...
Portugal, Portugal
Mindera is seeking a Senior AI Engineer for AI Systems to own the accelerator core, agent harness, and evaluation infrastructure. You will drive reliability, scalability, and observability for non-deterministic AI systems and empower delivery teams to trust outputs. You will design agents across coding, planning, and testing ...
Portugal, Coimbra
GBT Solutions, Lda
... Report progress, risks and blockers and collaborate with technical teams. Key Requirements - Strong hands-on experience in Python, machine learning, software engineering and applied AI. - Strong understanding of ML concepts, model evaluation, performance measurement and model improvement. - Practical experience with LLMs, ...
Portugal, Portugal
GBT Solutions, Lda
... Report progress, risks and blockers and collaborate with technical teams. Key Requirements - Strong hands-on experience in Python, machine learning, software engineering and applied AI. - Strong understanding of ML concepts, model evaluation, performance measurement and model improvement. - Practical experience with LLMs, ...
Portugal
... the engineering habits that go with it: git, code review, linters, unit tests and CI/CD pipelines are things you use daily. - Strong understanding of feature engineering, ML algorithms, model training and evaluation. - Solid experience across a modern ML stack: gradient boosting (LightGBM, XGBoost), scikit-learn, PyTorch, ...
Portugal, Porto
AdvanceWorks
Experienced AI Security Engineer to support client cybersecurity infrastructure, focusing on AI/ML systems, LLM risks, and incident investigation. Collaborates with engineering teams to embed security throughout the SDLC.
Portugal
Agile Engine
$2500 USD
Full Stack Engineer (Senior/Lead) ID41561 ($2,500 signing bonus) AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in application development and AI/ML, and our people-first culture has earned us multiple ...
Portugal, Estoril
ITSector
ITSector is seeking an experienced software engineer to design and implement AI-powered applications and intelligent agents. You will build integrations between AI models and enterprise systems through APIs and tools, while developing scalable pipelines for knowledge retrieval and https://jobeax.com/link/SIEkrz4VgCkhRIDm ...
Portugal, Lisboa
ScaleOps Search Ltd
Senior Data Engineer - Snowflake & Kafka Portugal, Mostly Remote We are working with an established technology consultancy that is looking to hire two Senior Data Engineers for a major data modernisation programme. You will join a team responsible for designing, building and optimising the data pipelines supporting a new ...
Portugal, Évora
INSCALE
... well, and we have an ambitious plan to grow our business to 1,000 stores by 2028. Role description and responsibilities Join a highly competent team of Data Engineers , where you will have the opportunity to bring value to data & participate in enabling data-driven business-critical decisions. Key responsibilities: - Thorough ...
Portugal, Lisboa