AI Research Peer Review Evaluator (ML/AI)

Lightly AG, a Zurich‑based AI spin‑off from ETH/HSG and backed by Y Combinator, develops machine‑learning and computer‑vision solutions used by leaders in autonomous driving, medical imaging and visual inspection. The company is now seeking researchers with solid ML/AI backgrounds to act as AI Research Peer Review Evaluators, a remote, project‑based contractor role focused on assessing AI‑generated scientific peer reviews. In this position you will read and scan ML/AI research papers to grasp their contributions, methodology, experiments and claims. You will examine the original human peer reviews to set an expert baseline, then evaluate AI‑generated reviews against that baseline using a structured scoring rubric. Your assessment will cover technical accuracy, analytical depth, constructive value and novelty judgments, while flagging hallucinations, unsupported claims or missed issues. You will also compare two AI‑generated reviews side‑by‑side, search and verify relevant literature through Google Scholar, arXiv or Semantic Scholar, and provide concise, evidence‑based rationales for every scoring decision, applying the rubric consistently across papers. The role requires a Master's, PhD, or ongoing graduate study in Machine Learning, Artificial Intelligence, Computer Science, Statistics or a closely related technical field. Candidates must have authored or co‑authored at least one scientific paper and possess experience critically reading ML/AI research, including evaluation of methodology, experimental design and scientific claims. Familiarity with major ML/AI venues such as NeurIPS, ICML, ICLR, ACL or CVPR is essential, as is the ability to apply detailed evaluation guidelines and scoring rubrics consistently. Strong analytical abilities and written communication skills are needed to distinguish meaningful technical concerns from superficial criticism and to craft clear, concise rationales. Attention to detail is crucial for spotting factual inaccuracies or hallucinated claims. Additional advantages include prior academic peer‑review experience for an ML/AI conference or journal and proficiency in conducting academic literature searches and verifying citations. The position is fully remote, part‑time, and offers flexible working hours, allowing you to contribute directly to the evaluation of cutting‑edge agentic AI systems that aim to improve scientific peer review. Interested applicants should submit a CV together with a brief note describing their research background, relevant publications and any previous peer‑review activities. What the role asks for: - Master's, PhD, or graduate study in ML/AI, CS, Statistics, or related field - Authored or co‑authored at least one scientific research paper - Experience critically reading ML/AI research papers - Familiarity with major ML/AI venues (NeurIPS, ICML, ICLR, ACL, CVPR) - Ability to apply detailed evaluation rubrics consistently - Strong analytical and written communication skills - Nice-to-have: prior academic peer‑review experience for ML/AI conference or journal - Nice-to-have: proficiency conducting academic literature searches and verifying citations