AI Research Peer Review Evaluator (ML/AI)

Lightly AG, a Zurich‑based AI start‑up spun out of ETH and HSG and backed by Y Combinator, builds machine‑learning and computer‑vision solutions used by leaders in autonomous driving, medical imaging and visual inspection. The company is now seeking researchers with solid ML/AI expertise to join a project that evaluates how well autonomous AI systems can perform scientific peer review of machine‑learning papers. In this remote, part‑time contractor role you will read and dissect ML/AI research articles, grasping their main contributions, methods, experiments and claims. You will compare the original human reviews with those generated by AI agents, scoring the AI output against a structured rubric that measures technical accuracy, depth of analysis, constructive value and novelty assessment. The work also involves spotting hallucinations, unsupported statements or missed technical issues, and highlighting any valuable insights the AI reviewers uncover. You will run side‑by‑side comparisons of two AI‑generated reviews, verify cited literature through sources such as Google Scholar, arXiv or Semantic Scholar, and write concise, evidence‑based explanations for each evaluation decision while consistently applying the project's scoring guidelines. The ideal candidate holds a Master's, PhD or is actively pursuing graduate studies in Machine Learning, Artificial Intelligence, Computer Science, Statistics or a closely related discipline. You must have authored at least one scientific paper and be comfortable critically analysing ML/AI publications, including methodology, experimental design and claim validity. Familiarity with top‑tier venues such as NeurIPS, ICML, ICLR, ACL or CVPR is required, as is the ability to conduct thorough literature searches and verify citation details. Strong analytical abilities, clear written communication, the capacity to produce concise, evidence‑backed rationales, consistent application of detailed rubrics, and meticulous attention to factual detail are all essential. While prior academic peer‑review experience for an ML/AI conference or journal is strongly preferred, it is not mandatory. The position offers fully remote work with flexible hours, allowing you to contribute from any location. You will directly influence the assessment of cutting‑edge agentic AI systems and help shape the future quality of AI‑generated scientific peer review. Interested applicants should submit a CV together with a brief note describing their research background, relevant publications and any peer‑review experience. What the role asks for: - Master's/PhD or graduate study in ML/AI, CS, Stats, or related - Authored at least one scientific research paper - Experience critically reading ML/AI research papers - Familiar with major ML/AI venues (NeurIPS, ICML, ICLR, ACL, CVPR) - Comfortable conducting academic literature searches and verifying citations - Strong analytical and written communication skills - Provide concise, evidence‑based rationales for decisions - Apply detailed evaluation rubrics consistently across papers - Strong attention to detail for factual accuracy - Prior academic peer‑review experience for ML/AI conference or journal (nice-to-have)