LLM Evaluator (Model Response Analyst)

About the role

We are looking for a detail‑oriented LLM Evaluator to assess and improve the performance of large language models. You will work remotely, reviewing AI‑generated content for accuracy, coherence, factual reliability, bias, safety, and alignment with our guidelines.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple model responses to the same prompt, select the preferred output, and justify your choice.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Design adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines for ambiguous or edge‑case scenarios.
• Participate in cross‑checking sessions to calibrate scoring standards and ensure inter‑rater reliability.
• Analyze underperforming outputs, hypothesize root causes, and flag novel model behaviors for research.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP/AI quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Proven experience with Reinforcement Learning from Human Feedback (RLHF) data collection.
• Ability to explain why a model output is good or bad based on logic, tone, factuality, and instruction adherence.

Required skills

• Prompt engineering
• Reinforcement Learning from Human Feedback (RLHF)
• Data analysis
• A/B testing
• Computational linguistics

What we offer

• Fully remote work with a global, collaborative team.
• Opportunities to influence cutting‑edge AI model development.
• Continuous learning and professional growth in the AI safety domain.