LLM Evaluation Engineer Python & Docker (3 Months Contract)

Note

This is a contract-based, fully remote opportunity. Payments are made based on actual hours worked. No paid leaves or benefits are included. The contractor must handle their own taxes and compliances.

Job Overview

We are seeking a highly experienced Senior Software Engineer (Python) to work on a project that focuses on building and evaluating datasets for large language model (LLM) training. This role combines real-world software engineering with AI evaluation, involving analysis of public GitHub repositories, environment setup, code testing, and validation of AI performance in software development tasks.

Key Responsibilities

• Analyze and triage issues from public GitHub repositories
• Set up and automate development environments using Docker and related tools
• Assess and report on unit test coverage, code quality, and bug complexity
• Run, debug, and modify real-world codebases to evaluate LLMs
• Collaborate with research engineers to design and select projects that challenge LLM performanc
• Optionally lead and guide junior engineers on related tasks
  
  

Must-Have Skills

• Strong proficiency in Python (hands-on coding and debugging
• Expertise with Git and Docker for managing and containerizing development environments
• Proven ability to work with and analyze complex open-source codebases
• Familiarity with software QA techniques, unit testing, and test coverage tools
• Comfortable running and modifying real-world applications in local environments
• Clear written and verbal communication in a remote team setting
  
  

Must-Have Qualifications

• Minimum 5+ years of professional software engineering experience
• Bachelor’s degree or higher in Computer Science, Software Engineering, or a related field
• Prior experience working with public repositories or open-source codebases
• Hands-on experience in setting up and troubleshooting development environments
• Ability to analyze and triage real-world issues and bugs efficientlySelf-motivated, independent, and proactive in a distributed/remote environment