Freelance AI Evaluation Engineer - Python, Docker, CI/CD (Remote)

Location: Remote (India preferred)

Job Type: Freelance / Contract - Long-term project work

Experience: 1- 10 years

What You'll Do

You'll work with our team to design realistic engineering problems for AI evaluation, including:

Build small isolated test environments using Docker and Python
Design automated tests with pytest to verify expected behavior
Create reproducible bug scenarios (broken builds, dependency conflicts, configuration issues, CI failures)
Validate test scenarios run correctly end-to-end before delivery
Work with our QA pipeline to ensure submission quality

This is hands-on engineering work. You will be writing real Python code, debugging Docker containers, writing pytest assertions, and thinking deeply about how to design clean reproducible test scenarios.

Required Skills

Strong Python - comfortable writing scripts, debugging, working with packages
Docker hands-on - writing Dockerfiles, building images, debugging containers
Linux command line - comfortable in terminal-based development
Automated testing - pytest or equivalent, writing strict assertions
Git - branching, merging, version control basics

Bonus Skills

Experience with CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)
Familiarity with package managers (pip, npm, uv, poetry)
Background in DevOps, SRE, backend engineering, or QA automation
Prior work on AI evaluation, benchmarking, or annotation projects

Why This Role Matters

Your work directly shapes how the next generation of AI handles real software engineering tasks.

This is not annotation work. This is engineering work that requires real Python skills, real Docker debugging, and real test design.

Engagement Model

This is long-term freelance work, not full-time employment. You can expect:

4 to 8 hours of focused work per day
Flexible hours, work on your own schedule
Long-term engagements with frontier AI clients
Multiple delivery sprints across 3 to 6+ months
Direct feedback from senior reviewers (not faceless rejection emails)

Jobs

Freelance AI Evaluation Engineer - Python, Docker, CI/CD (Remote, Immediate Joiners)