Software Engineer

This is a fully remote position, open to applicants in New York.

📋 Description

• Analyze and evaluate AI-generated responses to technical prompts regarding their accuracy, clarity, tone, and usefulness.

• Examine technical documentation, error messages, code comments, architectural explanations, instructions, and other software engineering materials.

• Assess whether the generated responses effectively address the underlying technical questions or requirements.

• Identify factual, logical, or technical inaccuracies within fluent or convincing outputs.

• Evaluate if technically accurate responses are communicated clearly and suitably for their intended audience.

• Review technical writing, design documents, API references, specifications, and similar technical materials for precision, thoroughness, organization, and instructional value.

• Detect unclear explanations, ambiguous phrasing, misleading statements, inconsistencies, and any missing information.

• Assign analytical quality scores to AI-generated technical responses.

• Compose detailed rationales that explain evaluation decisions.

• Accurately and consistently document subtle errors, inconsistencies, or ambiguities.

• Provide actionable written feedback to enhance AI model performance.

• Operate independently within structured remote evaluation workflows.

• Collaborate asynchronously with project managers, reviewers, and other expert contributors.

• Adhere to established evaluation guidelines, rubrics, and project-specific quality standards.

• Complete assigned tasks within agreed project requirements and specified timeframes.

⛳️ Requirements

• Professional experience as a Software Engineer utilizing a modern technology stack.

• Native-level fluency in English with outstanding written communication abilities.

• Proven track record of authoring or reviewing technical documentation, design documents, API references, specifications, or similar content.

• Experience in conducting code reviews, design reviews, or giving structured feedback on technical writing.

• Strong capability to evaluate technical communication for accuracy, audience suitability, and instructional value.

• Excellent analytical skills coupled with meticulous attention to detail.

• Ability to identify subtle factual inaccuracies, logical inconsistencies, ambiguous explanations, and misleading technical language.

• Strong skills in working independently and comfort in remote, asynchronous project settings.

• Familiarity with AI model evaluation, annotation, RLHF, or related structured evaluation workflows is a plus but not mandatory.

• No prior experience in AI is necessary.

• Work must be conducted without utilizing confidential or proprietary information belonging to any employer, client, institution, or any third party.

🏝️ Benefits

• Part-time independent contractor engagement.

• Fully remote work environment.

• Compensation ranging from $90–$140 per hour.

• Flexible project scope, workload, timing, and duration based on project needs.