Senior Python Backend Developer, ML Engineer

This is a fully remote position, open to applicants in Brazil, +3 more countries.

📋 Description

• Develop, enhance, and utilize ML Engineering platforms and components; create scalable backend systems, APIs, and microservices utilizing FastAPI.

• Implement MLOps practices, including model KPI measurement, tracking, model drift detection, and feedback loops for models.

• Deploy and operationalize machine learning and deep learning models, with a focus on LLMs and Generative AI.

• Integrate Azure OpenAI, GPT-4, GPT-4 Vision, and other LLM providers, ensuring retry logic and effective error handling.

• Stay updated on cutting-edge LLM, GenAI, and transformer technologies.

• Scale machine learning algorithms to handle large datasets while adhering to strict SLAs.

• Construct and manage feature engineering, inferencing, and continuous model training pipelines.

• Write production-quality backend code in Python and SQL, employing object-oriented and asynchronous programming techniques.

• Apply dependency injection patterns and layered architectural approaches.

• Establish LLM observability to monitor prompts, tokens, costs, and latency metrics.

• Create prompt management systems that include versioning and fallback solutions.

• Utilize Celery or similar workflows for asynchronous task processing and complex pipelines.

• Design multi-tenant architectures that ensure client data isolation.

• Execute LLM cost optimization strategies.

• Integrate third-party APIs and services, such as document/OCR services, cloud storage, and enterprise systems.

• Collaborate with client-facing teams to understand business context and technical requirements.

• Produce testable, maintainable production code that addresses edge cases and errors.

• Adhere to architecture and coding guidelines, participating in design and code reviews.

• Create unit tests and higher-level tests.

• Troubleshoot issues using structured logging and distributed tracing methodologies.

• Employ bug tracking, code review, version control, and related delivery tools effectively.

• Engage in scrum calls and agile ceremonies.

• Document application modifications and API specifications using OpenAPI/Swagger.

• Investigate emerging architectural patterns and technologies through proofs-of-concept and prototypes.

⛳️ Requirements

• Bachelor’s or Master’s degree in Computer Science or a related discipline.

• Proficient in Python coding with a minimum of 7 years of experience.

• At least 2 years of practical experience in machine learning and production LLM systems.

• Over 3 years of experience in building backend APIs using FastAPI, async patterns, rate limiting, and SQLAlchemy.

• Skilled in designing maintainable and extensible systems utilizing dependency injection, interfaces, and abstract base classes.

• Familiarity with vector databases such as Pinecone, Weaviate, or Chroma, along with hybrid search techniques.

• Strong knowledge of RAG architectures, including retrieval, reranking, context assembly, and response generation.

• Practical experience with LangChain and LangGraph for developing and orchestrating LLM workflows.

• Advanced Python capabilities, including async/await, type hints, Pydantic, and SOLID principles.

• MLOps experience with MLflow, model versioning, and A/B testing; familiarity with Langfuse is advantageous.

• Background in NLP and computer vision, encompassing document understanding, OCR, and GPT-4 Vision.

• Proven experience in constructing feature pipelines, real-time and batch inference systems, and model serving.

• Required hands-on experience with Hugging Face; experience with LlamaIndex is a plus.

• Knowledge of database technologies such as SQL.

• Strong problem-solving abilities and capacity to thrive in a fast-paced, team-oriented environment.

• Understanding of DevOps, CI/CD practices including Docker containerization, Azure DevOps pipelines or GitHub Actions, and Kubernetes (preferred).

• Expertise in data security, encompassing multi-tenant data isolation, secure key management (Azure Key Vault), and audit trail implementation.

• Experience in designing solutions on cloud platforms such as Azure, AWS, or GCP.

• Experience in data engineering within Big Data systems, including large-scale data processing and ETL/ELT pipelines.

• Knowledge of rate limiting and quota management for high-throughput API usage.

• Skills in cost management and optimization for LLM usage at scale.

• Proficiency in document processing, including PDF extraction and OCR tooling.

• Experience in production incident management and on-call responsibilities.

• Familiarity with testing strategies for non-deterministic LLM outputs.

• Domain expertise in regulated industries is a plus.

🏝️ Benefits

• Engaging projects that make a difference.

• Access to Udemy courses of your choice.

• Team-building activities, events, marathons, and charity initiatives to foster connections and rejuvenate.

• Workshops, training sessions, and expert knowledge-sharing opportunities for continuous growth.

• Clearly defined career progression.

• Absence days to support work-life balance.

• Flexible working hours and setup - work from anywhere and manage your schedule your way.