Data Engineer, Elasticsearch, Data Warehousing

This is a fully remote position, open to applicants in India.

📋 Description

• Design, implement, and enhance Elasticsearch clusters for optimal querying and data retrieval performance.

• Develop and oversee Elasticsearch indexes to ensure efficient data storage, indexing, and querying.

• Create and refine data storage solutions such as data lakes and warehouses.

• Integrate structured and unstructured data from various internal and external sources to establish a cohesive analytical view.

• Guarantee data accuracy, consistency, and completeness through thorough validation, cleansing, and transformation processes.

• Maintain detailed documentation for data processes, tools, and systems while advocating for best practices to promote efficient workflows.

• Collaborate with product managers and other stakeholders to gather requirements and convert them into technical solutions.

• Engage in requirement analysis sessions to comprehend business needs and user expectations.

• Provide technical insights and recommendations during the requirement-gathering phase.

• Participate in Agile development practices, including sprint planning, daily stand-ups, and sprint reviews.

• Work closely with Agile teams to deliver software solutions on schedule and within the defined scope.

• Adjust to evolving priorities and requirements in a dynamic Agile setting.

• Conduct extensive testing and debugging to ensure the reliability, security, and performance of applications.

• Write unit tests and verify the functionality of developed features and individual components.

• Create integration tests to ensure that various components within an application operate as intended and meet specified requirements.

• Identify and rectify software defects, code smells, and performance bottlenecks.

• Stay informed about the latest technologies and trends in full-stack development.

• Suggest innovative solutions to enhance the performance, security, scalability, and maintainability of applications.

• Continuously seek chances to optimize and refactor the existing codebase for increased efficiency.

• Keep current with cloud platforms such as AWS, Azure, or Google Cloud Platform.

• Effectively collaborate with cross-functional teams, including testers and product managers.

⛳️ Requirements

• Bachelor's degree in Computer Science, Engineering, or a related field.

• Demonstrated experience as a Data Engineer, with a minimum of 3 years in the field.

• Proficiency in Elasticsearch and the Python programming language is essential.

• Experience with database technologies such as SQL (e.g., MySQL, PostgreSQL) and NoSQL (e.g., MongoDB) databases.

• Strong understanding of programming libraries/frameworks and technologies including Flask, API frameworks, data warehousing/lakehouse principles, ORM, data analysis, Databricks, Pandas, Spark, PySpark, machine learning, OpenCV, and Scikit-learn.

• Utilize Java to build and improve backend systems, particularly for integration with Elasticsearch and databases.

• Develop APIs, microservices, and automation scripts as required.

• Utilities & Tools: logging, requests, subprocess, regex, pytest.

• Familiarity with the ELK stack, Redis, and distributed task queues.

• Strong grasp of data warehousing/lakehouse principles and concurrent/parallel processing concepts.

• Experience with at least one cloud data engineering stack (Azure, AWS, or GCP) and the ability to quickly learn and adapt to new ETL/ELT tools across different cloud providers.

• Familiarity with version control systems such as Git and collaborative development workflows.

• Competence in working with the Linux OS and creating shell scripts.

• Solid understanding of software engineering principles, design patterns, and best practices.

• Exceptional problem-solving and analytical abilities, with a strong attention to detail.

• Effective communication skills, both written and verbal, and the capability to collaborate within a team environment.

• Openness to learning new technologies and tools as necessary.

🏝️ Benefits

• Competitive salary and performance-based incentives.

• Opportunities for professional growth and career advancement.

• Comprehensive health and wellness benefits.

• Flexible work hours and remote work options.

• Collaborative and inclusive company culture.