• Support and operate MSK clusters by integrating services and features as necessary to make them fully self-service and self-managing.
• Operating and managing Databricks clusters, ensuring it as a stable, secure platform for large-volume streaming data processing and analysis.
• Build and maintain the infrastructure needed to support a wide range of streaming use cases both within the team and externally.
• Work closely with other teams to evangelize best practices through documentation, tutorials, examples, and complete implementations
• Responsible for the implementation of data-centric business deliverables such as data manipulation, building data systems and data pipelines. 
• Contribute to data solution design as well as articulate this back to the business with clarity. 
• Ensure that all solutions are performing as efficiently and effectively as possible, apply monitoring and respond to incidents when needed. 
• Collaboration and support for technology teams, data researchers, data analysts and data scientists are expected on daily basis as well as sharing knowledge and skills.
• Build new data sources and data pipelines that deliver key data and insights to the business
• Optimise data pipelines from data extraction to processing

Skills: MSK Cluster Management, Databricks Operations, Infrastructure Development, Best Practices Evangelism, Data Pipeline Implementation, Data Solution Design, Performance Monitoring and Incident Response, Collaborative Support

• Play a critical role in helping the Insight engineering team to build a world-class API platform
• Develop and implement tools to further optimize production workflow and governance of API platform
• Creating code examples, documentation, and other tools necessary to support partners at scale
• Play a key role in onboarding new API consumers and governing the existing usage
• Work with internal and external teams to validate and troubleshoot API-related issues
• Building relationships with various partnering groups and becoming a trusted advisor
• Consult in-depth with developers who are planning or already using Insight API
• Design, build, and run proof of concepts with prospective API consumers
• Support business development partners with technical consultations pre and post-implementation
• Optimize data storage systems with the business to scope and deliver new data engineering projects and requirements

Skills: API Platform Development, Production Workflow Optimization, Technical Documentation Creation, API Consumer Onboarding, API Issue Troubleshooting, Relationship Management, Technical Consultation, Data Engineering Support

• Design data processing systems
• Build and operationalize data processing systems
• Design, build, operationalize, secure, and monitor data processing systems
• Create robust and automated pipelines to ingest and process structured and unstructured data from source systems into analytical platforms using batch and streaming mechanisms leveraging cloud-native toolset
• Implement custom applications using tools such as Kinesis, Lambda and other cloud native tools to address streaming use cases
• Engineers and supports data structures including but not limited to SQL and NoSQL databases
• Engineers and maintain ELT processes for loading Snowflake “share house” and AWS data lake
• Engineers API’s for returning data from these structures to the Enterprise Applications
• Leverages the right tools for the right job to deliver testable, maintainable, and modern data solutions
• Respond to customer/team inquiries and assist in troubleshooting and resolving challenges
• Works with other scrum team members to estimate and deliver work inside of a sprint
• Research data questions, identifies root causes, and interact closely with business users and technical resources

Skills: Data Processing System Design, Operationalization of Data Systems, Data Processing System Security and Monitoring, Automated Data Ingestion Pipelines, Custom Application Implementation, Data Structure Engineering, ELT Process Maintenance, API Development for Data Access

• Design, develop, and maintain the data solutions and data tools for client
• Extend and evolve the data solution by researching and applying new big data technologies and stacks to solve business problems
• Maintain the data solutions by diagnosing, predicting and correcting scaling problems
• Work with large streams of data with tools like Spark, S3, Kinesis, DynamoDB etc
• Contribute to team’s growing set of development platforms, tools, and processes
• Build automation and tools that will increase the productivity of teams located in different GEOs.
• Provide input and feedback to teams regarding decisions surrounding topics such as infrastructure, data architectures, resiliency etc.
• Maintain and build on existing data infrastructure and tools
• Support the internationalization of data infrastructure as we continue to grow globally
• Design and development of BI and Data solutions

Skills: Data Solutions Development, Big Data Technology Research, Scaling Problem Maintenance, Stream Processing, Development Tools Contribution, Productivity Automation, Infrastructure Consultation, BI Solutions Design

• Build and expand data platform and services
• Help us identify the best technologies for evolving data needs
• Collaborate with product team to deliver user-facing features
• Monitor and improve data pipeline performance
• Write unit, integration, and load tests
• Actively participate in code reviews
• Write and review technical proposals
• Improve engineering standards, tooling, and processes
• Build the required infrastructure to support the team’s production jobs and development initiatives using AWS technologies.
• Sharing knowledge through code review, presenting, and training.
• Advise and provide support to junior engineers.

Skills: Data Platform Development, Technology Evaluation, User-Facing Feature Collaboration, Data Pipeline Performance Monitoring, Testing and Quality Assurance, Code Review Participation, Technical Proposal Writing, Infrastructure Support with AWS

• Knowledge of security and privacy 'by design’ frameworks
• Strong programming experience in SQL
• Programming experience in Python
• Familiarity with Snowflake
• Familiarity with Airflow
• Experience with AWS data-related products such as EMR, Glue, S3, Lambda
• Understanding of data engineering fundamentals (design patterns, common practices) and what it takes to build high-volume data products and services.
• Knowledge of OS, networking and distributed computing
• Experience with one or more of the following technologies such as Spark, Kafka, Nifi, HBase, Hive, Hadoop, Kudu, Solr/ElasticSearch
• Certification in the above technologies
• Experience in supporting data platforms with multi-terabyte or billions of records
• Experience in troubleshooting and resolving issues (hardware, networking, OS and software) with self-hosted Hadoop cluster

Qualifications: BA in Data Science with 3 years of Experience

• Experience in software development.
• Excellent communication skills and the ability to work well in a distributed team.
• Able to code an application from scratch following best practices such as writing clean code with unit tests, and using continuous integration.
• Experience in designing and developing web services and REST APIs.
• Advanced knowledge of relational databases such as PostgreSQL, and being capable of writing non-trivial SQL.
• Experience in data modeling, ETL development, and data warehousing.
• Experience with AWS big data technologies such as Redshift, Athena, and EMR
• Experience with MongoDB.
• Experience with CI/CD, with emphasis on Docker and Kubernetes.
• Experience with Airflow, with emphasis on developing ETL pipelines using Airflow.
• ETL development experience on large datasets (terabyte scale or larger)
• Experience with Apache Kafka

Qualifications: BS in Computer Science with 5 years of Experience

• Hands-on experience implementing and maintaining scalable, highly available, high load systems
• Proficiency in any backend programming language and willingness to learn Golang
• Proficiency in any database and willingness to learn MongoDB and BigQuery
• Understanding of ETL principles
• Hands-on experience with Kubernetes
• Fluency in English on technical topics.
• Any experience writing automated tests or willingness to learn it
• Any experience in CI/CD or willingness to learn it
• Any experience in software development using Agile methodology
• Knowledge of BigQuery and GCP ecosystem
• Knowledge of DBT 

Qualifications: BA in Software Engineering with 4 years of Experience

• Extensive experience in the deployment of data platforms and tools in a cloud environment
• Experience working with cloud provider tools and architecture (Azure & AWS)(AWS preferred)
• Experience using scripting languages such as bash, SQL and/or Python, Terraform
• Experience in management and tuning of data platforms (Databricks/Delta Lake required)
• Experience with a broad set of data ingestion and data pipeline patterns, and associated tools like (Teradata, SQL Server, Oracle, Snowflake, and Databricks)
• Experience managing or administering orchestration tools such as Apache Airflow
• Experience with designing data ingestion solutions from a wide range of data sources and connections like SaaS API connections, SFTP, real-time streaming and Change Data Capture
• Preferred experience with working in an agile product development
• Preferred experience working with and maintaining CI/CD pipelines and versioning tools such as Git
• Good communication skills and ability to articulate system designs and patterns to varying levels of leadership.

Qualifications: BS in Information Technology with 3 years of Experience

• Excellent problem-solving skills able to own RCA, outcome management and continual learning
• Strong stakeholder and customer management skills.
• Solid understanding of database management systems including IAAS, PAAS and other cloud-based technologies
• A strong background in platform and product support, optimization and performance tuning
• Experience in administering large ETL and data warehouse applications
• Experience working in a dynamic operational environment
• Experience in Cloud environments such as Azure, Snowflake, AWS etc.
• Understanding of Azure DB administration and the Azure data platform
• Experience with GIT source control
• Experience with CI/CD processes

Qualifications: BA in Data Analytics with 4 years of Experience