BIG DATA ENGINEER JOB DESCRIPTION

When raw data volumes scale into petabytes and pipelines are missing or broken, analytics teams go dark and business decisions stall on incomplete information. The Big Data Engineer exists to prevent that failure, owning the design, development, and operation of end-to-end data pipelines that move data from ingestion through transformation to consumption. Sitting within data platform or engineering teams, the role spans both cloud and on-premise environments and requires fluency in distributed compute systems. Few individual contributors touch as many downstream stakeholders - data scientists, product owners, and business analysts all depend on the reliability of what this engineer ships.

As the Big Data Engineer, you design and maintain the data pipeline infrastructure that turns large-scale raw data into structured, analysis-ready assets powering business intelligence and machine learning initiatives. You operate within cross-functional engineering teams, partnering with data scientists, platform architects, and business stakeholders across cloud and on-premises environments to ensure data reaches the right destination with the right quality.

Career Impact: Mastering distributed compute systems, Data Lakehouse architecture, and end-to-end ETL pipeline design positions a Big Data Engineer as one of the most sought-after technical profiles across cloud-first organizations.

Business Impact: Without this role, petabyte-scale data lakes produce no usable signal - the Big Data Engineer is the direct reason analytics teams, ML models, and operational dashboards function at all.

Growth Opportunity: Engineers who build competency in ML Ops, streaming architectures, and cloud-native data platforms consistently move into Senior or Architect-level roles with expanded scope over platform strategy and team mentorship.

• Design and build data ingestion pipelines that collect structured and unstructured data from multiple source systems at scale.
• Develop ETL and ELT workflows that transform raw data into clean, validated, analysis-ready datasets meeting functional and non-functional requirements.
• Architect and maintain data pipeline infrastructure spanning batch and real-time processing to support analytics and machine learning workloads.
• Implement CI/CD pipelines and automated testing frameworks to ensure reliable, repeatable deployment of data engineering artifacts.
• Monitor data pipeline performance and resolve infrastructure issues, including root cause analysis of data quality failures and bottlenecks.
• Collaborate with data scientists, product owners, and business stakeholders to translate functional requirements into scalable technical solutions.
• Enforce data governance standards covering lineage tracking, access controls, schema management, and data retention policies.
• Support cloud migration initiatives by designing reference architecture documents and participating in proof-of-concept evaluations.

• Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical field, or equivalent work experience.
• 5 or more years of data engineering experience, with hands-on work across large-scale data lake or data warehouse environments.
• Demonstrated ability to design and implement distributed data processing systems handling high-volume, high-velocity workloads.
• Proficiency in writing, optimizing, and debugging complex SQL queries across relational and columnar data stores.
• Experience developing and maintaining ETL and data pipeline solutions using Python or Scala in production environments.
• Solid understanding of cloud platform services including storage, compute, and managed data processing offerings from at least one major provider.
• Familiarity with CI/CD practices, version control workflows, and automated testing in a software development environment.
• Strong written and verbal communication skills, with the ability to explain technical architecture decisions to non-technical stakeholders.

• Experience applying ML Ops principles to support model versioning, auditability, and deployment of machine learning systems at scale.
• Hands-on background in streaming data architectures, including real-time ingestion, event-driven processing, and low-latency pipeline design.
• Working knowledge of Infrastructure-as-Code or container orchestration practices used to automate deployment and scaling of data platform components.
• Exposure to data modelling techniques including schema evolution, partitioning strategies, and MPP optimization for high-volume analytical workloads.

• Pipeline uptime rate, measuring the percentage of scheduled jobs completing without failure across the production environment.
• Data freshness lag in minutes or hours, tracking how closely available data reflects the most recent source state.
• ETL job execution time, benchmarking transformation throughput against defined SLA thresholds for downstream consumers.
• Data quality error rate per pipeline run, counting schema violations, null failures, and rejected records as a proportion of total processed volume.
• Time to deploy new pipeline from requirements sign-off to production, reflecting engineering velocity and CI/CD maturity.
• Typical tools: Distributed processing frameworks (commonly Spark or Flink), workflow orchestration (commonly Airflow), and cloud data platforms (commonly Databricks or Snowflake).

• Base Salary Range: $115,000 to $165,000 per year depending on experience and location.
• Bonus: 5 to 15 percent annual performance bonus, common at mid to large technology employers.
• Equity: RSU grants typical at growth-stage and public technology companies, vesting over 4 years.
• Health Benefits: Medical, dental, and vision coverage, typically employer-subsidized for employee and dependents.
• PTO: 15 to 20 days per year, plus public holidays and sick leave.
• Common Perks: Remote or hybrid work options, annual learning and development budget, and conference attendance support.

Figures are estimates based on general US market benchmarks and may be outdated. Adjust based on location, company size, and seniority level.

Background checks, including verification of employment history and education, are a standard condition of hire for this position. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, disability, veteran status, or any other characteristic protected under applicable federal, state, and local law. Reasonable accommodations are available to applicants and employees with disabilities upon request. Candidates must be authorized to work in the United States without sponsorship unless otherwise stated in the specific posting.

The Big Data Engineer owns the full pipeline lifecycle, from defining data requirements and mining large-scale structured and unstructured data to deploying proof-of-concept machine learning systems in Azure-governed environments. Working across research teams and business units, the engineer governs DS and ML workflows to ensure auditability, version control, and data security at scale.

Key Responsibilities

• Define data requirements, gather and mine large scales of structured and unstructured data, and validate data using various data tools in the Big Data Environment.
• Design and develop big data applications based on business requirements defined by research teams or business units.
• Migrate or refactor big data proof of concept engagements to production applications using best practices and CI/CD standardization.
• Develop and orchestrate big data pipelines that turn large scale raw data into meaningful units of analysis pertaining to specific research or data science problems.
• Apply software engineering rigor and best practices to machine learning, including version control, CI/CD, and automation.
• Facilitate the development and deployment of proof-of-concept machine learning systems.
• Support model development, with an emphasis on auditability, versioning, and data security.
• Leverage cloud technologies, namely Azure, to develop, deploy, manage, and govern DS and ML workflows or supporting resources.

Required Qualifications

• Bachelor's degree in Computer Science, Math, or Scientific Computing preferred.
• At least 5 years of recent experience in data engineering or data-oriented software development using Big Data frameworks and tools.
• Fluency in Python, Bash, Pyspark, and SQL.
• Extensive experience in distributed compute environments, preferably Spark.
• Experience with Big Data Cloud Environments such as Snowflake and Databricks.
• Strong understanding of software testing, benchmarking, and continuous integration, with comfort in Linux administration.
• Experience developing and maintaining ML systems built with open-source tools, with MLFlow as a strong plus.
• Experience building custom integrations between cloud-based systems using APIs, with strong software engineering skills in complex multi-language systems.
• Experience working within Agile Software Development cycles.

Embedded within the Data and Design team ecosystem, the Middle Big Data Engineer builds and optimizes big data pipelines, jobs, and data sets while assembling complex datasets that meet functional and non-functional business requirements. Working closely with the Product Owner and cross-functional stakeholders, the engineer drives process improvements and scalable architecture that accelerate data delivery across the organization.

Core Functions

• Build and optimize big data pipelines, jobs, and data sets.
• Create and maintain an optimal data pipeline architecture.
• Assemble large and complex data sets that meet functional and non-functional business requirements.
• Identify, design, and implement internal process improvements including automating manual processes, optimizing data delivery, and re-designing jobs for greater scalability.
• Collaborate with stakeholders including the Product Owner and Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
• Propose and implement solutions for manipulating, processing, and extracting value from large datasets and message queuing, stream processing, and highly scalable big data stores.

Qualifications and Experience

• Skilled in working with SQL, RDBMS, Hadoop, and NoSQL databases.
• Strong background with Hadoop, Spark, and Kafka.
• Experience working with AWS and GCP.
• Skilled in Airflow, Spark-Streaming, and Flink.
• Experienced in supporting and working with cross-functional teams in a dynamic environment.
• Strong analytical skills related to working with structured and unstructured datasets.
• Able to propose and implement solutions effectively.

Reporting to architecture and engineering leadership, the Big Data Engineer translates complex functional and business requirements into technical solutions while building and maintaining technology platform architecture for both on-premise and cloud-based environments. Partnering with engineers and business stakeholders across teams, the engineer delivers data insights that enable next-generation Demand Planning forecasting and Supply Chain Management outcomes.

Primary Duties

• Build and maintain the technology platform architecture for both on-premise and cloud-based solutions.
• Participate in cross-functional efforts to design and develop the next generation of data platform for Demand Planning forecasting and Supply Chain Management.
• Perform requirements engineering to translate complex functional and business requirements into technical requirements, solutions, and design.
• Partner with engineers and business stakeholders from various teams to build data insights and help them achieve their business goals.
• Demonstrate experience in real-world IT solutions environments, including creating a product or IT solution in Big Data and Analytics primarily using Cloudera distribution.

Skills and Qualifications

• Bachelor's degree or equivalent in a technical field.
• Proficiency in solution architecture and expertise in big data processing and ETL technologies, with at least 3 years required.
• Experience in high-level programming languages using Scala and Java, mandatory, with Python as an added advantage.
• Proficiency with databases, data warehouse, data management, and SQL.
• Experience building high-quality end-to-end data solutions in an agile environment, from requirements through production.
• Experience with complex event processing using Kafka, Pulsar, and Spark Streaming, and batch processing using Hive, Impala, Kudu, and Spark.
• Experience in Snowflake as an added advantage.
• Strong communication skills and technical team leadership, mentorship, and collaboration.

Sitting at the intersection of cloud infrastructure and big data architecture, the Big Data Engineer conceptualises, implements, and automates both structured and non-structured data using tools such as Spark, Hadoop, Cassandra, Kafka, and Elastic. Operating across managed cloud services including AWS and Azure, the engineer supports renowned customers in agile environments where data extraction, warehousing, and semantic storage capabilities are central to business value.

Duties

• Conceptualise, implement, and automate both structured and non-structured data.
• Use data extraction systems for SQL and NoSQL databases, data lakes, and associated semantic data storage and data analysis systems for data warehouses and big data stores.
• Use associated managed cloud services such as AWS and Azure.
• Use the latest Big Data tools such as Spark, Hadoop, Cassandra, Kafka, and Elastic.
• Work in close cooperation with renowned customers across an agile environment.

Experience and Qualifications

• Bachelor's degree in Computer Science or a comparable field, with a focus on Big Data and Data Science during studies.
• First practical experience in a corporate environment gained through studies or work.
• Experience with SQL and NoSQL databases, data lakes, data warehouses, and big data stores.
• Experience with cloud services including AWS and Azure.
• Familiarity with Big Data tools such as Spark, Hadoop, Cassandra, Kafka, and Elastic.
• Experience in an agile working environment.

A key member of the data engineering team, the Big Data Engineer manages end-to-end data lifecycle across ingestion, transformation, and consumption using Databricks, Spark, and AWS cloud ecosystems handling multiple TBs and PBs of data volume. Collaborating across platform, ML, and DevOps disciplines, the engineer applies machine learning and statistical techniques to telemetry anomaly detection problems that directly inform business and operational decisions.

Functions

• Manage end-to-end data lifecycle including data ingestion, transformation, and consumption using Databricks, Spark, and cloud ecosystems.
• Build and maintain data pipelines for telemetry, logs, and real-time data ingestion through APIs.
• Apply machine learning and statistical techniques to time series classification and telemetry anomaly detection problems.
• Work with storage and ETL tools to handle multiple TBs and PBs of data volume use cases from ingestion to consumption.
• Deploy and manage solutions using AWS frameworks including SQS, Stream, Kubernetes, EC2, S3, Docker, and containers.
• Implement and maintain DevOps pipelines and CI/CD practices across code bases.
• Work with analytics and visualization toolsets and support monitoring using Elastic, Kibana, and Grafana.

Minimum Qualifications

• Minimum 10 years of Data Engineering experience, with at least 5 years in large-scale Data Lake and Platform ecosystems.
• Expertise in Data Lakehouse architecture and end-to-end Databricks techniques including Data Science components.
• Expert proficiency with Databricks, Spark, Python, SQL, Scala, Kafka, and Streaming.
• Strong knowledge of ML Ops techniques and model building at scale.
• Expertise in MapReduce, Hadoop, Hive, and Presto.
• Experience with structured data formats such as Avro, Parquet, Protobuf, and Thrift, and schema evolution concepts.
• Experience with analytics and visualization tools such as PowerBI and Tableau.
• Excellent oral and written communication skills, with a self-motivated and independent working style.

Scalable business intelligence across disparate systems depends on the Big Data Engineer, who manages data integration and analysis while building extensible data acquisition and integration solutions to meet functional and non-functional client requirements. Based within cross-functional product and engineering teams, the engineer delivers high-performance distributed data services that connect diverse sources and support advanced analytics for human resources and related domains.

Accountabilities

• Manage data integration and data analysis of disparate systems.
• Build extensible data acquisition and integration solutions to meet functional and non-functional client requirements.
• Implement processes and logic to extract, transform, and distribute data across one or more data stores from a wide variety of sources.
• Provide problem-solving expertise and complex analysis of data to develop business intelligence integration designs.
• Interface with internal product development teams and cross-functional teams including Product Management, Integration Engineering, Quality Engineering, and System Admin Teams.
• Work with remote and geographically distributed teams to build the right products using the right building blocks.

Professional Experience

• Minimum 6 to 8 years of experience in data integration projects, preferably related to human resources analytics.
• Strong knowledge of Data Warehousing and Data Lake concepts.
• Hands-on experience with Spark and Scala for a minimum of 5 years.
• Strong knowledge of Big Data querying tools such as Pig, Hive, and Impala.
• Strong experience integrating data from different file-storage formats like Parquet, ORC, Avro, and Sequence files.
• Strong experience with Java open-source and API standards, and experience with Oracle, SQL, and NoSQL data stores such as MongoDB.
• Experience in AWS Data Warehousing and database platforms preferred.
• Excellent communication, presentation, interpersonal, and analytical skills, with the ability to communicate complex concepts clearly to different audiences.

As the Big Data Engineer, this role designs and develops data ingestion and processing code using Python, Pyspark, Language R, and Hive on the Cloudera CDH Platform while supporting Cloud Migration POC work and automating deployment through shell scripts and Oozie. The data engineering team relies on this work to maintain reference architecture documents, ensure code reusability through peer reviews, and advance collaboration with the Apache open-source community.

Key Deliverables

• Design and develop data ingestion and processing code using Python, Pyspark, Language R, and Hive on Cloudera CDH Platform.
• Design and support Cloud Migration POC work.
• Create and update design specs and reference architecture documents to enable acceleration in solution development.
• Innovate new ideas, research related technology, develop new concepts, prototype, and deliver implementations on the Cloudera data platform.
• Participate in testing and peer code reviews to identify bugs and ensure reusability of code.
• Automate the deployment of solutions using shell scripts, Python, and Oozie.
• Work with IT change management groups to promote developed code and scripts from non-production to production environments.
• Collaborate with Apache community on Hadoop and other related open-source projects.

Background and Experience

• 7 or more years of work experience on Hadoop and Cloudera-based Data Lake solutions.
• 5 or more years of data ingestion and processing experience using Shell scripting, Python, Scala, Hive, Spark, and Pyspark.
• 1 to 2 years of work experience on cloud-native and cloud-agnostic data lake and data warehousing solutions, preferably AWS S3, AWS Redshift, and Snowflake.
• Proficient in Agile-based delivery approach.
• Work experience on AWS Glue, AWS Data Pipeline, Snowpipe, SSIS, and Java is a plus.
• Ability to work as part of a team, with self-motivation, adaptability, and a positive attitude.

Big Data Engineer delivers software components across the full development lifecycle, from reviewing process and data requirements through building, testing, deploying, and documenting solutions that meet FINRA standards. The work directly supports cross-team peer review processes, security policy adherence, and continuous improvement initiatives within a regulated financial technology environment.

Role Responsibilities

• Review and analyze process, system, and data requirements and specifications as part of a group.
• Review and contribute to component designs for a project, product, and program.
• Build, test, deploy, and document software components of a project, product, and program-level solution.
• Interface with other team members and teams in peer review of requirements, specifications, and software.
• Document and communicate development status promptly.
• Participate in collaborative resolution of defects and internal process improvement initiatives, providing feedback and suggestions.
• Assist with adherence to technology policies and comply with all security controls.

Education and Experience

• Bachelor's degree in Computer Science, Information Systems, or related discipline with at least 3 years of related experience, or equivalent training and work experience.
• Exposure to cloud-based Big Data technologies like Hadoop, Hive, and Spark.
• Experience with one or more programming languages such as Scala, Python, and Java.
• Familiarity with development best practices such as code reviews and unit testing.
• Strong written and verbal technical communication skills, with the ability to maintain focus and develop proficiency in new skills rapidly.

The Big Data Engineer owns the translation of functional requirements into technical specifications and supports the design and development of high-performing end-to-end big data applications that process batch and real-time massive volumes of data in a multi-tenancy cloud environment. Working across consulting project teams and diverse client settings, the engineer experiments with emerging technologies to bring early-adoption business value to wide and varied audiences.

Day-to-Day Responsibilities

• Translate functional requirements into technical requirements.
• Assist with design and development of high-performing and stable end-to-end big data applications to perform complex processing of batch and real-time massive volumes of data in a multi-tenancy cloud environment.
• Develop and implement tools for data acquisition, extraction, transformation, management, and manipulation of large and complex data sets.
• Participate in all aspects of development including design, development, build, deployment, monitoring, and operations.
• Research and experiment with emerging technologies and industry trends with a view to bringing business value through early adoption.

Requirements

• Bachelor's degree in Computer Science or a directly related IT discipline.
• Strong understanding of key concepts in data engineering as a foundation on which to build.
• Excellent communications skills, with an ability to communicate with impact and articulate complex information meaningfully to wide and varied audiences.
• A passion for technology and a strong interest in becoming a Big Data Engineer.
• Flexibility regarding local and international travel, and a flexible attitude towards working across different client projects.

Embedded within data platform teams, the Big Data Engineer designs, codes, and optimizes big data processes using Apache Spark while building low-latency streaming pipelines and batch processing systems that track data lineage and ensure data quality. Working closely with engineering and operations counterparts, the engineer applies lambda architecture principles and MPP optimization to deliver reliable, discoverable, high-scale data infrastructure.

Scope of Work

• Design, code, and optimize big data processes using Apache Spark or similar technologies with expert-level SQL skills.
• Build data pipelines and applications to stream and process datasets at low latency.
• Track data lineage, ensure data quality, and improve discoverability of data.
• Design and implement batch and stream data processing pipelines, optimizing distribution, partitioning, and MPP of high-level data structures.
• Apply knowledge of Engineering and Operational Excellence using standard methodologies.

Technical Qualifications

• Bachelor's degree or equivalent in Computer Science or a related technical field, with 5 or more years of meaningful experience.
• 5 or more years of experience with ETL, Data Modeling, and Data Architecture.
• Expert-level skills in writing and optimizing SQL.
• Experience with Big Data technologies such as Hive and Spark.
• Proficiency in one scripting language such as Python, Ruby, or Linux.
• Experience operating very large data warehouses or data lakes.
• Sound knowledge of distributed systems and data architecture including lambda architecture design.

Reporting to the development lead, the Big Data Engineer develops data pipelines in the Big Data technology stack using Python, PySpark, Hadoop, HDFS, and Kafka while supporting the definition of the development roadmap and its breakdown into user stories. Partnering with DevOps teams across build and run phases, the engineer ensures all implemented artifacts are covered with automated tests and integrated in the CI/CD pipeline to maintain reliable, production-grade data delivery.

Job Functions

• Develop data pipelines in the Big Data technology stack using Python, PySpark, Hadoop, HDFS, and Kafka.
• Implement connections to new source systems including Kafka, relational databases, REST API-based services, and file-based data sources.
• Ensure implemented artifacts are covered with automated tests and integrated in the CI/CD pipeline.
• Support the development lead in defining the development roadmap and breaking it down to user stories.
• Support the implementation of a generic ETL framework using Python, including test automation.

Knowledge, Skills and Abilities

• Several years of experience in Python and respective packages and frameworks in Data Engineering, including PySpark and PyTest.
• Experience in the Hadoop ecosystem as a developer, including Spark, HDFS, Hive, Impala, and Kafka.
• Experience with common interfaces to data sources and formats including Kafka, REST API, relational databases, file shares, JSON, Protocol Buffers, and Parquet.
• Familiarity with CI/CD toolchains including GIT and Jenkins.
• Familiarity with relational databases and SQL.
• Proactive, supportive, and open to new approaches, with a sense of ownership across build and run phases in a DevOps team.

Sitting at the intersection of streaming data engineering and cloud platform development, the Big Data Engineer develops and enhances Big Data pipelines using Nifi, Kafka, Spark, and Flink-SQL for processing structured and unstructured data while automating monitoring and alerting based on workflow needs. Operating across data science, visualization, and web development teams, the engineer advances CI/CD best practices and explores emerging technologies aligned with the big data technology roadmap.

Activities

• Develop and enhance Big Data pipelines using Nifi, Kafka, Spark, and Flink-SQL for processing of structured and unstructured data.
• Proactively automate monitoring and alerting based on workflow needs.
• Develop and enhance Hadoop workflows including data ingestion, extraction, and processing of structured and unstructured data.
• Engage in proof of concepts, technical demos, and interaction with customers and other technical teams.
• Work to utilize common CI/CD best practices across different teams including data scientists, data visualization analysts, and web developers.
• Explore and develop skills in newer technologies according to the big data technology roadmap.

Position Requirements

• Working experience with Big Data event streaming technologies.
• Applied experience with Kafka and Nifi.
• Applied experience with Scala and Python.
• Familiarity with SQL, Azure, and AWS.
• Experience with Spark, Flink-SQL, and the Cloudera platform or products such as CDP, Nifi, Kafka, and Minifi as a plus.
• Familiarity with Azure DevOps, Git, and UNIX or UNIX-like systems.
• Experience with SCRUM or other similar Agile frameworks.

A key member of the Azure cloud engineering team, the Big Data Engineer develops data processing pipelines and tests using Spark and Scala while monitoring and managing the Azure cloud platform and associated components for data ingestion, transformation, and processing. Collaborating across software engineering disciplines, the engineer diagnoses and resolves complex data infrastructure problems, including performance tuning and optimization, to maintain stable and well-documented production systems.

Work Activities

• Develop data processing pipelines and tests using Spark and Scala.
• Monitor and manage the Azure cloud platform and associated components for data ingestion, transformation, and processing.
• Diagnose, isolate, and resolve complex problems pertaining to data infrastructure, including performance tuning and optimization.
• Design and write programs according to functional and non-functional requirements.
• Develop and maintain technical documentation, and follow established configuration and change control processes.

Qualifications and Experience

• University-level education or equivalent.
• 2 or more years of experience working as a Software Engineer.
• 1 or more years of experience in functional programming, preferably Scala.
• 1 or more years of experience in Apache Spark 2.0.
• Knowledge of design patterns and ability to write complex SQL queries.
• Good understanding of distributed computation and Bash scripting.

Sitting at the intersection of software engineering and big data application development, the Big Data Engineer builds and delivers complete applications by writing high-quality source code, performing unit and integration testing, and troubleshooting and debugging within Agile and CI/CD frameworks. Operating across a team-oriented and collaborative environment, the engineer evaluates and reprogram existing applications to add new features and maintain system stability across Core Java, Hadoop, Spark, Hive, and Kafka stacks.

Core Responsibilities

• Understand business requirements and how they translate into application features.
• Write high-quality source code to program complete applications within deadlines.
• Perform unit, integration, functional, and non-functional testing.
• Troubleshoot and debug applications.
• Evaluate existing applications to reprogram, update, and add new features.

Minimum Qualifications

• Strong object-oriented skills including strong design pattern knowledge.
• Experience working with Agile and CI/CD.
• Core Java, Hadoop, Spark, Hive, HBase, Phoenix, Kafka, and SQL expertise required.
• DevOps experience is a major plus.
• Ability to work in a team-oriented and collaborative environment, connecting with people and quickly building trust with others.

As the Big Data Engineer in R&D Innovation, this role designs and implements the Enterprise Big Data Platform for GSK using container and cloud technologies, including auto-healing automated infrastructures and secure anonymization data systems built with declarative programming languages. The GSK R&D team relies on this work to create holistic data views that power AI and ML workflows for drug discovery acceleration and manufacturing process optimization across an Agile development lifecycle.

Leadership Responsibilities

• Design automated infrastructures that create new auto-healing capabilities.
• Create and integrate storage technology and DFS independency into the solution landscape.
• Develop data pipelines leveraging DevOps standards and CI/CD to build Data Engines.
• Create secure and private anonymization data systems using declarative programming languages to interface between Data Silos, Data Engines, and Graph Databases for AI and ML workflows.
• Create holistic data views through ingestion, cleaning, linking, harmonization, and contextualization of multiple systems to enable AI and ML work on complex high-value problems.
• Participate actively in all stages of the project lifecycle from ideation to industrialization in an Agile development environment, including creating POCs, POVs, and MVPs.

Experience and Qualifications

• Bachelor's degree in Engineering, Mathematics, Statistics, or Computer Science.
• Extensive experience as a full-time software engineer with expertise in Data Engineering or Site Reliability Engineering.
• Expertise with non-imperative paradigms including Scala, Haskell, F#, TypeScript, or OPA Rego.
• Experience working on Big Data platforms, preferably Spark, and deploying solutions on Cloud Platforms, preferably Azure or GCP.
• Infrastructure-as-Code experience with Terraform, Ansible, or Cloud templates.
• Expertise with container technologies including Kubernetes, Helm, and Docker, and professional DevOps experience with Jenkins, Azure DevOps, and CI/CD.
• Experience building and maintaining APIs, streaming data with Apache Kafka, and operating in a highly regulated and secure environment.
• Experience with cryptography and cybersecurity, and ability to design and implement logging, tracing, and application monitoring systems.

Big Data Engineer delivers value by designing, developing, constructing, and maintaining complete data management and processing systems, from raw data ingestion through transformation and storage, to fulfill functional and non-functional business needs. Serving as the infrastructure foundation for analytics and business intelligence, the role integrates diverse programming languages and tools to improve data quality, reduce system complexity, and enable data-driven decisions across the organization.

Operational Focus

• Design, develop, construct, install, test, and maintain complete data management and processing systems including data pipelines.
• Aggregate and transform raw data from a variety of data sources to fulfill functional and non-functional business needs.
• Discover opportunities for data acquisition and explore new ways of using existing data.
• Create data models to reduce system complexity and increase efficiency and reduce cost.
• Optimize and monitor performance by automating processes, optimizing data delivery, and re-designing architecture to improve performance.
• Propose ways to improve data quality, reliability, and efficiency of the whole system.
• Build complete infrastructure to ingest, transform, and store data for further analysis and business requirements.
• Create complete solutions by integrating a variety of programming languages and tools together.

Background and Experience

• Bachelor's degree in Computer Science, Computer Engineering, or other technical discipline, or equivalent work experience.
• Experience in software development or database work, with familiarity in data modelling, big data frameworks, ETL, and SQL.
• Familiarity with agile or other rapid application development methods.
• Experience with object-oriented coding in a variety of languages, and knowledge of key data structures and algorithms.
• Experience with relational database internals including query processing and query planning, or other data processing infrastructure.
• Knowledge of data modelling and different data structures, and familiarity with monitoring, backup, and disaster recovery of data systems.
• Strong analytical and communication skills for a collaborative environment.

The Big Data Engineer refines raw data at scale by gathering, processing, and converting unstructured data into structured formats while managing schema, data access, and retention policies in close collaboration with operation and engineering teams. Working closely with both operational and product teams, the engineer monitors data performance, tracks product quality trends, and advises on scalability solutions that keep production systems healthy and analytically actionable.

Performance Expectations

• Gather and process raw data at scale, and process unstructured data into structured data while managing schema of new data.
• Manage data access to protect data in a safe way and define data retention policies.
• Read, extract, transform, stage, and load data to selected tools and frameworks as required.
• Perform tasks such as writing scripts and SQL queries to analyze processed data.
• Work closely with the operation team to advise on solutions for service scalability, healthy monitoring, and refining optimization by data analysis.
• Work closely with the engineering team to monitor product performance and track product quality trends.
• Monitor data performance and modify infrastructure as needed.

Required Qualifications

• 3 or more years of recent experience in data engineering.
• Bachelor's degree or more in Computer Science or a related field.
• Experience on Cloudera CDH platform, Spark programming, Impala SQL, and data analysis via Hive.
• Strong knowledge of and experience with statistics and data management showing flawless execution and attention to detail.
• Programming experience in Python, Java, or Scala, with willingness to learn new programming languages.
• Deep knowledge of data features engineering, data mining, machine learning, and information retrieval.
• Experience processing large amounts of structured and unstructured data from multiple sources, and in production support and troubleshooting.
• Experience in C, Linux Shell, JavaScript, or other programming languages is a plus.
• English language required, Mandarin is a plus.

The Big Data Engineer advances enterprise analytics capabilities by designing connected data solutions that translate business requirements into innovative technical implementations, with heavy emphasis on automation, continuous process improvement, and streamlined data ingestion and curation processes. Working across business teams and various data domains, the engineer mentors team members, enables self-service analytics for business partners, and defines standards that support production SLAs and long-term development lifecycle maturity.

Strategic Responsibilities

• Work closely with business teams and various data domains to design connected data solutions that enable advanced analytics.
• Design, develop, and implement complex Big Data projects and technical solutions in support of maturing business intelligence and analytic capabilities.
• Translate business requirements into innovative technical solutions with a heavy emphasis on automation, continuous process improvement, reusability, and streamlining data ingestion and curation processes.
• Mentor and guide team members to develop their technical competency and work with business partners to enable self-service capabilities to speed business insights.
• Provide ongoing operations and support for production systems to meet defined SLAs.
• Research modern technologies to solve unique challenges, and define standards and best practices for an end-to-end development lifecycle.

Education and Experience

• Bachelor's degree and 5 years of experience, or high school diploma and 9 years of experience.
• 2 or more years of experience in supporting and developing data ingestion solutions using StreamSets.
• 5 or more years of experience in supporting and developing data pipelines using Spark and PySpark.
• 6 or more years of experience with Python.
• 4 or more years of experience with SQL, Hive, Impala, and Kudu.

Senior Big Data Engineer builds the infrastructure and abstractions that enable engineers and data scientists to craft scalable ETL pipelines for metrics, analysis, machine learning, and dashboard visualizations across a resilient, high-availability platform. The work directly supports ops team monitoring, data collection from multiple sources and stores, and the delivery of analytics and machine learning platforms that inform business decisions at scale.

Ownership Areas

• Build infrastructure and abstractions that enable engineers and data scientists to craft scalable ETL pipelines for metrics, analysis, machine learning, and dashboard visualizations.
• Work closely with the ops team to monitor and tune existing infrastructure.
• Make intuitive decisions about what services, frameworks, and capabilities need to be in place before they are needed.
• Build and maintain a data collection system that robustly extracts meaningful data from multiple sources and data stores.
• Build analytics and machine learning platforms to collect, store, process, and analyze huge sets of data.

Technical Qualifications

• Bachelor's degree in Computer Science, Information Systems, or a related technical field.
• 3 or more years of experience in Data Infrastructure, with Cloud Software experience a plus.
• Experience in crafting and scaling data infrastructure, models, and pipelines.
• Hands-on experience with processing tools including Spark, Flink, Hadoop, and Lambda, messaging tools including Kafka, Zookeeper, and Pulsar, and storage tools including Hive, MongoDB, Athena, Phoenix, Splice, Redshift, and DynamoDB.
• Experience with machine learning platforms such as Sagemaker, H2O, Keras, and NumPy.
• Programming experience in Scala, Java, or Python, with the ability to own a project from inception to completion.
• Open and active in sharing knowledge, with excellent communication skills.

Embedded within product and specialist department teams, the Big Data Engineer conceives, plans, and develops Big Data solutions while porting on-premises systems to the cloud and supporting the implementation of machine learning solutions. Working closely with departments and product teams, the engineer evaluates and optimises existing solutions, cleanses and enriches incoming data streams, and integrates results into a central database that enables analysis, reporting, and machine learning at scale.

Delivery Expectations

• Conceive, plan, and develop Big Data solutions.
• Answer a wide range of questions from specialist departments and product teams.
• Analyse and optimise existing solutions and adapt them to changing requirements.
• Use Big Data tools to automatically evaluate incoming data streams and make recommendations and control activities.
• Cleanse, enrich, and integrate incoming data into the central database to make it available for analysis, reporting, and machine learning.
• Help port on-premises solutions to the cloud and support the implementation of machine learning solutions.

Skills and Qualifications

• Knowledge of messaging systems like Kafka or RabbitMQ, and various SQL and NoSQL databases.
• Experience with data integration from different sources and practical experience with programming languages such as Java, Python, Scala, PHP, or JavaScript.
• Experience with Big Data tools like MapReduce and Hive or similar.
• Familiarity with the design and optimization of complex data flows, data models, and architectures for data processing.
• Ability to handle complex tasks while keeping the bigger picture in view, questioning existing methods and analysing potential risks.
• A collaborative team player who enjoys contributing skills and working with others.

The Big Data Engineer guides data ingestion and ETL development within a game development studio environment, writing code to ingest data from various sources, clean and transform it into clear data models, and implement algorithms at scale as needed. Working closely with game development studios, the engineer ensures data is logged consistently and pipelines are optimized for performance to support BI and data warehouse operations.

What You'll Do

• Write code to ingest data from various sources.
• Work with game development studios to log data in a consistent and complete manner.
• Write ETLs to clean and transform data into clear data models.
• Optimize pipelines for performance and implement algorithms at scale as needed.
• Implement new technologies as needed and work with large volumes of data.

Experience and Qualifications

• 2 or more years of total DWH and BI experience with Oracle.
• Expertise in ETL processes design and implementation, and data lifecycle management.
• Confident knowledge in data querying and transformation using SQL and Oracle PL/SQL.
• Experience with versioning systems such as SVN and Git.
• Experience with Hadoop ecosystem including HDFS, Impala, and Hive as a strong plus.
• Performance tuning skills in any DBMS.
• Communicative level of Russian.
• Ability to work with and finalize requestor's requirements.