BIG DATA SOFTWARE ENGINEER JOB DESCRIPTION TEMPLATE

Big Data Software Engineer means owning the infrastructure that turns raw, high-volume data into decisions. Not dashboards - the pipelines, processing logic, and distributed systems underneath them. This role sits within platform or data engineering teams at technology companies that manage petabyte-scale workloads across cloud and on-premise environments, often serving both internal analysts and external client-facing reporting products. The engineer answers to product and engineering leadership while working directly alongside data scientists, BI teams, and infrastructure architects to keep latency low and data fidelity high.

As the Big Data Software Engineer, you design and maintain distributed data pipelines and analytical infrastructure that directly determine the speed and reliability of data delivery across the organization. You operate within a cross-functional engineering team, translating business requirements from stakeholders into scalable, production-grade solutions that support both real-time and batch processing needs.

Career Impact: Engineers who build production-ready distributed systems at scale develop a rare combination of pipeline architecture and machine learning integration experience that commands premium market value across cloud-native technology companies.

Business Impact: The data pipelines and processing systems this role owns determine whether client-facing reporting, financial analytics, and internal product decisions are grounded in accurate and timely information.

Growth Opportunity: Hands-on ownership of end-to-end data platform components - from ingest optimization to microservice architecture - creates a direct path toward senior data engineering, data architecture, or engineering lead roles.

• Design and implement distributed data processing pipelines to support real-time and batch analytical workloads at scale.
• Build and maintain platform infrastructure including databases and internal tooling to enable sustained growth and system reliability.
• Optimize data ingest, transformation, and filtering processes to improve throughput and reduce processing latency.
• Collaborate with data scientists and BI teams to integrate machine learning models and deliver analytical outputs to production.
• Review and troubleshoot existing system architectures to identify performance bottlenecks and data quality issues.
• Develop internal microservices and APIs to enable reliable data delivery across products and consumer teams.
• Partner with business stakeholders to translate reporting and analytics requirements into scalable technical solutions.
• Onboard new teams to platform capabilities and support the evaluation of technologies to meet evolving business needs.

• Bachelor's degree in Computer Science, Information Management, or a related field, or equivalent work experience.
• 2 or more years of software engineering experience, with hands-on work building or maintaining large-scale data processing systems.
• Demonstrated ability to write production-quality code in at least one statically typed language such as Java or Scala.
• Solid understanding of distributed systems concepts including fault tolerance, horizontal scaling, and data partitioning strategies.
• Experience designing and optimizing complex SQL queries and working with both relational and non-relational data storage models.
• Ability to design and implement real-time data processing logic using stream-based architectural patterns.
• Strong problem-solving skills with the ability to diagnose data pipeline failures and architect solutions independently.
• Effective communication skills to translate complex technical findings for non-engineering stakeholders including product and business teams.

• Experience supporting or executing a cloud migration initiative, including re-platforming data workloads to a major cloud provider.
• Familiarity with CI/CD methodology and microservices development practices applied to data engineering contexts.
• Exposure to machine learning model integration within production data pipelines, including feature engineering and model serving patterns.
• Experience in an agile engineering environment with demonstrated ability to contribute to cross-functional delivery teams alongside data scientists and QA engineers.

• Pipeline uptime percentage, measuring reliability of real-time and batch data delivery to downstream consumers.
• Mean latency per pipeline run, tracking processing speed against defined SLA thresholds for production workloads.
• Data ingest error rate, reflecting accuracy and completeness of transformed records entering the platform.
• Number of teams successfully onboarded to platform services within a defined rollout period.
• Code review pass rate on first submission, indicating engineering quality and adherence to production standards.
• Typical tools: Stream processing engines (commonly Spark Streaming or Flink), distributed storage (commonly Hadoop HDFS or HBase), query engines (commonly Hive or Presto).

• Base Salary Range: $115,000 to $165,000 depending on experience and location.
• Bonus: annual performance bonus typically 8 to 15 percent of base salary.
• Equity: stock options or RSUs common at growth-stage and public technology companies.
• Health Benefits: medical, dental, and vision coverage standard across most employers.
• PTO: 15 to 20 days annually, with many technology employers offering unlimited PTO policies.
• Common Perks: Remote or hybrid work options, conference and continuing education stipends, home office equipment allowance.

Figures are estimates based on general US market benchmarks and may be outdated. Adjust based on location, company size, and seniority level.

Background checks, including verification of employment history and education, are a standard condition of employment for this role. All qualified applicants will receive consideration without regard to race, color, religion, sex, national origin, disability, veteran status, age, or any other characteristic protected under applicable federal, state, and local law. Candidates requiring a reasonable accommodation during the application or interview process may request one at any time. Applicants must be authorized to work in the United States.

The Big Data Software Engineer owns the design and development of both bare-metal and cloud-based analytical solutions, building production-quality pipelines with Spark, Flink and Kafka Streams to extract optimal value from data. Working closely with customers and stakeholders in an agile environment, this engineer delivers measurable impact by reviewing, auditing and troubleshooting existing architectures while exploring emerging technologies to support client business needs.

Key Responsibilities

• Write and tune complex near-real-time and batch data pipelines with Spark, Flink and Kafka Streams.
• Apply knowledge of programming, machine learning and data modeling to build production-quality analytical solutions.
• Design and develop both bare-metal and cloud-based analytical solutions leveraging state-of-the-art technologies, tools and algorithms.
• Work closely with customers and stakeholders in an agile environment to extract optimal value from data.
• Review, audit and troubleshoot existing solutions and system architectures.
• Explore available technologies to provide business support to clients.

Required Qualifications

• University degree in Computer Science or equivalent.
• 2+ years of industrial experience as a software engineer.
• Familiarity with big data ecosystems including Spark, HBase and Hadoop (HDFS and Yarn).
• Strong ability to write production-quality code in Java, with basic knowledge of real-time processing systems like Spark Streaming, Kafka or Flink.
• Experience with Snowflake, Cloudera or cloud technologies is a plus, experience with Python or Scala is a plus.
• Strong problem-solving skills, proactive team player able to work independently in interdisciplinary projects.
• Strong command of English, German is a plus.

Embedded within a large-scale data engineering organization, the Big Data Software Engineer develops and implements code for big data development, data pipeline management and data security across a strategic cloud migration initiative. Working closely with business stakeholders and cross-functional big data teams, this engineer advances platform scalability by delivering production-ready end-to-end solutions and translating complex requirements into practical, innovative outcomes.

Core Functions

• Develop and implement code for big data development, data pipeline management and data security.
• Work with business stakeholders to understand requirements and design appropriate solutions.
• Engage with other big data teams to share and develop best practices.
• Participate in AWS POC and support big data to AWS migration as part of a strategic re-platform initiative.
• Provide problem-solving expertise and complex data analysis to understand data issues and deliver solutions.
• Support the production platform.

Qualifications and Experience

• Bachelor's degree in Computer Science, Information Management or related field, or equivalent IT work experience.
• Minimum 6 years in data engineering platforms, with at least 3 years hands-on using Hadoop Cloudera Platform, Spark, Scala, Hive, Elastic Search, Impala, SQOOP and Oozie.
• Proven experience implementing production-ready and highly scalable end-to-end solutions on a big data platform, experience in the HR domain is a strong plus.
• Advanced SQL knowledge, data analysis and data mining experience, understanding of CI/CD methodology and microservices development techniques.
• Experience with AWS or other cloud platforms and familiarity with BI tools is a strong plus.
• Ability to collaborate with business teams to translate requirements into practical and innovative solutions.

Reporting to the platform engineering leadership, the Big Data Software Engineer builds and maintains a large ClickHouse database and develops internal tools to automate system management, enabling faster growth and platform scalability. Partnering with product and data teams, this engineer creates microservices for data delivery and enhances observability UI, driving measurable improvements in customer support and data insight quality.

Primary Duties

• Build and maintain a large ClickHouse database to enable faster growth and platform scalability.
• Develop internal tools to automate and manage the system.
• Optimize data ingest, filtering and improvement.
• Utilize machine learning and data science principles to gain valuable insights from data.
• Onboard new teams to the platform and integrate third-party database visualization tools.
• Create microservices to enable data delivery.
• Enhance existing observability UI to better support customers.

Skills and Qualifications

• Strong knowledge of data structures, design patterns, and algorithms, with experience in OOP languages.
• Strong design and problem-solving skills with a bias for architecting at scale.
• Adaptable and proactive with a willingness to take ownership, open mind and passion for coding excellence.
• Strong communication skills.
• Ability to articulate complex issues clearly and engagingly.

Sitting at the intersection of data engineering and business intelligence, the Big Data Software Engineer powers client-facing, product performance and financial reporting by owning the design, implementation and support of major analytics components built on MapReduce and scalable data architectures. Operating across product, analytics and technology teams, this engineer shapes platform direction by selecting the right technologies and building tools and data sets that enable new analytics and media product offerings.

Duties

• Aggregate data sets to build out new product offerings related to analytics and media.
• Build tools, data sets and access methodologies to support business cases in a scalable way.
• Own design, implementation and support of major analytics components including definition of data to be captured, aggregated and made available.
• Help select the right technologies for the platform.
• Complete large-scale analytics enhancements using MapReduce and make incremental changes to existing analytics jobs.

Experience and Qualifications

• 3-5 years with highly scalable distributed systems using open source tools, with in-depth knowledge of the software development lifecycle including design, build, test, deploy and support.
• Strong software engineering fundamentals including OO design, unit testing, code reuse and code reviews.
• Experience building large-scale data processing systems and data warehousing solutions.
• Proficiency in Java, C++, PHP, Ruby or Python, familiarity with Hadoop, HBase, MapReduce, Kafka or Cassandra, experience with Maven, Hudson and GitHub.
• Knowledge of high-performance algorithm design, data ETL and data modeling, familiarity with Linux and Agile development methodology.

A key member of an interdisciplinary engineering team, the Big Data Software Engineer creates software to automate the distillation of massive data volumes and develops analytics for large heterogeneous data sets under moderate supervision. Collaborating across end-user groups and a fast-moving product team, this engineer enables better customer outcomes by applying creative approaches to distributed data processing and transforming raw data into focused, meaningful knowledge.

Functions

• Create software to automate the distillation of massive amounts of data into understandable information.
• Develop analytics for large heterogeneous data sets.
• Apply creative and innovative approaches to processing complex data in distributed systems.
• Work with end-users to ensure analytics transform data into focused and meaningful knowledge.
• Collaborate with a fast-moving team open to new ideas that improve customer products.

Education and Experience

• Bachelor's degree and a minimum of 2 years of prior related experience, or equivalent combination of education and experience.
• Working knowledge of Java, Python, C++ or C#, interest in Apache Spark, Apache Flink and Scala functional programming.
• Interest in microservice design principles, familiarity with REST, gRPC, AkkaHttp and Websockets.
• Interest in Docker, PCF, DC/OS or Kubernetes, interest in Git and Agile team collaboration tools.
• Ability to solve new problems collaboratively and creatively.
• Ability to obtain and maintain a DOD Top Secret/SCI Clearance requiring US Citizenship, applicable internship or research experience is a plus.

Scalable real-time infrastructure and high-quality data pipelines depend on the Big Data Software Engineer, who designs and implements distributed data processing pipelines using Hadoop/Spark, Hive and Presto while working closely with data scientists to develop machine learning models and algorithms. Based within a cross-functional team of developers, data scientists, QA engineers and BI professionals, this engineer strengthens platform performance by maintaining architecture focused on throughput, latency and continuous integration of new technologies.

Accountabilities

• Assist the team in designing and developing a scalable real-time infrastructure and data pipelines.
• Work closely with data scientists to develop machine learning models and algorithms.
• Design and implement distributed data processing pipelines using Hadoop/Spark, Hive and Presto.
• Integrate the analytics platform internally across products and teams.
• Focus on performance, throughput and latency to build and maintain architecture.
• Write APIs, tools and scripts, and stay current with new technologies and potential solutions.

Background and Experience

• At least 2 years of software development experience with hands-on work on large and complex datasets.
• Familiarity with big data architectures such as SMACK and Lambda Architecture.
• Strong SQL skills including query optimization, familiarity with Scala, Python or Java.
• Experience with Hadoop/Spark, Hive and Presto, knowledge of NoSQL databases such as Cassandra/ScyllaDB, ClickHouse, Elasticsearch and MongoDB.
• Experience with real-time processing frameworks like Spark Streaming, Flink or AWS Kinesis.
• True problem solver, proactive and self-directed, passionate about big data, data mining and data analysis technologies.

As the Big Data Software Engineer, this role leads the development of Data Analytics solutions that support existing business units and enable new services, encompassing data wrangling, statistical analysis, machine learning model support and prototype development on Linux-based infrastructure. The engineering team relies on this work to deliver actionable insights to stakeholders ranging from engineers to senior managers, while maintaining knowledge-sharing practices in close cooperation with the Airbus DTO Team.

Job Functions

• Install and manage new Linux and web services and big data tools on Linux servers.
• Collect data and perform data wrangling including transforming, cleansing and linking with other data.
• Provide data sets for machine learning models and determine relationships between data source attributes.
• Apply data mining techniques to perform statistical analysis.
• Develop and implement program prototypes in C#, Python, Java or similar languages.
• Support proof-of-concept projects and present results to different target groups from engineers to managers.
• Share and exchange knowledge with other teams in close cooperation with the Airbus DTO Team.

Minimum Qualifications

• Completed studies in Computer Science, Mathematics or equivalent engineering degree.
• Experience with data manipulation, cleansing and feature engineering, strong knowledge of statistical analysis including mean, standard deviation and correlations.
• Strong knowledge of machine learning, data mining, statistical analysis and modeling, programming skills in R, Python, Java, Matlab or similar, basic knowledge of Hadoop infrastructure including HDFS and Spark.
• Solid experience with SQL technologies such as SQL Server, MySQL, Oracle, SAP and BW, experience with NoSQL databases including HBase, MongoDB, ArangoDB and Cassandra, ETL concepts and tools.
• Knowledge of visualization tools such as RShiny, Spotfire and Tableau.
• Strong collaboration and knowledge-sharing skills across teams.
• Fluent in German and English.