BIG DATA DEVELOPER JOB DESCRIPTION

A Big Data Developer writes the code that decides how terabytes of data move, transform, and land reliably. The role owns the design and maintenance of distributed data pipelines, data lake infrastructure, and ingestion frameworks that serve downstream analytics and application teams. Governance standards such as data lineage documentation and privacy compliance are also in scope, not a side concern. Few engineering seats carry this combination of architectural ownership and daily pipeline accountability.

As the Big Data Developer, you design and maintain enterprise-scale distributed data pipelines and data lake infrastructure that deliver reliable, governed data to analytics and application consumers across the organization. You work alongside data architects, data scientists, and business analysts, contributing pipeline architecture decisions that directly shape the performance and coverage of the company's analytical capabilities.

Career Impact: Hands-on ownership of distributed ingestion frameworks and data lake architecture builds the kind of depth that advances a developer toward Data Architect or Principal Engineer roles in the distributed systems market.

Business Impact: When pipelines fail or data quality degrades, every downstream report, model, and application that depends on them breaks — this role keeps that from happening at enterprise scale.

Growth Opportunity: Exposure to both batch and real-time streaming workloads, combined with cloud infrastructure experience, expands market value into the high-demand intersection of data engineering and platform engineering.

• Design and maintain distributed data pipeline architecture spanning batch and real-time streaming workloads to meet functional and non-functional requirements.
• Build ingestion frameworks and ETL processes that move and transform large data volumes from diverse source systems into the data lake.
• Architect data structures, schemas, and data models that support analytics consumption and downstream application needs.
• Implement data governance practices including lineage documentation, data quality controls, and privacy and security requirements.
• Collaborate with data architects, data scientists, and business analysts to translate requirements into engineering deliverables.
• Review code quality and enforce technical standards across development and testing cycles, including unit, integration, and UAT phases.
• Monitor and performance-tune long-running queries, jobs, and pipeline components to meet throughput and latency targets.
• Evaluate emerging big data technologies and lead proof-of-concept work to drive adoption of improved approaches.

• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or equivalent work experience.
• 3 or more years of hands-on data pipeline development experience, with demonstrated delivery of production-grade distributed data solutions.
• Proficiency in distributed data processing concepts including batch processing, stream processing, and data lake architecture patterns.
• Strong command of SQL and the ability to write and optimize complex queries, stored procedures, and performance-tuned transformation logic.
• Experience with data modeling, schema design, and data governance practices including lineage and data quality management.
• Demonstrated ability to work within Agile or Scrum delivery teams, contributing to sprint ceremonies, estimation, and iterative delivery.
• Strong analytical and problem-solving skills with attention to detail and accuracy in data handling.
• Effective verbal and written communication skills for cross-functional collaboration with technical and non-technical stakeholders.

• Experience designing or operating cloud-based data infrastructure on a major public cloud provider, with exposure to managed storage and compute services.
• Background in containerization, orchestration platforms, or infrastructure-as-code practices that support reproducible pipeline deployment.
• Familiarity with machine learning model integration or ML pipeline support as a consumer or provider of feature data.
• Experience mentoring junior developers, conducting code reviews, or providing technical guidance within a delivery team.

• Pipeline reliability rate, measuring the percentage of scheduled pipeline runs completing without failure or manual intervention.
• Data freshness lag, tracking average time between source event and landing in the consumption layer across ingestion pipelines.
• Query and job optimization improvement, measured by reduction in average run time after tuning interventions.
• Data quality pass rate, reflecting the percentage of records meeting defined schema, completeness, and accuracy rules on ingest.
• Sprint delivery rate, measuring the percentage of committed story points completed per Agile sprint.
• Typical tools: Distributed processing frameworks (commonly Spark or Hadoop); workflow orchestration (commonly Airflow or Autosys); cloud storage and compute (commonly AWS or Azure)

• Base Salary Range: $110,000 to $155,000 per year depending on seniority and location.
• Bonus: Annual performance bonus of 5 to 15 percent of base salary.
• Equity: Stock options or RSUs offered at mid-to-senior level in growth-stage and public companies.
• Health Benefits: Medical, dental, and vision coverage for employee and dependents.
• PTO: 15 to 20 days annually plus standard public holidays.
• Common Perks: Remote or hybrid work flexibility, professional development budget, and conference attendance support.

Figures are estimates based on general US market benchmarks and may be outdated. Adjust based on location, company size, and seniority level.

Work authorization in the United States is required, and employment is contingent on successful completion of a background check. Reasonable accommodations for qualified individuals with disabilities are available throughout the application and employment process upon request. All applicants will be considered without regard to race, color, religion, sex, national origin, age, disability, veteran status, genetic information, sexual orientation, gender identity, or any other characteristic protected under applicable federal, state, or local law.

The Big Data Developer owns the design and implementation of SQL-driven regulatory reporting pipelines within the Wholesale Banking Credit Technology team. Working closely with data architects, senior leadership, and global team members, the developer delivers data mapping specifications and lineage documentation that support compliant, end-to-end credit data solutions.

Key Responsibilities

• Perform in-depth data analysis activities and develop low-level and high-level design strategies.
• Implement complex regulatory driven enhancements and develop SQL scripts for reporting and ad-hoc analysis.
• Collaborate with data architects and global team members to build effective solutions.
• Document lineage and data processes, design and develop for regulatory deliverables in Wholesale Banking Credit Technology.
• Work with Senior Leadership Team to define the Program Roadmap and organize for execution.
• Design data mapping specifications, data feed specifications and perform data analysis and comparison.
• Connect across various applications and business processes, document data flows and identify gaps.

Required Qualifications

• 5+ years of experience with data analysis and various databases.
• 3+ years of experience with big data platform Hadoop and tool sets.
• Strong PL/SQL and SQL development skills to aid data analysis.
• Ability to write and analyze complex queries and stored procedures.
• Experience working in Agile/Scrum teams.
• Good verbal, written and interpersonal communication skills.
• Attention to detail and accuracy with excellent analytical and problem-solving skills.

Embedded within a global engineering team, the Big Data Developer designs and develops analytical models using Hadoop, Spark/Scala, Hive, Python and Kafka to build scalable data and graph ingestion processes. Working closely with onshore and offshore developers, data scientists, and business analysts, the developer delivers robust ETL pipelines and graph user interface solutions that advance consistent, high-quality data outcomes across the organization.

Core Functions

• Design and develop analytical models using big data technologies like Hadoop, Spark/Scala, Hive, Python and Kafka.
• Build data and graph ingestion processes and support graph user interface development using DataStax, Gremlin, D3 and Angular.
• Design robust code for performance, reuse and supportability with appropriate documentation.
• Support Hadoop cluster including ingestion and migration of feeds, ETL processes, scripts, Autosys and machine learning models.
• Implement organization-mandated SDLC and Agile processes with built-in controls for consistent delivery.
• Train new members including code reviews and mentoring for onshore and offshore teams.
• Work closely with offshore and onsite development teams.

Qualifications and Experience

• Hands-on experience as a big data developer with a solid record of multiple successful deliveries.
• Solid exposure to Hadoop, Spark/Scala, Python, Angular, graph technologies and Kafka.
• Knowledge of UNIX, Autosys and Java.
• Experience with Hive, Impala, Oracle or SQL Server database technologies.
• Hands-on experience in Ab-Initio ETL tool and Tableau business intelligence tool.
• Experience working with global teams including data scientists, business analysts and developers.
• Ability to influence architecture and design and communicate effectively to senior management and stakeholders.

Reporting to the engineering leadership, the Big Data Developer builds an event sourcing platform that serves as the company's main information hub, designing data structures, ETL pipelines, and a big data analytics environment to store and process terabytes of data. Partnering with vendors, external consultants, and internal stakeholders, the developer executes data governance practices that keep company data secure, transparent, and high quality.

Primary Duties

• Build an event sourcing platform as the main information hub in the company.
• Design and develop data structures, pipelines and ETL processes to move and transform terabytes of data between applications.
• Develop a big data analytics environment including data lake and computation infrastructure.
• Execute data governance practices to keep data secure, transparent and in good quality.
• Cooperate with vendors and external consultants.

Skills and Qualifications

• 2+ years of experience in development, deployment and operation of IT systems.
• 1+ years of professional programming in Scala, with Python as a nice-to-have.
• Practical experience with big data technologies from Apache stack such as Spark, Hadoop or Kafka.
• Excellent understanding of relational databases and practical SQL.
• Basic experience with Linux systems, especially CLI and basic bash.
• Knowledge of system integration techniques such as web services, API and messaging.
• Good understanding of computer science fundamentals including algorithms, complexity and performance optimization.

Sitting at the intersection of infrastructure evaluation and big data processing, the Big Data Developer converts complex functional requirements into detailed designs and assesses existing solutions for AWS-hosted environments. Operating across data processing, aggregation, and transformation workstreams using PySpark and Apache Hadoop, the developer delivers performance-optimized Python scripts and Hive and Impala solutions within an Agile team setting.

Duties

• Assess requirements and evaluate existing solutions for infrastructure support.
• Convert complex functional requirements into detailed designs.
• Evaluate and use hosted solutions on AWS.
• Handle exception handling and performance optimization on Python scripts.
• Perform data processing, aggregation and transformation using PySpark.

Experience and Qualifications

• 4 to 8 years of experience in a related role.
• Knowledge of Hive and Impala.
• Experience working within an Agile environment.
• Working experience in PySpark for data processing, aggregation and transformation.
• Familiarity with data loading tools including Sqoop.
• Required skills: Apache Hadoop, with AWS Big Data and PySpark as nice-to-have.
• High-level analytical, problem-solving, project management and communication skills.

A key member of the Commercial Lending Services Technology team, the Big Data Developer analyzes business and technical requirements to design CI pipelines and big data solutions supporting the Wholesale Lending Transformation Asset Management effort. Collaborating across business and technology partners, DevOps leads, and data engineering stakeholders, the developer assures quality, security, and compliance across large-scale SOR applications and ML/AI platform environments.

Accountabilities

• Analyze business and technical requirements to determine system design, potential issues and impact analysis.
• Provide individual and project support for complex projects and adhere to project timelines and technical deliverables.
• Assure quality, security and compliance requirements are met for developed applications.
• Work with business and technology partners to prevent or solve availability and performance problems.
• Research new tools and technologies and lead proof-of-concepts to drive adoption.
• Design and develop CI pipelines with Jenkins, Ansible, uDeploy and other DevOps tools.

Technical Qualifications

• Big data skills working in Spark, data ingestion and DevOps on Cloudera and Hortonworks.
• ML/AI experience building Conda environments and working with IBM Watson ML product.
• Python and Unix scripting experience including deploying Python packages and building Anaconda environments.
• Knowledge of admin monitoring tools for Spark and data ingestion.
• Knowledge of Jupyter, JupyterLab kernels, R Studio and H2O.

Scalable and compliant data infrastructure depends on the Big Data Developer, who contributes to the core design of data architecture, models, and schemas while building and maintaining ETL pipelines that serve real-time streaming analytics and big data capabilities. Based within a cross-functional stakeholder environment, the developer implements data privacy requirements, manages database governance, and drives internal process improvements that strengthen the organization's data foundation.

Role Responsibilities

• Contribute to the core design of data architecture, data models and schemas.
• Design, build and maintain optimal data pipeline architecture for ETL from external APIs, data streams and data stores.
• Design and maintain the foundation for ingesting data and providing frameworks for operating on that data.
• Design and maintain the foundation for real-time streaming analytics and big data analytics capabilities.
• Identify and implement internal process improvements including automating manual processes and optimizing data delivery.
• Work with stakeholders to assist with data-related technical issues and support their data foundation needs.
• Implement data privacy and data security requirements to ensure solutions comply with security standards.
• Manage database governance and support team to improve database management and resolve related issues.

Position Requirements

• Proficient with ANSI SQL relational databases including Oracle, SQL Server, PostgreSQL and MySQL.
• Proficient with NoSQL databases including MongoDB and Cassandra.
• Proficient in ETL development standards, data modeling, schema design, data governance and OLAP analytic capability.
• Proficient in SQL optimization, performance tuning for big data and storage procedures.
• Proficient in Python for data pre-processing across text, CSV, Excel and web data sources.
• Experience with big data frameworks including Hadoop, HBase, Hive and Pig is a plus.
• Experience with Spark including RDDs, DataFrames, Spark Structured Streaming and SparkSQL is a plus.
• Agile mindset with good teamwork spirit and good English reading and writing skills.

As the Big Data Developer, this role develops and supports Cloudera Data Platform applications and translates business requirements into robust technology solutions for a global financial advisory firm. The IT department relies on this work to maintain cluster performance, expand Hadoop and Spark ecosystem capabilities, and ensure adherence to SDLC, information security, and compliance standards across the organization.

What You'll Do

• Take part in development projects on Cloudera Data Platform.
• Support existing Cloudera applications and monitor performance and tune clusters including Hive, ETL jobs, Sqoop, NiFi and Spark.
• Translate business requirements into efficient and robust technology solutions with business and technology partners.
• Adhere to policies and procedures including SDLC, architecture standards, information security and compliance.
• Implement additional Hadoop and Spark ecosystem components to meet evolving business needs.
• Develop training materials and provide training to internal IT and business users.
• Evaluate new releases of Cloudera platform and provide recommendations on upgrades and impact analysis.

Education and Experience

• Bachelor's degree or equivalent in MIS, Computer Science, Engineering or related discipline.
• Cloudera or Hortonworks Developer Certification such as CCP Data Engineer, CCA Spark and Hadoop Developer or Hortonworks Certified Developer.
• Demonstrated experience in data modelling, design and implementation of adding new datasets into the cluster.
• Knowledge of Linux and Windows servers and bash shell scripting.
• Knowledge of relational database structures, theories and practices.
• Proficiency in Cloudera Data Platform, Sqoop, NiFi, Hive, HiveQL, Impala, Ranger, Spark and Spark SQL on Microsoft Azure.
• Proficiency in one or more of Java, Scala or Python, with CI/CD, Kafka, tokenization or machine learning as a plus.

Big Data Developer Lead produces and delivers production-quality software solutions by analyzing business requirements alongside business analysts and conducting code reviews across the full SDLC. The work directly supports a financial services technology platform, enabling scalable Hadoop-based data pipelines, Kafka streaming integration, and Azure-connected architectures that meet evolving non-functional and operational requirements.

Strategic Responsibilities

• Analyze business requirements with business analysts in a small and dynamic team.
• Design, plan and deliver production quality solutions using modern programming languages.
• Write automated tests as part of the software lifecycle.
• Continuously improve and drive non-functional requirements of platforms for sustainability and maintainability.
• Provide technical expertise and recommendations in assessing new software projects and initiatives.
• Conduct code reviews and test software, and participate in application architecture and design across the SDLC.
• Ensure proper operational controls and procedures are implemented to facilitate the move from test to production.

Background and Experience

• Excellent hands-on programming skills in Hadoop and related ecosystem components including HDFS, Hue, Hive, Impala and Spark.
• Advanced working knowledge of SQL and experience in DWH and ETL implementation in Hadoop and traditional RDBMS.
• Experience building and optimizing big data pipelines, architectures and data sets in batch and real-time workflows on Hadoop.
• Experience and knowledge in Python, Kafka streaming integration and Azure preferred.
• Strong analytical, problem-solving and synthesizing skills with excellent verbal and written communication.
• Experience with Agile development methodologies across multiple substantial project tasks.
• Comfortable working independently and in a team-oriented, collaborative environment.

The Big Data Developer owns the design and maintenance of PySpark, Kafka, and Spark Streaming pipelines on Hortonworks, developing batch and real-time data load jobs from diverse sources into Hadoop data lake environments. Working closely with QA, UAT, and scrum teams, the developer participates in story point estimation, unit and integration testing, and production deployment to deliver high-quality, performance-tuned data solutions.

Key Deliverables

• Design, architect and support data and ETL pipelines and recommend improvements.
• Develop and maintain optimal data pipeline architecture using PySpark, Kafka and Spark Streaming on Hortonworks.
• Design ETL jobs to read data from Hadoop and pass to downstream applications.
• Develop batch and real-time data load jobs from a broad variety of data sources into Hadoop.
• Analyze vast data stores, uncover insights and performance-tune long-running queries and jobs.
• Participate in scrum calls, story point estimation and own the development piece.
• Develop test cases, perform unit testing and integration testing, and support QA, UAT and production deployment.

Minimum Qualifications

• Bachelor's degree in Computer Science, Information Technology or Engineering, with a Master's degree in Finance or IT as a plus.
• Hands-on experience designing and programming big data tools and technologies including Hortonworks distribution.
• Must have hands-on experience in PySpark, Kafka and Spark Streaming for ETL on big data lake.
• Must have data architecting and data modeling skills with use of Erwin as a data modeling tool.
• Strong UNIX shell scripting and Python scripting experience.
• Exposure to IAAS providers such as Google Compute Engine, Microsoft Azure or Amazon AWS is a plus.
• Experience in Agile methodology with knowledge of standard methodologies and best practices in a big data environment.

Embedded within the Bell Business Intelligence Big Data Team, the Big Data Developer develops high-performance data processing pipelines on the Hadoop platform while engaging in peer code reviews and proactively recommending emerging technologies. Working closely with business analysts, internal customers, and senior developers, the developer enables improved data coverage and analytic capabilities that support defect-free, production-ready outcomes.

Day-to-Day Responsibilities

• Develop high-performance data processing pipelines.
• Partner with business analysts and internal customers to improve data coverage and analytic capabilities.
• Aim for defect-free programming, create and maintain quality code, and provide support during testing cycles and post-production deployment.
• Engage in peer code reviews and address highly complex development-related issues independently.
• Research, learn and recommend emerging technologies proactively.

Professional Experience

• 2+ years of experience with multiple mainstream programming languages such as Python, Java, C++, C# or Go.
• Experience working with Apache Spark in Scala or PySpark, Kafka and other big data technologies.
• Experience developing big data ingestion frameworks or working with ingestion tools.
• Experience with data pipeline ETL tools such as Talend as an added advantage.
• Experience with virtualization, container-based and cloud platforms such as Kubernetes, OpenShift, Swarm and Docker.
• Demonstrated analytical and problem-solving skills in a big data environment.
• Strong communication skills, self-motivation, willingness to learn and excellent planning and organizational skills.

Reporting to engineering leadership, the Big Data Developer creates optimal data pipeline architecture using Apache Airflow and Azure technologies, assembling large complex data sets and ingesting data into the data lake via Spark and Databricks. Partnering with infrastructure and analytics stakeholders, the developer builds and maintains ETL pipelines at enterprise scale, enabling high-performance, scalable analytics solutions that support machine learning and traditional data warehousing needs.

Scope of Work

• Design, architect and support new and existing data and ETL pipelines and recommend improvements.
• Create optimal data pipeline architecture and systems using Apache Airflow.
• Assemble large, complex data sets meeting functional and non-functional business requirements.
• Ingest data into the data lake and provide frameworks and services for operating on that data using Spark and Databricks.
• Analyze, debug and correct issues with data pipelines.
• Build infrastructure for optimal ETL from a wide variety of data sources using SQL, Spark and Azure technologies.

Knowledge Skills and Abilities

• Bachelor's degree in Computer Science, Software Engineering or equivalent experience.
• 8+ years of experience building high-performance scalable enterprise analytics or data-centric solutions.
• At least 5 years of experience implementing complex ETL pipelines preferably with Hadoop or Spark.
• Exceptional coding and design skills in Java/Scala or C# with expertise in Python and strong knowledge of algorithms and data structures.
• Hands-on experience with Azure including Azure Data Lake.
• Experience with Spark data pipelines, streaming, Apache Airflow, C# .NET Core and traditional data marts and warehouses.
• Machine learning expertise and ability to lead or have led teams before.

As the Big Data Developer, this role delivers and integrates analytics-driven solutions with end business applications through flexible microservice architecture and big data ecosystem innovation. The data science and business analytics teams rely on this work to scale proof-of-concept adoptions and design analytics delivery systems that meet evolving business and architecture requirements.

Job Functions

• Contribute to design and implementation of various interfaces to deliver and integrate analytics-driven solutions with end business applications.
• Implement a flexible and highly scalable analytics delivery mechanism through microservice architecture.
• Work with data science and business analytics teams to understand business context and design analytics solution delivery systems.
• Work with external and internal stakeholders to participate in architecture design discussions.
• Bring in new ideas and innovation from the big data ecosystem and help drive adoption of newer technology through proof of concepts.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering or a related area.
• 3+ years of professional experience in engineering and technology development and delivery.
• Experience with enterprise Java, web application design and microservice architecture.
• Experience in refactoring and re-engineering of enterprise systems and code performance tuning and debugging.
• Experience in big data technology stack including Hadoop, MapReduce, Hive, Spark, Scala, NoSQL, Kafka and open source tools is highly desirable.
• Demonstrable excellence in innovation, problem-solving and analytical skills.

Big Data Developer guides global engineering teams by providing technical leadership, conducting design and code reviews, and ensuring adherence to technical standards for product development. The work directly supports high-volume stream processing environments, enabling reliable real-time pipelines built on Java, Flink, Kafka Streams, and Spark Streaming across Agile, cross-regional teams.

Operational Focus

• Develop and provide technical leadership.
• Guidance to the team including design and code reviews.
• Ensure adherence to technical standards for product development.
• Accountability for quality of team deliveries.
• Collaborate and share work with teams around the globe.

Experience and Qualifications

• Bachelor's degree in Computer Science, Information Technology or a related field.
• 5+ years in Java programming.
• 2+ years of experience with stream processing tools such as Flink, Kafka Streams, Spark Streaming, Storm or Samza in a high-volume environment.
• Deep understanding of stream processing in a real-time production environment.
• Deep knowledge of message queues such as Kafka and RabbitMQ as an advantage.
• Knowledge and interest in financial markets as an advantage.
• Experience in Agile teams with ability to work and solve large-scale problems independently.