BIG DATA SPECIALIST JOB DESCRIPTION

A Big Data Specialist is the engineer who keeps large-scale data pipelines honest. When batch jobs silently degrade or streaming throughput drops under load, downstream teams lose the reliable data they need to make decisions - and recovery can take days. This role owns the design, development, and architectural governance of distributed data platforms processing millions of transactions, operating within global engineering teams that span eCommerce, cloud infrastructure, and Agile delivery environments. Cloudera-certified practitioners and those holding Apache Spark credentials are the profile most commonly sought for this work.

As the Big Data Specialist, you design and maintain distributed data platforms that support high-volume batch and real-time processing for enterprise-scale operations. You collaborate with product owners, DevOps engineers, and platform architects, working within Agile delivery cycles to ensure platform stability, performance, and coverage across cloud and on-premises environments.

Career Impact: Hands-on ownership of Spark and Hadoop infrastructure at enterprise scale - paired with mandatory platform certifications like Cloudera CCA or Databricks - builds a technical profile that commands strong market positioning in cloud data engineering.

Business Impact: The platforms this role maintains process millions of transactions and support product decisions across analytics teams, meaning degraded pipeline performance has direct, measurable consequences for revenue and operational continuity.

Growth Opportunity: Experience spanning batch processing, real-time streaming, microservices architecture, and cloud-based data solutions accelerates progression toward senior architect and platform lead roles within three to five years.

• Design and implement distributed data processing solutions spanning batch and real-time streaming pipelines to support enterprise-scale workloads.
• Architect backend systems and refactor existing components to improve resilience, scalability, and availability across platform layers.
• Monitor platform processes and activities across analytics teams to detect and resolve stability issues before they escalate.
• Coordinate with DevOps engineers, infrastructure teams, and production support to triage incidents and manage resource capacity.
• Collaborate with product owners and scrum masters to gather requirements, assess current systems, and translate business problems into technical solutions.
• Review code, enforce development best practices, and implement deployment pipelines to ensure release quality within Agile sprint cycles.
• Define and maintain architectural standards, technical documentation, and project scope to keep delivery on schedule and within budget.
• Validate data access patterns, streaming performance, and cost optimization across ingestion, storage, and processing layers.

• Bachelor's degree in Computer Science, Computer Engineering, or a related discipline, or equivalent work experience.
• 5 or more years of hands-on experience in Big Data platform development, distributed systems architecture, and scalable backend engineering.
• Demonstrated ability to design and develop batch and real-time data processing solutions using distributed computing frameworks.
• Strong SQL skills with experience in large-scale relational and NoSQL database environments.
• Solid understanding of data structures, algorithms, microservices architecture patterns, and event-driven design.
• Proficiency in Scala and Python for data engineering and pipeline development.
• Experience maintaining Hadoop-based infrastructure including HDFS, YARN, and cluster resource management.
• Excellent verbal and written communication skills, including technical documentation and stakeholder reporting.

• Active certification in a recognized Big Data platform such as Cloudera CCA Spark and Hadoop Developer, Databricks Certified Developer Apache Spark, or an equivalent Hortonworks credential.
• Experience with ETL tooling, data federation or virtualization technologies, and XML and JSON data formats.
• Familiarity with containerization, orchestration, and cloud-based data solution deployment across web, batch, and streaming application types.
• Working knowledge of continuous integration tooling including build, release, and deployment automation for full-stack and incremental pipelines.

• Pipeline uptime rate, reflecting how consistently batch and streaming jobs complete without failure.
• Mean time to resolution for platform incidents, measuring speed of triage and recovery across production environments.
• Sprint delivery rate, tracking the percentage of committed items completed within each Agile cycle.
• Code review turnaround time, indicating responsiveness and throughput within the engineering team.
• Platform resource utilization ratio, measuring efficiency of compute and storage capacity relative to workload volume.
• Typical tools: Distributed processing frameworks (commonly Apache Spark or Apache Flink); cluster management platforms (commonly Cloudera or Hortonworks); version control and CI systems (commonly Git and Jenkins)

• Base Salary Range: $120,000 to $165,000 annually depending on experience and location.
• Bonus: Annual performance bonus typically 8 to 15 percent of base salary.
• Equity: RSUs or stock options common at mid-to-large technology employers.
• Health Benefits: Medical, dental, and vision coverage for employee and dependents.
• PTO: 15 to 20 days annually plus public holidays and sick leave.
• Common Perks: Remote or hybrid flexibility, professional certification reimbursement, and conference attendance budget.

Figures are estimates based on general US market benchmarks and may be outdated. Adjust based on location, company size, and seniority level.

Reasonable accommodations are available to qualified individuals with disabilities throughout the application and employment process in accordance with the Americans with Disabilities Act. All applicants are considered without regard to race, color, religion, sex, national origin, age, disability, veteran status, or any other characteristic protected by applicable federal, state, or local law. Employment is contingent on successful completion of a background check. Candidates must be authorized to work in the United States.

The Big Data Specialist owns end-to-end system analysis, architecture, and deployment of platforms handling millions of transactions, directly generating significant revenue at scale. Working closely with product management and automation teams, this role shapes technology strategy, establishes eCommerce architectural standards, and ensures quality delivery across full project life cycles.

Key Responsibilities

• Design and build cutting-edge solutions at scale to impact millions of customers and generate significant revenue.
• Engage with product management to translate product vision and business problems into technical definitions, implementations, and successful deployments.
• Conduct system analysis, design, and architecture for entire products and platforms handling millions of transactions.
• Refactor backend and frontend layers to improve resilience, scalability, availability, serviceability, and usability.
• Define project scope, deliverables, dependencies, resources, timelines, and budgets to ensure on-time delivery within full project life cycles.
• Establish architectural and development standards around eCommerce platforms.
• Develop project technology strategy by mapping product requirements, articulating solution risks and barriers, and recommending project approaches.
• Prepare technical write-ups for programming features and ensure team members understand deliverables and task lists.
• Ensure documentation, logs, and reports are complete, current, and stored appropriately.
• Collaborate with automation teams to determine testing requirements and ensure full coverage and product quality.
• Maintain excellent communication with stakeholders while managing expectations throughout all project phases.

Required Qualifications

• Bachelor's or Master's degree in Computer Science or Software Engineering.
• 5 to 9 or more years of experience in technical design, software development, and architecture for scalable backend platforms.
• 7 or more years of hands-on experience with Apache Spark or Apache Flink, RDBMS and NoSQL databases, and Elasticsearch or Solr.
• Proficiency in Scala and Python.
• Good knowledge of data security practices.
• Excellent code version control and conflict management capabilities with Git.
• Strong build, release, and deployment management capabilities including full stack and incremental builds.
• Good working knowledge of Jenkins, Maven, and Ansible for continuous integration.
• Excellent verbal and written communication skills including facilitation, presentation, and documentation.

Embedded within a data engineering environment, the Big Data Specialist delivers solutions across large-scale data ingestion, batch and real-time processing, movement, storage, and access while applying organizational architecture patterns such as Microservices and Event Driven design. Working closely with project teams, this role advances data platform quality by identifying business problems, designing testable software, and optimizing data performance and cost.

Core Functions

• Apply organizational architecture patterns such as Microservices and Event Driven design effectively in projects.
• Work with large volumes of data including ingestion, batch and real-time processing, movement, storage, and access.
• Test, debug, and fix issues within established SLAs.
• Design software that is easily testable and observable.
• Identify business problems at the project level and provide solutions.
• Understand data access patterns, streaming technology, data validation, data performance, and cost optimization.

Qualifications and Experience

• Experience in Scala programming.
• Experience in Big Data technologies including Spark and Kafka.
• Good understanding of data structures, algorithms, and organizational strategy.
• Strong SQL skills.
• Experience with ETL tools, preferably Talend.
• Experience with Linux OS at user level.
• Python or R programming skills are a plus.

Reporting to the product owner and scrum master, the Big Data Specialist executes backend performance improvements across batch and streaming components while working within an Agile team to transform innovative ideas into robust software. Partnering with development and operations stakeholders, this role drives full delivery cycles from requirements gathering through API implementation and deployment pipeline setup.

Primary Duties

• Work with Big Data technologies including Hadoop and Spark.
• Support backend performance improvement across batch jobs, streaming jobs, and other product components.
• Support sprint development and releases.
• Conduct code reviews and implement best practices.
• Work in an Agile team to turn innovative ideas into robust software and solve complex design and implementation problems.
• Collaborate with product owner and scrum master to define and implement requirements.
• Gather requirements and functional specifications, assess current software systems, and identify areas for improvement.
• Implement APIs and deployment pipelines for applications.

Skills and Qualifications

• Bachelor degree in Computer Science, Computer Engineering, or a software-related discipline.
• Hands-on experience in Big Data technologies including Hadoop, HBase, Hive, and Scala.
• Hands-on experience in Spark development, Sqoop, Flume, Kafka, NiFi, and Python.
• Experience creating web services for high availability cloud environments with monitoring capabilities.
• Hands-on experience in Java, Spring, microservices, REST API, Spring Boot, Postgres, Docker, and Kubernetes.
• Proven problem-solving skills and ability to identify, formulate, and resolve engineering problems.

Sitting at the intersection of platform engineering and cross-functional collaboration, the Big Data Specialist refines infrastructure stability by monitoring Cloudera Hadoop environments, coordinating with DevOps and production support teams, and managing resource capacity across the Big Data platform. Operating across global development and analytics teams, this role strengthens architectural quality and ensures reliable data platform performance end to end.

Duties

• Collaborate within the organization on current and future state architecture.
• Contribute to hands-on development within a global development team.
• Monitor platform processes and activities across analytics teams to ensure platform stability.
• Coordinate with DevOps teams, infrastructure engineers, and production support to triage and resolve issues.
• Manage resource capacity of the Big Data platform efficiently.

Experience and Qualifications

• 5 to 7 years of appropriate technical experience.
• Proficiency in Scala and Python.
• Experience with Spark for data processing.
• Experience maintaining Cloudera Hadoop infrastructure including HDFS, YARN, Spark, Impala, and edge nodes.
• Strong SQL skills with experience in large database platforms.
• Knowledge of high-quality software architecture, design methodologies, and complete SDLC and Agile processes.
• Experience with Core Java, XML and JSON technologies, and Data Federation or Virtualization technologies.
• Strong oral and written communication skills.

A key member of the risk management function, the Big Data Specialist leads credit risk policy development including scorecards and credit decision logic optimization, supported by SQL and Snowflake-based reporting across data platforms. Collaborating across business project teams and working groups, this role enables data-driven risk decisions that directly strengthen the organization's retail lending outcomes.

Functions

• Develop credit risk policy including scorecards, credit decision logic, and optimization.
• Develop risk reporting using SQL and Snowflake.
• Identify potential risk management opportunities through data mining across data platforms.
• Provide advice and add value to business projects requiring risk management input.
• Present concise summaries of results to relevant working groups.

Professional Experience

• Previous experience within retail credit risk and broad knowledge of retail lending products.
• Strong SQL skills with Snowflake experience preferred.
• Experience with data processing tools such as R, Python, or SAS.
• Very good English communication skills.
• Strong organizational skills with ability to prioritize workload and work accurately under pressure.
• Proven track record of working independently and introducing fresh thinking to the role.