BIG DATA ARCHITECT JOB DESCRIPTION

A Big Data Architect owns the platform decisions that determine whether an enterprise's data is usable at all. Most organizations operating distributed cloud infrastructure across AWS, Azure, or GCP accumulate data that no team can query reliably until someone designs the ingestion, storage, and processing layers that make it coherent. This role holds that architectural authority, sitting above the engineering layer and answerable to both technology leadership and the business units that depend on data for decisions. The scope runs from data lake design to real-time pipeline governance.

As the Big Data Architect, you define the structural foundation of an enterprise data platform, translating business requirements into scalable architecture decisions that support analytics, AI, and operational workloads. You work across data engineering, solutions architecture, and product teams, with direct accountability for platform integrity, performance standards, and data governance across cloud environments.

Career Impact: Architects who have delivered large-scale data lake and pipeline systems across multiple cloud providers hold some of the most durable technical credentials in the enterprise market, with demand that consistently outpaces supply at the senior level.

Business Impact: When platform architecture is sound, analytics teams can deliver on time and data scientists can move from prototype to production, so the quality of this role's decisions directly determines how fast the organization converts raw data into revenue-generating intelligence.

Growth Opportunity: Experience owning data governance frameworks, real-time stream processing at scale, and cloud-native architecture patterns positions a Big Data Architect for advancement into Principal Architect, Head of Data Engineering, or Chief Data Officer tracks.

• Architect enterprise-scale data platforms spanning cloud, on-premises, and hybrid environments to meet performance and reliability requirements.
• Design data pipelines, data lake structures, and warehouse solutions that support analytics and AI workloads across business units.
• Define architectural standards and governance frameworks covering data lineage, data security, and access control.
• Lead technical evaluations of big data frameworks, cloud services, and ingestion tools to inform platform roadmap decisions.
• Collaborate with data engineers, data scientists, solutions architects, and product managers to align platform design with business requirements.
• Monitor platform performance and drive infrastructure tuning to maintain throughput, latency, and cost targets.
• Develop and maintain architecture documentation including data models, data flow diagrams, and decision records.
• Guide engineering teams on distributed computing principles, design patterns, and modern SDLC practices to raise platform quality.

• Bachelor's degree in Computer Science, Data Science, Engineering, or a related field, or equivalent work experience.
• 7 or more years of data engineering or platform architecture experience, with at least 4 years in a lead or architect-level capacity.
• Demonstrated ability to design distributed, fault-tolerant data systems across structured and unstructured data sources at enterprise scale.
• Deep knowledge of ETL and ELT pipeline design, data modeling, data governance, and data security principles.
• Proven experience working with major public cloud platforms and their native analytics, storage, and compute services.
• Strong understanding of real-time and batch processing architectures, including stream-based data patterns and distributed messaging systems.
• Experience with containerization, orchestration, and CI and CD practices as they apply to data platform deployments.
• Effective written and verbal communication skills, with ability to present architectural trade-offs to both technical and non-technical audiences.

• Cloud architecture certification at the associate or professional level, with preference for platforms relevant to the organization's primary stack.
• Experience delivering big data solutions in a customer-facing consulting or professional services environment.
• Familiarity with data catalogue, data lineage tooling, and enterprise metadata management practices.
• Exposure to machine learning infrastructure and the data platform requirements that support model training and deployment pipelines.

• Data pipeline availability rate, measuring uptime and reliability of ingestion and processing layers across production environments.
• Platform query performance percentile, tracking p95 and p99 latency against defined SLAs for analytics workloads.
• Data governance coverage ratio, reflecting the percentage of datasets with documented lineage, ownership, and access policy.
• Architecture review cycle time, measuring days from requirements intake to approved design for new platform components.
• Infrastructure cost per processed terabyte, tracking efficiency of storage and compute resource utilization over time.
• Typical tools: Cloud data platforms (commonly AWS EMR, Azure Databricks, or GCP Dataproc); pipeline orchestration (commonly Apache Spark or Kafka-based streaming).

• Base Salary Range: $155,000 to $210,000 annually depending on seniority and location.
• Bonus: 10 to 20 percent annual performance bonus, common at architect level.
• Equity: stock options or RSUs typical at growth-stage and public technology employers.
• Health Benefits: medical, dental, and vision coverage for employee and dependents.
• PTO: 15 to 25 days annually, with many employers offering flexible or unlimited PTO.
• Common Perks: Remote or hybrid work options, conference and certification reimbursement, home office stipend.

Figures are estimates based on general US market benchmarks and may be outdated. Adjust based on location, company size, and seniority level.

Work authorization in the United States is required for this position, and employment is contingent on successful completion of a background check. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, disability, veteran status, or any other characteristic protected under applicable federal, state, and local law. Reasonable accommodations are available to individuals with disabilities throughout the application and employment process upon request.

The Big Data Architect owns the design and delivery of enterprise-scale data pipelines, data lakes, and warehouse solutions across cloud and on-premises environments. Working with engineers, data scientists, and business analysts, this role shapes data platform architecture to ensure scalability, performance, and reliable reporting outcomes across the organization.

Key Responsibilities

• Architect enterprise-scale big data platforms and distributed solutions across cloud and on-premises environments.
• Design and develop scalable data pipelines, data lakes, and data warehouse solutions across the entire data supply chain.
• Lead analytics and redesign of business data analytics processes and reporting solutions.
• Manage end-to-end technology program and project life cycles including requirements, UAT, and deployment.
• Evaluate, select, and integrate big data frameworks, tools, and cloud services based on project requirements.
• Collaborate with engineers, data scientists, business analysts, and stakeholders to define data structures and architecture.
• Monitor data platform performance and tune infrastructure components to ensure optimal results.
• Create and maintain architecture documents, technical specifications, and user-focused documentation.

Required Qualifications

• Degree in Computer Science, Data Science, Mathematics, Engineering, or a related field required; MBA or PhD is a plus.
• AWS Certified Solutions Architecture (Associate or Professional) or other cloud specialty certification preferred.
• 7+ years of experience in data engineering and 4+ years architecting large-scale data solutions.
• Proven experience integrating data from multiple heterogeneous and distributed sources at enterprise scale.
• Deep understanding of big data platform architecture, scalability, performance, and distributed computing principles.
• Strong knowledge of ETL and ELT processes, data modeling, data governance, data lineage, and data security practices.
• Proficiency in Python, Scala, Java, JavaScript, or Go; experience with Spark, Kafka, and Hadoop ecosystems.
• Experience with cloud platforms including AWS, Azure, or GCP and related analytics and database services.
• Expertise in SQL and NoSQL databases, graph databases, time series databases, and relational database systems.
• Knowledge of container technologies such as Docker and Kubernetes and experience with CI and CD processes.
• Excellent verbal and written communication skills with the ability to tailor technical content to any audience.

Embedded within a global professional services practice, the Big Data Architect designs enterprise-scale distributed solutions using compute, container, serverless, storage, analytics, and database services. Working closely with partners and customers, this role delivers technical engagements and packaged big data offerings that guide organizations through their cloud transformation journey.

Core Functions

• Design enterprise-scale globally distributed and highly available solutions using compute, container, serverless, storage, analytics, database, and network services.
• Build hands-on products and solutions leveraging AI and machine learning, serverless, IoT, analytics, data warehouse, BI, and security services.
• Deliver on-site technical engagements with partners and customers including pre-sales visits and consulting proposals.
• Create packaged big data service offerings and deliver them as part of customer projects.
• Guide customers through their cloud journey and transformation.
• Engage with the global professional services community to learn and share expertise.

Qualifications and Experience

• 3+ years of hands-on experience in IT implementation or leading IT projects in architecture, data engineering, or development.
• AWS Certified Solutions Architecture (Associate or Professional) or other specialty certification preferred.
• Demonstrated industry leadership in database, big data, data warehousing, or data science.
• Deep understanding and implementation experience of data lakes, stream-based analytics, MPP databases, NoSQL storage, data warehouse design, BI reporting, and dashboard development.
• Experience implementing big data, analytics, AI, and machine learning solutions on AWS.
• Hands-on experience leading large-scale global data warehousing and analytics projects.
• Experience designing and coding in Python or Scala or building ETL and data ingestion pipelines.
• Business fluent verbal and written communication skills in English and German.

Reporting to senior technology leadership, the Big Data Architect drives and owns architectural design strategies and their realization for the big data platform. Partnering with PMs, engineers, and TechOps teams, this role ensures architectural integrity and builds a strong engineering culture that enables the platform to scale and perform reliably.

Primary Duties

• Drive and own architectural design strategies and their realization for the big data platform.
• Identify big data platform areas that can be improved, propose better solutions, and drive implementation.
• Create and maintain alignment between business needs and the big data platform environment and technology.
• Work with PMs, engineers, and TechOps to maintain the integrity of tech infrastructure.
• Provide hands-on technical and execution leadership for engineering teams.
• Define, execute, and maintain architectural integrity throughout the big data platform environment.
• Help build a strong engineering culture by partnering closely with engineering teams and developing their expertise.

Skills and Qualifications

• 10+ years of industry experience working with big data platforms.
• Deep understanding of big data platform architecture, scalability, and performance.
• Expert-level hands-on experience in cloud technologies, preferably AWS.
• Deep understanding of data formats including Hudi, Iceberg, and Delta Lake.
• Deep expertise in real-time and batch data processing at large scale.
• Expert understanding and use of standard software design patterns and modern SDLC processes including Agile, CI, and CD.
• Strong technical leadership with change management skills and ability to communicate to any level from executives to junior engineers.

Sitting at the intersection of data engineering and enterprise architecture, the Big Data Architect designs and develops distributed, fault-tolerant, and scalable data solutions using big data technologies. Operating across internal architect communities and cross-functional teams, this role ensures data pipelines, APIs, and platform components meet organizational best practices and deliver reliable, high-performance results.

Duties

• Architect, design, and develop distributed, fault-tolerant, and scalable data solutions using big data technologies.
• Collect and process data at scale from a variety of sources for different project needs.
• Design, develop, and maintain data pipelines and data platforms using selected frameworks and tools.
• Develop and maintain data APIs and integrate data from various sources using federation and virtualization techniques.
• Monitor data platform performance and tune infrastructure and platform components on a regular basis.
• Collaborate with the internal architect community to ensure organizational best architectural practices are followed.
• Maintain a high level of expertise in data technologies and stay current on the latest developments.

Experience and Qualifications

• 15+ years of experience in software design, architecture, and development; 7+ years in data engineering; 4+ years architecting large-scale data solutions.
• Proficient understanding of distributed computing principles and experience with containers and API design and implementation.
• Prior experience designing big data platform components that are scalable, high-performing, and cost-efficient in operations.
• Experience processing large amounts of structured and unstructured data including data modeling, cleaning, visualization, and reporting.
• Excellent knowledge of ETL techniques and frameworks, messaging systems, stream-processing systems, big data ML toolkits, and big data querying tools.
• Experience with NoSQL, graph, relational, and time series databases.
• Proficiency in Python, Go, Perl, JavaScript, Kafka, Spark, and Kubernetes.
• Excellent interpersonal and communication skills with proven experience managing teams across multiple geographies.

A key member of the data and AI delivery team, the Big Data Architect architects and develops Data and AI solutions across cloud and on-premises environments, covering requirements analysis through deployment. Collaborating across analytical and engineering teams, this role enables high-performance data integration systems and supports complex customer projects managed with agile methods.

Functions

• Architect, design, and develop Data and AI solutions on cloud and on-premises environments.
• Conduct requirements analysis, platform selection, technical architecture design, application design and development, testing, and deployment.
• Implement security and encryption best practices for Data and AI environments.
• Assist analytical and engineering teams in designing and implementing high-performance systems with large-volume data integration, storage, processing, and provisioning.
• Provide technical pre-sales support and organize and manage complex customer projects using agile methods.

Requirements

• Degree in Computer Science or equivalent required.
• Hands-on experience with data and AI applications including Hadoop stack, Kafka, Databricks, Azure Synapse, and GCP AutoML.
• Experience designing solutions for large data warehouses with strong understanding of clusters, parallel architecture, high-scale distributed RDBMS, and NoSQL platforms.
• Proficiency in Java, Linux, Python, and Scala.
• Knowledge of container technologies including Docker, Kubernetes, and OpenShift.
• Experience with data catalogue, data governance, and data lineage practices and tools.
• Understanding of machine learning theory and practice.
• Strong problem-solving skills and strong teaching and coaching skills.

Reliable data infrastructure for renewable energy operations depends on the Big Data Architect, who processes and prepares solar and wind park data in the Microsoft Azure Cloud using ETL and ELT processes. Based within an energy data team, this role develops new data sources, builds microservices and API connections, and advances a group-wide Data Lakehouse concept.

Accountabilities

• Process and prepare solar and wind park data in the Microsoft Azure Cloud.
• Develop ETL and ELT processes for needs-based data processing and report creation.
• Develop new data sources and analyze and process structured and unstructured data.
• Develop applications for data transformation, microservices, and API connections for data exchange between existing applications and databases.
• Create a group-wide data catalogue.
• Participate in establishing and further developing a Data Lakehouse concept.

Technical Qualifications

• Project experience as a big data architect required.
• Good understanding and practical experience in big data technologies as well as SQL and NoSQL database systems; CosmosDB experience is desirable.
• Experience with ETL and ELT tools; Azure Data Factory experience is desirable.
• Experience with modern concepts for structured and unstructured data, big data, and data lakes.
• Knowledge of Microsoft Azure services including Azure Functions, Databricks, Event Hub, and Data Lake Store Gen 2.
• Knowledge of NodeJS, JavaScript (ES6+), and Python.
• Experience with RESTful web services and Power BI desirable.
• Familiarity with DevOps concepts and agile working methods.
• Very good communication skills in English required, German is desirable.

What You'll Do

• Manage data-related requests, analyze issues, and provide efficient resolution.
• Design all program specifications and perform required tests.
• Monitor all production issues and inquiries and provide efficient resolution.
• Evaluate all functional requirements, map documents, and troubleshoot all development processes.
• Collaborate with application groups to prepare effective solutions and document technical specifications and project deliverables.
• Collaborate with technology teams and business executives to conceptualize new application projects and recommend technologies, design patterns, and implementation strategies.
• Help with onboarding cloud and data engineering technologies including Spark and Kafka.

Background and Experience

• BS in Engineering or a related field with at least 10 years of professional IT experience required.
• Experience in data platform architecture and design for cloud data engineering or big data ecosystems.
• Experience building scalable data ingestion frameworks and curation pipelines in any cloud platform.
• Hands-on experience in Scala and Spark and PySpark programming.
• Experience with batch data ingestion strategy using Spark streaming and Kafka.
• Exposure to real-time and event-based data processing using streaming including DStreams and Structured Streaming in Spark.
• Good understanding of security and enterprise data governance tools.
• Experience in NoSQL using MongoDB or Cassandra.

Big Data Architect leads the design and implementation of sustainable tools and processes for the big data ecosystem while developing the technology and capability roadmap in collaboration with business analysts and solutions architects. The work directly supports data-driven decision making across a global food enterprise by enabling reliable big data infrastructure and emerging tool adoption across technical teams.

Strategic Responsibilities

• Act as a key Data and Analytics technical leader within the organization.
• Collaboratively develop the technology and capability roadmap for the big data ecosystem.
• Lead the design and implementation of sustainable tools and processes to support the big data ecosystem.
• Generate and implement ideas to improve the operational and strategic health of the big data ecosystem.
• Participate in the evaluation, implementation, and deployment of emerging tools and processes in the big data space.
• Develop communication and education plans for technical teams on technologies and processes in the big data ecosystem.
• Partner with business analysts and solutions architects to develop technical architectures for strategic enterprise projects.
• Collaboratively troubleshoot technical and performance issues in the big data ecosystem.

Education and Experience

• Bachelor's degree required; Computer Science, MIS, or Engineering preferred.
• Minimum 5 years of IT experience; 7+ years preferred.
• Experience working directly with business clients to design solutions that meet business requirements.
• Experience with data and analytics concepts including dimensional modeling, ETL, reporting tools, data governance, data warehousing, and structured and unstructured data.
• Database development experience using Oracle, SQL Server, SAP BW, or SAP HANA.
• Big data development experience using Hive and Spark.
• Experience in Google Cloud Services and familiarity with Kafka and Linux.
• Python, Scala, or Java development experience.
• Effective analytical, verbal, and written communication and influencing skills with ability to work in a team environment.

The Big Data Architect owns the definition and implementation of end-to-end modern data platforms supporting analytics and AI use cases, serving as the technical liaison between customers and engineering teams. Collaborating with enterprise architects, data scientists, and sales teams, this role delivers AWS-based big data solutions that address data privacy, modeling, virtualization, and API integration across client organizations.

Leadership Responsibilities

• Lead, define, and implement end-to-end modern data platforms in support of analytics and AI use cases.
• Collaborate with enterprise architects, data architects, ETL developers, data scientists.
• Designers to define required data structures, formats, pipelines, metadata, and workload orchestration capabilities.
• Address data privacy and security, data ingestion and processing, data storage and compute, analytical and operational consumption, data modeling, data virtualization, self-service analytics, AI enablement, and API integrations.
• Serve as the technical liaison between customers and engineering teams.
• Collaborate with the sales team to formulate and execute a sales strategy for adoption of AWS and big data technologies.

Qualifications and Experience

• Bachelor's degree or higher in Computer Science, Information Management, Big Data and Analytics, or a related field preferred.
• AWS certifications in architecture, data engineering, and development preferred.
• Subject matter data expertise in Financial Services, Consumer Products, Energy and Resources, Life Sciences, or Government industries preferred.
• Strong SQL, database, data modeling, data warehousing, and development skills.
• Strong programming and scripting experience in Java, .NET, Python, Scala, or JavaScript.
• Experience with cloud big data and analytics services on AWS including S3, Redshift, Athena, EMR, Glue, and QuickSight.
• Experience with dashboarding and reporting tools such as Tableau and Qlik, and industry ETL tools such as Informatica, Talend, and SSIS.
• Experience with data security, encryption, PII and PSI legislation, and identity and access management.
• Knowledge of software configuration management tools such as JIRA, Git, Jenkins, and Bitbucket.
• Strong people management skills including leading teams, training, onboarding, and offboarding.

The Big Data Architect delivers scalable big data warehouse solutions across the entire data supply chain, designing strategies for real-time data analysis and extending the business data lake. Working with management, business partners, analysts, architects, and the Data Science team, this role ensures data quality, actionable intelligence, and well-governed integration across multiple systems.

Day-to-Day Responsibilities

• Design and develop scalable big data warehouse solutions across the entire data supply chain.
• Create and implement solutions for metadata management.
• Create and review technical and user-focused documentation including data models, data dictionaries, business glossaries, process and data flows, and architecture diagrams.
• Extend and enhance the business data lake and solve complex data integrations across multiple systems.
• Design and execute strategies for real-time data analysis and decisioning.
• Collaborate with management, business partners, analysts, developers, architects, and engineers to support data quality efforts.
• Work closely with the Data Science team to improve actionable data.

Professional Experience

• Bachelor's or Master's degree in Computer Science, data processing, or equivalent required.
• Experience in data warehousing or similar analytic data environments.
• Experience with Java programming and developing frameworks.
• Experience with Hadoop, Spark, Amazon EMR, and EC2 or equivalent.
• Experience with AWS technologies including Aurora, Athena, EMR, Redshift, and S3.
• Experience with Postgres and MySQL, Bitbucket, and Git.
• Proficiency in data architecture, data assembly, data governance, data security, and data integration tools such as Talend and Cascading.
• Knowledge of Business Intelligence, MDM, XML, and SOA and web services.
• Familiarity with Linux, Jenkins, and CI and CD concepts.
• Excellent organizational, project management, and communication skills.