BIG DATA JOB DESCRIPTION

Big Data Analyst Functions:

• Gathering data from various sources, and then cleansing, organizing, processing, and analyzing it to extract valuable insights and information
• Working with various database types to handle large amounts of data, in structured, semi-structured and unstructured form
• Developing reports and dashboards (charts, tables, ...) with data analysis results to help different audiences make better decisions
• Identifying, analyzing, and interpreting trends or patterns in complex data sets
• Understanding sources and lineage of data to control the quality of our data sets
• Collaborating with team members to prioritize business and information requirements
• Improving the company data lake and data marts
• Performing routine analysis tasks to support day-to-day business
• Applying statistical analysis methods for consumer data research and analysis purposes
• Closely collaborating with both the IT teams and customer-facing teams to accomplish company goals and prove value to our customers

Big Data Analyst Knowledge, Experience and Requirements:

• Experience in data analysis
• Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy
• Experience with some programming languages: Python, Java, C#, ...
• Experience with databases and SQL/NoSQL
• Working proficiency in written and spoken English
• Enthusiasm for teamwork, constant learning, and adapting to new circumstances
• Technical expertise with data models, database design, data mining and segmentation techniques
• Experience with the Python data science stack (pandas, numpy, ...) and/or writing complex SQL queries
• Knowledge of statistics and experience with statistical packages for analyzing datasets

To showcase analytical expertise effectively, a strong Big Data Resume highlights technical skills and improves chances of securing data analyst roles.

Big Data Architect Responsibilities:

• Defines technology roadmap in support of product development roadmap
• Lead the design, architecture and development of multiple real time streaming data pipelines encompassing multiple product lines and edge devices
• Ensure proper data governance policies are followed by implementing or validating data lineage, quality checks, classification, etc.
• Provide technical leadership to agile teams onshore and offshore: Mentor junior engineers and new team members, and apply technical expertise to challenging programming and design problems
• Resolve defects/bugs during QA testing, pre-production, production, and post-release patches
• Have a quality mindset, squash bugs with a passion, and work hard to prevent them in the first place through unit testing, test-driven development, version control, continuous integration and deployment.
• Ability to lead change, be bold, and have the ability to innovate and challenge status quo
• Conduct design and code reviews
• Analyze and improve efficiency, scalability, and stability of various system resources
• Operate within Agile Development environment and apply the methodologies
• Track technical debt and ensure unintentional technical debt is not created
• Recommends improvements to the software delivery cycle to help remove waste and impediments for the team
• Drives, promotes and measures team performance against the sprint and project goal.
• Works with the team to continuously improve in development practices and process
• Troubleshoots complex problems with existing or newly-developed software
• Mentoring and coaching of Software Engineers

Big Data Architect Qualifications:

• Expert knowledge of data architectures, data pipelines, real time processing, streaming, networking, and security
• Proficient understanding of distributed computing principles
• Advanced knowledge of Big Data querying tools, such as Pig or Hive
• Expert understanding of Lambda Architecture, along with its advantages and drawbacks
• Proficiency with MapReduce, HDFS
• Experience with integration of data from multiple data sources and multiple data types

Big Data Consultant Details:

• Work with business and functional stakeholders to understand data requirements and downstream analytics needs.
• Responsible for ratifying technology solutions, producing concise Design documents, including Source to Target mappings, data manipulation / processing logic, contributing to work estimates
• Create Design Documentation and approach for successful program delivery.
• Understand different project methodologies, project lifecycles, major phases, dependencies and milestones within a project, and the required documentation needs.
• Translate business requirements & E2E designs into technical implementations based on system capabilities.
• Implement Data integration and Data Warehouse based solutions using open source big data tools and cloud-native technologies
• Create a solution for data and process migration from source to destination (on-premise to cloud, cloud to cloud, or to a data platform) without business impact.
• Ensure consistency in approach, design, and output to deliver quality and concise design solution documentation.
• Define, and promote re-usable, extendible, scalable, maintainable solutions considering the trade-off for cost vs benefit
• Communicate at all levels in a clear and credible way about the importance of solution design.
• Lead developers in the team and foster a shared vision for design.

Big Data Consultant Requirements:

• Prior exposure in successfully leading the development & delivery of data applications
• Exposure to Software Development Solutions and Design specification/documentation
• Experience in Python, SQL, PySpark, Map-reduce, Hive, Hbase, Airflow
• Good understanding of big data workloads on one or more cloud platforms like AWS, GCP, Azure
• Solution delivery for data processing components in larger End to End projects
• Stakeholder management includes persons working at higher grade
• Understanding data transformation, MI/BI reporting, Data Warehouse Concepts, Data Relational Models
• Ability to think logically and articulate process designs
• Good attention to detail/hands-on with excellent organizational skills
• Ability to collaborate across teams to deliver complex systems and components and manage stakeholder’s expectations well
• Experienced with planning, estimating, organizing, and working on multiple projects

Strong collaboration and delivery capabilities require solid Big Data Skills and Experience to effectively manage stakeholders and execute complex data projects.

Big Data Developer Key Responsibilities:

• Hands on Big Data Sr developer role (Spark Structured streaming + Spark SQL + Kafka)
• Actively participate in scrum calls, story points, estimates and own the development piece.
• Provide technical assistance to team and involve in efficient design
• Analyze the user stories, understand the requirements and develop the code as per the design
• Develop test cases, perform unit testing and integrating testing
• Support QA Testing, UAT and production deployment
• Develop batch and real-time data load jobs from a broad variety of data sources into Hadoop. And design ETL jobs to read data from Hadoop and pass to variety of consumers / destinations.
• Perform analysis of vast data stores and uncover insights.
• Analyze the long running queries and jobs, performance tune them by using query optimization techniques and Spark code optimization.

Big Data Developer Skills, Experience and Qualifications:

• 8 to 10 years of total IT experience including 5+ years of Big Data experience
• Experience in Spark Structured Streaming, Kafka, Spark SQL, Scala are must
• Experience on Databricks, Azure Data Factory, Azure Datalake service
• Experience in building real time data streaming pipelines from Kafka (or any message broker) using Spark Structured Streaming
• Hands on functional programming like Scala, Python or Java 8 prior to Big Data projects.
• Experience in designing Big Data projects, designing data models on Hive and HBase for high-performance and storage.
• Executed at least one end to end Hadoop data lake projects (streaming real time data) and led the developers’ team.
• Proficient in Linux/Unix scripting.
• Experience in Agile methodology is a must.
• Knowledge of standard methodologies, concepts, best practices, and procedures within HDF Big Data environment
• Self-starter and able to independently implement the solution.
• Good problem-solving techniques and communication
• Hands on experience with project management software, like MS Project
• Excellent verbal and written communication
• Excellent organization and time management skills

Big Data Engineer Duties:

• Use big data technologies to develop distributed, fault-tolerant scalable data solultions.
• Collect and process data at scale from a variety of sources for different project needs.
• Participate in identifying, evaluating, selecting and integrating big data frameworks and tools required for the big data platform.
• Design, develop, and maintain data pipelines , data platforms using selected frameworks and tools based on requirements from different projects.
• Convert structured and unstructured data in to the form that is suitable for processing. Provide support to different teams in analysizing data.
• Design, develop and maintain data API’s.
• Integrate data from variety of data sources using federation techniques.
• Develop solutions independently based on high-level design and architecture with minimal supervision.
• Monitor the performance of the big data platform on a regular basis and tune the infrastructure and platform components accordingly to ensure the best performance.
• Maintain a high level of expertise in data technologies and stay current on latest data technologiess.

Big Data Engineer Experience and Requirements:

• Bachelor’s Degree or higher in Computer Science or a related field.
• Overall 7+ year experience in software development.
• 3+ years of experience in data engineering.
• Prior experience with implementing big data platform components that are scalable, high performing, and lower in operations cost.
• Proven experience with integration of data from multiple heterogeneous and distributed data sources.
• Experience with processing large amounts of data (structured and unstructured.
• Experience in production support and troubleshooting.
• Hands-on knowledge of containers, API designing and implementing is a must.
• Experience with NoSQL databases, Graph databases, relataional databases, time series databases.
• Excellent knowledge of various ETL techniques and frameworks, various messaging systems, stream-processing systems, Big data ML toolkits, big data querying tools
• Experience in Python, Go, Perl, Javascript, Kafka, Spark, Kubernetees.
• Good knowledge of Agile software development methodology.
• Excellent interpersonal, communication (verbal and written) skills.
• Proven experience in managing and working with teams based in multiple geograhies.

Big Data Software Engineer Functions:

• Design and build massive Big Data analytical solutions utilizing graph, machine learning and text mining algorithms.
• Design and build Data infrastructures and tools leveraging Big Data industry standards and cutting edge frameworks 

• Work side by side with analysts to extract meaningful and actionable insights from PayPal data.
• Lead analytical projects from inception through research, development and all the way to production on PayPal’s data processing infrastructure
• passionate for about technology and for developing robust, scalable, state of the art software systems
• highly motivated, goal driven and have posses a can-do approach
• Innovative, entrepreneurial, team player, great at Ability to multi-tasking, curious and open minded

Big Data Software Engineer Requirements and Qualifications:

• B.Sc. in computer sciences/ mathematics or equivalent; or IDF technological unit technology experience
• Proven development experience in Java or/ Scala
• 4+ years' experience building production software systems
• Linux / other *nix - hands-on experience
• Hands on Some experience with different databases solutions (SQL/NOSQL)
• Excellent English (written and verbal)
• Hands on experience with Big Data and Streaming technologies: Hadoop / Spark / ElasticSearch
• Design and architecture experience, as well as knowledge and experience with object oriented design patterns
• Experience working on large-scale application deployments and performance tuning.
• Experience in text mining/machine learning / graph algorithms

Professionals with strong distributed systems experience can follow the Big Data Career Guide to advance into scalable data engineering roles.

Senior Big Data Software Engineer Duties:

• Define data requirements, gather and mine large scales of structured and unstructured data, and validate data using various data tools in the Big Data Environment
• Design and develop big data applications based on business requirements defined by research teams or business units
• Migrate or refactor big data proof of concept engagements to production applications using best practices and CI/CD standardization
• Develop and orchestrate big data pipelines that turn large scale raw data into meaningful units of analysis that pertain to specific research or data science problems.
• Apply software engineering rigor and best practices to machine learning, including version control, CI/CD, automation, etc.
• Facilitate the development and deployment of proof-of-concept machine learning systems
• Support model development, with an emphasis on auditability, versioning, and data security
• Leverage cloud technologies, namely Azure, to develop, deploy, manage, and govern DS and ML workflows or supporting resources

Senior Big Data Software Engineer Experience and Qualifications:

• Bachelors’ in Computer Science, Math or Scientific computing preferred.
• At least 5 years of recent experience in data engineering or data-oriented software development using Big Data frameworks and tools
• Fluency in Python, Bash, Pyspark and SQL
• Extensive experience in distributed compute environments, preferably Spark
• Experience with Big Data Cloud Environments such as Snowflake and Databricks
• Comfort with Linux administration
• Strong understanding of software testing, benchmarking, and continuous integration
• Ability to translate business needs to technical requirements
• Experience developing and maintaining ML systems built with open-source tools (MLFlow, specifically, is a big plus)
• Experience building custom integrations between cloud-based systems using APIs
• Strong software engineering skills in complex, multi-language systems
• Experience working within Agile Software Development cycles

 Data Solution Architect Roles:

• Manage and lead the Global Prospect database Design and Development
• Build and Enhance the existing capabilities and provide ongoing support
• Discover and Analyze the prioritized features to be implemented on an ongoing basis
• Maintain, Support, and continuously enhance prospect matching algorithms based on evolving needs
• Partner with business, analytics and machine learning teams to identify business problems and design big data and/or real-time solutions.
• Manage and execute on the opportunity backlog, analyze, and lead the build of prioritized stories
• Define technical architecture for new and existing solutions, and inform all development activities to align with the same
• Create a culture of innovation and experimentation, support full software development lifecycle that incorporates the best of technology approaches and delivery methodologies
• Lead a team of developers, engaging with them in day to day activities and helping in review of design and codes.

 Data Solution Architect Experience and Requirements:

• Bachelor’s in Information Technology, Computer Science, Mathematics, Engineering or equivalent
• 3+ years of experience in managing product or engineering teams, delivering business solutions across a variety of platforms and technologies such as Big Data, Pyspark, Hive, Scala, Python
• Experience in Agile Scrum methodology or the Software Delivery Life Cycle.
• Experience in designing scalable solutions and lead implementation of complex data products
• Hands on Experience with reporting, designing APIs, developing user interfaces, web services application architectures and microservices application architecture is preferred
• Strong program management, analytical & problem-solving skills
• Ability to think abstractly and deal with ambiguous/under-defined problems

Build a Job-Winning Big Data Resume Builder Today