Data Engineer Roles:

• Architects data structures; set and monitor standards.
• Great time management and self-direction and able to work independently while contributing on team efforts with clear communication
• Can scope, estimate, develop, document, and test data/database/dataset functionality
• Incorporate disparate data sources into a database/data lake for consumption via analytics tools such as Tableau or Looker
• Ability to work with technical team to resolve data discrepancies, and work within project management tool like Jira to track work items and defects
• Design, create, modify and review database objects (tables, views, indexes, keys, stored procedures, functions, DB links, etc.) to support development projects.
• Troubleshoot production issues related to data and SQL code
• Coordinate with VP of technology to manage projects/priorities

Data Engineer Education and Experience:

• 3+ years of practical experience building and supporting Postgres databases
• Expert understanding of SQL including stored procedures and functions, permissions, bulk load/export
• Insistence on DRY methodology
• Understanding of normalization and its tradeoffs
• Documentation experience for both engineering and cross functional documentation
• Ability to manage, describe maintain and document disparate data flows and sources
• Experience automating and routine tasks and processes
• Familiarity with HIPAA compliance requirements
• Coding and scripting experience using Python
• Experience using an ORM such as Django ORM or SqlAlchemy
• Experience with data operations tools (Keboola and Snowflake)

Data Engineer Responsibilities:

• Partner with senior executives on customer opportunities to gather customer requirements, develop designs and present potential data, DBAaaS  (Database Administrator as a Service), DBaaS (Database as a Service) solutions.
• Provide expert technical consulting to internal teams and customers; serve as technical contact to outside vendors as required.
• Provide training to Atmosera personnel as required.
• Participate in a walkthrough of deliverable data and application components.
• Provide appropriate feedback to ensure the technical quality of the application or system.
• Provide technical solutions to design, program, and process issues.
• Research, evaluate and recommend the use of new products and services to support customer applications and system software.
• Coordinate and participate in the implementation of new support and system software and ensure compliance with Atmosera standards and processes where they exist or are defined.
• Define and maintain all design documentation on Internal and External (customer) DBs and data pipleines including schema design, HA, backup, and replication.
• Ad-hoc break/fix escalation support for data designs and DBs which may include on-call and after-hours support.
• Development of proposals for data design and application refactoring for clients in cases of performance issues or modernization initiatives.
• Define and maintain a clear scope on project assignments, including coordinating resolution of technical practices and issues with all levels of management, including executive management.
• Prepare and present periodic progress or status reports on project and/or work assignments, including keeping project stakeholders informed of significant issues that may impact assignment completion as early as possible.
• Participate in project budget and work assignment preparation by assessing complex requirements, completing cost estimates, and outlining proposed expenditures.
• Plan, prepare, participate, and facilitate work sessions such as meetings, brainstorming, requirements gathering, application design & modeling sessions. 
• Ensure that applicable documentation is complete, accepted, and that all participants are actively involved.
• Gain, maintain and apply effective technical, functional, and interpersonal management techniques.
• Keep abreast of industry hardware/software technology and software development and support techniques in order to improve internal processes and application efficiency.
• Participate in the research of the feasibility of new technologies and leveraging uses for current technologies.
• Direct or perform the life-cycle activities of projects to ensure the accurate, timely and cost-effective implementation of specific systems. 
• Ensure appropriate mechanisms exist for resolving issues, setting technical and design directions.
• Participate in Agile SCRUM or Kanban planning, standups, and retrospectives.
• Estimating work, tracking progress, and implementing continuous improvement through retrospectives.

Data Engineer Knowledge, Skills and Abilities:

• Expertise in Microsoft data products
• Experience in Azure PaaS SQL services including Azure Data Warehouse as well as Azure Data Lake.
• Experience with BI tools such as Dynamics.
• Experience with data visualization technologies such as PowerBI and SSIS.
• Experience in designing, building and maintaining large scale MS-SQL geographically diverse HA clusters.
• Excellent oral and written communication skills; including documentation and presentations.
• Excellent organizational and time management skills; including tracking of billable hours.
• Ability to perform well in a dynamic environment while maintaining schedules.
• Ability to generate presentation materials suited to many different audiences, i.e. end users, TACC, NOC, Executives, etc.
• Require limited supervision and direction while driving results and managing priorities independently.
• Experience with non MS-SQL SQL products.
• Prior tools development in C# and the ability to program in other languages like: Python, Django, etc.

Responsibilities of Data Engineer:

• Developing scalable ETL jobs and managing data pipelines
• Using SQL queries / Python and API calls to stand up and merge datasets across disparate systems
• Designing a data architecture that can support both storage of high volumes of low-level log and vision data as well as speed-of-thought analytics (manual and machine).
• This entails selecting the correct services optimized for data storage and querying, and creating pipelines that synchronize and cache these services. This also entails  selecting the location of those services--edge, on-prem, and cloud.
• Ensuring very large databases and compute clusters operate optimally via sharding, index design, and materializing views
• Implementing and maintaining database structures, security, permissions, and general governance
• Developing/maintaining documentation on databases and production tables
• Rapidly gain experience in the general behavior of Lamwork’s robotic systems, data collection, logging, and be an effective all-around contributor to Lamwork’s data logging and extraction efforts at scales from low-level robotics to high-level customer inventory scheduling.
• Be able to connect high-level misbehavior or errors of robotic systems to relevant data that needs to be extracted.
• Be capable of working with Lamwork’s existing data pipeline and logging infrastructure (access and/or edit and improve if required) and extracting any relevant data
• Be able to concretely summarize and inter-relate complex technical information.
• Build concrete causal hypotheses to guide data extraction.
• Extract data, test hypotheses, and present findings in high level human-readable reports.
• Understand that logging and data extraction itself can be the cause of problems in real-time systems, and be able to work closely with engineers to ensure that systems work smoothly.
• E.g., A robot dropping a tray could be related to the computer logging too much data & causing latency in the controller.
• Gain an in-depth understanding of the utility of data to particular decisions being made by the engineering and business teams
• Be able to extract relevant information, summarize it, and present it to relevant engineering, product, sales, and management personnel

Qualifications of Data Engineer:

• BA / MS degree in Computer Science, Statistics, or related discipline
  
• MS in Computer Science strongly preferred
• Basic understanding of statistics and data analysis methods including regression, confidence intervals, and hypothesis testing
• Competence in dealing with multiple types of data: real-time data, asynchronous data, bulk logs etc.
• Experience with standing up ETL pipelines to transfer massive volumes of data
• Experience working with Hadoop or Spark-based data platforms
• Experience processing and manipulating data, preferably in Python (e.g., with Pandas)
• Experience in building data visualization applications
• Strong proficiency in SQL, Python, and working with REST APIs
• 2-5+ years experience as a data scientist, data engineer, or related role
• Preferred: Some experience with BI tools, preferably in Tableau
• Preferred: Experience with ElasticSearch

Data Engineer Functions:

• Develop technical solutions using proven techniques in data and analytics processes
• Develop, prototype, and build frameworks based on open source and commercially available tools
• Orchestrate and maintain data pipelines that meet security standards and ensure the integrity and quality of data
• Demonstrate a passion for serving the needs of internal and external customers by enabling them with self-service reporting tools and analytics capabilities
• Drive the execution of data initiatives that provide key performance metrics
• Understand the data-related challenges, nuances, and requirements to identify and recommend the optimal technical approach
• Train and educate team members as well as stakeholders about best practices in data engineering and governance
• Collaborate closely with the engineering and devops team to implement DataOps, thus reducing our analytics development cycle
• Research and improve our data platform to ingest, process, transform, and distribute insightful data to our audience ranging from executives, analysts, and engineers to customers, vendors, and partners
• Evangelize data-driven culture by breaking down silos and encouraging data sharing

Data Engineer Qualifications & Experience:

• 3+ years of hands-on experience in data engineering for a SaaS company or a mature startup
• Proven experience working with various tools but more importantly, familiarity with how to best assemble and deploy production ready data stack to any cloud environment
• BS in a quantitative or scientific field such as computer science, computer engineering or equivalent experience
• Experience in applying agile software development approach - Git, CI/CD, Jira, etc - to data engineering
• Familiarity with popular programming languages (such as Ruby, Python, .NET, etc)
• Exceptional fluency with SQL; you conquered the join venn diagram long ago and have moved on to explaining cost based optimization to your peers on the engineering team
• Some level of experience working in the cloud - AWS, Azure, or GCP
• Experience with ingesting, processing, and visualizing data sources of varying types - structured/relational and unstructured
• Experience in developing, managing, and manipulating large, complex datasets
• Data-driven, detail-oriented individual with excellent storytelling and problem-solving abilities
• Ability to work independently and autonomously, as well as part of a team
• Superb time management, prioritization of tasks and ability to meet deadlines with little supervision

Data Engineer Roles:

• Develop processes and pipelines for cleaning, validating, and uploading data to Excel-readable Write reproducible scripts (Python preferred) that will classify, clean, and visualize unstructured text Develop scripts to extract data from text files, PDFs, reports, bank statements, financial statements, etc.
• Load large datasets received from clients into MS-SQL databases
• Liaise with clients and IT to determine ideal mode of delivery of digitized data/information from Maintain an internal repository of scripts for common work to be used throughout the firm
• Train firm employees on technical resources and documentation developed by Data Engineering team
• Develop and test algorithms and re-evaluate on an on-going basis to identify opportunity for increased efficiency
• Maintain and build features for a custom document search tool built in Python, R and SQL
• Keeps abreast of new tools, software, and technologies and make recommendations to project teams for future deployment

Data Engineer Qualifications:

• Bachelor’s Degree in Computer Science, Computer Engineering, or related field
• Deep fluency with Python
• Proficiency with SQL and utilizing SQL in data solutions
• Experience with R is preferred
• Experience with a compiled language, such as Java, C#, or C++, is a plus
• Web development skills is a plus
• Experience with cloud computing services also a plus (Microsoft Azure or AWS)
• Working knowledge of visualization platforms, e.g. Power BI, D3, etc.
• Ability to build rapport and maintain professional client relationships
• Ability to set and convey project timelines
• Fast-paced office environment handling multiple tasks and demands
• Must be able to exercise appropriate judgment as necessary
• Requires the ability to concentrate and pay close attention to detail
• Requires flexibility to work long hours, occasional nights/weekends, including travel

Data Engineer Duties and Roles:

• Work with product, engineering & business teams to deliver complex data analysis requests
  
• Visualize datasets across multiple databases & warehouses using tools such as Tableau, D3, Looker, etc.
• Build financial models & growth projections for new products and business initiatives
• Build ETL pipelines for regular reporting on business and operational KPIs
• Help business understand key trends by executing complex analysis via Tableau or ad-hoc SQL queries
• Coordinate within cross-functional teams such as engineering, product, marketing, customer experience for various data analysis needs
• Proactively build data and event-driven dashboard for real-time business operations and consumer insights

Data Engineer Minimum Requirements:

• Bachelors in CS, Statistics, Economics or Engineering, Masters preferred
• 3+ years of hands-on SQL experience
• 2+ years of experience in using data visualization tools such as Tableau, Looker, PowerBI
• 2+ years of experience in building financial models, growth projections & ETL data pipelines
• Experience either in R or Python and working with data warehousing solution such as AWS Redshift or Google BigQuery
• Curious, proactive, fast learner able to quickly picking up new areas
• Experience with agile software development methodologies and tools
• Good communication skills

Data Engineer Duties:

• Partner with the product teams to develop data testing strategies
• Create test plans and test cases
• Develop data quality automation tests and data pipelines
• Test large and complex data sets
• Ensure a high level of data integrity
• Define strategies for continuous improvement of data quality
• A passion for working with large data sets
• Experience with either SQL or noSQL databases
• Proficient with Python and SQL scripting
• Strong analytical and problem-solving skills
• Exceptional attention to detail and a positive attitude

Data Engineer Requirements:

• A Bachelors, Masters, or PhD in Computer Science, Information Technology, Computer Engineering or some related discipline
• Experience with AWS (especially RedShift)
• Familiarity with Snowflake
• Knowledge of workflow management platforms (especially Airflow)
• Experience in software engineering best practices such as code reviews, testing frameworks, maintainability and readability
• Ability to create cloud architecture diagrams, and experience in a consulting environment
• Docker and containerized deployment
• Non-relational database knowledge

Data Engineer Responsibilities:

• Managing standardization, migration, transformation, validation, and quality assurance of data within multi-database platforms
  
• Leveraging internal and external ETL tools for data processing and publishing Identifies
• Maintains company databases, including data sources, data structures, data organization, and data optimization
• Identifying complex issues proactively and is responsible to see them through resolution, including identifying trends through data analysis and manipulation
• Specific client data life-cycles from discovery to implementation to maintenance
• Formulating and monitors policies, procedures, and standards relating to database management
• Responding to production defects and relays information back to the Operations Manager to communicate to clients
• Contributing in all phases of the data and software development lifecycle when needed
• Creating and maintaining code through GitHub repository for change control
• Supporting off-hours data processing and emergency requests as needed

Data Engineer Skills, Experience, Qualifications:

• Bachelor's degree (B. A. / B. S.) from four-year college or university;
  
• Two to four years related experience and/or training; or equivalent combination of education and experience.
• Experience with SQL, database design, optimization, and tuning
• Experience with Postgresql
• Experience using Github
• Experience in Shell Scripting and one other object oriented language such as Python, or PhP.
• Experience in continuous integration and development methodologies tools such as Jenkins
• Experience in an Agile development environment
• Time management skills
• Programming skills particularly SQL, Shell Scripting, and Python

Data Engineer Duties and Responsibilities:

• Building and maintaining pipelines to gather, enrich and merge public and private data sources, making them ready to sing harmony to a customer dataset’s melody
  
• Redesigning and better automating our customer dataset analysis and delivery process
• Handle everything from custom scoring algorithms to anomaly detection to monitoring pipeline health to machine learning solutions
• Extending our pipelines to work with occasionally-quirky client sources, mapping schemas and working within their security requirements
• Improving data storage, indexing and API-based service of data to user interfaces
• Working in an agile team in a scrum process, collaborating closely with other software engineers and product management

Data Engineer Skills, Experience, Qualifications:

• Bachelor’s degree or equivalent experience in mathematics, statistics, economics, computer science, or similar
• Experience in data engineering; likely other software engineering experience as well
• Understanding of statistics
• Experience with using and maintaining relational database systems
• Familiarity with the AWS ecosystem
• Authorized to work in the United States
• Experience in or exposure to the constraints of a startup environment
• Highly attentive to detail, with a skeptical sixth sense about data quality
• Ability to work independently in a challenging, fast-paced environment with several ongoing concurrent projects
• A can-do mentality, with the willingness to roll up your sleeves and take initiative to solve something when necessary
• Basic knowledge of common machine learning techniques
• Curious and eager problem solver, able to self-teach new skills when needed, with a hunger for building well-designed, high-quality solutions.
• Recognition that there are always multiple answers to a problem and the ability to engage in a constructive dialogue to find the best path forward.
• Ability to commute to the Pittsburgh, PA office daily (once current pandemic constraints are lifted)

Data Engineer Roles and Responsibilities:

• Automate the processing of raw data at scale as part of an end-to-end analytics solution
• Work as part of a cross-functional team to build scalable high quality data solutions
• Serve as the team’s subject matter expert to the full suite of data tools and resources
• Work with product owners in an agile environment to analyze requirements and translate them into functioning software
• Gain deeper insight into the data and find ways to best apply it to solving our customers’ problems
• Dive into difficult problems, and successfully deliver results on schedule

Data Engineer Skills, Experience, Qualifications:

• Bachelor’s, master’s, or doctorate degree in a related field, or an intriguing reason for not having one
• Expertise in data modeling, ETL development, and data warehousing plus hands-on experience with different data warehouse 
• Processing technologies such as AWS Redshift, Oracle, PostgreSQL, Hadoop, Spark, etc.
• Proficiency in SQL
• Hands-on experience with cloud computing and Linux-based systems
• Shows curiosity and an ability to learn quickly, especially new technology and processes
• Approaches all work with a team-based/collaborative orientation
• High-level scripting language (e.g., Python, Ruby, etc.)
• Experience with ETL tools including Pentaho, Talend, or Informatica
• Hands-on experience with benefits data
• Experience with JavaScript
• Experience with AWS technologies including Redshift, RDS, S3
• Demonstrable skills and experience using SQL with large data sets (e.g. Redshift, Oracle, SQL Server)

Data Engineer Roles:

• Develop and maintain effective database solutions and models to store and retrieve company data.
  
• Working with the data architects to implement tools to clean, match, and transform data from different sources
• Develop and maintain data standards as well as managing the data libraries and dictionaries at all times
• Work with business and technology stakeholders to translate business problems into data insights
• Creates optimal designs (PowerBI) through iterative development cycles, aligning user needs with organisational objectives and system requirements

Data Engineer Skills & Experience:

• Experience of driving change in a software-driven business
• Demonstrable experience in leading the delivery of significant and complex projects and programmes
• Knowledge of Azure Data Management tools (e.g. Azure SQL, Azure DW, Azure Data Factory, Spark, HDInsight, Power BI)
• Experience with data visualization tools
• Experience in implementing operational automation using scripts
• Evidence of tangible business change both directly and through others
• Able to manage and influence stakeholders at all levels of the organisation
• In depth knowledge of all project management methodologies and techniques
• Proven application of project management methodologies and techniques such as Scrum\Kanban