Data Engineer
Technology
Montreal
Mar 19 2021
Agile
Apache
Python
Linux
SQL
MySQL
MongoDB
Jira
GitHub
AWS
Agile
We are currently looking for a Data Engineer for a SaaS company located in Montreal.
The Team
Joining a dynamic and ambitious team of tech hungry developers, you will be working in an agile environment building, maintaining,
evolving and creating big data solutions.
Our team works at a fast pace on awesome products that are constantly evolving in feature technology and traffic resulting in increase
volume of existing and new data to the pipelines. So we're looking for someone who can pick things up quickly.
Your job will be dynamic, interesting and challenging
The Challenge
• Implement in Apache Spark & Python environment
• Clean & prepare data for the Bi team
• Supporting and improving scripts for data ingestion and maintaining data in the data lake
• Monitor performance and recommend/implement any necessary infrastructure
changes
• Define data retention and access policies
• Automate data pipeline infrastructure in a Linux environment
• Design systems that are cost-aware and which provide best value
• Manage cluster and environment and related scripts/tools
• Work with developers and other infrastructure engineers to deliver complimentary solution
• Work with a skilled team of developers and architects within a group dynamic setting.
• Bring innovations and new solution to current and future challenge
As a Data Engineer you will design, develop, document, and test big data solutions. You will work on the collecting, storing, processing, and analyzing
of large (10^10) sets of data.
You are a fan of Hadoop technologies such as Apache Spark and HDFS this position is for you.
The team is seeking a talented, passionate, roll-up-your-sleeves, hands-on, highly independent data engineer that will participate in:
data pipelines
You are a great fit if
• 2+ years of experience as a Data Engineer
• 2+ years of experience with production Hadoop pipelines and tools
• 3+ years of experience in Python
• Experience with Scala and Spark
• 3+ years of experience working in a Linux environment
• Proficient understanding of distributed computing principles
• Practical experience with SQL and or NoSQL (MySQL | MongoDB | Elastic Search | MemSQL).
• Ability to understand and troubleshoot Big Data issues at many layers of the stack
• Ability to multi-task, prioritize and estimate effort
• Experience establishing integration standards and related processes.
• Detail oriented
• Good documentation habits
• Experience with workflow management tools (Jira, GitHub Workflow)
Even better if you have
• Self-supporting (able to set up and maintain dev environment)
• Experience with AWS EMR and AirFlow
• A worshiper of TDD
• Conceptual tool experience (UML Diagramming, Flow)
• Experience with containers and container-based architectures.
• Experience with R
• Experience with Machine Learning algorithms
• Experience with Apache Spark Dataframes|Datasets
• Experience working in an Agile environment
• Bilingual Preferred (English & French)
Company benefits
• Flexible hours
• 3 weeks of vacation plus 1 week at Christmas;
• Health insurance with dental;
• Life insurance;
• Public transportation card;
• RRSP contribution match program.
Create a Job Alert
Spam free job openings in your inbox. Only ones that are relevant to your interests will be sent - Unsubscribe at any time.
Related Jobs