Senior Data Engineer

We are looking for a Senior Data Engineer! Reach out if you are interested and feel free to refer friends/colleagues!

Type of Employment: Contract
Title: Senior Data Engineer
Term: 12-month –  40 hours
Location: Toronto, Mississauga, Etobicoke and GTA – Hybrid, must be okay with eventually coming into office 1-2/ week
Job ID number: C1027

Brief description of duties:

• Some of the challenges the candidate would tackle are:
• Design, implement and operate the client’s core datasets that have extreme requirements on scalability, reliability, maintainability, flexibility, auditability, and quality
• Design large scale streaming and batch data pipelines (billions of data points) that involve analyzing transactions over a long period without reprocessing vast amounts of data
• Perform data profiling and source to target mapping analysis for the best design
• Collaborate with business solution analysts, other engineers, solution architects, and other team members to innovate and evolve our datasets to data products to create a single coherent platform with sources of truth that serve a plethora of stakeholders
• Apply your expertise in data and software engineering to design and implement data products that meet extreme requirements on scalability, reliability, maintainability, flexibility, auditability, and quality
• Be T-Shaped. Your primary area is data engineering, but you are comfortable working in a secondary area of expertise such as data presentation/visualization, backend engineering, or data modelling (SQL, NoSQL, Graph & Time-series)
• Work closely with cross-functional teams of data, backend and frontend engineers, product owners, technical product owners, and technical support personnel
• Gaining technical expertise in building a data platform at scale to solve business, product, and technical use cases
• Getting hands-on experience with technologies such as Elasticsearch, Apache Airflow, Apache Kafka, Apache Beam, Apache Spark, Hive, HDFS, Kubernetes (Openshift)
• Getting hands-on experience with Google Cloud Platform and technologies such as BigQuery, Cloud Composer, Pub/Sub, Dataflow, Dataproc, GCS, Looker, and other cloud-native offerings in GCP

MUST haves:

• 6+ years of professional software engineering and programming experience (Java, Python) with a focus on designing and developing complex data-intensive applications
• 3+ years of architecture and design (patterns, reliability, scalability, quality) of complex systems
• Advanced coding skills and practices (concurrency, distributed systems, functional principles, performance optimization)
• Professional experience working in an agile environment
• Strong analytical and problem-solving ability
• Strong written and verbal communication skills
• Experience in operating and maintaining production-grade software
• Comfortable with tackling very loosely defined problems and thrive when working on a team which has autonomy in their day to day decisions
• In-depth knowledge of software and data engineering best practices
• Experience in mentoring and leading junior engineers
• Experience in serving as the technical lead for complex software development projects
• Experience with large scale distributed data technologies and tools
• Experience pulling data from a variety of data source types including Mainframe (EBCDIC), Fixed Length and delimited files, databases (SQL, NoSQL, Time-series)
• Strong coding skills for analytics and data engineering (Java, Python, and Scala)
• Experience performing analysis with large datasets in a cloud-based environment, preferably with an understanding of Google’s Cloud Platform (GCP)
• Understands how to translate business requirements to technical architectures and designs
• Comfortable communicating with various stakeholders (technical and non-technical)

Nice to haves:

• Exposure to data-science or machine-learning packages (Pandas, Pytorch, Keras, TensorFlow, etc…)
• Contributions to open-source software (code, docs, or mailing list posts)
• GCP Professional Data Engineer Certification
• Confluent Certified Developer for Kafka