ETL Azure Developer

General Responsibilities

• Design, develop and implement ingestion framework from Oracle source to Azure Data Lake – initial load and incremental ETL. Used tools are:
  • Azure Data Factory (expert knowledge) to maintain pipeline from Oracle to Azure Data Lake
  • Azure Synapse to build stored procedures and read data from data lake
• Review the requirements, database tables, and database relationships – Identify gaps and inefficiencies in current production reporting environment and provide recommendations address them in the new platform.
• Continue to evolve and design ingesting framework and CDC
  • Prepare design artifacts
• Analysis of data – physical model mapping from data source to reporting destination.
  • Understand the requirements. Recommend changes to the Physical model.
  • Develop the scripts of physical model, and create DB.
  • Access Oracle DB environments, use SSIS, SQL Server and other development tools for developing solution.
  • Proactively communicate with business on any changes required to conceptual, logical and Physical models, communicate and review dependencies and risks.
• Development of ETL strategy and solution based on different set of modules
  • Understand the Tables and Relationships.
  • Create low level design documents and unit test cases.
  • Create the workflows of package design
• Development and testing of data with Incremental and Full Load.
  • Develop high quality ETL mappings/scripts/jobs
  • ETL data from Applications to Data Warehouse
  • ETL data from Data Warehouse to Data Mart
  • Perform unit tests.
• Performance Review, data Consistency checks
  • Troubleshoot performance issues, ETL Load issues, log activity for each Individual package and transformation.
  • Review Performance of ETL Overall.
• End to end Integrated testing for Full Load and Incremental Load
• Plan for Go Live, Production Deployment.
  • Create production deployment steps.
  • Configure parameters, scripts for go live. Test and review the instructions.
  • Create release documents and help build and deploy code across servers.
• Go Live Support and Review after Go Live.
  • Review existing ETL process, tools and provide recommendation on improving performance and reduce ETL timelines.
  • Review Infrastructure and any pain points for overall process improvement
• Knowledge Transfer to Ministry staff, development of documentation on the work completed.
  • Document share and work on the ETL end to end working knowledge, Troubleshooting steps, configuration and scripts review.
  • Transfer documents, scripts and review of documents.

Skills

Experience and Skill Set Requirements

Please note this role is part of a Hybrid Work Arrangement and resource(s) will be required to work at a minimum of 3 days per week at the Office Location.

Experience:

• Experience of 7+ years of working with SQL Server, ADF and T-SQL Development (Must Have)
• Experience working with building Databases, Data Warehouse and Data Mart and working with delta / incremental and full loads (Must Have)
• Experience with any ETL tools such as SQL Server, ADF, Cloud tools (Must Have)
• Experience working with MSSQL Sever on premise and within Azure Environment (Must Have)
• Experience on Data modeling, and tools – e.g. SAP Power Designer,
• Experience with snowflake and star schema model. Experience in designing data warehouse solutions using slowly changing dimensions.
• Experience working with SQL Server SSIS and other ETL tools, solid knowledge and experience with SQL, other RDBMS (SQL Server, SSIS)
• Understanding data warehouse architecture with a data vault, dimensional data and fact model.
• Analyze, design, develop, test and document ETL programs from detailed and high-level specifications, and assist in troubleshooting.
• Utilize SQL to perform tasks other than data transformation (DDL, complex queries)
• Good knowledge of database performance optimization techniques
• Ability to assist in the requirements analysis and subsequent developments
• Ability to conduct unit tests and assist in test preparations to ensure data integrity
• Work closely with Designers, Business Analysts and other Developers
• Liaise with Project Managers, Quality Assurance Analysts and Business Intelligence Consultants
• Design and implement technical enhancements of Data Warehouse as required.

Skills:

• 7+ years in ETL tools such as Microsoft SSIS, stored procedures (Must Have)
• 2+ Azure Data Lake and Data Warehouse, and building Azure Data Factory pipelines (Must Have)
• 2+ years Python (nice to have)
• Databricks
• Synapse (nice to have)
• SQL Server
• Oracle
• Ability to present technical requirements to the business

Evaluation Criteria

Design Documentation and Analysis Skills (45%)

• Demonstrated experience in creating both Functional Design Documents (FDD) & Detailed Design Documents (DDD).
• Experience in Fit-Gap analysis, system use case reviews, requirements reviews, coding exercises and reviews.
• Experience in the development and maintaining a plan to address contract deliverables, through the identification of significant milestones and expected results with weekly status reporting.
• Work with the Client & Developer(s) assigned to refine/confirm Business Requirements
• Participate in defect fixing, testing support and development activities for ETL tool. Assist with defect fixing and testing support for Power BI reports.
• Analyze and document solution complexity and interdependencies by function including providing support for data validation.

Development, Database and ETL experience (45%)

• Demonstrated experience in Microsoft specific software development and a number of years of practical experience (minimum 7+ years)
• Proven experience in developing in Azure DevOps
• Experience in application mapping to populate data vault and dimensional data mart schemas
• Demonstrated experience in Extract, Transform & Load and Extract, Load and Transforms software development and a number of years of practical experience (minimum 7+ years)
• Experience in providing ongoing support on Azure pipeline/configuration and SSIS development
• Experience building data ingesting and change data capture using Golden Gate (an asset but not mandatory)
• Assist in the development of the pre-defined and adhoc reports and meet the coding and accessibility requirements.
• Demonstrated experience with Oracle and Microsoft interfaces
• Proficient in SQL and Azure DevOps
• Implementing logical and physical data models

Knowledge Transfer (10%)

• The Developer must have previous work experience in conducting Knowledge Transfer and training sessions, ensuring the resources will receive the required knowledge to support the system. The resource must develop learning activities using review-watch-do methodology & demonstrate the ability to prepare and present.
• Development of documentation and materials as part of a review and knowledge transfer to other members
• Development and facilitation of classroom based, or virtual instructor demo led sessions for Developers
• Monitor identified milestones and submission of status reports to ensure Knowledge Transfer is fully completed