Limited Time Offer!

For Less Than the Cost of a Starbucks Coffee, Access All DevOpsSchool Videos on YouTube Unlimitedly.
Master DevOps, SRE, DevSecOps Skills!

Enroll Now
Uncategorised

Azure DevOps Pipelines Fundamental Tutorials

September 24, 2024 Leave a Comment

Here is an in-depth, step-by-step tutorial on Azure DevOps Pipelines that covers everything from the fundamentals to advanced concepts. This guide is designed to help you understand the basics of pipelines, how to set them up, and how to configure them for advanced use cases such as continuous integration (CI) and continuous deployment (CD).

1. Introduction to Azure DevOps Pipelines

Azure DevOps Pipelines allow you to automate builds, tests, and deployments in a repeatable and scalable manner. It integrates well with various development languages and platforms such as .NET, Java, Node.js, Python, and more.

Key Concepts of Pipelines:

  • Pipelines: Automates building, testing, and deploying your applications.
  • Continuous Integration (CI): Automates building and testing code whenever a developer commits changes.
  • Continuous Deployment (CD): Automates deploying the code to production or other environments after successful builds and tests.
  • YAML Pipelines: The pipeline is defined in YAML, making it version-controllable and portable.

2. Setting up Azure DevOps Pipelines (Fundamentals)

Step 1: Creating a Project in Azure DevOps

  1. Sign in to your Azure DevOps organization or create a new organization.
  2. Click on New Project.
  3. Name your project, select visibility (Public or Private), and click Create.

Step 2: Setting Up a Pipeline

  1. Navigate to Pipelines → Create Pipeline.
  2. Select your repository where the code is stored (e.g., GitHub, Azure Repos, Bitbucket).
  3. Choose YAML or Classic Editor for pipeline creation:
    • YAML: Recommended for flexibility and version control.
    • Classic Editor: GUI-based setup, easier for beginners.

Step 3: Configuring a YAML Pipeline

  1. Azure DevOps will automatically detect your project type (e.g., .NET Core, Python, Node.js) and suggest a pipeline template.
  2. Customize your YAML file. Example for a .NET Core project:
trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: UseDotNet@2
  inputs:
    packageType: 'sdk'
    version: '6.x'

- script: dotnet build
- script: dotnet test

3. Pipeline Architecture & Components

Triggers:

Triggers are used to specify when the pipeline should run. There are two primary types:

  • CI Trigger: Automatically runs the pipeline when code is pushed or merged to a specific branch.
trigger:
  branches:
    include:
      - main

Scheduled Trigger: Runs the pipeline at specific times.

schedules:
  - cron: "0 0 * * *"
    branches:
      include:
        - main

Pools:

A pool defines the agent (machine) that will run your pipeline. Example:

pool:
  vmImage: 'ubuntu-latest'

Azure DevOps supports various agents such as Microsoft-hosted agents (Linux, Windows, macOS) or self-hosted agents.

Steps:

Each pipeline is made of steps. These can be scripts or tasks.

  • Script: Custom commands.
steps:
  - script: echo Hello, World!

Task: Predefined tasks like build, test, or deploy.

steps:
  - task: DotNetCoreCLI@2
    inputs:
      command: 'build'
      projects: '**/*.csproj'

4. Advanced Features in Azure Pipelines

Multi-Stage Pipelines (CI/CD):

Multi-stage pipelines split the build, test, and deployment phases across multiple stages. Each stage can be executed sequentially or in parallel.

Example:

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        steps:
          - script: echo Building...

  - stage: Deploy
    dependsOn: Build
    jobs:
      - job: DeployJob
        steps:
          - script: echo Deploying...

Environments and Approvals:

Environments in Azure Pipelines represent the different deployment environments (e.g., Dev, QA, Prod). You can set up approvals for sensitive environments like Production.

Example:

environments:
  - name: 'production'
    resourceName: 'ProdEnvironment'
    approval:
      steps:
        - approval: Manual
          approvers:
            - user@company.com

Conditional Logic:

You can set conditions in the pipeline to determine when certain steps or stages should run. Example:

- script: echo This runs only on main branch
  condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')

5. Pipeline Testing Strategies

Unit Tests:

Run unit tests in your pipeline to ensure that changes do not break the application. Example for .NET Core:

steps:
  - task: DotNetCoreCLI@2
    inputs:
      command: 'test'
      projects: '**/*.Tests.csproj'

Code Coverage:

You can integrate code coverage tools like Cobertura or JaCoCo to report on how much of your code is covered by tests. Example (JaCoCo for Java projects):

steps:
  - task: Maven@3
    inputs:
      goals: 'verify'
      options: '-Djacoco.skip=false'

6. Continuous Deployment (CD)

Step 1: Configuring Deployments to Azure

Azure DevOps integrates easily with various deployment platforms such as Azure App Services, AKS (Kubernetes), Virtual Machines, or Containers.

Example for deploying to Azure App Service:

steps:
  - task: AzureRmWebAppDeployment@4
    inputs:
      azureSubscription: '<your-service-connection>'
      appName: '<your-app-service-name>'
      package: '$(System.DefaultWorkingDirectory)/drop/*.zip'

Step 2: Rolling Back Deployments

If the deployment fails, you can automatically roll back to the previous version by setting rollback conditions.

7. Pipeline Variables & Secrets Management

Variables:

Variables in Azure Pipelines allow you to store values that can be reused across multiple steps.

variables:
  buildConfiguration: 'Release'

Secret Variables:

Sensitive values like API keys can be stored as secret variables.

variables:
  - name: 'API_KEY'
    value: '$(API_KEY)'  # Referenced securely

8. Best Practices for Azure DevOps Pipelines

  • Use YAML Pipelines: YAML pipelines are easier to version and maintain than classic pipelines.
  • Break Down Pipelines: Split pipelines into smaller stages for faster feedback.
  • Parallel Jobs: Use parallel jobs for steps that are independent, reducing pipeline execution time.
  • Environment-Specific Variables: Use variable groups or pipeline environments for managing configuration values for different environments.
  • Automate Rollbacks: Implement automatic rollback strategies in case a deployment fails.

1. Advanced Pipeline Triggers

While basic triggers (CI and scheduled triggers) are common, advanced use cases require more sophisticated triggering mechanisms.

Path-Based Triggers:

You can configure pipelines to trigger only when changes occur in specific files or directories.

trigger:
  branches:
    include:
      - main
  paths:
    include:
      - src/**
    exclude:
      - docs/**

Tag-Based Triggers:

To trigger pipelines based on Git tags, useful for triggering deployment pipelines after tagging a release.

trigger:
  tags:
    include:
      - v*

Pipeline Triggering Other Pipelines:

You can chain multiple pipelines together by triggering one pipeline after another.

resources:
  pipelines:
    - pipeline: buildPipeline
      source: 'BuildPipeline'
      trigger: true

2. Multi-Stage Pipelines for End-to-End CI/CD

Multi-stage pipelines are a crucial feature that allows splitting the pipeline into various build, test, and deployment stages. This ensures clear separation of concerns and flexibility in the deployment process.

Example: Multi-Stage CI/CD Pipeline for a Web Application

trigger:
  branches:
    include:
      - main

stages:
  - stage: Build
    jobs:
      - job: BuildJob
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: DotNetCoreCLI@2
            inputs:
              command: 'build'
              projects: '**/*.csproj'
          - task: PublishBuildArtifacts@1
            inputs:
              pathToPublish: $(Build.ArtifactStagingDirectory)
              artifactName: drop

  - stage: Test
    dependsOn: Build
    jobs:
      - job: TestJob
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: DotNetCoreCLI@2
            inputs:
              command: 'test'
              projects: '**/*.Tests.csproj'

  - stage: Deploy
    dependsOn: Test
    condition: succeeded()
    jobs:
      - job: DeployJob
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - script: echo Deploying to production

This example demonstrates how to break the pipeline into build, test, and deployment stages, with dependencies and conditions to ensure smooth flow.

Advanced Stage Dependencies:

You can create complex dependencies between stages using conditions. For example, skip a stage if the previous one fails.

condition: always()

3. Deployment Strategies

Azure Pipelines support a variety of deployment strategies to ensure safe and efficient releases.

Rolling Deployments:

Rolling deployments gradually replace instances of your service with the new version, reducing downtime.

steps:
  - task: AzureRmWebAppDeployment@4
    inputs:
      azureSubscription: '<subscription>'
      WebAppName: '<app-service>'
      DeploymentMethod: 'rolling'

Canary Deployment:

Canary deployments send a small percentage of traffic to the new version to ensure the changes work as expected before rolling them out entirely.

This often involves external services like Azure Traffic Manager or custom traffic-routing logic. Integration can be done with custom pipeline logic or by using additional Azure App Services for traffic splitting.

4. Pipeline Templates for Reusability

For teams working on multiple pipelines, it is useful to create templates that can be reused across pipelines. Templates allow you to define jobs, steps, or entire pipelines that can be imported into other pipelines.

Pipeline Template Example:

Define a reusable build step:

# templates/build-template.yml
parameters:
  - name: buildConfiguration
    type: string
    default: 'Release'

steps:
  - script: |
      echo Build project in ${{ parameters.buildConfiguration }} mode.
      dotnet build --configuration ${{ parameters.buildConfiguration }}

Using the Template:

# azure-pipelines.yml
stages:
  - stage: Build
    jobs:
      - template: templates/build-template.yml
        parameters:
          buildConfiguration: 'Debug'

5. Advanced Environment Management & Approvals

Azure DevOps pipelines support environments that represent different stages in the deployment lifecycle (e.g., Dev, QA, Staging, Production).

Defining Environments:

environments:
  - name: 'production'
    resourceName: 'ProdAppService'
    protectionRules:
      - approvals:
          steps:
            - approvers: [user1@company.com, user2@company.com]
              timeoutInMinutes: 30
              approvalType: 'Manual'

This ensures that any deployment to production requires a manual approval from specified stakeholders.

Advanced Approvals and Gates:

Gates provide checks that must be passed before deployment can proceed. This can include external APIs, system health checks, or compliance scans.

Example of using a REST API gate:

gates:
  - gate: RestApiGate
    inputs:
      method: GET
      url: 'https://healthcheck.yourdomain.com/api/status'
      successCriteria: eq(root['status'], 'healthy')

6. Secret and Secure File Management

Handling sensitive data is critical in CI/CD pipelines. Azure DevOps offers Azure Key Vault integration to securely manage secrets and credentials.

Accessing Secrets from Azure Key Vault:

variables:
  - group: MyVariableGroup
  - name: 'API_KEY'
    value: $(API_KEY)
  - name: 'KEY_VAULT_SECRET'
    value: $(MySecret)

7. Parallel Jobs & Matrix Builds

In many cases, especially when working on large projects, running builds in parallel can significantly reduce build time.

Matrix Builds:

Matrix builds allow you to run the same job with different configurations in parallel.

strategy:
  matrix:
    Linux:
      vmImage: 'ubuntu-latest'
      configuration: 'Debug'
    Windows:
      vmImage: 'windows-latest'
      configuration: 'Release'

8. Pipeline Caching and Artifacts

Azure Pipelines support caching to reduce pipeline run times by storing dependencies between builds.

Caching Dependencies:

steps:
  - task: Cache@2
    inputs:
      key: 'npm | "$(Agent.OS)" | package-lock.json'
      path: 'node_modules'
      restoreKeys: |
        npm | "$(Agent.OS)"

Using Build Artifacts:

Artifacts are files or packages generated by your build that can be shared or deployed.

steps:
  - task: PublishBuildArtifacts@1
    inputs:
      pathToPublish: $(Build.ArtifactStagingDirectory)
      artifactName: 'drop'

9. Integrating with External Tools

Azure Pipelines can be integrated with third-party services like Slack, Jira, or ServiceNow.

Slack Notifications:

Azure Pipelines can send build notifications directly to your Slack channels.

steps:
  - task: SlackNotification@1
    inputs:
      slackConnection: 'MySlackServiceConnection'
      message: 'Build completed'
      channel: '#devops'

0. Optimizing Pipeline Performance

Best Practices:

  • Use Hosted Agents for Speed: Leverage Microsoft-hosted agents for faster setup and execution, or use self-hosted agents for specialized environments.
  • Limit Resource Usage: Avoid unnecessary resource allocation by using pipeline stages and jobs efficiently.
  • Parallel Execution: Run independent jobs in parallel to speed up builds.
  • Cache External Dependencies: Use caching for dependencies like npm, NuGet, or Docker images.

Resource Limiting:

To prevent resource waste:

jobs:
  - job: Build
    pool:
      vmImage: 'ubuntu-latest'
    timeoutInMinutes: 20
    cancelTimeoutInMinutes: 5
Rajesh Kumar
Follow me
Latest posts by Rajesh Kumar (see all)
Subscribe
Notify of
guest
0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x