github/super-linter - Azure DevOps Pipelines

Linting is a form of static code analysis. It analyzes the code written against  rules for stylistic or programmatic errors. GitHub Super Linter is a docker image combining multiple commonly used linters which can be use very easily for automated lint validations/fixes. Super-Linter lets you set up automated runs for these linters, as well as manage multiple linters in a single project!

Below is an Azure DevOps Pipeline YAML which can be used with Azure Repo for running github/super-linter on SQL with SQLFluff linter (easy to add on additional linters or change it to another) for your DevOps organization. If you use Github and Github Actions you will have even more possibilities of automating pipelines with Github repos.

pool:
  vmImage: ubuntu-latest
 
stages:
- stage: PreDeployment
  jobs:
    - job: Code_Validation
      steps:
      - powershell: |
            ## remove 'refs/head' that is prefixed to branch names
            $sourceBranch = "$(System.PullRequest.SourceBranch)"
            $sourceBranch = $sourceBranch.Replace("refs/heads/", "")
            $targetBranch = "$(System.PullRequest.TargetBranch)"
            $targetBranch = $targetBranch.Replace("refs/heads/", "")

            ## auth headers
            $headers=@{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("azdo:$(System.AccessToken)")) }

            ## get PR diff changes from API
            $response = (Invoke-WebRequest -Uri "$(System.CollectionUri)$(System.TeamProject)/_apis/git/repositories/$(Build.Repository.ID)/diffs/commits?baseVersion=$targetBranch&targetVersion=$sourceBranch&api-version=5.1" -Method GET -Headers $headers  | ConvertFrom-Json)

            ## get path to changed files only, join them by spaces
            $paths = $response.changes | Where-Object { -not $_.item.isFolder -or $_.item.isFolder -eq $false } | ForEach-Object { $_.item.path }
            echo "$paths"

            ## IF not using PR but trigger from repo commit, the following can be used to find commited changed files
            ## $files=$(git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion))
            ## $files=$(git diff --name-only HEAD~ HEAD)
            
            $temp=$paths -split ' '
            $count=$temp.Length
            echo "Total changed $count files"
            
            ## create a temp folder to hold the changed files in the PR
            New-Item -Path . -Name "lintfiles" -ItemType "directory" -verbose | Out-Null
            
            ## Copy in super-linter configurations - see doc in github/super-linter
            Copy-Item -Path .sqlfluff -Destination "lintfiles" -verbose
            Copy-Item -Path .sqlfluffignore -Destination "lintfiles" -verbose

            ## copy changed files in the PR to lintfiles for lint validation 
            foreach($file in $temp){
              if(Test-Path -path $file.substring(1)){
                Copy-Item -Path $file.substring(1) -Destination "lintfiles" -verbose
              }
            }
            echo " "
            echo "Files for linting:"
            echo " "
            echo (Get-ChildItem -Recurse -Path "lintfiles" | select-Object -ExpandProperty Name) 
      - script: |
          docker pull github/super-linter:latest    
        displayName: Pull github/super-linter docker image
      - script: |
          echo -e "## PR CHANGED FILES #####################################\n"

          search_dir=lintfiles
          for entry in "$search_dir"/*
          do
            echo "$entry"
          done                  

          echo -e "\n\n###########################################################"
          # github/super-linter expects a git repo, therefore initializing git for files to be linted from PR
          cd lintfiles
          git config --global user.email "you@example.com"
          git config --global user.name "Your Name"
          git config --global init.defaultBranch main
          git init
          git add --all 
          git commit -m 'local dummy lint commit'
          cd ..

          docker run \
            -e RUN_LOCAL=true \
            -e VALIDATE_SQLFLUFF=true \
            -e SQLFLUFF_CONFIG_FILE=.sqlfluff \
            -v $(System.DefaultWorkingDirectory)/lintfiles:/tmp/lint \
            github/super-linter
        displayName: super-linter validation
- stage: Publish
  jobs:
    - job: Publish
      steps:
      - publish: $(Build.SourcesDirectory)
        artifact: RELEASE
        displayName: Publish artifact

Credits for parts of script: https://stackoverflow.com/questions/65088433/how-to-get-only-changed-files-using-azure-devops-pipelines