Pipelines: Azure DevOps (ADO) & Ansible AWX Integration

These are the essential Azure DevOps and Ansible components required for a robust, automated workflow—whether integrating with AWX or using a direct runner:

• Azure Boards: Agile project management for tracking features, user stories, bugs, tasks, sprints, and Kanban boards. Ensures traceability from planning through deployment.
• Azure Repos: Git repositories for source control of Ansible playbooks, variable files, pipeline YAML, and application code. Supports branch policies, pull requests, and code reviews.
• Azure Pipelines: CI/CD automation for building, testing, and deploying code and infrastructure. Pipelines can trigger on PR merges, variable file changes, or manual runs, and orchestrate calls to AWX or direct runners.
• Variable Groups & Secure Files: Centralized, secure management of environment variables and secrets (API tokens, SSH keys) for use in pipelines and playbooks.
• Service Connections: Secure, managed authentication to external systems (AWX API, Azure, AWS, etc.), enabling pipelines to trigger AWX jobs or access cloud resources.
• Agent Pools & Agents: Compute environments (Microsoft-hosted or self-hosted Linux VMs) where pipeline jobs run. Self-hosted agents must have Ansible and dependencies installed for direct playbook execution.
• Azure Artifacts: Package management for sharing and consuming dependencies (e.g., Python roles/modules for Ansible), versioned and integrated with pipelines.
• Azure Test Plans: Manual and automated test management, supporting quality gates and validation within the pipeline.
• Dashboards & Wiki: Custom dashboards for monitoring pipeline status, test results, and deployment health; Wiki for documenting processes and runbooks.
• Branch Policies & Approvals: Enforce code review, PR approval, and build validation before merging changes to protected branches.
• Variable Management Logic: Detect changes to variable files, set pipeline variables dynamically, and map them to Ansible extra_vars or AWX job parameters.
• AWX Integration Components
• AWX API Endpoint: REST API for triggering AWX job templates from ADO pipelines, passing variables as JSON payloads.
• AWX Job Templates & Projects: Define which playbook to run, inventory to use, and how to accept external variables; projects sync playbooks from Azure Repos.
• AWX Credentials: Secure storage of SSH keys and cloud credentials, managed within AWX and referenced by job templates.
• Direct Runner (Agent-First) Components
• Ansible-Enabled Self-Hosted Agent: Linux VM with Ansible, Python, and SSH client installed, running ansible-playbook commands directly from pipeline tasks.
• Bash/PowerShell Pipeline Tasks: Pipeline steps that invoke ansible-playbook with variables injected from ADO variable groups or pipeline context.
• Secure File Mounting: Use of Azure Secure Files to provide SSH keys and sensitive files to the agent at runtime, ensuring secrets are not exposed in code.

These components must be in place before configuring pipelines:

• Azure DevOps Organization: Active ADO organization with contributor permissions for pipeline creation, repository management, and service connection configuration.
• Version Control Repository: Azure Repos or GitHub repository storing Ansible playbooks, inventories, variable files, and pipeline YAML definitions. Branch protection must be enabled.
• Agent Infrastructure:
  • Microsoft-Hosted Agents: Enabled parallel jobs for Linux/Windows workloads.
  • Self-Hosted Agents: Linux VM (Ubuntu 20.04+/RHEL 8+) with Ansible 2.12+, Python 3.8+, and SSH client installed.
• Authentication Mechanisms:
  • ADO service principal for Azure resource access.
  • SSH key pairs for target node authentication.
  • Personal Access Tokens (PAT) for repository access.
• Secure Storage:
  • Variable Groups: Azure Key Vault-linked groups for secrets.
  • Secure Files: Encrypted storage for certificates/SSH keys.
• Branch Policies: Configured for protected branches (e.g., main) requiring PR approvals and build validation.
• AWX-Specific Prerequisites
• AWX Deployment: Running AWX instance (Kubernetes/VM) with network access to ADO agents and target infrastructure. Minimum 4vCPU/8GB RAM.
• AWX API Access: Valid API token with permissions to launch job templates. Service connection configured in ADO.
• Project Synchronization: AWX project linked to your Azure Repos/GitHub repository for playbook syncing.
• Direct Runner (Agent-First) Prerequisites
• Persistent Agent VM: Linux controller machine with static IP/DNS, Ansible dependencies, and persistent storage for inventories/collections.
• Network Access: Agent VM must have SSH access to all target nodes (cloud/on-prem).
• Pipeline Execution Permissions: ADO agent service account granted sudo rights for playbook execution.

Variable Management Strategy

Variable Storage & Passing Mechanisms

Approach 1: Direct Runner Execution

Step-by-Step Setup

1. Deploy Self-Hosted Agent
   • Create Ubuntu 22.04 VM in Azure
   • Install dependencies:

     sudo apt update && sudo apt install -y ansible sshpass python3-pip
     pip3 install ansible[azure] f5networks.f5_bigip cisco.asa paloaltonetworks.panos

   • Register ADO agent:

     ./config.sh --unattended --url https://dev.azure.com/your-org \
       --auth pat --token $(PAT_TOKEN) --pool "Ansible-Agents"

2. Create SSH Service Connection
   • ADO → Project Settings → Service Connections → New service connection
   • Select SSH type
   • Provide VM details: Hostname, Port (22), Username, SSH Key
   • Save as "Ansible-SSH-Connection"
3. Pipeline Configuration with Variables

   # azure-pipeline.yml
   trigger:
     branches:
       include: [ main ]
   
   variables:
     - group: NetworkConfig  # Contains F5_POOL, CISCO_VLAN
     VENDOR: "f5"
     CONFIG_VERSION: $(Build.BuildId)
   
   stages:
   - stage: Deploy
     jobs:
     - job: Run_Ansible
       pool: Ansible-Agents
       steps:
       - checkout: self
       - task: Bash@3
         inputs:
           targetType: 'inline'
           script: |
             ansible-playbook -i inventory/prod.yml deploy.yml \
               -e "vendor_type=$(VENDOR)" \
               -e "config_version=$(CONFIG_VERSION)" \
               -e "pool_name=$(F5_POOL)" \
               -e "vlan_id=$(CISCO_VLAN)" \
               -e "@group_vars/prod.yml"
         env:
           SSH_KEY: $(SECURE_SSH_KEY)

Vendor Playbook Examples

- name: Configure F5 VIP
  hosts: f5_devices
  tasks:
    - f5networks.f5_bigip.virtual_server:
        name: "{{ vip_name }}"
        destination: "{{ vip_ip }}"
        port: "{{ vip_port }}"

- name: Update Cisco ACL
  cisco.asa.asa_acl:
    name: "{{ acl_name }}"
    lines: "{{ acl_rules }}"

- name: Add Security Rule
  paloaltonetworks.panos.panos_security_rule:
    rule_name: "{{ rule_name }}"
    source_zone: "{{ src_zones }}"
    destination_zone: "{{ dest_zones }}"

Approach 2: AWX API Integration

Step-by-Step Setup

1. Configure AWX
   • Create project synced to Azure Repos
   • Define job template with:
     • Playbook: deploy.yml
     • Inventory: Dynamic Azure inventory
     • Enable Prompt on Launch for variables
   • Create survey in job template for variables:
     • VENDOR (text)
     • CONFIG_VERSION (integer)
     • F5_POOL (password encrypted)
2. Create ADO Service Connection
   • ADO → Project Settings → Service Connections
   • Type: Generic REST API
   • URL: https://<awx-host>/api/v2/job_templates/
   • Authentication: Bearer Token (AWX API token)
   • Name: "AWX_API"
3. Pipeline Configuration with Variables

   # awx-pipeline.yml
   variables:
     - group: AWX-Vars  # Contains AWX_TOKEN, F5_POOL
     VENDOR: "cisco"
     CONFIG_VERSION: $(Build.BuildId)
   
   stages:
   - stage: Trigger_AWX
     jobs:
     - job: AWX_Integration
       pool: ubuntu-latest
       steps:
       - task: RESTAPI@1
         inputs:
           connectionType: connectedServiceName
           serviceConnection: AWX_API
           method: POST
           headers: '{"Content-Type":"application/json"}'
           body: |
             {
               "extra_vars": {
                 "vendor": "$(VENDOR)",
                 "config_version": "$(CONFIG_VERSION)",
                 "f5_pool": "$(F5_POOL)",
                 "vlan_id": "$(CISCO_VLAN)"
               }
             }
           urlSuffix: "/14/launch/"

Vendor API Payload Examples

"extra_vars": {
  "pool_name": "prod_servers",
  "members": ["10.1.1.10:80"]
}

"extra_vars": {
  "vlan_id": 200,
  "vlan_name": "dmz"
}

"extra_vars": {
  "group_name": "azure_servers",
  "ip_list": ["10.3.3.0/24"]
}

Hybrid Approach: Agent + AWX

- script: |
    az login --identity
    export AZURE_TOKEN=$(az account get-access-token --query accessToken -o tsv)
    ansible-playbook -i azure_rm.yml trigger_awx.yml \
      -e "config_version=$(Build.BuildId)" \
      -e "resource_group=$(RESOURCE_GROUP)"
  displayName: "Hybrid Azure-AWX Execution"

Hybrid Variable Flow

1. Azure credentials via Managed Identity
2. Resource group from ADO variables
3. Config version from build ID
4. AWX inventory generated dynamically from Azure

Full Variable Workflow Example

# Complete pipeline with variables
trigger:
  - main

variables:
  - group: Network-Secrets  # Key Vault linked
  - name: BUILD_ENV
    value: prod

stages:
- stage: Validate
  jobs:
  - job: Check_Variables
    steps:
    - script: |
        echo "Using F5 pool: $(F5_POOL)"
        echo "Cisco VLAN: $(CISCO_VLAN)"
      displayName: "Verify Variables"

- stage: Deploy
  jobs:
  - deployment: Production
    environment: 'prod'
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
          - task: Bash@3
            inputs:
              targetType: 'inline'
              script: |
                ansible-playbook -i inventory/prod.yml \
                  -e "env=$(BUILD_ENV)" \
                  -e "build_id=$(Build.BuildId)" \
                  -e "pool_name=$(F5_POOL)" \
                  site.yml
            env:
              PANOS_API_KEY: $(SECURE_PAN_KEY)

Ansible Variable Structure

# site.yml
- hosts: "network_devices"
  tasks:
    - include_vars: "vars/{{ vendor }}.yml"
    - include_tasks: "tasks/{{ vendor }}.yml"

# vars/f5.yml
vip_name: "web_prod"
vip_ip: "10.1.1.100"
vip_port: 443

# tasks/f5.yml
- name: Configure VIP
  f5networks.f5_bigip.virtual_server:
    name: "{{ vip_name }}"
    destination: "{{ vip_ip }}"
    port: "{{ vip_port }}"

Validate both approaches using these comprehensive checks before production deployment:

• Agent Connectivity Tests:
  • Run ansible -m ping all from ADO agent to target nodes
  • Verify SSH connectivity: nc -zv target_host 22
  • Test Azure authentication: az account show
• Dry Run Execution:
  • Direct Runner: ansible-playbook --check --diff playbook.yml
  • AWX: Enable "TEST MODE" in job templates
• Direct Runner Validation
• Pipeline Smoke Test: Run minimal "Hello World" playbook:

  - name: Validation Playbook
    hosts: localhost
    tasks:
      - debug:
          msg: "ADO-Agent connection successful"

• Permission Verification:
  • Confirm agent service account has sudo rights
  • Test secret injection: echo $SSH_KEY | wc -c
• Vendor-Specific Checks:
  • F5: ansible -m bigip_device_info -a "provider: ..."
  • Cisco: ansible -m ios_command -a "commands: 'show version'"
  • Palo Alto: Validate API key with panos_op module
• AWX Integration Validation
• API Connectivity Test:

  curl -H "Authorization: Bearer $AWX_TOKEN" \
  https://awx.example.com/api/v2/ping/

  Expect response: {"active": true}
• Job Template Simulation:
  • Manually launch AWX job with test variables
  • Verify project sync status in AWX UI
  • Check "Recent Jobs" for success status
• Webhook Verification:
  • Use Postman to simulate ADO payload
  • Validate extra_vars parsing in AWX job output
• Common Validation Tools
• Ansible Linting:

  ansible-lint playbooks/

• Pipeline Debugging:
  • Enable system diagnostics: system.debug: true
  • Add debug steps: env | sort
• Log Analysis:
  • Direct Runner: ADO pipeline logs
  • AWX: Job output view with timestamp filtering

Validation Checklist

Follow these step-by-step troubleshooting procedures for both direct runner and AWX integration approaches to quickly identify and resolve common issues:

• General Troubleshooting Steps:
  • Check pipeline logs in Azure DevOps for error messages and failed tasks.
  • Enable detailed logging by setting system.debug: true in pipeline variables.
  • Review agent status and ensure the agent is online and healthy in Project Settings > Agent Pools.
    Tip: Self-hosted agents must be running and registered; Microsoft-hosted agents must be available.
  • Validate pipeline triggers and YAML location if the pipeline isn’t starting as expected.
    See: trigger block in YAML and Project Settings > Pipelines > Triggers
  • Check for authentication and permission errors, especially with service connections, managed identities, or service principals.
    Ensure correct permissions for Azure resources, Key Vault, and target VMs.
  • Isolate the problem by disabling non-essential tasks and rerunning the pipeline.
• Direct Runner Troubleshooting
• SSH Connection Issues:
  • Check SSH key permissions: chmod 600 ~/.ssh/id_rsa
  • Add SSH key to the agent: ssh-add ~/.ssh/id_rsa
  • Verify host key: ssh-keygen -R <host> to remove old keys if "Host key verification failed"
  • Disable host key checking for non-interactive sessions by adding to ansible.cfg:

    [defaults]
    host_key_checking = False

  • Check for SSH timeout or unreachable host:
    • Ping target: ping <target_host>
    • Test SSH port: nc -zv <target_host> 22
    • Increase SSH timeout in ansible.cfg:

      [defaults]
      timeout = 30

• Permission & Environment Issues:
  • Ensure agent service account has sudo rights if needed.
  • Check for missing dependencies (e.g., Ansible, Python, vendor collections).
  • Validate secret injection: print or echo environment variables (never print secrets directly in logs).
  • Check working directory: Azure DevOps agents may run in /home/vsts/work/1/s/ for Microsoft-hosted agents.
  • If you see "permission denied" errors, review file and directory permissions on the agent and target hosts.
• Pipeline and Playbook Issues:
  • Lint playbooks: ansible-lint playbooks/
  • Test playbooks with --check and --diff flags before real execution.
  • For variable issues, echo variables in pipeline steps and ensure -e flags are set correctly.
  • Review Ansible logs in the pipeline output for detailed task failures.
• AWX Integration Troubleshooting
• AWX Project Sync Issues:
  • Check AWX project credentials: PATs must be valid and have access to Azure DevOps repos.
  • For SSH, ensure correct SSH key and config are set in AWX credentials.
  • Review AWX logs for errors like "not authorized to access this collection" or "authentication failed".
  • If using HTTPS, ensure the PAT is passed as an Authorization header if required by Azure DevOps.
    Manual test: git --config http.extraheader="Authorization: Basic <base64PAT>" clone <repo>
  • If using SSH, update ansible.builtin.git module's ssh_opts as needed.
• API and Job Trigger Issues:
  • Test AWX API connectivity: curl -H "Authorization: Bearer $AWX_TOKEN" https://awx.example.com/api/v2/ping/
  • Check for correct urlSuffix and job template ID in pipeline REST API tasks.
  • Verify extra_vars are passed and parsed correctly in AWX job output.
  • Check for 401/403 errors: ensure ADO service connection has the right token and permissions.
• AWX Job Execution Issues:
  • Manually trigger job in AWX UI using same variables as pipeline.
  • Review job output and events for failed tasks or misconfigured variables.
  • Check project sync status; ensure latest playbooks are pulled from Azure DevOps.
  • For inventory or credential errors, verify all referenced resources exist and are valid in AWX.
• Common Authentication Issues:
  • Expired or revoked PATs or SSH keys in AWX or ADO service connections.
  • Service accounts disabled or lacking permissions (common after personnel changes).
  • Wrong branch or repository reference in AWX project settings.
• General Pipeline Debugging Tips
• Log Analysis:
  • Review ADO pipeline logs and AWX job output for error details.
  • Use env | sort or printenv to show available environment variables.
  • Enable Azure Monitor or Application Insights for enhanced diagnostics.
• Agent and Build Queue Issues:
  • Ensure agents are online and not blocked by parallel job limits or approvals.
  • Check for agent capability mismatches (e.g., missing Ansible/Python on agent).

Troubleshooting Checklist