Ansible AWX: Deep Dive

These are the essential building blocks that make Ansible AWX a powerful platform for managing and orchestrating automation:

• Projects: Structured containers for Ansible playbooks, typically synchronized with version control systems like GitHub or GitLab. Projects centralize playbook management, enforce consistency, and simplify team collaboration by automatically pulling updates from linked repositories.
• Inventories: Define the infrastructure targets for your automation. Inventories list hosts and group them logically for streamlined task execution. AWX supports both static inventories (manually defined host lists) and dynamic inventories that adapt to infrastructure changes via external sources.
• Credentials: Securely manage authentication data such as SSH keys, API tokens, and cloud provider credentials. Credentials are encrypted and safely stored, allowing seamless and secure connections to remote systems while reducing manual input and risk of unauthorized access.
• Job Templates: Preconfigured blueprints for running Ansible jobs. Job templates connect projects (playbooks), inventories (target hosts), and credentials (access). They can be customized and reused, enabling reliable, hands-off task execution and scheduling across your infrastructure.
• Workflows: Chains of job templates and other workflows, allowing you to orchestrate complex, multi-step automation processes. Workflows enable conditional logic, approvals, and branching, making it possible to automate sophisticated operational tasks.
• Schedules: Allow you to automate the execution of job templates and workflows at specified times or intervals, supporting routine maintenance, compliance checks, and recurring operational tasks without manual intervention.
• Role-Based Access Control (RBAC): Provides granular permissions and delegation, ensuring that users have appropriate access to projects, inventories, credentials, and jobs. RBAC supports secure, multi-user environments and compliance requirements.
• Notifications: Integrated notification system that can alert users or external systems about job status, failures, or completions via email, Slack, webhooks, and other channels.
• REST API: A comprehensive API for integrating AWX with other tools, enabling automation and control from external systems and CI/CD pipelines.
• Logging & Auditing: Detailed logging and audit trails for all automation activity, supporting troubleshooting, compliance, and operational transparency.

Before you can deploy and start using Ansible AWX, make sure you have the following prerequisites in place:

• Supported Operating System: A modern Linux distribution such as CentOS, Red Hat Enterprise Linux, Ubuntu, or Fedora. AWX is typically deployed on Linux servers.
• Container Platform: AWX is distributed as a set of containers. You need Docker, Podman, or a Kubernetes/OpenShift cluster to run these containers.
• Python: Python 3.8 or newer is required on the host system for running installation scripts and AWX CLI utilities.
• Resource Requirements: Sufficient CPU, RAM (at least 4 GB recommended), and disk space to support AWX containers and your automation workloads.
• Network Access: Ensure the server has access to your target infrastructure, source control repositories (e.g., GitHub, GitLab), and any external services AWX will interact with.
• Database: AWX requires a PostgreSQL database for storing configuration and job data. You can use the built-in containerized database or connect to an external PostgreSQL instance.
• Admin Access: Sudo or root privileges on the host system are needed to install required packages, configure containers, and manage system services.
• Firewall Rules: Open necessary ports (default 80/443 for web UI, and any ports needed for SSH or API access to managed hosts).
• Version Control System (VCS) Access: Access credentials and network connectivity to your version control repositories where Ansible playbooks and roles are stored.
• Basic Ansible Knowledge: Familiarity with writing and running Ansible playbooks, as well as basic YAML syntax, is recommended for effective use of AWX.

Follow these detailed steps to configure Ansible AWX for both standalone operation and CI/CD integration. This guide covers variable management and execution paths for F5, Palo Alto, and Cisco environments:

• Step 1: Set Up Credentials
  - Navigate to Credentials in AWX
  - Add device credentials:
    • Cisco: SSH credentials
    • Palo Alto: API key or username/password
    • F5: API username/password
  - For CI/CD: Add API credentials for GitHub Actions/Azure DevOps
• Step 2: Create Inventory
  - Define target devices with host-specific variables:
  

  # Cisco IOS Inventory
  [ios_devices]
  R1 ansible_host=192.168.24.250
  R2 ansible_host=192.168.24.251
  
  [ios_devices:vars]
  ansible_network_os=ios
  ansible_connection=network_cli

  # Palo Alto Inventory
  [palo_alto]
  PA-1 ansible_host=10.0.0.1
  PA-2 ansible_host=10.0.0.2

• Step 3: Configure Project
  - Link to Git repository containing playbooks
  - Enable automatic sync to pull latest changes
  - Store base variables in group_vars/ directory
• Step 4: Create Job Template
  - Select inventory, project, playbook, credentials
  - Enable Prompt on Launch for Extra Variables
  - Add Survey for interactive variable prompts
• Step 5: Variable Management

  Method	Standalone (No CI/CD)	CI/CD Integrated
  Input Location	AWX UI (Surveys/Extra Variables)	Pipeline variables file (e.g., vars.yml)
  Execution Trigger	Manual launch from AWX dashboard	Pipeline job triggers AWX via API
  Variable Passing	Direct entry in AWX UI	API call with extra_vars parameter

• Step 6: Launch Execution
  • Standalone:
    - Click Launch in AWX UI
    - Enter variables via Survey prompts
    - Monitor execution in AWX dashboard
  • CI/CD Integrated:
    - Pipeline updates variables file
    - Triggers AWX job via API:
    

    # GitHub Actions Example
    - name: Trigger AWX Job
      uses: fitbeard/action-trigger-awx@v24.6.3
      with:
        extra_vars: '@update_vars.yml'

    - AWX merges pipeline variables with base variables
• Step 7: Monitoring & Validation
  - View real-time logs in AWX dashboard
  - Verify changes on target devices
  - Audit trail maintained in AWX and CI/CD systems

Key Benefits:

• Unified Interface: Single AWX instance supports both workflows
• Variable Flexibility: Use Surveys for manual runs, file-based vars for pipelines
• Audit Trails: Full visibility in AWX logs and CI/CD pipeline history
• Security: Sensitive variables stored in CI/CD secrets or AWX credentials

Once your AWX project is linked to your Git repository, any updates you make to your playbooks or roles in the repo (such as adding new files or modifying existing ones) will not appear in AWX until you perform a project sync.

• Manual Sync: After pushing changes to your Git repository, go to the Projects section in AWX and click the sync icon next to your project. This pulls the latest content from your Git repo into AWX and makes new or updated playbooks available for job templates.
• Automatic Sync (Best Practice): To ensure AWX always matches your repository, schedule regular project syncs. In the AWX UI, you can set up a Schedule for your project to sync at defined intervals (e.g., every hour or after work hours). This is especially useful for teams or environments with frequent updates.
• Update Revision on Launch: For critical workflows, enable the Update Revision on Launch option in your project settings. This ensures AWX automatically syncs the latest revision from your Git repo every time a job template is launched, guaranteeing jobs always use the most recent playbook version.

Best Practice: Always use source control (like Git) for your playbooks and roles, and schedule regular project syncs in AWX. This ensures your automation environment is always up-to-date, auditable, and consistent with your source repository.

Use this checklist to validate your Ansible AWX setup before running playbooks. Ensure all components work correctly for both standalone and CI/CD workflows:

• 1. Credentials Validation
  • ✅ SSH/API credentials tested with ansible -m ping from AWX CLI
  • ✅ CI/CD tokens (GitHub/ADO) validate with API connection test
  • ✅ Credential permissions verified for target devices
• 2. Inventory Validation
  • ✅ Hosts respond to AWX ping tests
  • ✅ Dynamic inventories sync correctly (test with awx-manage inventory_import)
  • ✅ Group/host variables resolve as expected
• 3. Project & Playbook Validation
  • ✅ Git sync status shows "Successful" in AWX UI
  • ✅ Playbook linting passes (ansible-lint in pipeline)
  • ✅ Syntax check passes: ansible-playbook --syntax-check
• 4. Job Template Configuration
  • ✅ Survey variables appear at launch
  • ✅ "Prompt on Launch" works for extra variables
  • ✅ Test run completes with successful status
• 5. Execution & Monitoring
  • ✅ Real-time logs show detailed output
  • ✅ Notifications deliver to configured channels (Slack/Email)
  • ✅ Audit logs capture user/job history
• 6. CI/CD Integration Checks

  Validation Point	Standalone	CI/CD Integrated
  Variable Passing	Survey inputs reflect in playbook	Pipeline variables merge correctly
  Execution Trigger	Manual launch works	API call from pipeline triggers job
  Secret Management	AWX credentials encrypted	Sensitive vars stored in CI/CD secrets

• 7. Vendor-Specific Validation

  Cisco IOS

  • ✅ Interface changes appear in show run
  • ✅ Configuration backups exist in target directory
  • ✅ VLAN configurations persist after reload

  Palo Alto

  • ✅ Security rules appear in GUI/CLI
  • ✅ Commit duration < 30 seconds
  • ✅ Object groups reference correct members

  F5 BIG-IP

  • ✅ VIPs respond to test traffic
  • ✅ Pool member status shows online
  • ✅ iRules execute as expected
• 8. Final Verification Tests
  • ✅ Dry-run mode: --check shows expected changes
  • ✅ Idempotency test: Second run shows "changed=0"
  • ✅ Rollback procedure tested (configuration restore)

Validation Tools & Commands:

# API Connectivity Test
curl -k -H "Authorization: Bearer $TOKEN" https://awx.example.com/api/v2/ping/

# CLI Syntax Check
awx-cli job_templates launch $JT_ID --monitor --extra_vars '@test_vars.yml'

# Pipeline Test Command (GitHub Actions)
- name: Test AWX Connection
  run: |
    curl -k -H "Authorization: Bearer ${{ secrets.AWX_TOKEN }}" \
    https://${{ secrets.AWX_HOST }}/api/v2/ping/

Common Failure Points:

• ❌ Firewall blocks AWX-to-device communication
• ❌ Missing Python dependencies on managed nodes
• ❌ Permission issues with credentials
• ❌ Variable precedence conflicts (group_vars vs extra_vars)

Follow this step-by-step troubleshooting guide to resolve common issues in Ansible AWX deployments:

• Step 1: Check AWX Service Status
  • ✅ Verify AWX services are running: kubectl get pods (K8s) or systemctl status receptor
  • ❌ If down: Restart with ansible-tower-service restart or systemctl restart receptor
• Step 2: Validate Network Connectivity
  • ✅ Test AWX-to-target connectivity: ansible -m ping
  • ✅ Check firewall rules (port 27199 for Receptor)
  • ❌ If blocked: Use receptor traceroute: receptorctl traceroute <node>
• Step 3: Verify Credentials and Permissions
  • ✅ Confirm credentials are correctly stored
  • ✅ Test SSH key permissions on managed nodes
  • ❌ Permission issues: Run chown awx -R /var/lib/awx/projects/
• Step 4: Check Playbook and Project Configuration
  • ✅ Validate YAML syntax: ansible-playbook --syntax-check
  • ✅ Ensure playbooks appear: Project sync status must be "Successful"
  • ❌ If missing: Check /var/lib/awx/projects permissions
• Step 5: Analyze Job Execution Failures
  • ✅ Review real-time job logs in AWX UI
  • ✅ Check for Failed to create temporary directory errors
  • ❌ Temp directory fix: Set remote_tmp = /tmp/ansible-${USER} in ansible.cfg
• Step 6: CI/CD Pipeline Debugging
  • ✅ Test API connectivity: curl -k -H "Authorization: Bearer $TOKEN" https://awx.example.com/api/v2/ping/
  • ✅ Verify variable passing: Use extra_vars: '@vars.yml'
  • ❌ If job doesn't trigger: Check CI/CD token permissions

Common Errors and Solutions

Variable-Specific Issues (New Section)

Advanced Debugging Tools

• Enable debug logging: Set LOG_AGGREGATOR_LEVEL='DEBUG' in AWX settings
• Retain job pods: Disable "Release Receptor Work" in Troubleshooting settings
• Check receptor status: receptorctl --socket /var/run/awx-receptor/receptor.sock status