Most teams don’t realize they’re choosing an automation architecture every time they run a playbook. They think they’re choosing a tool. They’re actually choosing a pattern.

In my opinion, there are only two patterns that matter:

1. Pipeline‑Driven Execution
2. Controller‑Driven Execution (AWX/AAP)

Both work. Both fail. Both solve different problems.

This article breaks down the patterns, the guidance for using them, the anti‑patterns that cause outages, and when combining them actually makes sense.

Pattern 1: Pipeline‑Driven Execution

“Automation lives in Git. Execution happens in CI.”

This is the alleged cloud‑native. I hate the term ‘cloud-native’, it’s just a modern pattern and can equally apply to on-premises, but I’ll use it as it’s become an industry norm.

The pipeline:

• checks out the repo
• runs ansible-playbook directly
• injects secrets at runtime
• uses inventories stored in Git
• uses vars stored in Git
• uses execution environments defined in Git
• tests automation before deploying
• runs deterministically and reproducibly

This pattern treats Ansible like any other automation tool: code —> test —> deploy —> verify.

When to use it

Pipeline execution is the right answer when you need:

• reproducibility
• portability
• deterministic execution
• Git as the single source of truth
• cloud‑neutral automation
• ephemeral runners
• multi‑cloud or hybrid cloud
• developer‑friendly workflows
• automation that can run anywhere

This is the pattern used by ‘cloud‑native’ teams, platform teams, and anyone who values Git truth over controller truth.

Strengths

• Fully reproducible
• Fully portable
• Fully version‑controlled
• No hidden state
• No GUI configuration
• No controller dependency
• Works in any CI/CD system
• Works locally
• Works in DR
• Works in multi‑cloud

Weaknesses

• Pipelines must understand environment boundaries
• Pipelines must manage secrets securely
• Pipelines must enforce guardrails
• Pipelines can become too flexible
• Pipelines can accidentally bypass governance

This is why enterprises often avoid this pattern, not because it’s wrong, but because they fear losing control. The fear is real.

Pattern 2: Controller‑Driven Execution (AWX/AAP)

“Automation runs inside a governed platform.”

This is the typical enterprise pattern.

AWX/AAP:

• pulls playbooks from Git
• stores inventories
• stores credentials
• enforces RBAC
• provides audit trails
• standardizes execution environments
• triggers automation from events
• provides multi‑team visibility

This pattern treats Ansible as a governed automation platform, not a CLI tool.

When to use it

Controller execution is the right answer when you need:

• RBAC
• credential isolation
• auditability
• inventory synchronization
• standardized execution environments
• event‑driven automation
• multi‑team governance
• compliance and regulatory controls

This is the pattern used by large enterprises, regulated industries, and teams with strict governance requirements.

Strengths

• Centralized governance
• Centralized credentials
• Centralized inventory
• Centralized audit
• Standardized execution environments
• Event‑driven automation
• Multi‑team visibility
• Strong guardrails

Weaknesses

• Hidden state
• GUI configuration
• Non‑portable automation
• Not fully reproducible
• Not fully version‑controlled
• Controller dependency
• Harder to test automation before deployment
• Harder to run automation outside AWX

This is why ‘cloud‑native’ teams avoid this pattern, not because it’s wrong, but because it’s not portable.

Anti‑Patterns

These are the patterns that break automation, cause outages, and create drift.

Anti‑Pattern 1: AWX as the “automation store”

Pipelines call AWX job templates as if AWX stores automation.

It doesn’t.

AWX stores configuration, not automation.

Automation lives in Git. AWX overrides Git with controller‑side state.

This creates split‑brain automation, if you think git is the source of truth, surprise, it isn’t.

Anti‑Pattern 2: Pipelines storing credentials

This is a governance failure.

Pipelines should inject secrets at runtime, not store them.

AWX should manage credentials. Actually, just about anything else should manage credentials.

Anti‑Pattern 3: AWX storing inventories that drift from Git

Inventories should be version‑controlled.

If AWX is the inventory source of truth, Git is no longer authoritative.

This breaks reproducibility.

Anti‑Pattern 4: Pipelines bypassing RBAC

If pipelines can run any playbook against any host using any credential, you’ve lost governance.

This is dangerous in enterprise environments.

Anti‑Pattern 5: AWX storing controller‑side vars that override Git

This creates non‑deterministic execution.

Playbooks behave differently depending on controller configuration.

This breaks portability.

Does the Combination Make Sense?

Here’s the part most teams miss:

The two patterns are not mutually exclusive. They are complementary.

The Practical Guidance

Use pipeline execution when:

• automation must be reproducible
• automation must be portable
• automation must be version‑controlled
• automation must run anywhere
• automation must be tested before deployment

Use controller execution when:

• governance matters
• RBAC matters
• credential isolation matters
• auditability matters
• inventory sync matters
• event‑driven automation matters

Use both when:

• you need reproducibility and governance
• you need portability and control
• you need Git truth and environment truth
• you need cloud‑native execution and enterprise guardrails

This is the two‑layer architecture.

Final Takeaway

There are two patterns.

Trying to make one do both is how teams create drift, outages, and brittle automation.

Following the very simple instructions at https://www.manageiq.org/docs/reference/latest/doc-Managing_Providers/miq/index I proceeded to add the new provider.

It’s all pretty straight forward. Putting in the URL to my Tower server I get greeted with a bunch of errors.

Credential validation was not successful:  {:headers=>{"server"=>"nginx", "date"=>"Mon, 13 Jan 2020  23:42:12 GMT", "content-type"=>"text/html; charset=utf-8",  "content-length"=>"3873", "connection"=>"close",  "vary"=>"Cookie, Accept-Language, Origin",  "content-language"=>"en", "x-api-total-time"=>"0.046s"
.
.
.
blah blah
.
.
status=>404

By default ManageIQ and Cloudforms search for /api/v1 on the Tower server. /api/v1 was deprecated and removed as of Ansible Tower 3.6, see https://docs.ansible.com/ansible-tower/latest/html/towerapi/conventions.html

So, what is a person to do? Hit the google. Eventually I came across this bugzilla item https://bugzilla.redhat.com/show_bug.cgi?id=1740860 and it gave a hint as to just specifying the /api/v2 in the URL I gave to ManageIQ rather than just the base hostname. eg. https://blah…./api/v2

Tried it, it worked! My credential validated and a provider refresh was automatically initiated and all my Ansible Tower templates and inventories were discovered correctly.

The section ‘Authenticating with Azure‘ sounds like the right place, but you can’t use your AD username / password from Ansible because you turned on 2FA – You turned it on RIGHT? So the option left to you is to create a Service Principal (SP).

Note: having 2FA on your account is what you should be doing, so don’t turn it off.

It’s quite simple to create a credential for Ansible to use when connecting to Azure. Simply, fire up the Cloud Shell (awesome feature BTW Microsoft) and create a Service Principal (SP).

But Hang On, what is a Service Principal? The Ansible guide refers you to the Azure documentation over at https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal which you will read, and if you’re like me, you’ll wonder what you actually just read. Have no fear. As I mentioned above you can use a simple Azure CLI command (via the Cloud Shell you just started) and create the Service Principal. Think of the Service Principal as a credential an application (in this case Ansible) can use to access the Azure service(s).

geoff@Azure:~$ az ad sp create-for-rbac --name svc-ansible-azure  # (optional if not specified one will be generated)  --password 'ALovelyComplexPasswor@'
Changing "svc-ansible-azure" to a valid URI of "http://svc-ansible-azure", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/88888888-4444-4444-4444-cccccccccccc"
  Retrying role assignment creation: 1/36
  Retrying role assignment creation: 2/36
{
  "appId": "appid888-4444-4444-4444-cccccccccccc",
  "displayName": "svc-ansible-azure",
  "name": "http://svc-ansible-azure",
  "password": "password-4444-4444-4444-cccccccccccc",
  "tenant": "tenant88-4444-4444-4444-cccccccccccc"
}
geoff@Azure:~$

If you want to see what that command just did in the Azure portal, head over to the Azure Active Directory -> App registrations blade.

and then you can see the Service Principal you just created.

So what do you do with the new credential.

The Ansible Azure scenario guide has a section on what to do, however, it’s a bit too vague for me.

Using Environment Variables

To pass service principal credentials via the environment, define the following variables:

• AZURE_CLIENT_ID
• AZURE_SECRET
• AZURE_SUBSCRIPTION_ID
• AZURE_TENANT

Azure has given me :

“appId”: “appid888-4444-4444-4444-cccccccccccc”,
“displayName”: “svc-ansible-azure”,
“name”: “http://svc-ansible-azure”,
“password”: “password-4444-4444-4444-cccccccccccc”,
“tenant”: “tenant88-4444-4444-4444-cccccccccccc”

For your sanity,
AZURE_CLIENT_ID ==> appId
AZURE_SECRET ==> password
AZURE_TENANT ==> tenant

The remaining item, AZURE_SUBSCRIPTION_ID is exactly that, you can also get from the Cloud Shell as follows

geoff@Azure:~$ az account list
[
  {
    "cloudName": "AzureCloud",
    "id": "subscrip-4444-4444-4444-cccccccccccc
    "isDefault": true,
.
.
.

In this case AZURE_SUBSCRIPTION_ID ==> id , whichever id in your account that is valid for your use case.

If you want to add these credentials into Ansible Tower, simply create a Credential of type Microsoft Azure Resource Manager and use the values you’ve deduced above. Ansible Tower will automatically translate them into Environment Variables for your Tower template execution.

Enjoy Ansible and Azure!

Ansible Tower neatly stores credentials within it – or externally if that fills you with joy. There isn’t a ServiceNow credential type in Ansible Tower. Undeterred, I thought I would use machine credentials, but tower has an annoying behavior of only allowing 1 instance of each credential type attached to a tower template and I am already using machine credentials in my template.

Fortunately on the left hand side of the tower ui there’s an entry labelled credential types

When creating the credential type you need to supply two (2) pieces of information. The first piece is called the Input configuration – or what the fields look like on the web ui when you create a credential of this type and secondly, the Injector configuration which details what do do with thew credentials.

In my case, the new credential type is called SNOW and i’m providing the instance name, username and password as part of the structure for this credential – via the Input configuration and then I detail that I want to store this data in environment variables that will be accessible from my playbook when run in tower.

Input configuration

fields:
  - id: instance
    type: string
    label: Instance
  - id: username
    type: string
    label: Username
  - id: password
    type: string
    label: Password
    secret: true
required:
  - instance
  - username
  - password

Injector configuration

env:
  SN_INSTANCE: '{{instance}}'
  SN_PASSWORD: '{{password}}'
  SN_USERNAME: '{{username}}'

The way you use them in your playbook is quite simple. The following is a snippet of playbook showing that.

   - name: Create an incident
     snow_record:
       username: '{{ lookup("env", "SN_USERNAME") }}'
       password: '{{ lookup("env", "SN_PASSWORD") }}'
       instance: '{{ lookup("env", "SN_INSTANCE") }}'
       state: present
       data:
         short_description: "This is a test incident opened by Ansible"
         severity: 3
         priority: 2
     register: new_incident

This is normally achieved by specifying become as part of your playbook, such as this snippet.

---
- name: Patch Linux
  hosts: all
  gather_facts: true
  become: true

Typically, as part of a patching playbook, you would reboot the system and wait for the reboot to finish using a code fragment like this :

 - name: Wait for server to restart
   local_action:
     module: wait_for
       host={{ ansible_ssh_host }}
       port=22
       delay=60
       timeout=300

This local_action inherits the become: true from the parent definition and this is where Tower starts to complain. Remember, with Ansible Tower, it’s the tower server itself where the local_action will run. You can expect to see something like this :

"module_stderr": "sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?\n",

No, you SHOULD NOT enable the awx user to use sudo on the Tower system as the AWX service user is intentionally restricted from sudo operations. The best approach is to de-privilege the local_action. Fortunately, local_action has it’s own become capability so you can turn off the request for privileged access as you don’t need it.

The above code block is now :

 - name: Wait for server to restart
   become: false
   local_action:
     module: wait_for
       host={{ ansible_ssh_host }}
       port=22
       delay=60
       timeout=300

and the tower job template will execute without any errors.