Molecule + Podman: Why It’s a Pain, but here’s an Example That Finally Works

Ansible Podman No Comments

If you’ve ever tried to run Molecule with Podman instead of Docker, you already know the truth: it never “just works.” Between dynamic inventories, missing Python interpreters, inconsistent container images, and Molecule’s habit of generating YAML that looks like it was written by a committee, you spend more time debugging the framework than testing your Ansible roles.

This post walks through a real, working Molecule + Podman setup that supports multiple OS containers (Ubuntu, Rocky, CentOS Stream), uses Podman natively, and avoids the usual landmines:

  • Python missing inside containers
  • Fact‑gathering failures
  • Molecule’s dynamic inventory breaking
  • Destroy steps leaving orphaned containers
  • Podman connection plugin quirks

If you’ve been fighting Molecule + Podman, this example will save you hours.

The Problem: Molecule + Podman Is Not a Drop‑In Replacement for Docker

Molecule’s Docker driver is mature. Podman’s driver… well… it isn’t.

If you’re testing roles across multiple OSes, these problems multiply and you eventually just give up.

The Working Example: Multi‑OS Molecule Scenario Using Podman

Here’s the structure:

molecule/default/
├── create.yml
├── prepare.yml
├── converge.yml
├── verify.yml
├── destroy.yml
├── molecule.yml
└── requirements.yml

Key idea:

We start our own containers with podman and generate our own dynamic inventory inside create.yml and force Molecule to use it.

This avoids Molecule’s broken Podman inventory plugin entirely.

1. create.yml Start Containers + Build Inventory

We define our Podman containers, start them and force feed the inventory to molecule:

- name: Create
  hosts: localhost
  gather_facts: false
  vars:
    molecule_inventory:
      all:
        hosts: {}
        molecule: {}
    
    podman:
      - name: ubuntu-2404
        image: ubuntu:24.04
        command: /bin/bash
        tty: true
      - name: rocky9
        image: quay.io/rockylinux/rockylinux:9  
        command: /bin/bash
        tty: true
      - name: rocky10
        image: quay.io/rockylinux/rockylinux:10
        command: /bin/bash
        tty: true
      - name: cs9
        image: quay.io/centos/centos:stream9
        command: /bin/bash
        tty: true
  tasks:

    - name: Start Podman container(s)
      containers.podman.podman_container:
        name: "{{ item.name }}"
        image: "{{ item.image }}"
        state: started
        command: "{{ item.command | default(omit) }}"
        tty: "{{ item.tty | default(omit) }}"
      register: result
      loop: "{{ podman }}"
      loop_control:
        label: "{{ item.name }}"

    - name: Add container to molecule_inventory
      vars:
        inventory_partial_yaml: |
          all:
            children:
              molecule:
                hosts:
                  "{{ item.name }}":
                    ansible_connection: containers.podman.podman
      ansible.builtin.set_fact:
        molecule_inventory: >
          {{ molecule_inventory | combine(inventory_partial_yaml | from_yaml, recursive=true) }}
      loop: "{{ podman }}"
      loop_control:
        label: "{{ item.name }}"

    - name: Dump molecule_inventory
      ansible.builtin.copy:
        content: |
          {{ molecule_inventory | to_yaml }}
        dest: "{{ molecule_ephemeral_directory }}/inventory/molecule_inventory.yml"
        mode: "0600"

    - name: Force inventory refresh
      ansible.builtin.meta: refresh_inventory

    - name: Fail if molecule group is missing
      ansible.builtin.assert:
        that: "'molecule' in groups"
        fail_msg: |
          molecule group was not found inside inventory groups: {{ groups }}
      run_once: true # noqa: run-once[task]

# we want to avoid errors like "Failed to create temporary directory"
- name: Validate that inventory was refreshed
  hosts: molecule
  gather_facts: false
  tasks:
    - name: Check uname
      ansible.builtin.raw: uname -a
      register: result
      changed_when: false

    - name: Display uname info
      ansible.builtin.debug:
        msg: "{{ result.stdout }}"

It’s important to note, we added our containers to the inventory with the podman ansible_connection, containers.podman.podman_container.

Out inventory file under the covers looks like this:

all:
  children:
    molecule:
      hosts:
        ubuntu-2404:
          ansible_connection: containers.podman.podman
        rocky9:
          ansible_connection: containers.podman.podman
        rocky10:
          ansible_connection: containers.podman.podman
        cs9:
          ansible_connection: containers.podman.podman

And note that we force Molecule to refresh:

- meta: refresh_inventory

This is the magic step most examples miss.

2. prepare.yml Only Touch Ubuntu Containers

Instead of conditionals, we use a host pattern and fix the missing python3 in the Ubuntu containers.

- hosts: "ubuntu*"
  gather_facts: false
  tasks:
    - name: Refresh apt cache Ubuntu container
      raw: |
        apt-get update
    - name: Install Python inside Ubuntu container
      raw: |
        apt-get install -y python3
    - name: Set correct Python interpreter for Ubuntu
      set_fact:
        ansible_python_interpreter: /usr/bin/python3

Why this works:

  • Molecule names your Ubuntu containers ubuntu-*
  • No need for OS detection
  • No need for fact gathering

This is the cleanest possible prepare step.

3. converge.yml Run Tasks on All Containers

This is the thing we’re testing

- hosts: all
  tasks:
    - shell: "cat /etc/*release*"
      register: result
    - debug: msg="{{ result }}"

Simple, predictable, works across all OSes or well, it does what you want your code to do.

4. verify.yml OS‑Specific Assertions

- hosts: rocky9
  tasks:
    - assert:
        that:
          - "'dnf' in ansible_facts.packages"

- hosts: ubuntu-2404
  tasks:
    - assert:
        that:
          - "'apt' in ansible_facts.packages"

This is how you validate OS‑specific behaviour cleanly, again, you do you.

5. destroy.yml Clean Container Removal

- name: Destroy molecule containers
  hosts: molecule
  gather_facts: false
  tasks:
    - name: Delete Podman container(s)
      containers.podman.podman_container:
        name: "{{ item }}"
        state: absent
      loop: "{{ groups['all'] | difference(['localhost']) }}"
      delegate_to: localhost

- name: Remove dynamic molecule inventory
  hosts: localhost
  gather_facts: false
  tasks:

    - name: Remove dynamic inventory file
      ansible.builtin.file:
        path: "{{ molecule_ephemeral_directory }}/inventory/molecule_inventory.yml"
        state: absent

This avoids:

  • Molecule leaving orphaned containers
  • Podman complaining about missing names
  • localhost being treated as a container

Why This Example Works When Others Don’t

1. We bypass Molecule’s broken Podman inventory plugin

We generate our own containers and inventory and force Molecule to use it.

2. We use host patterns instead of OS detection

Fact gathering doesn’t work until Python is installed.

3. We use Podman’s connection plugin correctly

Every host gets:

ansible_connection: containers.podman.podman

4. We refresh inventory explicitly

Without this, Molecule uses stale inventory.

Conclusion: Molecule + Podman Is Painful …. But Fixable

Podman is great. Molecule is great. Together? They fight.

But with:

  • explicit container creation
  • explicit inventory generation
  • host patterns
  • correct Podman connection plugin
  • correct Ubuntu prepare steps
  • correct destroy logic

You can run a multi‑OS Molecule test matrix under Podman reliably.

This example proves it.

If you’re building cloud automation, CI pipelines, or multi‑distro Ansible roles, this pattern will save you hours of frustration.

The Two Patterns of Ansible Automation

Ansible Ansible Tower AWX No Comments

Pipeline Execution vs Controller Execution

Most teams don’t realize they’re choosing an automation architecture every time they run a playbook. They think they’re choosing a tool. They’re actually choosing a pattern.

In my opinion, there are only two patterns that matter:

  1. Pipeline‑Driven Execution
  2. Controller‑Driven Execution (AWX/AAP)

Both work. Both fail. Both solve different problems.

This article breaks down the patterns, the guidance for using them, the anti‑patterns that cause outages, and when combining them actually makes sense.

Pattern 1: Pipeline‑Driven Execution

“Automation lives in Git. Execution happens in CI.”

This is the alleged cloud‑native. I hate the term ‘cloud-native’, it’s just a modern pattern and can equally apply to on-premises, but I’ll use it as it’s become an industry norm.

The pipeline:

  • checks out the repo
  • runs ansible-playbook directly
  • injects secrets at runtime
  • uses inventories stored in Git
  • uses vars stored in Git
  • uses execution environments defined in Git
  • tests automation before deploying
  • runs deterministically and reproducibly

This pattern treats Ansible like any other automation tool: code —> test —> deploy —> verify.

When to use it

Pipeline execution is the right answer when you need:

  • reproducibility
  • portability
  • deterministic execution
  • Git as the single source of truth
  • cloud‑neutral automation
  • ephemeral runners
  • multi‑cloud or hybrid cloud
  • developer‑friendly workflows
  • automation that can run anywhere

This is the pattern used by ‘cloud‑native’ teams, platform teams, and anyone who values Git truth over controller truth.

Strengths

  • Fully reproducible
  • Fully portable
  • Fully version‑controlled
  • No hidden state
  • No GUI configuration
  • No controller dependency
  • Works in any CI/CD system
  • Works locally
  • Works in DR
  • Works in multi‑cloud

Weaknesses

  • Pipelines must understand environment boundaries
  • Pipelines must manage secrets securely
  • Pipelines must enforce guardrails
  • Pipelines can become too flexible
  • Pipelines can accidentally bypass governance

This is why enterprises often avoid this pattern, not because it’s wrong, but because they fear losing control. The fear is real.

Pattern 2: Controller‑Driven Execution (AWX/AAP)

“Automation runs inside a governed platform.”

This is the typical enterprise pattern.

AWX/AAP:

  • pulls playbooks from Git
  • stores inventories
  • stores credentials
  • enforces RBAC
  • provides audit trails
  • standardizes execution environments
  • triggers automation from events
  • provides multi‑team visibility

This pattern treats Ansible as a governed automation platform, not a CLI tool.

When to use it

Controller execution is the right answer when you need:

  • RBAC
  • credential isolation
  • auditability
  • inventory synchronization
  • standardized execution environments
  • event‑driven automation
  • multi‑team governance
  • compliance and regulatory controls

This is the pattern used by large enterprises, regulated industries, and teams with strict governance requirements.

Strengths

  • Centralized governance
  • Centralized credentials
  • Centralized inventory
  • Centralized audit
  • Standardized execution environments
  • Event‑driven automation
  • Multi‑team visibility
  • Strong guardrails

Weaknesses

  • Hidden state
  • GUI configuration
  • Non‑portable automation
  • Not fully reproducible
  • Not fully version‑controlled
  • Controller dependency
  • Harder to test automation before deployment
  • Harder to run automation outside AWX

This is why ‘cloud‑native’ teams avoid this pattern, not because it’s wrong, but because it’s not portable.

Anti‑Patterns

These are the patterns that break automation, cause outages, and create drift.

Anti‑Pattern 1: AWX as the “automation store”

Pipelines call AWX job templates as if AWX stores automation.

It doesn’t.

AWX stores configuration, not automation.

Automation lives in Git. AWX overrides Git with controller‑side state.

This creates split‑brain automation, if you think git is the source of truth, surprise, it isn’t.

Anti‑Pattern 2: Pipelines storing credentials

This is a governance failure.

Pipelines should inject secrets at runtime, not store them.

AWX should manage credentials. Actually, just about anything else should manage credentials.

Anti‑Pattern 3: AWX storing inventories that drift from Git

Inventories should be version‑controlled.

If AWX is the inventory source of truth, Git is no longer authoritative.

This breaks reproducibility.

Anti‑Pattern 4: Pipelines bypassing RBAC

If pipelines can run any playbook against any host using any credential, you’ve lost governance.

This is dangerous in enterprise environments.

Anti‑Pattern 5: AWX storing controller‑side vars that override Git

This creates non‑deterministic execution.

Playbooks behave differently depending on controller configuration.

This breaks portability.

Does the Combination Make Sense?

Here’s the part most teams miss:

The two patterns are not mutually exclusive. They are complementary.

The Practical Guidance

Use pipeline execution when:

  • automation must be reproducible
  • automation must be portable
  • automation must be version‑controlled
  • automation must run anywhere
  • automation must be tested before deployment

Use controller execution when:

  • governance matters
  • RBAC matters
  • credential isolation matters
  • auditability matters
  • inventory sync matters
  • event‑driven automation matters

Use both when:

  • you need reproducibility and governance
  • you need portability and control
  • you need Git truth and environment truth
  • you need cloud‑native execution and enterprise guardrails

This is the two‑layer architecture.

Final Takeaway

There are two patterns.

Trying to make one do both is how teams create drift, outages, and brittle automation.

Proudly powered by WordPress & Skyrocket Themes

MENU

Navigation