Categories
Ansible AWX Red Hat

Fatal glibc error: CPU does not support x86-64-v2

I’m just writing this down in case anyone has a similar issue.

As per Building Red Hat Enterprise Linux 9 for the x86-64-v2 microarchitecture level | Red Hat Developer, back in 2020, AMD, Intel, Red Hat, and SUSE collaborated to define three x86-64 microarchitecture levels on top of the x86-64 baseline. The three microarchitectures group together CPU features roughly based on hardware release dates:

  • x86-64-v2 brings support (among other things) for vector instructions up to Streaming SIMD Extensions 4.2 (SSE4.2)  and Supplemental Streaming SIMD Extensions 3 (SSSE3), the POPCNT instruction (useful for data analysis and bit-fiddling in some data structures), and CMPXCHG16B (a two-word compare-and-swap instruction useful for concurrent algorithms).
  • x86-64-v3 adds vector instructions up to AVX2, MOVBE (for big-endian data access), and additional bit-manipulation instructions.
  • x86-64-v4 includes vector instructions from some of the AVX-512 variants.

This is a great idea and goal except when you have perfectly good old hardware that, while end-of-life is still working and you find it doesn’t support the new compile target.

This nice little awk script from the fine folks over at stackexchange will show you what microarchitecture your cpu supports by looking at the /proc/cpuinfo flags. I’ve included a local copy here and as you can see it’s pretty simple.

#!/usr/bin/awk -f

BEGIN {
    while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1
    if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1
    if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2
    if (level == 2 && /avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3
    if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4
    if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 }
    exit 1
}

Running the awk script on my test system reveals :

$ ./testarch.awk
CPU supports x86-64-v1

The implications of this are annoying for me. I was trying to get awx to work on my little play system, but as the awx container is based on centos9 and compiled requiring at least x86-64-v2 then the awx container just wont start – yes I know there is more to awx than just this container, but it highlights the point nicely in the following command.

$ docker run --rm  ghcr.io/ansible/awx:latest
Fatal glibc error: CPU does not support x86-64-v2

This seems to have started somewhere after awx release 19.5.0

Categories
Ansible Ansible Tower Cloudforms ManageIQ

Ansible Tower provider for Cloudforms / ManageIQ

Always fun to strike problems in what should be the simplest things. I wanted to add Ansible Tower as a service into ManageIQ. Cloudforms would have a similar result.

Following the very simple instructions at https://www.manageiq.org/docs/reference/latest/doc-Managing_Providers/miq/index I proceeded to add the new provider.

It’s all pretty straight forward. Putting in the URL to my Tower server I get greeted with a bunch of errors.

Credential validation was not successful:  {:headers=>{"server"=>"nginx", "date"=>"Mon, 13 Jan 2020  23:42:12 GMT", "content-type"=>"text/html; charset=utf-8",  "content-length"=>"3873", "connection"=>"close",  "vary"=>"Cookie, Accept-Language, Origin",  "content-language"=>"en", "x-api-total-time"=>"0.046s"
.
.
.
blah blah
.
.
status=>404

By default ManageIQ and Cloudforms search for /api/v1 on the Tower server. /api/v1 was deprecated and removed as of Ansible Tower 3.6, see https://docs.ansible.com/ansible-tower/latest/html/towerapi/conventions.html

So, what is a person to do? Hit the google. Eventually I came across this bugzilla item https://bugzilla.redhat.com/show_bug.cgi?id=1740860 and it gave a hint as to just specifying the /api/v2 in the URL I gave to ManageIQ rather than just the base hostname. eg. https://blah…./api/v2

Tried it, it worked! My credential validated and a provider refresh was automatically initiated and all my Ansible Tower templates and inventories were discovered correctly.

Categories
Ansible Ansible Tower Azure Red Hat

Azure Credentials for Ansible

So, you need Ansible to connect to Azure. Congrats, Ansible is awesome for managing Azure resources. The Ansible team has already put together a scenario on how to integrate Ansible with Azure over at https://docs.ansible.com/ansible/latest/scenario_guides/guide_azure.html

The section ‘Authenticating with Azure‘ sounds like the right place, but you can’t use your AD username / password from Ansible because you turned on 2FA – You turned it on RIGHT? So the option left to you is to create a Service Principal (SP).

Note: having 2FA on your account is what you should be doing, so don’t turn it off.

It’s quite simple to create a credential for Ansible to use when connecting to Azure. Simply, fire up the Cloud Shell (awesome feature BTW Microsoft) and create a Service Principal (SP).

But Hang On, what is a Service Principal? The Ansible guide refers you to the Azure documentation over at https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal which you will read, and if you’re like me, you’ll wonder what you actually just read. Have no fear. As I mentioned above you can use a simple Azure CLI command (via the Cloud Shell you just started) and create the Service Principal. Think of the Service Principal as a credential an application (in this case Ansible) can use to access the Azure service(s).

geoff@Azure:~$ az ad sp create-for-rbac --name svc-ansible-azure  # (optional if not specified one will be generated)  --password 'ALovelyComplexPasswor@'
Changing "svc-ansible-azure" to a valid URI of "http://svc-ansible-azure", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/88888888-4444-4444-4444-cccccccccccc"
  Retrying role assignment creation: 1/36
  Retrying role assignment creation: 2/36
{
  "appId": "appid888-4444-4444-4444-cccccccccccc",
  "displayName": "svc-ansible-azure",
  "name": "http://svc-ansible-azure",
  "password": "password-4444-4444-4444-cccccccccccc",
  "tenant": "tenant88-4444-4444-4444-cccccccccccc"
}
geoff@Azure:~$

If you want to see what that command just did in the Azure portal, head over to the Azure Active Directory -> App registrations blade.

and then you can see the Service Principal you just created.

So what do you do with the new credential.

The Ansible Azure scenario guide has a section on what to do, however, it’s a bit too vague for me.

Using Environment Variables

To pass service principal credentials via the environment, define the following variables:

  • AZURE_CLIENT_ID
  • AZURE_SECRET
  • AZURE_SUBSCRIPTION_ID
  • AZURE_TENANT

Azure has given me :

“appId”: “appid888-4444-4444-4444-cccccccccccc”,
“displayName”: “svc-ansible-azure”,
“name”: “http://svc-ansible-azure”,
“password”: “password-4444-4444-4444-cccccccccccc”,
“tenant”: “tenant88-4444-4444-4444-cccccccccccc”

For your sanity,
AZURE_CLIENT_ID ==> appId
AZURE_SECRET ==> password
AZURE_TENANT ==> tenant

The remaining item, AZURE_SUBSCRIPTION_ID is exactly that, you can also get from the Cloud Shell as follows

geoff@Azure:~$ az account list
[
  {
    "cloudName": "AzureCloud",
    "id": "subscrip-4444-4444-4444-cccccccccccc
    "isDefault": true,
.
.
.

In this case AZURE_SUBSCRIPTION_ID ==> id , whichever id in your account that is valid for your use case.

If you want to add these credentials into Ansible Tower, simply create a Credential of type Microsoft Azure Resource Manager and use the values you’ve deduced above. Ansible Tower will automatically translate them into Environment Variables for your Tower template execution.

Enjoy Ansible and Azure!

Categories
Ansible

Ansible, more than just SSH

I often see the statement Ansible manages clients using SSH or WinRM. While this is a true statement, it is also incomplete.

Ansible currently has 26 connection types which you can find at https://docs.ansible.com/ansible/latest/plugins/connection.html

For me personally, some of the other interesting connection types are :

  • netconf
  • network_cli

    netconf and network_cli are commonly used to perform network device automation.
  • psrp

    psrp is similar to WinRM however it has the added benefit of being used via a proxy which is very useful when you have to consider bastian hosts.
  • vmware_tools

    vmware_tools is a relatively new addition to the ansible family and allows you to execute commands, transfer files to vSphere based systems without using the VM network.

Most Ansible developers will have also used connection type local in many of their playbooks, probably without realizing that it was a different connection type.

Ansible is also extensible. If you need to connect to something weird and wacky (but of great importance to you) then you can develop your own modules and connection plugin (or other sorts of plugins) – see https://docs.ansible.com/ansible/latest/plugins/plugins.html

Ansible versatility doesn’t end there though and many newcomers to ansible don’t realise that you can also manage multiple clouds, container platforms and virtualisation platforms – it’s the Swiss Army knife of IT automation.

Categories
Ansible Ansible Tower ServiceNOW

Adding a custom credential type in Ansible Tower for ServiceNOW

It’s been one of those weeks and I needed to get some more experience with the ansible ServiceNOW modules, specifically within Ansible Tower. It looked pretty simple and in fact it really was quite simple.

Ansible Tower neatly stores credentials within it – or externally if that fills you with joy. There isn’t a ServiceNow credential type in Ansible Tower. Undeterred, I thought I would use machine credentials, but tower has an annoying behavior of only allowing 1 instance of each credential type attached to a tower template and I am already using machine credentials in my template.

Fortunately on the left hand side of the tower ui there’s an entry labelled credential types

When creating the credential type you need to supply two (2) pieces of information. The first piece is called the Input configuration – or what the fields look like on the web ui when you create a credential of this type and secondly, the Injector configuration which details what do do with thew credentials.

In my case, the new credential type is called SNOW and i’m providing the instance name, username and password as part of the structure for this credential – via the Input configuration and then I detail that I want to store this data in environment variables that will be accessible from my playbook when run in tower.

Input configuration

fields:
  - id: instance
    type: string
    label: Instance
  - id: username
    type: string
    label: Username
  - id: password
    type: string
    label: Password
    secret: true
required:
  - instance
  - username
  - password

Injector configuration

env:
  SN_INSTANCE: '{{instance}}'
  SN_PASSWORD: '{{password}}'
  SN_USERNAME: '{{username}}'

The way you use them in your playbook is quite simple. The following is a snippet of playbook showing that.


   - name: Create an incident
     snow_record:
       username: '{{ lookup("env", "SN_USERNAME") }}'
       password: '{{ lookup("env", "SN_PASSWORD") }}'
       instance: '{{ lookup("env", "SN_INSTANCE") }}'
       state: present
       data:
         short_description: "This is a test incident opened by Ansible"
         severity: 3
         priority: 2
     register: new_incident

Categories
Ansible ESXi Red Hat

ESXi 6+ PXE Boot from Centos 8 – Nope?

I was rebuilding some Lab ESXi physical hosts, but also thought i’d upgrade my ‘builder’ system to Centos 8. My builder system uses a bunch of Ansible playbooks to create the necessary DHCP, TFTP etc configuration to support PXE booting multiple OS types – including ESXi 6.5/6.7.

I started with test builds of Centos 7/8 using my now Centos 8 build server and it was all fine.

However….. when I tried to build ESXi 6.5+ the TFTP delivered the ESXi mboot.c32 file to the host (via syslinux 6.04 which is new to Centos 8) but it couldn’t be loaded. After several hours of frustration I tried downgrading to the syslinux 3.86 version mentioned in https://www.vmware.com/techpapers/2015/installing-vmware-esxi-6.0-using-pxe-10508.html . Sadly you can’t install that version on Centos 8 without considerable grief.

I was able to install syslinux 4.05 on Centos 8 and lo and behold the build process works. Clearly something in syslinux 6 doesn’t like PXE booting ESXi. I’m not sure what yet, but hopefully this blog post at least gives people a workaround to a frustrating problem.

Categories
Ansible Ansible Tower Red Hat

Ansible Tower – Local_Action + Sudo ?

There are many times when you run an Ansible playbook through Ansible Tower and you have to become a privileged user on the target system. This is business as usual for Ansible and Ansible Tower.

This is normally achieved by specifying become as part of your playbook, such as this snippet.

---
- name: Patch Linux
  hosts: all
  gather_facts: true
  become: true

Typically, as part of a patching playbook, you would reboot the system and wait for the reboot to finish using a code fragment like this :

 - name: Wait for server to restart
   local_action:
     module: wait_for
       host={{ ansible_ssh_host }}
       port=22
       delay=60
       timeout=300

This local_action inherits the become: true from the parent definition and this is where Tower starts to complain. Remember, with Ansible Tower, it’s the tower server itself where the local_action will run. You can expect to see something like this :

"module_stderr": "sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?\n",

No, you SHOULD NOT enable the awx user to use sudo on the Tower system as the AWX service user is intentionally restricted from sudo operations. The best approach is to de-privilege the local_action. Fortunately, local_action has it’s own become capability so you can turn off the request for privileged access as you don’t need it.

The above code block is now :

 - name: Wait for server to restart
   become: false
   local_action:
     module: wait_for
       host={{ ansible_ssh_host }}
       port=22
       delay=60
       timeout=300

and the tower job template will execute without any errors.