As per Building Red Hat Enterprise Linux 9 for the x86-64-v2 microarchitecture level | Red Hat Developer, back in 2020, AMD, Intel, Red Hat, and SUSE collaborated to define three x86-64 microarchitecture levels on top of the x86-64 baseline. The three microarchitectures group together CPU features roughly based on hardware release dates:

• x86-64-v2 brings support (among other things) for vector instructions up to Streaming SIMD Extensions 4.2 (SSE4.2)  and Supplemental Streaming SIMD Extensions 3 (SSSE3), the POPCNT instruction (useful for data analysis and bit-fiddling in some data structures), and CMPXCHG16B (a two-word compare-and-swap instruction useful for concurrent algorithms).
• x86-64-v3 adds vector instructions up to AVX2, MOVBE (for big-endian data access), and additional bit-manipulation instructions.
• x86-64-v4 includes vector instructions from some of the AVX-512 variants.

This is a great idea and goal except when you have perfectly good old hardware that, while end-of-life is still working and you find it doesn’t support the new compile target.

This nice little awk script from the fine folks over at stackexchange will show you what microarchitecture your cpu supports by looking at the /proc/cpuinfo flags. I’ve included a local copy here and as you can see it’s pretty simple.

#!/usr/bin/awk -f

BEGIN {
    while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1
    if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1
    if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2
    if (level == 2 && /avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3
    if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4
    if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 }
    exit 1
}

Running the awk script on my test system reveals :

$ ./testarch.awk
CPU supports x86-64-v1

The implications of this are annoying for me. I was trying to get awx to work on my little play system, but as the awx container is based on centos9 and compiled requiring at least x86-64-v2 then the awx container just wont start – yes I know there is more to awx than just this container, but it highlights the point nicely in the following command.

$ docker run --rm  ghcr.io/ansible/awx:latest
Fatal glibc error: CPU does not support x86-64-v2

This seems to have started somewhere after awx release 19.5.0

You need to get to the latest update level on the CentOS systems. If the systems have been unloved you will likely find that they can no longer access the repos servers.

Change the baseurl to http://vault.centos.org/, comment out the mirrorlist as per this image.

You’ll need to do this in at least :

/etc/yum.repos.d/CentOS-Linux-BaseOS.repo
/etc/yum.repos.d/CentOS-Linux-AppStream.repo

Then you can perform the required upgrade :

dnf update
dnf upgrade

Then I suggest re-booting and you can then perform the AlmaLinux migration by :

curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
bash almalinux-deploy.sh

Then the following should show that you’ve converted OK

cat /etc/os-release

cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.6 (Sky Tiger)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.6 (Sky Tiger)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.6"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

Installing on a Red Hat clone (eg. Centos 8)

This wont take long.

curl https://packages.microsoft.com/config/rhel/7/prod.repo | sudo tee /etc/yum.repos.d/microsoft.repo
sudo yum install -y powershell

That’s it

Installing on Ubuntu 20.04 and above

sudo snap install powershell --classic

Again, that’s it.

In both cases you can then launch the shell via :

$ pwsh
PowerShell 7.2.0
Copyright (c) Microsoft Corporation.

https://aka.ms/powershell
Type 'help' to get help.

PS /home/gocallag> 

Apply all the latest patches to your Centos 8 systems

dnf update -y
reboot

Then install the Centos 8 stream repo’s

dnf install -y centos-release-stream

Then swap from Centos Linux repo’s to Centos stream repo’s

dnf swap -y centos-{linux,stream}-repos

Then do a distro sync to get everything back in sync

dnf distro-sync -y
reboot

You should be golden at this point

Firstly, VMware provides a fine Powercli container built on top of Photon OS , but being me I thought Hey I wonder if I can get the same thing with a Red Hat Universal Base Image (UBI)? And so, my journey began.

I decided i’d use the VMware Dockerfile as the starting point, but I want to build it using buildah and run it using podman – because I’d like to know (you can see a pattern here) .

The original Dockerfile is accessible here, or here’s a local copy.

FROM photon:3.0
  
LABEL authors="renoufa@vmware.com,jaker@vmware.com"

ENV TERM linux

WORKDIR /root

# Set terminal. If we don't do this, weird readline things happen.
RUN echo "/usr/bin/pwsh" >> /etc/shells && \
    echo "/bin/pwsh" >> /etc/shells && \
    tdnf install -y powershell-6.2.3-1.ph3 unzip && \
    pwsh -c "Set-PSRepository -Name PSGallery -InstallationPolicy Trusted" && \
    pwsh -c "\$ProgressPreference = \"SilentlyContinue\"; Install-Module VMware.PowerCLI -RequiredVersion 11.5.0.14912921" && \
    pwsh -c "\$ProgressPreference = \"SilentlyContinue\"; Install-Module PowerNSX -RequiredVersion 3.0.1174" && \
    pwsh -c "\$ProgressPreference = \"SilentlyContinue\"; Install-Module PowervRA -RequiredVersion 3.6.0" && \
    curl -o ./PowerCLI-Example-Scripts.zip -J -L https://github.com/vmware/PowerCLI-Example-Scripts/archive/03272c1d2db26a525b31c930e3bf3d20d34468e0.zip && \
    unzip PowerCLI-Example-Scripts.zip && \
    rm -f PowerCLI-Example-Scripts.zip && \
    mv ./PowerCLI-Example-Scripts-* ./PowerCLI-Example-Scripts && \
    mv ./PowerCLI-Example-Scripts/Modules/* /usr/lib/powershell/Modules/ && \
    find / -name "net45" | xargs rm -rf && \
    tdnf erase -y unzip && \
    tdnf clean all


CMD ["/bin/pwsh"]

I’ve made a few changes, some cosmetic due to the way I like to layout my docker file, but the outcome is similar. My Dockerfile is below or you can find it over at my github account. Using the default RHEL7 UBI (sadly Microsoft don’t have powershell for RHEL8 as yet) I was able to build the image at around 567 Mb, whereas the Photon OS image is around 362 Mb. Not a bad result given how little effort (none) i’ve put into making it as small as possible.

FROM registry.access.redhat.com/ubi7/ubi:latest

LABEL authors="geoffocallaghan@gmail.com"

WORKDIR /root

RUN curl https://packages.microsoft.com/config/rhel/7/prod.repo -o /etc/yum.repos.d/microsoft.repo && yum install -y powershell  unzip
RUN pwsh -c 'Set-PSRepository -Name PSGallery -InstallationPolicy Trusted; \
             $ProgressPreference = "SilentlyContinue"; \
             Install-Module VMware.PowerCLI -RequiredVersion 11.5.0.14912921; \
             Install-Module PowerNSX -RequiredVersion 3.0.1174; \
             Install-Module PowervRA -RequiredVersion 3.6.0'
RUN curl -o ./PowerCLI-Example-Scripts.zip -J -L https://github.com/vmware/PowerCLI-Example-Scripts/archive/03272c1d2db26a525b31c930e3bf3d20d34468e0.zip \
    && unzip PowerCLI-Example-Scripts.zip \
    && rm -f PowerCLI-Example-Scripts.zip \
    && mv ./PowerCLI-Example-Scripts-* ./PowerCLI-Example-Scripts \
    && mv ./PowerCLI-Example-Scripts/Modules/* /opt/microsoft/powershell/6/Modules/ \
    && find / -name "net45" | xargs rm -rf


CMD ["/bin/pwsh"]

As you can see in the Dockerfile, i’m simply installing powershell from the microsoft repository on top of the RHEL7 UBI image and then (via powershell) installed the PowerCLI, PowerNSX and PowervRA modules from the upstream powershell gallery.

Building it with buildah is trivial.

buildah build-using-dockerfile -t rcli  .

And to run it via podman (trivial example)

[gocallag@orac8 rhel7]$ podman run -it rcli pwsh
PowerShell 6.2.3
Copyright (c) Microsoft Corporation. All rights reserved.

https://aka.ms/pscore6-docs
Type 'help' to get help.

PS /root> Get-VM   # plus a couple of tabs to force auto-completion of the command 
Get-VM                                       Get-VmfsDatastoreInfo                        Get-VMHostPatch
Get-VMByToolsInfo                            Get-VMGuest                                  Get-VMHostPciDevice
Get-VMCCommand                               Get-VMHost                                   Get-VMHostProfile
Get-VMCEdge                                  Get-VMHostAccount                            Get-VMHostProfileImageCacheConfiguration
Get-VMCEdgeNic                               Get-VMHostAdvancedConfiguration              Get-VMHostProfileRequiredInput
Get-VMCEdgeNicStat                           Get-VMHostAttributes                         Get-VMHostProfileStorageDeviceConfiguration
Get-VMCEdgeStatus                            Get-VMHostAuthentication                     Get-VMHostProfileUserConfiguration
Get-VMCEdgeUplinkStat                        Get-VMHostAvailableTimeZone                  Get-VMHostProfileVmPortGroupConfiguration
Get-VMCFirewallRule                          Get-VMHostBirthday                           Get-VMHostRoute
Get-VMCLogicalNetwork                        Get-VMHostDiagnosticPartition                Get-VMHostService
Get-VMCOrg                                   Get-VMHostDisk                               Get-VMHostSnmp
Get-VMCPSettings                             Get-VMHostDiskPartition                      Get-VMHostStartPolicy
Get-VMCSDDC                                  Get-VMHostFirewallDefaultPolicy              Get-VMHostStorage
Get-VMCSDDCCluster                           Get-VMHostFirewallException                  Get-VMHostSysLogServer
Get-VMCSDDCDefaultCredential                 Get-VMHostFirmware                           Get-VMmaxIOPS
Get-VmcSddcNetworkService                    Get-VMHostFirmwareVersion                    Get-VMQuestion
Get-VMCSDDCPublicIP                          Get-VMHostHardware                           Get-VMResourceConfiguration
Get-VMCSDDCVersion                           Get-VMHostHba                                Get-VMStartPolicy
Get-VmcService                               Get-VMHostImageProfile                       Get-VMToolsGuestInfo
Get-VMCTask                                  Get-VMHostMatchingRules                      Get-VMToolsInfo
Get-VMCVMHost                                Get-VMHostModule                             Get-VMToolsInstallLastError
Get-VMEncryptionInfo                         Get-VMHostNetwork                            Get-VMToolsUpgradePolicy
Get-VMEvcMode                                Get-VMHostNetworkAdapter
Get-VmfsDatastoreIncrease                    Get-VMHostNtpServer

You’re likely, possibly, most likely not wondering if I have anything planned for this container. The answer is yes, but it will be the subject of later posts. I’m a big fan of the ability to run Powercli via powershell on linux, and doing it via a container is a very neat packaging solution. Sure, i’ve could’ve used the VMware container (kudos to them for creating it), but I now know more than I did this morning and that’s the result I was aiming for.

The section ‘Authenticating with Azure‘ sounds like the right place, but you can’t use your AD username / password from Ansible because you turned on 2FA – You turned it on RIGHT? So the option left to you is to create a Service Principal (SP).

Note: having 2FA on your account is what you should be doing, so don’t turn it off.

It’s quite simple to create a credential for Ansible to use when connecting to Azure. Simply, fire up the Cloud Shell (awesome feature BTW Microsoft) and create a Service Principal (SP).

But Hang On, what is a Service Principal? The Ansible guide refers you to the Azure documentation over at https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal which you will read, and if you’re like me, you’ll wonder what you actually just read. Have no fear. As I mentioned above you can use a simple Azure CLI command (via the Cloud Shell you just started) and create the Service Principal. Think of the Service Principal as a credential an application (in this case Ansible) can use to access the Azure service(s).

geoff@Azure:~$ az ad sp create-for-rbac --name svc-ansible-azure  # (optional if not specified one will be generated)  --password 'ALovelyComplexPasswor@'
Changing "svc-ansible-azure" to a valid URI of "http://svc-ansible-azure", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/88888888-4444-4444-4444-cccccccccccc"
  Retrying role assignment creation: 1/36
  Retrying role assignment creation: 2/36
{
  "appId": "appid888-4444-4444-4444-cccccccccccc",
  "displayName": "svc-ansible-azure",
  "name": "http://svc-ansible-azure",
  "password": "password-4444-4444-4444-cccccccccccc",
  "tenant": "tenant88-4444-4444-4444-cccccccccccc"
}
geoff@Azure:~$

If you want to see what that command just did in the Azure portal, head over to the Azure Active Directory -> App registrations blade.

and then you can see the Service Principal you just created.

So what do you do with the new credential.

The Ansible Azure scenario guide has a section on what to do, however, it’s a bit too vague for me.

Using Environment Variables

To pass service principal credentials via the environment, define the following variables:

• AZURE_CLIENT_ID
• AZURE_SECRET
• AZURE_SUBSCRIPTION_ID
• AZURE_TENANT

Azure has given me :

“appId”: “appid888-4444-4444-4444-cccccccccccc”,
“displayName”: “svc-ansible-azure”,
“name”: “http://svc-ansible-azure”,
“password”: “password-4444-4444-4444-cccccccccccc”,
“tenant”: “tenant88-4444-4444-4444-cccccccccccc”

For your sanity,
AZURE_CLIENT_ID ==> appId
AZURE_SECRET ==> password
AZURE_TENANT ==> tenant

The remaining item, AZURE_SUBSCRIPTION_ID is exactly that, you can also get from the Cloud Shell as follows

geoff@Azure:~$ az account list
[
  {
    "cloudName": "AzureCloud",
    "id": "subscrip-4444-4444-4444-cccccccccccc
    "isDefault": true,
.
.
.

In this case AZURE_SUBSCRIPTION_ID ==> id , whichever id in your account that is valid for your use case.

If you want to add these credentials into Ansible Tower, simply create a Credential of type Microsoft Azure Resource Manager and use the values you’ve deduced above. Ansible Tower will automatically translate them into Environment Variables for your Tower template execution.

Enjoy Ansible and Azure!

I started with test builds of Centos 7/8 using my now Centos 8 build server and it was all fine.

However….. when I tried to build ESXi 6.5+ the TFTP delivered the ESXi mboot.c32 file to the host (via syslinux 6.04 which is new to Centos 8) but it couldn’t be loaded. After several hours of frustration I tried downgrading to the syslinux 3.86 version mentioned in https://www.vmware.com/techpapers/2015/installing-vmware-esxi-6.0-using-pxe-10508.html . Sadly you can’t install that version on Centos 8 without considerable grief.

I was able to install syslinux 4.05 on Centos 8 and lo and behold the build process works. Clearly something in syslinux 6 doesn’t like PXE booting ESXi. I’m not sure what yet, but hopefully this blog post at least gives people a workaround to a frustrating problem.

Fortunately, it wasn’t far away – it’s been moved to the PowerTools repository which you can enable with a quick:

dnf config-manager --set-enabled PowerTools

Then you can install my favourite little text based web browser again.

In order to make this more reasonable, such as the 1920×1080 full screen resolution on my laptop you have to set the hyper-v framebuffer at boot time.

sudo grubby --update-kernel=ALL --args="video=hyperv_fb:1920x1080"

You’ll likely need to do this after each kernel update.

May the full screen be with you.

This is normally achieved by specifying become as part of your playbook, such as this snippet.

---
- name: Patch Linux
  hosts: all
  gather_facts: true
  become: true

Typically, as part of a patching playbook, you would reboot the system and wait for the reboot to finish using a code fragment like this :

 - name: Wait for server to restart
   local_action:
     module: wait_for
       host={{ ansible_ssh_host }}
       port=22
       delay=60
       timeout=300

This local_action inherits the become: true from the parent definition and this is where Tower starts to complain. Remember, with Ansible Tower, it’s the tower server itself where the local_action will run. You can expect to see something like this :

"module_stderr": "sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?\n",

No, you SHOULD NOT enable the awx user to use sudo on the Tower system as the AWX service user is intentionally restricted from sudo operations. The best approach is to de-privilege the local_action. Fortunately, local_action has it’s own become capability so you can turn off the request for privileged access as you don’t need it.

The above code block is now :

 - name: Wait for server to restart
   become: false
   local_action:
     module: wait_for
       host={{ ansible_ssh_host }}
       port=22
       delay=60
       timeout=300

and the tower job template will execute without any errors.