OCI: No route to host?

I’ve been doing some work on Oracle’s Cloud as they provide a decent free tier to experiment with. I’ve been very pleasantly surprised with OCI and will likely move some of my personal workloads there.

It wasn’t without a bit of a head scratching experience though when I was trying to get application connectivity between two OCI images on the same private network I had created.


curl: (7) Failed to connect to port 80: No route to host

My first thought was the cloud ingress rules, but i’d added the following as a first desperate attempt to get things working.

Try again, Still no route!

What I discovered is the OCI supplied images (I was using the Ampere Ubuntu 20.04 image in this case) have an interesting set of iptables rules baked into the image.

root@blog:~# cat /etc/iptables/rules.v4
# CLOUD_IMG: This file was created/modified by the Cloud Image build process
# iptables configuration for Oracle Cloud Infrastructure

# See the Oracle-Provided Images section in the Oracle Cloud Infrastructure
# documentation for security impact of modifying or removing these rule

:OUTPUT ACCEPT [463:49013]
:InstanceServices - [0:0]
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p udp --sport 123 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
#-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -d -j InstanceServices

I’ve commented out the offending line. With OCI supplied images, even though the default filter is ACCEPT, they place a reject-with icmp-host-prohibited at the end of the INPUT table, which effectively rejects everything not specifically allowed (such as the port 22 rule the line before).

My two options were to either put in my specific allows (the right thing to do) or remove the reject and just use the INPUT filter default. I chose the latter as I was experimenting in this case and kept the information at my finger tips for more ‘production-like’ deployments.

The end result, communication between the 2 OCI Ubuntu instances over the private network now works fine.

Caveat: In my case I understood the risks associated with removing the reject for my Use Case. Please perform your own due diligence for your Use Case, you’re probably better off specifically adding the communication rules you want to allow.

Getting started with Powershell on Linux

First of all, simply don’t believe anyone who says that it’s hard to install Powershell on Linux.

Installing on a Red Hat clone (eg. Centos 8)

This wont take long.

curl https://packages.microsoft.com/config/rhel/7/prod.repo | sudo tee /etc/yum.repos.d/microsoft.repo
sudo yum install -y powershell

That’s it

Installing on Ubuntu 20.04 and above

sudo snap install powershell --classic

Again, that’s it.

In both cases you can then launch the shell via :

$ pwsh
PowerShell 7.2.0
Copyright (c) Microsoft Corporation.

Type 'help' to get help.

PS /home/gocallag> 

Converting Centos 8 to Centos 8 Stream – because you know you want to!

This is more so I can remember, but it’s basically 3 steps.

Apply all the latest patches to your Centos 8 systems

dnf update -y

Then install the Centos 8 stream repo’s

dnf install -y centos-release-stream

Then swap from Centos Linux repo’s to Centos stream repo’s

dnf swap -y centos-{linux,stream}-repos

Then do a distro sync to get everything back in sync

dnf distro-sync -y

You should be golden at this point

Fresh Openshift Cluster install (4.6) on vSphere doesn’t complete – Cluster Monitoring Operator stuck

It’s a simple issue to resolve, but just a little annoying.

The CVO doesn’t complete because the Cluster-monitoring-operator pod rollout stuck with error message CreateContainerConfigError.

The actual error shows that :

Error: container has runAsNonRoot and image has non-numeric user (nobody), cannot verify user is non-root

This is still an open issue with Red Hat and it’s being tracked via this BZ. It is however easily corrected by deleting the offending pod and letting it get re-created

Dotnet core 3.1 on Raspberry Pi 4

Don’t believe sites telling you it’s hard to install Dotnet core on a Raspberry Pi 4 – it isn’t and hopefully I haven’t tempted fate by showing how simple it is 🙂

pi@raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10.3 (buster)
Release:        10.3
Codename:       buster

Wander over to https://dotnet.microsoft.com/download/dotnet-core/scripts and you’ll find the installation script. You can download the bash script and execute it.

pi@raspberrypi:~ $ bash dotnet-install.sh
dotnet_install: Warning: Unable to locate zlib. Probable prerequisite missing; install zlib.
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/3.1.201/dotnet-sdk-3.1.201-linux-arm.tar.gz
dotnet-install: Extracting zip from https://dotnetcli.azureedge.net/dotnet/Sdk/3.1.201/dotnet-sdk-3.1.201-linux-arm.tar.gz
dotnet-install: Adding to current process PATH: `/home/pi/.dotnet`. Note: This change will be visible only when sourcing script.
dotnet-install: Installation finished successfully.

Then it’s just a simple case of adding /home/pi/.dotnet to your system PATH variable.

pi@raspberrypi:~ $ dotnet --version

Wondering about the missing pre-requisite for zlib – so was I. The installer is looking for zlib via a ldconfig command. It seems package zlib1g-dev installs it as libz so the check fails. I’ll update this post if I run into any actual problems.

Ansible Tower provider for Cloudforms / ManageIQ

Always fun to strike problems in what should be the simplest things. I wanted to add Ansible Tower as a service into ManageIQ. Cloudforms would have a similar result.

Following the very simple instructions at https://www.manageiq.org/docs/reference/latest/doc-Managing_Providers/miq/index I proceeded to add the new provider.

It’s all pretty straight forward. Putting in the URL to my Tower server I get greeted with a bunch of errors.

Credential validation was not successful:  {:headers=>{"server"=>"nginx", "date"=>"Mon, 13 Jan 2020  23:42:12 GMT", "content-type"=>"text/html; charset=utf-8",  "content-length"=>"3873", "connection"=>"close",  "vary"=>"Cookie, Accept-Language, Origin",  "content-language"=>"en", "x-api-total-time"=>"0.046s"
blah blah

By default ManageIQ and Cloudforms search for /api/v1 on the Tower server. /api/v1 was deprecated and removed as of Ansible Tower 3.6, see https://docs.ansible.com/ansible-tower/latest/html/towerapi/conventions.html

So, what is a person to do? Hit the google. Eventually I came across this bugzilla item https://bugzilla.redhat.com/show_bug.cgi?id=1740860 and it gave a hint as to just specifying the /api/v2 in the URL I gave to ManageIQ rather than just the base hostname. eg. https://blah…./api/v2

Tried it, it worked! My credential validated and a provider refresh was automatically initiated and all my Ansible Tower templates and inventories were discovered correctly.

Powercli via RHEL7 UBI container images

So yes, that is quite a specific title for a blog post. The path leading to it wasn’t as succinct, but it was an enjoyable journey.

Firstly, VMware provides a fine Powercli container built on top of Photon OS , but being me I thought Hey I wonder if I can get the same thing with a Red Hat Universal Base Image (UBI)? And so, my journey began.

I decided i’d use the VMware Dockerfile as the starting point, but I want to build it using buildah and run it using podman – because I’d like to know (you can see a pattern here) .

The original Dockerfile is accessible here, or here’s a local copy.

FROM photon:3.0
LABEL authors="renoufa@vmware.com,jaker@vmware.com"

ENV TERM linux


# Set terminal. If we don't do this, weird readline things happen.
RUN echo "/usr/bin/pwsh" >> /etc/shells && \
    echo "/bin/pwsh" >> /etc/shells && \
    tdnf install -y powershell-6.2.3-1.ph3 unzip && \
    pwsh -c "Set-PSRepository -Name PSGallery -InstallationPolicy Trusted" && \
    pwsh -c "\$ProgressPreference = \"SilentlyContinue\"; Install-Module VMware.PowerCLI -RequiredVersion" && \
    pwsh -c "\$ProgressPreference = \"SilentlyContinue\"; Install-Module PowerNSX -RequiredVersion 3.0.1174" && \
    pwsh -c "\$ProgressPreference = \"SilentlyContinue\"; Install-Module PowervRA -RequiredVersion 3.6.0" && \
    curl -o ./PowerCLI-Example-Scripts.zip -J -L https://github.com/vmware/PowerCLI-Example-Scripts/archive/03272c1d2db26a525b31c930e3bf3d20d34468e0.zip && \
    unzip PowerCLI-Example-Scripts.zip && \
    rm -f PowerCLI-Example-Scripts.zip && \
    mv ./PowerCLI-Example-Scripts-* ./PowerCLI-Example-Scripts && \
    mv ./PowerCLI-Example-Scripts/Modules/* /usr/lib/powershell/Modules/ && \
    find / -name "net45" | xargs rm -rf && \
    tdnf erase -y unzip && \
    tdnf clean all

CMD ["/bin/pwsh"]

I’ve made a few changes, some cosmetic due to the way I like to layout my docker file, but the outcome is similar. My Dockerfile is below or you can find it over at my github account. Using the default RHEL7 UBI (sadly Microsoft don’t have powershell for RHEL8 as yet) I was able to build the image at around 567 Mb, whereas the Photon OS image is around 362 Mb. Not a bad result given how little effort (none) i’ve put into making it as small as possible.

FROM registry.access.redhat.com/ubi7/ubi:latest

LABEL authors="geoffocallaghan@gmail.com"


RUN curl https://packages.microsoft.com/config/rhel/7/prod.repo -o /etc/yum.repos.d/microsoft.repo && yum install -y powershell  unzip
RUN pwsh -c 'Set-PSRepository -Name PSGallery -InstallationPolicy Trusted; \
             $ProgressPreference = "SilentlyContinue"; \
             Install-Module VMware.PowerCLI -RequiredVersion; \
             Install-Module PowerNSX -RequiredVersion 3.0.1174; \
             Install-Module PowervRA -RequiredVersion 3.6.0'
RUN curl -o ./PowerCLI-Example-Scripts.zip -J -L https://github.com/vmware/PowerCLI-Example-Scripts/archive/03272c1d2db26a525b31c930e3bf3d20d34468e0.zip \
    && unzip PowerCLI-Example-Scripts.zip \
    && rm -f PowerCLI-Example-Scripts.zip \
    && mv ./PowerCLI-Example-Scripts-* ./PowerCLI-Example-Scripts \
    && mv ./PowerCLI-Example-Scripts/Modules/* /opt/microsoft/powershell/6/Modules/ \
    && find / -name "net45" | xargs rm -rf

CMD ["/bin/pwsh"]

As you can see in the Dockerfile, i’m simply installing powershell from the microsoft repository on top of the RHEL7 UBI image and then (via powershell) installed the PowerCLI, PowerNSX and PowervRA modules from the upstream powershell gallery.

Building it with buildah is trivial.

buildah build-using-dockerfile -t rcli  .

And to run it via podman (trivial example)

[gocallag@orac8 rhel7]$ podman run -it rcli pwsh
PowerShell 6.2.3
Copyright (c) Microsoft Corporation. All rights reserved.

Type 'help' to get help.

PS /root> Get-VM   # plus a couple of tabs to force auto-completion of the command 
Get-VM                                       Get-VmfsDatastoreInfo                        Get-VMHostPatch
Get-VMByToolsInfo                            Get-VMGuest                                  Get-VMHostPciDevice
Get-VMCCommand                               Get-VMHost                                   Get-VMHostProfile
Get-VMCEdge                                  Get-VMHostAccount                            Get-VMHostProfileImageCacheConfiguration
Get-VMCEdgeNic                               Get-VMHostAdvancedConfiguration              Get-VMHostProfileRequiredInput
Get-VMCEdgeNicStat                           Get-VMHostAttributes                         Get-VMHostProfileStorageDeviceConfiguration
Get-VMCEdgeStatus                            Get-VMHostAuthentication                     Get-VMHostProfileUserConfiguration
Get-VMCEdgeUplinkStat                        Get-VMHostAvailableTimeZone                  Get-VMHostProfileVmPortGroupConfiguration
Get-VMCFirewallRule                          Get-VMHostBirthday                           Get-VMHostRoute
Get-VMCLogicalNetwork                        Get-VMHostDiagnosticPartition                Get-VMHostService
Get-VMCOrg                                   Get-VMHostDisk                               Get-VMHostSnmp
Get-VMCPSettings                             Get-VMHostDiskPartition                      Get-VMHostStartPolicy
Get-VMCSDDC                                  Get-VMHostFirewallDefaultPolicy              Get-VMHostStorage
Get-VMCSDDCCluster                           Get-VMHostFirewallException                  Get-VMHostSysLogServer
Get-VMCSDDCDefaultCredential                 Get-VMHostFirmware                           Get-VMmaxIOPS
Get-VmcSddcNetworkService                    Get-VMHostFirmwareVersion                    Get-VMQuestion
Get-VMCSDDCPublicIP                          Get-VMHostHardware                           Get-VMResourceConfiguration
Get-VMCSDDCVersion                           Get-VMHostHba                                Get-VMStartPolicy
Get-VmcService                               Get-VMHostImageProfile                       Get-VMToolsGuestInfo
Get-VMCTask                                  Get-VMHostMatchingRules                      Get-VMToolsInfo
Get-VMCVMHost                                Get-VMHostModule                             Get-VMToolsInstallLastError
Get-VMEncryptionInfo                         Get-VMHostNetwork                            Get-VMToolsUpgradePolicy
Get-VMEvcMode                                Get-VMHostNetworkAdapter
Get-VmfsDatastoreIncrease                    Get-VMHostNtpServer

You’re likely, possibly, most likely not wondering if I have anything planned for this container. The answer is yes, but it will be the subject of later posts. I’m a big fan of the ability to run Powercli via powershell on linux, and doing it via a container is a very neat packaging solution. Sure, i’ve could’ve used the VMware container (kudos to them for creating it), but I now know more than I did this morning and that’s the result I was aiming for.

Azure Credentials for Ansible

So, you need Ansible to connect to Azure. Congrats, Ansible is awesome for managing Azure resources. The Ansible team has already put together a scenario on how to integrate Ansible with Azure over at https://docs.ansible.com/ansible/latest/scenario_guides/guide_azure.html

The section ‘Authenticating with Azure‘ sounds like the right place, but you can’t use your AD username / password from Ansible because you turned on 2FA – You turned it on RIGHT? So the option left to you is to create a Service Principal (SP).

Note: having 2FA on your account is what you should be doing, so don’t turn it off.

It’s quite simple to create a credential for Ansible to use when connecting to Azure. Simply, fire up the Cloud Shell (awesome feature BTW Microsoft) and create a Service Principal (SP).

But Hang On, what is a Service Principal? The Ansible guide refers you to the Azure documentation over at https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal which you will read, and if you’re like me, you’ll wonder what you actually just read. Have no fear. As I mentioned above you can use a simple Azure CLI command (via the Cloud Shell you just started) and create the Service Principal. Think of the Service Principal as a credential an application (in this case Ansible) can use to access the Azure service(s).

geoff@Azure:~$ az ad sp create-for-rbac --name svc-ansible-azure  # (optional if not specified one will be generated)  --password 'ALovelyComplexPasswor@'
Changing "svc-ansible-azure" to a valid URI of "http://svc-ansible-azure", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/88888888-4444-4444-4444-cccccccccccc"
  Retrying role assignment creation: 1/36
  Retrying role assignment creation: 2/36
  "appId": "appid888-4444-4444-4444-cccccccccccc",
  "displayName": "svc-ansible-azure",
  "name": "http://svc-ansible-azure",
  "password": "password-4444-4444-4444-cccccccccccc",
  "tenant": "tenant88-4444-4444-4444-cccccccccccc"

If you want to see what that command just did in the Azure portal, head over to the Azure Active Directory -> App registrations blade.

and then you can see the Service Principal you just created.

So what do you do with the new credential.

The Ansible Azure scenario guide has a section on what to do, however, it’s a bit too vague for me.

Using Environment Variables

To pass service principal credentials via the environment, define the following variables:


Azure has given me :

“appId”: “appid888-4444-4444-4444-cccccccccccc”,
“displayName”: “svc-ansible-azure”,
“name”: “http://svc-ansible-azure”,
“password”: “password-4444-4444-4444-cccccccccccc”,
“tenant”: “tenant88-4444-4444-4444-cccccccccccc”

For your sanity,
AZURE_SECRET ==> password
AZURE_TENANT ==> tenant

The remaining item, AZURE_SUBSCRIPTION_ID is exactly that, you can also get from the Cloud Shell as follows

geoff@Azure:~$ az account list
    "cloudName": "AzureCloud",
    "id": "subscrip-4444-4444-4444-cccccccccccc
    "isDefault": true,

In this case AZURE_SUBSCRIPTION_ID ==> id , whichever id in your account that is valid for your use case.

If you want to add these credentials into Ansible Tower, simply create a Credential of type Microsoft Azure Resource Manager and use the values you’ve deduced above. Ansible Tower will automatically translate them into Environment Variables for your Tower template execution.

Enjoy Ansible and Azure!

Ansible, more than just SSH

I often see the statement Ansible manages clients using SSH or WinRM. While this is a true statement, it is also incomplete.

Ansible currently has 26 connection types which you can find at https://docs.ansible.com/ansible/latest/plugins/connection.html

For me personally, some of the other interesting connection types are :

  • netconf
  • network_cli

    netconf and network_cli are commonly used to perform network device automation.
  • psrp

    psrp is similar to WinRM however it has the added benefit of being used via a proxy which is very useful when you have to consider bastian hosts.
  • vmware_tools

    vmware_tools is a relatively new addition to the ansible family and allows you to execute commands, transfer files to vSphere based systems without using the VM network.

Most Ansible developers will have also used connection type local in many of their playbooks, probably without realizing that it was a different connection type.

Ansible is also extensible. If you need to connect to something weird and wacky (but of great importance to you) then you can develop your own modules and connection plugin (or other sorts of plugins) – see https://docs.ansible.com/ansible/latest/plugins/plugins.html

Ansible versatility doesn’t end there though and many newcomers to ansible don’t realise that you can also manage multiple clouds, container platforms and virtualisation platforms – it’s the Swiss Army knife of IT automation.

Adding a custom credential type in Ansible Tower for ServiceNOW

It’s been one of those weeks and I needed to get some more experience with the ansible ServiceNOW modules, specifically within Ansible Tower. It looked pretty simple and in fact it really was quite simple.

Ansible Tower neatly stores credentials within it – or externally if that fills you with joy. There isn’t a ServiceNow credential type in Ansible Tower. Undeterred, I thought I would use machine credentials, but tower has an annoying behavior of only allowing 1 instance of each credential type attached to a tower template and I am already using machine credentials in my template.

Fortunately on the left hand side of the tower ui there’s an entry labelled credential types

When creating the credential type you need to supply two (2) pieces of information. The first piece is called the Input configuration – or what the fields look like on the web ui when you create a credential of this type and secondly, the Injector configuration which details what do do with thew credentials.

In my case, the new credential type is called SNOW and i’m providing the instance name, username and password as part of the structure for this credential – via the Input configuration and then I detail that I want to store this data in environment variables that will be accessible from my playbook when run in tower.

Input configuration

  - id: instance
    type: string
    label: Instance
  - id: username
    type: string
    label: Username
  - id: password
    type: string
    label: Password
    secret: true
  - instance
  - username
  - password

Injector configuration

  SN_INSTANCE: '{{instance}}'
  SN_PASSWORD: '{{password}}'
  SN_USERNAME: '{{username}}'

The way you use them in your playbook is quite simple. The following is a snippet of playbook showing that.

   - name: Create an incident
       username: '{{ lookup("env", "SN_USERNAME") }}'
       password: '{{ lookup("env", "SN_PASSWORD") }}'
       instance: '{{ lookup("env", "SN_INSTANCE") }}'
       state: present
         short_description: "This is a test incident opened by Ansible"
         severity: 3
         priority: 2
     register: new_incident