How to Build an Ubuntu Bastion Host for VCFA 9.x and Holodeck

I started using HoloDeck in earnest late last year. The ease of deploying and redeploying VCF 9 environments has been a godsend as I learn some of the new VCF Automation features.

The Holorouter and Webtop were good enough to get started. They were close to the environment, already had useful tools, and saved me from building another machine before I understood what I was trying to reach.

I quickly discovered that neither of the available working environments was particularly well suited to development. I couldn’t install the applications I needed in the Webtop terminal, which made it difficult to work with Kubernetes, perform meaningful development, or even track and save my work with Git.

Using the router OS prompt came with its own risks. For example, don’t even think about installing VCF.PowerCLI—it can break the PowerShell scripts used to deploy HoloDeck.

Then there was the time-consuming process of copying information out of the environment. Whether it was a template, a certificate, or a piece of command output, getting it back to my Mac required a multistep workflow: save it to a text file, SSH into another system, transfer it with SCP, and then retrieve it locally.

The process worked, but it felt like using a filing cabinet to pass a sticky note across the room. To put it mildly, it made about as much sense as putting a screen door on a submarine.

Webtop got us moving, but using it for everyday file transfer felt a little like installing a screen door on a submarine.

This article keeps things practical. It explains why I built the bastion, how its two network connections work, which routes it uses, and how to connect through it.

The answer was a small Ubuntu bastion host: a normal SSH destination, a place to run the VCF CLI and kubectl, and an SSH SOCKS5 bridge for browser and API access from my Mac.

The addresses below are sanitized examples from a VCFA 9.1 HoloDeck environment. Replace them with the values from your own environment before running commands.

The Mac stays comfortable on the outside while the bastion handles the route into HoloDeck.

What the bastion is for

The bastion gives me one reliable place to work from:

Mac
|
| normal SSH, file transfer, and optional SOCKS5 tunnel
v
Ubuntu bastion
|
| ens34 on VM Network, static management address
|
| ens37 on Holo-PG-A, static transit routes
v
Holorouter -> HoloDeck environment

From my Mac, the bastion behaves like an ordinary Linux host. That fixes the copy-and-paste problem immediately. It also gives me a clean home for tools, scripts, temporary certificates, and command history without making the Holorouter or Webtop carry the entire workflow.

The bastion is not a replacement for the Holorouter. It is a carefully placed client of the HoloDeck routing path.

The two network connections

The primary interface connects to the vSphere port group named:

VM Network

For this article, use a static example address in a separate private management network:

192.168.50.10/24

The address is illustrative only. Replace it with an unused address, prefix, gateway, and DNS settings that belong to your own VM Network. Do not use 192.168.50.10/24 blindly, and do not choose a range that overlaps the HoloDeck routes or your local Mac network.

Use this side for ordinary SSH management from the Mac. A static address makes the bastion easier to find after a reboot, but it also makes address management your responsibility. Record the assignment in your normal IPAM or lab notes.

The second interface connects to:

Holo-PG-A

Holo-PG-A is the native VLAN for the site-a HoloDeck environment. This is the internal, Holorouter-facing connection. On the bastion, it appears as ens37 and connects to the following transit network:

10.1.10.128/25

The validated example used:

ens37 address: 10.1.10.229/25
Holorouter next hop: 10.1.10.129

The exact address can change. The interface role and next hop are the important parts.

One interface preserves normal management; the other owns the specific routes into HoloDeck.

Create the Ubuntu VM

Create a small Ubuntu VM using the normal VCFA or vSphere process for your environment. The bastion does not need to be large for this job; it needs reliable networking and enough disk for tools, logs, and temporary artifacts.

Attach two virtual network adapters:

  1. The first adapter to VM Network for static management access.
  2. The second adapter to Holo-PG-A for HoloDeck transit access.

Install Ubuntu and enable SSH. During the first login, update the system and install the small set of tools used throughout this article:

sudo apt update
sudo apt upgrade
sudo apt install -y \
curl \
dnsutils \
git \
jq \
netcat-openbsd \
network-manager \
openssh-server
  • curl tests HTTP and HTTPS endpoints.
  • dnsutils provides DNS troubleshooting commands such as dig and nslookup. The article uses getent, which Ubuntu provides separately through the base system.
  • git provides version control for scripts and configuration files.
  • jq filters and formats JSON returned by APIs and command-line tools.
  • netcat-openbsd performs simple TCP connectivity tests when an application-level request is not appropriate.
  • network-manager provides nmcli, which is used to configure the two interfaces and persistent routes.
  • openssh-server provides remote shell access, file transfers, and the SOCKS5 tunnel.

Only curl, network-manager, and openssh-server are essential to the connectivity workflow demonstrated here. Git supports the bastion’s development role, while dnsutils, jq, and netcat-openbsd are useful troubleshooting tools. Tools such as the VCF CLI and kubectl can be installed separately when their workflows are introduced.

I opted to use SSH key authentication primarily so I could give a Codex agent secure access to the environment for testing and developing scripts and solutions. That experience is a blog article in its own right—and one I’m looking forward to writing.

Identify the interfaces before adding routes

Ubuntu interface names depend on the image and installation. Do not assume that the management adapter is always ens34 or that the transit adapter is always ens37.

On the bastion, inspect the addresses and connection names:

ip -br address
nmcli device status
nmcli connection show

For the rest of this article, the example assumes:

  • ens34 is connected to VM Network and uses the static management address.
  • ens37 is connected to Holo-PG-A and uses the static transit address.

If your interface names differ, substitute them consistently.

Configure the management interface

Set the management address according to your local gateway and DNS design. The following is an illustrative NetworkManager example only:

MGMT_CONNECTION="$(
nmcli -g GENERAL.CONNECTION device show ens34
)"
sudo nmcli connection modify \
"${MGMT_CONNECTION}" \
ipv4.method manual \
ipv4.addresses '192.168.50.10/24' \
ipv4.gateway '192.168.50.1' \
ipv4.dns '192.168.50.1'

Replace 192.168.50.1 with the real management gateway and DNS server, or omit the DNS value if your environment supplies name resolution another way. Confirm the management route before applying changes over SSH.

Configure the HoloDeck-facing interface

First identify the NetworkManager connection attached to ens37:

ENS37_CONNECTION="$(
nmcli -g GENERAL.CONNECTION device show ens37
)"
printf 'ens37 connection: %s\n' "${ENS37_CONNECTION}"

Set the transit address and prevent this interface from installing a second default route:

sudo nmcli connection modify \
"${ENS37_CONNECTION}" \
ipv4.method auto \
ipv4.never-default yes

The ipv4.never-default yes setting matters. The bastion should keep its normal management default route on the VM Network interface. The HoloDeck-facing interface needs specific routes, not a competing default route.

Add all of the HoloDeck routes

The route table that matters is the one installed through ens37:

10.1.0.0/16 via 10.1.10.129 dev ens37 proto static metric 101
10.1.10.128/25 dev ens37 proto kernel scope link src 10.1.10.229 metric 101
10.2.0.0/16 via 10.1.10.129 dev ens37 proto static metric 101
10.100.0.0/27 via 10.1.10.129 dev ens37 proto static metric 101

The following output shows the Holorouter interface connected to the same transit network. Its address, 10.1.10.129, becomes the next hop for the bastion’s static routes:

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8000 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:bc:87:9f brd ff:ff:ff:ff:ff:ff
altname eno2
altname enp19s0
altname ens224
inet 10.1.10.129/25 scope global eth1
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

The connected route is created by the interface address. The three static routes reach the wider HoloDeck networks through the Holorouter at 10.1.10.129:

  • 10.1.0.0/16: HoloDeck site-a services and transit networks.
  • 10.2.0.0/16: future or current VCFA workload networks.
  • 10.100.0.0/27: Supervisor Kubernetes management-plane services in the validated lab.

Preserve any routes already present on the connection. The following example expresses the intended static route set:

The route table captures both validated service paths and future workload reachability.
sudo nmcli connection modify \
"${ENS37_CONNECTION}" \
ipv4.routes \
"10.1.0.0/16 10.1.10.129 101,10.2.0.0/16 10.1.10.129 101,10.100.0.0/27 10.1.10.129 101"
sudo nmcli device reapply ens37

If NetworkManager reports that a reapply is not possible, schedule a controlled connection restart or reboot. Do not casually cycle the only interface you are using for SSH access.

Check route ownership

The route table should make the intended ownership visible:

ip route
ip route get 10.100.0.2
ip route get 10.2.0.10

The Supervisor example should report a path similar to:

via 10.1.10.129 dev ens37 src 10.1.10.229

The last address is only an example from the 10.2.0.0/16 workload range. Do not treat it as proof that a workload exists. It proves that the route is selected; a real connection is needed to prove the destination path.

Also confirm that the management default route still belongs to the VM Network side:

ip route show default

Validate DNS and vCenter reachability

Test the path from the bastion before involving the Mac browser:

getent ahostsv4 vc-mgmt-a.site-a.vcf.lab
curl -skI \
--connect-timeout 5 \
--max-time 15 \
https://vc-mgmt-a.site-a.vcf.lab/

In the validated environment, vCenter returned an HTTPS response through the bastion, confirming that DNS resolution and network connectivity were working. For a trusted final configuration, install the appropriate vCenter CA certificate and repeat the request without -k. The insecure option is useful for isolating routing and certificate issues in a lab, but it should not become part of the permanent configuration.

You can also test connectivity to the Supervisor Kubernetes API server endpoint:

curl -sk \
--connect-timeout 5 \
--max-time 15 \
https://10.100.0.2/version

An authentication response can still prove that the network and TLS endpoint were reached. A timeout points to routing, firewall, or service availability rather than a bad Kubernetes command.

Use SSH normally from the Mac

Connect to the bastion through its static address on VM Network:

ssh ubuntu@<bastion-management-ip>

For repeat use, add a host entry to the Mac’s SSH configuration:

Host holodeck-bastion
HostName <bastion-management-ip>
User ubuntu
IdentityFile ~/.ssh/<bastion-key>

Now copying commands, editing files, and using kubectl feel like normal work again. Files can move directly with scp when needed:

scp ./local-file.txt holodeck-bastion:/tmp/local-file.txt
scp holodeck-bastion:/tmp/result.txt ./result.txt

Add a SOCKS5 tunnel when the browser needs the route

SSH can also create a local SOCKS5 listener on the Mac:

ssh -N \
-D 127.0.0.1:1080 \
holodeck-bastion

The listener is bound to localhost, so it is not exposed to the rest of the local network.

Configure Firefox or another browser with:

SOCKS host: 127.0.0.1
Port: 1080
SOCKS v5: enabled
Proxy DNS: enabled

Proxy-side DNS is important for names such as vc-mgmt-a.site-a.vcf.lab. Without it, the Mac may try to resolve the internal name locally before the request ever reaches the bastion.

Validated Firefox SOCKS Proxy Settings.

Test the tunnel from the Mac with socks5h, where the h means hostname resolution happens through the proxy:

curl -sk \
--proxy socks5h://127.0.0.1:1080 \
--connect-timeout 5 \
--max-time 15 \
https://vc-mgmt-a.site-a.vcf.lab/

This is the part Webtop never made especially pleasant. The browser stays on the Mac, copy and paste stays on the Mac, and the bastion supplies the route into the HoloDeck environment.

Here is a screenshot of the Firefox tabs on my Mac. As you can see, the SOCKS5 tunnel provides access to several deployed HoloDeck resources.

Remote Firefox access through a SOCKS proxy into the HoloDeck environment

Why this route is worth keeping

The bastion is deliberately uncomplicated:

  • Management arrives through VM Network using a static example address.
  • HoloDeck traffic leaves through Holo-PG-A and ens37.
  • The normal default route remains on the management side.
  • Specific HoloDeck networks use the Holorouter next hop.
  • SSH provides the working session and file transfer.
  • SOCKS5 provides browser and API access when the Mac cannot route directly.

This avoids changing the Holorouter’s VRF behavior, BGP, reverse-path filtering, proxy configuration, or system-wide socket behavior. Those components can remain responsible for the routing they already own.

What the bastion proves, and what it does not

A successful vCenter connectivity test proves that the bastion can resolve the vCenter hostname, route traffic to the destination, and receive an HTTPS response. It does not prove that every HoloDeck service or workload is reachable. Each destination still requires its own connectivity and application-level validation.

Similarly, the persistent 10.2.0.0/16 route proves that the bastion knows where future VCFA workload traffic should go. It does not prove SSH to a real workload until one exists and a connection has been tested.

Once workload access is configured, the same path can provide HTTP or SSH access to deployed machines. For example, the following screenshot shows access to a load-balanced vSphere Pod deployed through VCF Automation. That workflow will be the subject of a future article.

Keeping those evidence boundaries separate is the difference between “the route is configured” and “the application path works.”

Next steps

With the bastion in place, I now have a practical workstation for managing and developing against the HoloDeck environment. It provides a central location for running the VCF CLI, kubectl, diagnostic utilities, and other development tools without modifying the router OS or relying on Webtop’s cumbersome file-transfer process. It also gives me a clean foundation for testing additional services and workflows in future articles.

Disclaimer: Some of the stuff you see here has been checked out, tweaked, or even created by AI. Welcome to the new age, folks!

Mastering VCF 9 All Apps Organization Virtual Machine IP Discovery

VCF Automation CCI blueprint VM primary IP address

Getting a VM’s IP address sounds like it should be the easy part of an automation project. In my VCF Automation lab, it turned into one of those small details that consumed far more time than expected.

First some background on my lab environment. Using Holodeck I deployed a single Management Domain, Supervisor, Automation. On top of that I added a single All Apps Organization.

My goal was simple: Deploy an All Apps Organization virtual machine through a CCI.Supervisor.Resource, wait for the VM and its guest network to be ready, and return the assigned IPv4 address as a deployment output. I found an example that pointed me toward status.network.primaryIp4. It was close—but close does not count when a property name is case-sensitive.

My ultimate intent here, is to use that IP address or addresses as part of a three tier application blueprint. But that is another blog.

The path that worked in my environment was:

status.network.primaryIP4

The IP is uppercase. That capitalization was the difference between an empty or unusable value and the address I needed.

Following the VM’s actual status

The breakthrough came from looking at the status returned for the deployed VM instead of continuing to guess at the schema. The VM’s status.conditions array showed the conditions it passed through as provisioning progressed. The same status object exposed the network data under:

status:
network:
primaryIP4: 192.0.2.25

The lesson here is simple: Use the object returned by your own VCF Automation and VM Operator version as the source of truth. Blog posts and examples are useful starting points, but a small schema or capitalization difference can break a binding expression.

Steps to finding that information.

  1. Deploy the machine
  2. View YAML on the machine page
Steps to finding the deployed machine YAML.
  1. Walk the properties (I collapsed several sections to enhance readability). Here the path to primaryIP4 is status.network.primaryIP4.
All Apps status.network.primaryIP4 path.

How to find the conditions

Follow the same logic as finding primaryIP4. You will find the conditions or stages the machine went through along with the ‘status‘ and 'reason‘.

All Apps Org machine condition or state transition's properties.

Why the wait block matters

Reading the right path is only half of the solution. VCF Automation also needs to wait long enough for the Supervisor resource status to contain the network information.

Broadcom documents a status-collection race in VCF Automation 9.0.x. Without an explicit wait, the resource can be considered created before its reconciled status has been synchronized back to Automation. Broadcom’s minimum recommendation is to wait for VirtualMachineCreated=True.

For my blueprint, I waited for VM creation, VMware Tools, guest network synchronization, and a value matching an IPv4 pattern:

resources:
Supervisor_VM:
type: CCI.Supervisor.Resource
properties:
context: ${resource.Supervisor_Namespace.id}
manifest:
apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachine
metadata:
name: ${input.vmName}
spec:
# The rest of the sanitized VM manifest goes here.
wait:
timeoutSeconds: 1800
conditions:
- type: VirtualMachineGuestNetworkConfigSynced
status: 'True'
reason: Synced
- type: VirtualMachineCreated
status: 'True'
- type: VirtualMachineTools
status: 'True'
jsonPath:
- path: '{.status.network.primaryIP4}'
regex: \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

Return the address as a blueprint output

Once the wait completes, the address can be exposed as a deployment output. With the resource named Supervisor_VM, the binding is:

outputs:
vmIp:
type: string
title: VM primary IPv4 address
value: ${resource.Supervisor_VM.object.status.network.primaryIP4}

Again, the working property is primaryIP4, not primaryIp4.

This output can also be consumed by another resource binding. For example, an application VM could receive a database VM’s address during cloud-init. Make sure the upstream VM’s wait block completes before relying on that value.

Validation

After deploying a new version of the blueprint:

  1. Confirm the deployment completes without a wait timeout.
  2. Inspect the CCI.Supervisor.Resource object and verify that status.conditions includes the expected successful conditions.
  3. Verify that status.network.primaryIP4 contains the VM’s address.
  4. Confirm the deployment output shows the same address.
  5. From an appropriate network location, test the service that should be listening on that address.

The last step matters. VirtualMachineCreated=True, synchronized guest networking, and a populated primaryIP4 prove that the platform has an address for the VM. This does not prove that cloud-init finished successfully or that the application inside the guest is healthy.

Here is a screenshot of a deployed machine with the output vmIp set to 10.2.0.2.

 Displaying primaryIP4 as an vmIp output

The finished blueprint

The blueprint used in this blog is available here, VCF Automation Supervisor VM IP Discovery on GitHub

Take Aways

  1. Leverage AI to do the heavy research. One of the responses gave me the thought to check that case sensitivity.
  2. AI, also mean’s ‘Ain’t Intelligent’. It WILL send you down the wrong path eventually. As the developer you still need to know what it SHOULD look like in the end. In other words, trust but verify.
  3. Don’t get in a hurry. I worked on this on and off in my ‘spare time’ for a few days. Then finally it clicked at the end.
  4. All Apps Organizations are a new construct for most of us. The available public content and samples are few and far between. Please, Please, Please take some time to share your lessons learned with the community.

References

Nuff for now. Happy automating.

Managing Azure AKS clusters with VMWare Aria Automation

The use case presented to me for POC was to deploy a new Azure AKS cluster then install a basic application. Simple use case, but for those using VRA you know the kubernetes capabilities are all but non existent.

But after digging around and tinkering I figured a CodeStream (now called Pipelines) would probably fit the bill. The pipeline would run a terraform plan to build, and then destroy the deployment later on.

Keeping track of the state file between runs also presented a ‘problem’. After lots of kicking the tires I came up with a way to store the state file securing in an Azure Storage account. The state file in the container is simply the deployment name plus .tfstate. This allowed me to refer to it using day two actions and Event Broker Subscriptions (EBS).

Another issue that came up was deleting ‘codestream.execution’ resources when the deployment is deleted. Since these deployments are handled by terraform I needed another WF which called a pipeline to destroy the deployment when the eventType was DESTROY_DEPLOYMENT.

The files for this article can be found at azure-terraform-blog

Terraform is used to do the heavy lifting. The backend values get replaced with some pipeline inputs in the first pipeline task. The most important one is the deployment name. When destroying the deployment, terraform will pull the current state for that deployment and do its thing.

The CodeStream pipeline (Now Pipelines) uses a custom docker image. It includes the latest version of Terraform (Currently at 1.5.4), AZ CLI, Kubectl, and Helm (for another use case). It is stored on DockerHub as americanbwana/cas-terraform-154:latest.

I didn’t come up with the basic Template. I found this article on vEducate.co.uk. A very good starting point. ‘pipelineTask’ is used by the pipeline to either create (apply) or destroy the deployment. More on that later.

formatVersion: 1
inputs:
  pipelineTask:
    type: string
    title: Pipeline Task
    description: 'Create '
    readOnly: true
    default: create
resources:
  cs.pipeline:
    type: codestream.execution
    properties:
      pipelineId: 2b80427c...
      outputs:
        computed: true
      inputs:
        deploymentName: ${env.deploymentName}
        pipelineTask: ${input.pipelineTask}

vRA doesn’t delete the actual codestream.execution items when you destroy the deployment. A workflow called ‘Terraform delete AKS and Helm deployment’ is called by an Event Broker Subscription (EBS). Make sure to update the ‘codestreamPipelineId’ in the WF variables.

Event Broker Subscription

And finally on to the pipeline. The initialize task copies several variables into a file, which is then sourced by most stages. Terraform apply is only fired if the pipelineTask = ‘create’. And Terraform destroy is only fired when pipelineTask = ‘destroy’.

Pipeline

‘Get Service IP’ is also only fired if the pipelineTask = ‘create’. This task will get the IP address of WordPress and export it back to vRA.

Terraform Output
Assembler Output

Nuff for now. Happy coding.

CyberArk Ansible Integration

As an alternative to vRA Cloud Secrets

Well its been a while since I posted anything. To be honest, this site and posts were used to support my vExpert applications, but apparently blog content doesn’t count anymore. So…. now that I’m free from that obligation, I can just post because I want to.

This article details my efforts to understand how CyberArk and Ansible work together. My particular use case is to replace vRA Cloud secrets with variables stored in CyberArk. More specifically the issue with vRA secrets is they are limited to a single Project. This doesn’t work to well for a company with more than one project. Basically have one secret (mysecret) per project. Or if you have 10 projects, 10 secrets named mysecret (one for each project).

Now down to business. The first thing is to setup CyberArk following the instructions from their Quick Start tutorial. The basic setup is done by step 6, no real need to go past that unless you want to. A couple of notes here. First the Master Key (Step 2) and Admin api_key (Step 5) are saved to a text file on your docker host. And secondly, by default the SSL generated by the installer uses localhost, proxy, and 127.0.0.1 as the SAN. You can change this in conjur-quickstart/conf/tls/tls.conf. I’ll be using the default proxy as the hostname, along with some entries in /etc/hosts on my Mac and Ansible host.

Next I installed Cyberark CLI on my Mac. The instructions are available here. Note is is only supported on Windows, RHEL and Mac.

The setup file on my Mac for ~.conjurcli looks like this.

cert_file: /Users/me/conjur-server.pem
conjur_account: myConjurAccount
conjur_url: https://proxy:8443

Now to define some CyberArk Conjur (conjur) policy files. The first was to define a new clean branch for my ansible policies. I called it mybranch (Hey it was Friday and I already used my weekly good braincell quota). I even used a creative name, ‘create-ansible-branch.yaml’.

- !policy
  id: mybranch

And to apply it (assuming you’ve already logged in as Admin).

mymac>conjur policy replace -b root -f create-ansible-branch.yaml
mymac>conjur list
[
    "myConjurAccount:policy:mybranch",
    "myConjurAccount:policy:root"
]

Now on to defining the ansible host (ansible2)

- !layer

- !host ansible2

- !grant
  role: !layer
  member: !host ansible2

mymac>conjur policy load -b mybranch -f ansible2-host-policy.yaml

The result will contain an api_key for the new host. You’ll probably want to copy this into your scratch pad.

  {
      "created_roles": {
          "myConjurAccount:host:mybranch/ansiblehost": {
              "id": "myConjurAccount:host:mybranch/ansiblehost",
              "api_key": "1xgpkp02d8etyz2zb........" # <--- api_key
          }
      },
      "version": 2
  }

Now to create a new group, variable, and grant ansible2 permissions.

# Declare the secrets which are used to access the database
- &variables
  - !variable password2

# Define a group which will be able to fetch the secrets
- !group secrets-users

- !permit
  resource: *variables
  # "read" privilege allows the client to read metadata.
  # "execute" privilege allows the client to read the secret data.
  # These are normally granted together, but they are distinct
  #   just like read and execute bits on a filesystem.
  privileges: [ read, execute ]
  roles: !group secrets-users
# Entitlements

- !grant
  role: !group secrets-users
  member: !layer /mybranch

mymac>conjur policy load -b mybranch -f ansible2-access-policy.yaml
### Set the password variable value
mymac>conjur variable set -i mybranch/password2 -v "HelloWorld"

Our work with CyberArk is done for the time being. Now on to your ansible host. Here the assumption is our ansible host is setup properly. First install the Cyberark.conjur collection.

ubunutu@ansible2$ansible-galaxy collection install cyberark.conjur

Now to define some files on your ansible host. The file names and content are shown below. You can figure out how to get the contents of conjur.pem.

/etc/conjur.conf

account: myConjurAccount
appliance_url: https://proxy:8443
cert_file: /etc/conjur.pem
netrc_path: /etc/conjur.identity
plugins: []

/etc/conjur.identity

machine https://proxy:8443/authn
    login host/mybranch/ansible2
    password gybp2n1wssmh1fr8n5k27.........


/etc/conjur.pem

-----BEGIN CERTIFICATE-----
.......
-----END CERTIFICATE-----

Almost there, now to define and run a basic ansible playbook. And by basic, I mean basic.

# get_conjur_var.yaml

---
- hosts: localhost
  tasks:
  - name: Lookup variable in Conjur
    debug:
      msg: "{{ lookup('cyberark.conjur.conjur_variable', 'mybranch/password2') }}"

ubunutu@ansible2$ansible-playbook get_conjur_var.yaml

.... 
ok: [localhost] => {
    "msg": "HelloWorld"
}
....

The next article will demonstrate how to use this with vRA cloud to replace all those repetitive secrets (Per project, Yuk!)

vRO Action PowerShell Zip importing and use

One of my current tasks is to leverage vRealize Automation Orchestrator to meet the following use case.

  • Get the next available subnet from InfoBlox
  • Reserve the gateway and other IP’s in the new subnet
  • Create a new NSX-T segment
  • Create new NSX-T security groups
  • Discover the new segment in vRealize Automation Cloud
  • Assign the new InfoBlox to the discovered Fabric Network
  • Create a new Network Profile in vRA Cloud

This weeks goal was to get the InfoBlox part working. Well I had it working two years ago, but couldn’t remember how I did it (CRS).

Today I’ll discuss how to use vRO to get the next available subnet from InfoBlox. The solution uses a PSM I build, along with the PowerShell script which actually does the heavy lifting. One key difference between my solution and the one from VMware’s documentation is the naming of the zip file. This affects how to import and use it in vRO.

The code used in this example is available in this GitHub repo. Clone the repo, then run the following command to zip up the files.

zip -r -x ".git/*" -x "README.md" -X nextibsubnet.zip .

Next import the zip file into vRO, add some inputs, modify the output, then finally run it.

Within vRO, add a new Action. Then change the script type to “PowerCli 12 (PowerShell 7.1).

PowerShell Script Type

Change the Type to ‘Zip’ by clicking on the dropdown under ‘Type’ and selecting ‘Zip’

Click ‘Import’, then browse to the folder containing the zip file from earlier in the article.

You will notice the name is not ‘nextibsubnet.zip’ but InfobloxGetNextAvailableSubnet.zip. The imported zip assumes the name of the vRO Action.

Now the biggest difference between my approach and the VMware way. If you look at the cloned folder you will see a file named ‘getNextAvailableIbSubnet.ps1’. The VMware document called this file ‘handler.ps1’. Instead of putting in ‘handler.handler’ in ‘Entry Handler’, I’ll use ‘getNextAvailableIbSubnet.handler. This tells vRO to look inside ‘getNextAvailableIbSubnet.ps1’ for a function called ‘handler’.

Next we need to change the return type to Properties, and add a few inputs.

Save and run. And if everything is in order, you should get the next InfoBlox subnet from 10.10.0.0/24. The results from the action run.

So there you go. Now on to the next adventure.

Custom vSphere Template import into AWS as AMI

My current customer asked if they could use the same vSphere template as an AWS AMI. The current vSphere template has a custom disk layout to help them troubleshoot issues. The default single disk layout for AMI’s actually hinders their troubleshooting methodology.

Aside from the custom disk layout, I know VMtools would have to be replaced with cloud-init. Sure no problem. RIGHT! Well actually it wasn’t that hard.

Well I was finally able get it to work, and learned a bunch along the way. Those lessons include,

  • The RHEL default DHCP client is incompatible with AWS.
  • EFI bios is only supported in larger, more expensive instances.
  • AWS VM image import.
  • Make sure to enable ‘disable_vmware_customization’, if that made sense.

Requirements

  • AWS roles, policies and permissions per this document.
  • S3 bucket (packer-import-example) to store the VMDK until it is imported.
  • Basic IAM user (packer) with the correct permissions assigned (see above).
  • vSphere environment to build the image.
  • A RHEL 8.x DVD ISO for installation.
  • HTTP repo to store the kickstart file.

Now down to brass tacks. To be honest it took lots of trial and error (mostly error) to get this working right. For example, on one pass Cloud-Init wouldn’t run on the imported AMI. After looking at cloud.cfg I noticed ‘disable_vmware_customization’ was set to false instead of ‘true’. Another error occurred when my first import attempt failed as the machine did not have a ‘DHCP client’. That was odd as it booted up fine in vSphere and got an IP Address. Apparently AWS only supports certain DHCP clients. Go figure.

Eventually the machine booted properly in AWS, with the user-data applied correctly. The working user-data is in the repo’s cloud-init directory.

And my super simple vRAC blue print even worked. This simple BP adds a new user, assigns a password, and grants it SUDO permissions.

Successful vRA Cloud Deployment

A couple of notes on the packer amazon-import post processor. Those include,

  • The images are encrypted by default, even tho the default for ‘ami_encrypt’ is false by default.
  • ‘ami_name’ requires the AWS permission of ec2:CopyImage on the policy for the import role.
  • Don’t use the default encryption key if you wish to share this. You’ll need a Customer Managed Key (CMK). The import role (vmimport) will need to be a key user. You can set this with ‘ami_kms_key’ set to the Id of the CMK (i.e., ebea!!!!!!!!-aaaa-zzzz-xxxxxxxxxxxxxx)
  • The CMK needs to be shared with the target customer before sharing the AMI. ‘ami_org_arns’ allows you to set the organizations you’d like to share the AMI with.
  • There are lots of import options, you can check them out here.

This working example, plus others I’ve been working are available in this github repo.

Now onto another vRAC adventure.

Packer HCL and PVSCSI drivers

Just this last week I was updating an old Packer build configuration from JSON to HCL. But for the life of me could not get a new vSphere Windows 2019 machine to find a disk attached to Para Virtualized disk controller.

I repeatedly received this error after the machine new machine booted.

Error

In researching error 0x80042405 in C:\Windows\pather\setuperr.log, I found it simply could not find the attached disk.

setuperr.log

After some research I determined the PVSCSI drivers added to the floppy disk where not being discovered. Or more specifically the new machine didn’t know to search the floppy for additional drivers.

I finally found a configuration section for my autounattend.xml file which would fix it after an almost exhaustive online search.

The magic section reads as follows.

<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="windowsPE">        
       <component name="Microsoft-Windows-PnpCustomizationsWinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DriverPaths>
                <PathAndCredentials wcm:action="add" wcm:keyValue="A">
                    <!-- pvscsi-Windows8.flp -->
                    <Path>A:\</Path>
                </PathAndCredentials>
            </DriverPaths>
        </component>
...
    </settings>

After adding this section, the new vSphere Windows machine easily found the additional drivers.

This was tested against Windows 2019 in both AWS and vSphere deployments.

The vSphere deployment took an hour, mostly waiting for the updates to be applied. AWS takes significantly less time as I’m using the most recently updated image they provide.

The working files are located in the packer-hcl-vsphere-aws github repo.

Code Stream Nested Esxi pipeline Part 2

In this second part, I’ll discuss the actual Code Stream pipeline.

As stated before, the inspiration was William Lams wonderful Power Shell scripts to deploy a nested environment from a CLI. His original logic was retained as much as possible, however due to the nature of K8S a few things had to be changed. I’ll try to address those as they come up.

After some thought I decided to NOT allow the requester to select the amount of Memory, vCPU, or VSAN size. Each Esxi host has 24G of Ram, 4 vCPU, and contributes a touch over 100G to the VSAN. The resulting cluster has 72G of RAM, 12 vCPUs and a roughly 300G VSAN. Only Standard vSwitches are configured in each host.

The code, pipeline and other information is available on this github repo.

Deployment of the Esxi hosts is initiated by ‘deployNestedEsxi.ps1’. There are few changes from the original script.

  1. The OVA configuration is only grabbed once. Then only the specific host settings (IP Address and Name are changed.
  2. The hosts are moved into a vApp once built.
  3. The NetworkAdapter settings are performed after deployment.
  4. Persisted the log to /var/workspace_cache/logs/vsphere-deployment-$BUILDTIME.log.

Deployment of the vCSA is handled by ‘deployVcsa.ps1’ Some notable changes from the original code include.

  1. Hardcoded the SSO username to administrator@vsphere.local.
  2. Hardcoded the size to ‘tiny’.
  3. Save the log file to /var/workspace_cache/logs/NestedVcsa-$BUILDTIME.log.
  4. Save the configuration template to /var/workspace_cache/vcsajson/NestedVcsa-$BUILDTIME.json.
  5. Move the VCSA into the vApp after deployment is complete.

And finally ‘configureVc.ps1’ sets up the Cluster and VSAN. Some changed include.

  1. Hardcoded the Datacenter name (DC), and Cluster (CL1).
  2. Import the Esxi hosts by IP (No DNS records setup for the hosts or vCenter).
  3. Append the configuration results to /var/workspace_cache/logs/vsphere-deployment-$BUILDTIME.log.

So there you go, down and simple Code Stream pipeline to deploy a nested vSphere environment in about an hour.

Stay tuned. The next article will include an NSX-T deployment.

Code Stream Nested Esxi pipeline Part 1

Been a while since my last post. Over the last couple of months I’ve been tinkering with using Code Stream to deploy a Nested Esxi / vCenter environment.

My starting point is William Lams excellent PowerShell script (vsphere-with-tanzu-nsxt-automated-lab-deployment). I also wanted to use the official vmware/poweclicore docker image.

Well let’s just say it’s been an adventure. Much has been learned through trial and (mostly) error.

For example in Williams script, all of the files are located on the workstation where the script runs. Creating a custom docker image with those files would have resulted in a HUGE file, almost 16GB (Nested ESXi appliance, vCSA appliance and supporting files, and NSX-T OVA files). As one of my co-worker says, “Don’t be that guy”.

At first I tried cloning the files into the container as part of the CI setup. Downloading the ESXi OVA worked fine, but failed when I tried copying over the vCSA files. I think it’s just too much.

I finally opted to use a Kubernetes Code Stream instead of a Docker pipeline. This allowed me to use a Persistent Volume Claim.

Kubernetes setup

Some of the steps may lack details, as this has been an ongoing effort and just can’t remember everything. Sorry peeps!

Create two Name Spaces, codestream-proxy and codestream-workspace. Codestream-proxy is used by Code Stream to host a Proxy pod.

Codestream-workspace will host the containers running the pipeline code.

Next came the service account for Code Stream. The path of least resistance was to simply assign ‘cluster-admin’ to the new service account. NOTE: Don’t do this in a production environment.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cs-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: codestream
  apiGroup: ""
  namespace: default

Next came the Persistent Volume (pv) and Persistent Volume Claim (pvc). My original pv was set to 20GI, which after some testing was determined too small. It was subsequently increased it to 30GI. The larger pv allowed me to retain logs and configurations between runs (for troubleshooting).

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
  name: cs-persistent-volume-cw
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 30Gi
  hostPath:
    path: /mnt/nested
  persistentVolumeReclaimPolicy: Retain
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cs-pvc-cw
  namespace: codestream-workspace
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 30Gi
  volumeName: cs-persistent-volume-cw

The final step in k8s is to get the Service Account token. In this example the SA is called ‘codestream’ (So creative).

k get secret codestream-token-blah!!! -o jsonpath={.data.token} | base64 -d | tr -d "\n"

eyJhbGciOiJSUzI1NiIsImtpZCI6IncxM0hIYTZndS1xcEdFVWR2X1Z4UFNLREdQcGdUWDJOWUF1NDE5YkZzb.........

Copy the token, then head off to Code Stream.

Codestream setup

There I added a Variable to hold the token, called DAG-K8S-Secret.

Then went over to Endpoints, where I added a new Kubernetes endpoint.

Repo setup

The original plan was to download the OVA/OVF files from a repo every time the pipeline ran. However an error would occur on every VCSA file set download. Adding more memory to the container didn’t fix the problem, so I had to go in another direction.

The repo is well connected to the k8s cluster, so the transfer is pretty quick. Here is the directory structure for the repo (http://repo.corp.local/repo/).

NOTE: You will need a valid account to download VCSA and NSX-T.

NOTE: NSX-T will be added to the pipeline later.

Simply copying the files interactively on the k8s node seemed like the next logical step. Yes the files copied over nicely, but any attempt to deploy the VCSA appliance would throw a python error complaining about a missing ‘vmware’ module.

However I was able to run the container manually, copy the files over and run the scripts successfully. Maybe a file permissions issue?

Finally I ran the pipeline with a long sleep at the beginning. Using an interactive session, and copied the files over. This fixed the problem.

Here are the commands I used to copy the files over interactively.

k -n codestream-workspace exec -it po/running-cs-pod-id bash
wget -mxnp -q -nH http://repo.corp.local/repo/ -P /var/workspace_cache/ -R "index.html*"
# /var/workspace_cache is the mount point for the persistent volume
# need to chmod +x a few files to get the vCSA to deploy
chmod +x /var/workspace_cache/repo/vcsa/VMware-VCSA-all-7.0.3/vcsa/ovftool/lin64/ovftool*
chmod +x /var/workspace_cache/repo/vcsa/VMware-VCSA-all-7.0.3/vcsa/vcsa-cli-installer/lin64/vcsa-deploy*

This should do it for now. The next article will cover some of the pipeline details, and some of the changes I had to make to William Lams Powershell code.

Happy holidays.

vRA Cloud Day 2 Resource Action using a Polyglot workflow

One of my peers came up with an interesting use case today. His customer wanted to mount an existing disk on a virtual machine using a vRA Cloud day 2 action.

I couldn’t find an out of the box workflow or action on my vRO, which meant I had to do this thing from scratch.

After a quick look around I found a PowerCli cmdlet (New-Hardisk) which allowed me to mount an existing disk.

My initial attempts to just run it as a scriptable task resulted in the following error.

Hmm, so how do you increase the memory in a scriptable task? Simple, you can’t. Thus I had to move the script into an action, which does allow me to increase the memory. After some tinkering I found that 256M was sufficient to run the code.

function Handler($context, $inputs) {
    # $inputs:
    ## vmName: string
    ## vcName: string (in configuration element)
    ## vcUsername: string (in configuration element)
    ## vcPassword: secureString (in configuration element)
    ## diskPath: string. Example in code. 
    # output:
    ## actionResult: Not used
    $inputsString = $inputs | ConvertTo-Json -Compress

    Write-Host "Inputs were $inputsString"

    $output=@{status = 'done'}

    # connect to viserver
    Set-PowerCLIConfiguration -InvalidCertificateAction:Ignore -Confirm:$false
    Connect-VIServer -Server $inputs.vcName -Protocol https -User $inputs.vcUsername -Password $inputs.vcPassword

    # Get vm by name
    Write-Host "vmName is $inputs.vmName"
    $vm = Get-VM -Name $inputs.vmName

    # New-HardDisk -VM $vm -DiskPath "[storage1] OtherVM/OtherVM.vmdk"
    $result = New-HardDisk -VM $vm -DiskPath $inputs.diskPath 
    Write-Host "Result is $result"

    return "It worked!"
}

Looking at the code, you will notice an input of vmName (used by PS to find the VM). Getting the vmName is actually pretty stupid simple using JavaScript. My first task in the WF takes care of this.

// get the vmName
// $inputs.vm
// output: vmName
vmName = vm.name

The next step was to setup a resource action. The settings are shown in the following snapshot. Please note the setting within the green box. ‘vm’ is set with a binding action.

Changing the binding is fairly simple. Just click the binding link, then change the value to ‘with binding action’. The default values work just fine.

The disk I used in the test was actually a copy of another VM boot disk. It was copied over to another datastore, then renamed to ‘ExistingDisk2.vmdk’. The full diskPath was [dag-nfs] ExistingDisk/ExistingDisk2.vmdk.

Running the day 2 action on deployed machine seemed to work, as the WF logs show.

So there you have a basic PolyGlot vRO workflow using PowerCli and JavaScript.

I trust this quick blog was helpful in some small way.