One of my current tasks is to leverage vRealize Automation Orchestrator to meet the following use case.
Get the next available subnet from InfoBlox
Reserve the gateway and other IP’s in the new subnet
Create a new NSX-T segment
Create new NSX-T security groups
Discover the new segment in vRealize Automation Cloud
Assign the new InfoBlox to the discovered Fabric Network
Create a new Network Profile in vRA Cloud
This weeks goal was to get the InfoBlox part working. Well I had it working two years ago, but couldn’t remember how I did it (CRS).
Today I’ll discuss how to use vRO to get the next available subnet from InfoBlox. The solution uses a PSM I build, along with the PowerShell script which actually does the heavy lifting. One key difference between my solution and the one from VMware’s documentation is the naming of the zip file. This affects how to import and use it in vRO.
The code used in this example is available in this GitHub repo. Clone the repo, then run the following command to zip up the files.
zip -r -x ".git/*" -x "README.md" -X nextibsubnet.zip .
Next import the zip file into vRO, add some inputs, modify the output, then finally run it.
Within vRO, add a new Action. Then change the script type to “PowerCli 12 (PowerShell 7.1).
PowerShell Script Type
Change the Type to ‘Zip’ by clicking on the dropdown under ‘Type’ and selecting ‘Zip’
Click ‘Import’, then browse to the folder containing the zip file from earlier in the article.
You will notice the name is not ‘nextibsubnet.zip’ but InfobloxGetNextAvailableSubnet.zip. The imported zip assumes the name of the vRO Action.
Now the biggest difference between my approach and the VMware way. If you look at the cloned folder you will see a file named ‘getNextAvailableIbSubnet.ps1’. The VMware document called this file ‘handler.ps1’. Instead of putting in ‘handler.handler’ in ‘Entry Handler’, I’ll use ‘getNextAvailableIbSubnet.handler. This tells vRO to look inside ‘getNextAvailableIbSubnet.ps1’ for a function called ‘handler’.
Next we need to change the return type to Properties, and add a few inputs.
Save and run. And if everything is in order, you should get the next InfoBlox subnet from 10.10.0.0/24. The results from the action run.
My current customer asked if they could use the same vSphere template as an AWS AMI. The current vSphere template has a custom disk layout to help them troubleshoot issues. The default single disk layout for AMI’s actually hinders their troubleshooting methodology.
Aside from the custom disk layout, I know VMtools would have to be replaced with cloud-init. Sure no problem. RIGHT! Well actually it wasn’t that hard.
Well I was finally able get it to work, and learned a bunch along the way. Those lessons include,
The RHEL default DHCP client is incompatible with AWS.
EFI bios is only supported in larger, more expensive instances.
AWS VM image import.
Make sure to enable ‘disable_vmware_customization’, if that made sense.
Requirements
AWS roles, policies and permissions per this document.
S3 bucket (packer-import-example) to store the VMDK until it is imported.
Basic IAM user (packer) with the correct permissions assigned (see above).
vSphere environment to build the image.
A RHEL 8.x DVD ISO for installation.
HTTP repo to store the kickstart file.
Now down to brass tacks. To be honest it took lots of trial and error (mostly error) to get this working right. For example, on one pass Cloud-Init wouldn’t run on the imported AMI. After looking at cloud.cfg I noticed ‘disable_vmware_customization’ was set to false instead of ‘true’. Another error occurred when my first import attempt failed as the machine did not have a ‘DHCP client’. That was odd as it booted up fine in vSphere and got an IP Address. Apparently AWS only supports certain DHCP clients. Go figure.
Eventually the machine booted properly in AWS, with the user-data applied correctly. The working user-data is in the repo’s cloud-init directory.
And my super simple vRAC blue print even worked. This simple BP adds a new user, assigns a password, and grants it SUDO permissions.
Successful vRA Cloud Deployment
A couple of notes on the packer amazon-import post processor. Those include,
The images are encrypted by default, even tho the default for ‘ami_encrypt’ is false by default.
‘ami_name’ requires the AWS permission of ec2:CopyImage on the policy for the import role.
Don’t use the default encryption key if you wish to share this. You’ll need a Customer Managed Key (CMK). The import role (vmimport) will need to be a key user. You can set this with ‘ami_kms_key’ set to the Id of the CMK (i.e., ebea!!!!!!!!-aaaa-zzzz-xxxxxxxxxxxxxx)
The CMK needs to be shared with the target customer before sharing the AMI. ‘ami_org_arns’ allows you to set the organizations you’d like to share the AMI with.
Just this last week I was updating an old Packer build configuration from JSON to HCL. But for the life of me could not get a new vSphere Windows 2019 machine to find a disk attached to Para Virtualized disk controller.
I repeatedly received this error after the machine new machine booted.
Error
In researching error 0x80042405 in C:\Windows\pather\setuperr.log, I found it simply could not find the attached disk.
setuperr.log
After some research I determined the PVSCSI drivers added to the floppy disk where not being discovered. Or more specifically the new machine didn’t know to search the floppy for additional drivers.
I finally found a configuration section for my autounattend.xml file which would fix it after an almost exhaustive online search.
After adding this section, the new vSphere Windows machine easily found the additional drivers.
This was tested against Windows 2019 in both AWS and vSphere deployments.
The vSphere deployment took an hour, mostly waiting for the updates to be applied. AWS takes significantly less time as I’m using the most recently updated image they provide.
In this second part, I’ll discuss the actual Code Stream pipeline.
As stated before, the inspiration was William Lams wonderful Power Shell scripts to deploy a nested environment from a CLI. His original logic was retained as much as possible, however due to the nature of K8S a few things had to be changed. I’ll try to address those as they come up.
After some thought I decided to NOT allow the requester to select the amount of Memory, vCPU, or VSAN size. Each Esxi host has 24G of Ram, 4 vCPU, and contributes a touch over 100G to the VSAN. The resulting cluster has 72G of RAM, 12 vCPUs and a roughly 300G VSAN. Only Standard vSwitches are configured in each host.
The code, pipeline and other information is available on this github repo.
Deployment of the Esxi hosts is initiated by ‘deployNestedEsxi.ps1’. There are few changes from the original script.
The OVA configuration is only grabbed once. Then only the specific host settings (IP Address and Name are changed.
The hosts are moved into a vApp once built.
The NetworkAdapter settings are performed after deployment.
Persisted the log to /var/workspace_cache/logs/vsphere-deployment-$BUILDTIME.log.
Deployment of the vCSA is handled by ‘deployVcsa.ps1’ Some notable changes from the original code include.
Hardcoded the SSO username to administrator@vsphere.local.
Hardcoded the size to ‘tiny’.
Save the log file to /var/workspace_cache/logs/NestedVcsa-$BUILDTIME.log.
Save the configuration template to /var/workspace_cache/vcsajson/NestedVcsa-$BUILDTIME.json.
Move the VCSA into the vApp after deployment is complete.
And finally ‘configureVc.ps1’ sets up the Cluster and VSAN. Some changed include.
Hardcoded the Datacenter name (DC), and Cluster (CL1).
Import the Esxi hosts by IP (No DNS records setup for the hosts or vCenter).
Append the configuration results to /var/workspace_cache/logs/vsphere-deployment-$BUILDTIME.log.
So there you go, down and simple Code Stream pipeline to deploy a nested vSphere environment in about an hour.
Stay tuned. The next article will include an NSX-T deployment.
Been a while since my last post. Over the last couple of months I’ve been tinkering with using Code Stream to deploy a Nested Esxi / vCenter environment.
Well let’s just say it’s been an adventure. Much has been learned through trial and (mostly) error.
For example in Williams script, all of the files are located on the workstation where the script runs. Creating a custom docker image with those files would have resulted in a HUGE file, almost 16GB (Nested ESXi appliance, vCSA appliance and supporting files, and NSX-T OVA files). As one of my co-worker says, “Don’t be that guy”.
At first I tried cloning the files into the container as part of the CI setup. Downloading the ESXi OVA worked fine, but failed when I tried copying over the vCSA files. I think it’s just too much.
I finally opted to use a Kubernetes Code Stream instead of a Docker pipeline. This allowed me to use a Persistent Volume Claim.
Kubernetes setup
Some of the steps may lack details, as this has been an ongoing effort and just can’t remember everything. Sorry peeps!
Create two Name Spaces, codestream-proxy and codestream-workspace. Codestream-proxy is used by Code Stream to host a Proxy pod.
Codestream-workspace will host the containers running the pipeline code.
Next came the service account for Code Stream. The path of least resistance was to simply assign ‘cluster-admin’ to the new service account. NOTE: Don’t do this in a production environment.
Next came the Persistent Volume (pv) and Persistent Volume Claim (pvc). My original pv was set to 20GI, which after some testing was determined too small. It was subsequently increased it to 30GI. The larger pv allowed me to retain logs and configurations between runs (for troubleshooting).
The final step in k8s is to get the Service Account token. In this example the SA is called ‘codestream’ (So creative).
k get secret codestream-token-blah!!! -o jsonpath={.data.token} | base64 -d | tr -d "\n"
eyJhbGciOiJSUzI1NiIsImtpZCI6IncxM0hIYTZndS1xcEdFVWR2X1Z4UFNLREdQcGdUWDJOWUF1NDE5YkZzb.........
Copy the token, then head off to Code Stream.
Codestream setup
There I added a Variable to hold the token, called DAG-K8S-Secret.
Then went over to Endpoints, where I added a new Kubernetes endpoint.
Repo setup
The original plan was to download the OVA/OVF files from a repo every time the pipeline ran. However an error would occur on every VCSA file set download. Adding more memory to the container didn’t fix the problem, so I had to go in another direction.
The repo is well connected to the k8s cluster, so the transfer is pretty quick. Here is the directory structure for the repo (http://repo.corp.local/repo/).
NOTE: You will need a valid account to download VCSA and NSX-T.
NOTE: NSX-T will be added to the pipeline later.
Simply copying the files interactively on the k8s node seemed like the next logical step. Yes the files copied over nicely, but any attempt to deploy the VCSA appliance would throw a python error complaining about a missing ‘vmware’ module.
However I was able to run the container manually, copy the files over and run the scripts successfully. Maybe a file permissions issue?
Finally I ran the pipeline with a long sleep at the beginning. Using an interactive session, and copied the files over. This fixed the problem.
Here are the commands I used to copy the files over interactively.
k -n codestream-workspace exec -it po/running-cs-pod-id bash
wget -mxnp -q -nH http://repo.corp.local/repo/ -P /var/workspace_cache/ -R "index.html*"
# /var/workspace_cache is the mount point for the persistent volume
# need to chmod +x a few files to get the vCSA to deploy
chmod +x /var/workspace_cache/repo/vcsa/VMware-VCSA-all-7.0.3/vcsa/ovftool/lin64/ovftool*
chmod +x /var/workspace_cache/repo/vcsa/VMware-VCSA-all-7.0.3/vcsa/vcsa-cli-installer/lin64/vcsa-deploy*
This should do it for now. The next article will cover some of the pipeline details, and some of the changes I had to make to William Lams Powershell code.
One of my peers came up with an interesting use case today. His customer wanted to mount an existing disk on a virtual machine using a vRA Cloud day 2 action.
I couldn’t find an out of the box workflow or action on my vRO, which meant I had to do this thing from scratch.
After a quick look around I found a PowerCli cmdlet (New-Hardisk) which allowed me to mount an existing disk.
My initial attempts to just run it as a scriptable task resulted in the following error.
Hmm, so how do you increase the memory in a scriptable task? Simple, you can’t. Thus I had to move the script into an action, which does allow me to increase the memory. After some tinkering I found that 256M was sufficient to run the code.
function Handler($context, $inputs) {
# $inputs:
## vmName: string
## vcName: string (in configuration element)
## vcUsername: string (in configuration element)
## vcPassword: secureString (in configuration element)
## diskPath: string. Example in code.
# output:
## actionResult: Not used
$inputsString = $inputs | ConvertTo-Json -Compress
Write-Host "Inputs were $inputsString"
$output=@{status = 'done'}
# connect to viserver
Set-PowerCLIConfiguration -InvalidCertificateAction:Ignore -Confirm:$false
Connect-VIServer -Server $inputs.vcName -Protocol https -User $inputs.vcUsername -Password $inputs.vcPassword
# Get vm by name
Write-Host "vmName is $inputs.vmName"
$vm = Get-VM -Name $inputs.vmName
# New-HardDisk -VM $vm -DiskPath "[storage1] OtherVM/OtherVM.vmdk"
$result = New-HardDisk -VM $vm -DiskPath $inputs.diskPath
Write-Host "Result is $result"
return "It worked!"
}
Looking at the code, you will notice an input of vmName (used by PS to find the VM). Getting the vmName is actually pretty stupid simple using JavaScript. My first task in the WF takes care of this.
// get the vmName
// $inputs.vm
// output: vmName
vmName = vm.name
The next step was to setup a resource action. The settings are shown in the following snapshot. Please note the setting within the green box. ‘vm’ is set with a binding action.
Changing the binding is fairly simple. Just click the binding link, then change the value to ‘with binding action’. The default values work just fine.
The disk I used in the test was actually a copy of another VM boot disk. It was copied over to another datastore, then renamed to ‘ExistingDisk2.vmdk’. The full diskPath was [dag-nfs] ExistingDisk/ExistingDisk2.vmdk.
Running the day 2 action on deployed machine seemed to work, as the WF logs show.
So there you have a basic PolyGlot vRO workflow using PowerCli and JavaScript.
I trust this quick blog was helpful in some small way.
My current customer needs to use 172.18.0.0/16 for their new VMWare Cloud on AWS cluster. However we tried this in the past and were getting a “NO ROUTE TO HOST” error when trying to add the VMC vCenter as a cloud account.
The problem was eventually traced back to the ‘on-prem-collector’ (br-57b69aa2bd0f) network in the Cloud Proxy which also uses the same subnet.
Let’s say the vCenters IP is 172.18.32.10. From inside cloudassembly-sddc-agent container, I try to connect to the vCenter. Eventually getting a ‘No route to host’ error. Can anyone say classic overlapping IP space?
We reached out to our VMWare Customer Success Team and TAM, who eventually provided a way to change the Cloud Proxy docker and on-prem-collector subnets.
Now for the obligatory warning. Don’t try this in production without having GSS sign off on it.
In this example I’m going to change the docker network to 192.168.0.0/24 and the on-prem-collector network to 192.168.1.0/24.
First to update the docker interface range.
Add the following two lines to /etc/docker/daemon.json. Don’t forget to add the necessary comma(s). Then save and close.
Check to see which containers are using this network with docker network inspect on-prem-collector. Mine had two, cloudassembly-sddc-agent, cloudassembly-cmx-agent.
The second task consumes the headers produced by the first task, then GET(s) the Version Information from the vRA Cloud About route (‘/iaas/api/about’). The results are then returned as the vRacAbout (Properties) variable.
function Handler($context, $inputs) {
<#
.PARAMETER $inputs.headers (Properties)
vRAC Refresh Token
.PARAMETER $inputs.apiEndpoint (String)
vRAC Base API URL
.OUTPUT vRacAbout (Properties)
vRAC version information from the About route
#>
$requestUri += $inputs.apiEndpoint + "/iaas/api/about"
$requestResponse = Invoke-RestMethod -Uri $requestUri -Method Get -Headers $headers
$output=@{vRacAbout = $requestResponse}
return $output
}
Here, you can see the output variables for both tasks are populated. Pretty cool.
As you can see, using the vRO Properties type is fairly simple using the PowerShell on CEXP vRO.
First the bad news. The VMware Code Stream Cloud version has a limit of 300 saved Custom Integrations and versions. Your once working pipelines will all of sudden get a validation error of “The saved Custom Integration version is not longer available” if you exceed this limit.
Now the not so good news. They haven’t fixed it yet!
In this blog I’ll explore the new Secret capability in vRealize Automation Cloud. The use case includes the following:
Deploy a CentOS 8 machine
Store the new user password and SSH key in a Secret
Configure the machine using Cloud-Init
Assign the password and SSH key from a vRA Cloud Secret
Verify the password and SSH key assignment.
First, add two Secrets. Go to Infrastructure -> Secrets, then click NEW SECRET.
The first one will be the SSH key. Find your project, give it a name, then paste in the key. Click CREATE to save the values. Repeat the process for the Password secret.
The Cloud Template is fairly straight forward. The new user password will be assigned the secret.Blog_Password value, and the ssh_authorized_keys comes from secret.Blog_SSH_Key value.
Now a look at how the secret values are displayed on a deployed machine. Open the deployment, then click on the machine. Expand Cloud Config to view the secret values sent to the machine.
As you can see the values for the new user are encrypted, and do not match stored secret values (The user password is set to VMware1!). Good so far.
Now to see if the password and SSH key actually work. A quick SSH using the key should be sufficient.
Oops. Looks like the key didn’t work, but I was able to login using the password. Time for a bit of troubleshooting. Using elevated permissions (set in Cloud-Init), I take a look at the cloud-init config sent down to the machine.
#more /var/lib/cloud/instance/cloud-config.txt
Hmm, looks the key had a line return in it.
I’ll need to edit/update the Blog_SSH_Key secret. After finding my troublesome secret, I click Edit.
The previously stored value is not viewable, I can only update it.
The new value is viewable until I save it. I made sure this one didn’t have a line return in it. The changes are committed when I click Save.
Now to test the change on a newly deployed machine. I’ll use the same SSH command, with the exception of changing the IP address.
Success! I was able to log in using the key.
In this blog I explored a simple application using two vRA Cloud Secrets, troubleshooting, and updating a secret Value. The VMware developers did a great job. I’m sure the new feature will prove to be very valuable.
I’m not sure when this will get pushed down into vRA 8.x. Please contact your VMware team for more information.
{!aCoder} 