Unveiling the Hidden VIM API

Exploring the vSphere Web Services (VIM) API

Anyone who works with vSphere tools or automation inevitably encounters the VIM API. Whether you're using PowerCLI, the pyvmomi SDK for Python, vSphere Management SDK for Java, govmomi SDK for Go, or the vSphere Perl SDK, you're interacting with managed objects that are well documented. In fact, there is extensive documentation for these tools.

All these SDKs are built on a common foundation—the vCenter server-side implementation, which is a web service accessible via SOAP (Simple Object Access Protocol). The SDKs abstract away the complexity of communicating with this web service, providing an object-oriented view of vSphere's managed objects. This allows developers and administrators to focus on higher-level tasks without dealing with low-level XML, encoding, and marshaling.

Cross-Language Consistency: One API, Many SDKs

One of the great advantages of the VIM API is that VMware (now Broadcom) has designed SDKs across different programming languages in a similar way. This consistency makes it easy to transfer code that uses the VIM API between languages like Perl, Java, Python, and Go.

How is this possible? SOAP web services are typically described by a WSDL (Web Services Description Language) file, an XML document that defines the data types and operations offered by the web service. This WSDL file is used to generate code, making these data types and operations available in your language of choice. This consistency in SDK design stems from the shared foundation of the WSDL file.

However, after working with these SDKs for a while, you might start to notice some common issues. For instance, there are several well-documented problems in both Python and PowerCLI:

• Facing Parse error while querying config options of a VM using EnvironmentBrowser
• vCenter 6.5 / pyVmomi.VmomiSupport.UnknownWsdlTypeError: 'HostSubSpecificationUpdateEvent'
• Error in deserializing body of reply message for operation 'ReadNextEvents'
• Workaround for Error in deserializing body of reply message for operation 'RetrieveProperties'

Why Missing Objects in the SDK Matter

Missing objects in the SDK can cause your program to break! There are data types in the web service response that the SDK doesn't recognize, which leads to errors when the SDK tries to interpret the response. This can cause your program to fail unexpectedly.

In some cases, you might be able to work around this by modifying your code or guessing the missing data type and applying a monkey patch. However, there are times when you won't be able to find a solution, leaving you stuck with a broken program.

How VMware Handles the Undocumented Parts

If you've worked with esxcli before, you're familiar with its power to manage and gather information from ESXi hosts. There’s also a remote version of esxcli that you can install on Windows or Linux. This tool allows you to run commands and retrieve information from an ESXi host, even when connected indirectly through a vCenter.

Interestingly, esxcli was implemented in Python, yet the documentation doesn’t reveal much about how it works. Curious? Let's dig deeper by examining the esxcli tarball.

The contents are straightforward:

The package installs files for the main program, as well as pyVim and pyVmomi. Both pyVim and pyVmomi are part of the pyvmomi project on GitHub. To see what might be hidden, let's compare esxcli's pyVmomi/_typeinfo_vim.py with the version available on GitHub.

The differences are striking. A quick comparison shows many missing definitions in the GitHub version:

• vim.AgentManager
• vim.CbrcManager
• vim.DeltaMigrationManager
• vim.DrsStatsManager
• vim.EsxAgentConfigManager
• vim.ExternalStatsManager
• vim.ImageLibraryManager
• vim.NetworkManager
• vim.VRPResourceManager
• ...
• and of course, vim.vm.device.VirtualQAT (related to a known issue).

The esxcli version of pyVmomi/_typeinfo_vim.py is about 25% larger than the GitHub version. Additionally, there are files like esxcli/pyVmomi/ManagedMethodExecutorHelper.py in the esxcli version that are missing from the GitHub version.

Interestingly, three files—esxcli/pyVmomi/_typeinfos.py, esxcli/pyVmomi/StubAdapterAccessorImpl.py, and esxcli/pyVmomi/Version.py—are marked as confidential.

The confidentiality label points to the VMware Software Development Kit License Agreement, which mentions confidentiality terms. This isn't something to ignore. But are these terms applicable here? Let's check the LICENSE file in the package's top-level directory:

The Apache License 2.0 is a well-known open-source license. Legal experts may debate whether this license applies to the entire package or just to the files that aren’t marked confidential. However, since esxcli/pyVmomi/_typeinfo_vim.py isn’t marked confidential, it appears safe to use this file under the terms of the Apache License 2.0.

Key Takeaways

• VMware (now Broadcom) released esxcli under the Apache License 2.0.
• Many files in the esxcli package are not marked confidential and reveal parts of the vSphere Web Services (VIM) API that are undocumented and excluded from the pyvmomi GitHub package.
• Access to these undocumented API components can help developers troubleshoot crashes caused by unknown data returned by the web service.
• Furthermore, the package reveals how esxcli tunnels through vCenter to communicate with the ESXi host.