Recently on a customer site we had a peculiar scenario where we were deploying a VM into a NSX environment via vRA and the operation was failing due to an unknown reason. However, we noticed that the IP address that vRA had allocated for the new VM was still responding to our pings even though the provisioning process had failed and the VM was never actually deployed…… so what was responding to our pings and where was it????
We tried accessing it via telnet, ssh, RDP, VNC, SNMP and a few other methods to figure out what it was but had no luck (we were not the only ones working in the environment so there were machines deployed we knew nothing about). We knew it must be a VM as the particular subnet was associated with a VXLAN, and we could see within the vSphere Web Client all the VMs that were connected to the logical switch in question, but due to the fact the VM template used to build these test machines didn’t have VMware Tools installed, we couldn’t see the IP addresses on the VMs in the vSphere Web Client.
So the question asked was “how do you find out what VM an IP address belongs to, in an NSX-v environment?
In a traditional physical network, to figure out where an IP address is configured, I would normally jump on the gateway for the L3 subnet, and look at the ARP tables of the L3 device. On a Cisco device, the command I would use would be something along the lines of:
This would give me the MAC address associated to the IP address in question, and also the physical interface for which the router has associated the information to.
Next I would jump onto the L2 switch connected to the interface which the router has associated the ARP information to, and look at the mac-address tables to figure out which port the applicable MAC address is associated to.
Now you know the physical port the phantom device is connected to, and if your switch port descriptions are kept up to date, it should tell you what it is connected to the port. If it connects to another switch, you repeat the process until you find the end device and if required, trace the physical cable. If it went to a blade enclosure or a hypervisor, I would then go and bug the team who looked after those environments so they could tell me what the device is 😉
But what if there are no physical switch ports or cables like in a NSX-v environment and the IP in question is a VM connected to a logical switch?
First, we need to identify the master controller for the vni.
SSH into one of the controllers, or jump on the console of one, and run the following command:
We can see the master controller for vni 5001 is 10.29.90.61.
Now jump onto the controller 10.29.90.61 and we will look at the ARP tables associated with vni 5001. We can issue the following command on the master controller:
(the IP address we are going to follow in this example is 10.29.84.141)
Now we have the MAC address (00:50:56:be:81:d3) which belongs to the IP address 10.29.84.141, along with the Connection-ID (10183) of the ESXi host.
OPTIONAL: If we issue the following command we can see which VTEP the MAC address is mapped to.
You can see from both the commands above that IP address 10.29.84.141 has a MAC address 00:50:56:be:81:d3, is mapped to VTEP 10.29.97.1, which has a Connection-ID of 10183.
Next we need a way to figure out which host the Connection-ID belongs to.
Thanks to Dmitri Kalintsev and this post, we know that we can use the following command to display the connection-table, which will show the ESXi management host IP address associated to the Connection-ID:
We can see that Connection-ID 10183 belongs to Host-IP 10.29.91.3.
Now we can SSH into ESXi host 10.29.91.3 and run the following command to look at what VM and vNIC the MAC address belongs to:
And just in case you were wondering, the IP Address in question belonged to a machine which was manually provisioned into the wrong subnet/logical switch as part of some previous testing, but was forgotten about. We ended up deleting the machine 🙂
Hope this helps.