vExpert 2017 – William do Carmo

VMW-LOGO-vEXPERT-2017-k

Não sei dizer o quão estou feliz com a noticia que recebi da VMWARE essa semana!!! Fui um dos escolhidos como vExpert 2017. São quase 1500 vExperts espalhados no mundo e quase 20 aqui do Brasil! É um grande reconhecimento e é um honra fazer parte desse grupo seleto!!!

Aqui você encontra a lista dos vExperts de 2017
Aqui nós temos o diretório dos vExperts espalhados pelo mundo

Alguns dos beneficios de ser vExpert:

vExpert certificate
Permission to use the vExpert logo on cards, website, etc for one year
Access to a private directory for networking, etc.
Exclusive gifts from various VMware partners
Access to private betas (subject to admission by beta teams)
365-day eval licenses for most products
Private pre-launch briefings
Private briefings from tier 1 alliance partners
Blogger early access program for vSphere and some other products
Featured in a public vExpert online directory
Access to vetted VMware & Virtualization content for your social channels.

A VMWARE lança o NSX 6.3.0 !!!

Na primeira semana de fevereiro de 2017, tivemos um grande lançamento do NSX 6.3.0, com muitas novidades e correções.

Alguns sites que recomendo a leitura:

Release notes
Nosso Guru em VMWARE
Network Virtualization Blog, introdução ao NSX 6.3 e NSX-T 1.1
Documentação do NSX 6.3
Endereço para o download do NSX for vSphere 6.3.0

 

Abaixo as novidades:

Platform and Compliance Features

  • On the Platform side:
    • Controller Disconnected Operation (CDO) mode: A new feature called Controller Disconnected Operation (CDO) mode has been introduced. This mode ensures that data plane connectivity is unaffected when hosts lose connectivity with the controller. See the section Controller Disconnected Operation (CDO) Mode in the NSX Administration Guide.
    • Cross-vCenter NSX Active-Standby DFW Enhancements: NSX 6.3.0 has the following enhancements:
      • Multiple Universal DFW sections are now supported. Both Universal and Local rules can consume Universal security groups in Source, Destination, and AppliedTo fields.
      • Universal Security Groups: Universal Security Group membership can be defined in a static or dynamic manner. Static membership is achieved by manually adding a universal security tag to each VM. Dynamic membership is achieved by adding VMs as members based on dynamic criteria (VM name).
      • Universal Security Tags: You can now define Universal Security tags on the primary NSX Manager and mark for universal synchronization with secondary NSX Managers. Universal Security tags can be assigned to VMs statically, based on unique ID selection, or dynamically, in response to criteria such as antivirus or vulnerability scans.
      • Unique ID Selection Criteria: In earlier releases of NSX, security tags are local to a NSX Manager, and are mapped to VMs using the VM’s managed object ID. In an active-standby environment, the managed object ID for a given VM might not be the same in the active and standby datacenters. NSX 6.3.x allows you to configure a Unique ID Selection Criteria on the primary NSX Manager to use to identify VMs when attaching to universal security tags: VM instance UUID, VM BIOS UUID, VM name, or a combination of these options. See Unique ID Selection in the NSX Administration Guide for more information.
    • Control Plane Agent (netcpa) Auto-recovery: An enhanced auto-recovery mechanism for the netcpa process ensures continuous data path communication. The automatic netcpa monitoring process also auto-restarts in case of any problems and provides alerts through the syslog server. A summary of benefits:
      • automatic netcpa process monitoring
      • process auto-restart in case of problems, for example, if the system hangs
      • automatic core file generation for debugging
      • alert via syslog of the automatic restart event
    • vSphere 6.5 Compatibility: NSX 6.3.0 introduces support for vSphere 6.5a and later. NSX 6.3.0 retains compatibility with vSphere 5.5 and 6.0.
  • Compliance features:
    • FIPS: NSX 6.3.0 has a FIPS mode that uses only those cipher suites that comply with FIPS and Common Criteria standards. NSX Manager has a FIPS Mode that can be enabled via an NSX REST API call.VMware development partners are undergoing certification of new, FIPS-compliant partner solutions for use in NSX. Consult your partner documentation for details.
    • Common Criteria: For Common Criteria compliance, NSX has been tested for compliance with the EAL2+ level of assurance. Running a Common Criteria-compliant NSX installation requires that you configure NSX as explained in the document Configuring NSX for Common Criteria. as part of the NSX Administration Guide.
    • ICSA: This is an industry-wide accepted standard certification which tests and certifies products including anti-virus, firewall, IPSec VPN, cryptography, SSL VPN, network IPS, anti-spyware, and PC firewall products. Both Distributed Firewall and Edge Firewall are certified against ICSA Corporate Firewall criteria.
    • Change in DFW packet log format due to ICSA certification requirement: NSX 6.3.0 introduces a change to the DFW packet logs. In 6.3.0 and later, we include the ICMP type and code to satisfy ICSA certification requirements.This is how the pre-6.3.0 log looked, without ICMP code and type:

      2016-09-29T20:52:21.983Z 6673 INET6 match PASS domain-c27/1001 IN 96 ICMP
      fe80:0:0:0:21d:b502:f984:c601->ff02:0:0:0:0:0:0:1
      In 6.3.0 and later, it looks like the following with ICMP code and type. In this example, 8 is the code and 0 is the type:

      2016-09-29T20:54:16.051Z 42991 INET match PASS domain-c27/1001 IN 84 ICMP 8 0 10.113.226.5->10.28.79.55

Operations Enhancements

  • Troubleshooting Dashboard: NSX Dashboard is updated in NSX 6.3.0 to include more features such as service deployment status, NSX Manager backup status, and Edge Appliance notifications.
  • Security Tagging: This allows assigning and clearing multiple tags for a given VM through API calls.
  • Syslog Enhancements: A new syslog update is available specifically for Load Balancer.
  • Log Insight Content Pack: This has been updated for Load Balancer to provide a centralized Dashboard, end-to-end monitoring, and better capacity planning from the user interface (UI).
  • Role-Based Access Control: This feature restricts user management only to Enterprise Administrators, and as a result, the NSX Administrator will no longer have permission to create new users or assign roles to new users. From a security standpoint, this helps in creating a clear demarcation of these two admin roles.
  • Drain state for Load Balancer pool members: You can now put a pool member into Drain state, which forces the server to shutdown gracefully for maintenance. Setting a pool member to drain state removes the backend server from load balancing, but still allows the server to accept new, persistent connections.

Service and Routing Enhancements

  • 4-byte ASN support for BGP: BGP configuration with 4-byte ASN support is made available along with backward compatibility for the pre-existing 2-byte ASN BGP peers.
  • NAT enhancement for 5-tuple match: In order to offer more granular configuration and flexibility for NAT rules, a 5-tuple match support is available for NSX 6.3.0:
    • Match criteria is on the basis of five parameters – protocol, source IP, source port, destination IP, and destination port.
    • User interface (UI) changes have been provided for to help you more easily specify SNAT/DNAT configurations. When changing DNAT/SNAT configurations on older Edge versions, the UI continues to display the old style of panes.
    • The NSX REST API adds fields for the new parameters:
              <natRules>
                <natRule>
                {...}
              <!-- new fields applicable for DNAT -->
                  <dnatMatchSourceAddress>any</dnatMatchSourceAddress>
                  <dnatMatchSourcePort>any</dnatMatchSourcePort>
                </natRule>
      
                <natRule>
                {...}
              <!-- new fields applicable for SNAT -->
                  <snatMatchDestinationAddress>any</snatMatchDestinationAddress>
                  <snatMatchDestinationPort>any</snatMatchDestinationPort>
                </natRule>
              </natRules>
      
  • Improved Layer 2 VPN performance: Performance for Layer 2 VPN has been improved. This allows a single Edge appliance to support up to 1.5 Gb/s throughput, which is an improvement from the previous 750 Mb/s.
  • Improved Configurability for OSPF: While configuring OSPF on an Edge Services Gateway (ESG), NSSA can translate all Type-7 LSAs to Type-5 LSAs.

Security Enhancements

There are several improvements in the Distributed Firewall:

  • DFW timers: NSX 6.3.0 introduces Session Timers that define how long a session is maintained on the firewall after inactivity. When the session timeout for the protocol expires, the session closes. On the firewall, you can define timeouts for TCP, UDP, and ICMP sessions and apply them to a user defined set of VMs or vNICs. See Session Timers in the NSX Administration Guide.
  • New features to support micro-segmentation: To support micro-segmentation in visibility and planning tools, two new features have been introduced:
    • Application Rule Manager simplifies the process of creating security groups and whitelisting firewall rules for existing applications.
    • Endpoint Monitoring allows an application owner to profile their application and identify processes making network connections.
  • Linux support for Guest Introspection: NSX 6.3.0 enables Guest Introspection for Linux VMs. On Linux-based guest VMs, NSX Guest Introspection feature leverages fanotify and inotify capability provided by the Linux Kernel. See Install Guest Introspection for Linux in the NSX Administration Guide for more information. See Minimum Recommended Versions for a list of Linux flavors supported by NSX.
  • Publish Status for Service Composer: Service Composer publish status is now available to check whether a policy is synchronized. This provides increased visibility of security policy translations into DFW rules on the host.

Cloud Management Platform (CMP) and Partner Integration

  • Better interoperability between vCloud Director 8.20 and NSX 6.3.0 helps service providers offer advanced networking and security services to their tenants. vCD 8.20 with NSX 6.3.0 exposes native NSX capabilities supporting multiple tenants and tenant self-service.
  • NSX 6.3.0 supports the new vRO plugin version 1.1, which supports vRA and introduces the ability to support other, non-vRA applications.
  • NSX NetX 6.3.0 provides scale and performance improvements related to service insertion.

Install and Upgrade

  • NSX kernel modules now independent of ESXi version: Starting in NSX 6.3.0, NSX kernel modules use only the publicly available VMKAPI so that the interfaces are guaranteed across releases. This enhancement helps reduce the chance of host upgrades failing due to incorrect kernel module versions. In earlier releases, every ESXi upgrade in an NSX environment required at least two reboots to make sure the NSX functionality continued to work (due to having to push new kernel modules for every new ESXi version).
  • Rebootless upgrade and uninstall on hosts: On vSphere 6.0 and later, once you have upgraded to NSX 6.3.0, any subsequent NSX VIB changes will not require a reboot. Instead hosts must enter maintenance mode to complete the VIB change. This includes the NSX 6.3.x VIB install that is required after an ESXi upgrade, any NSX 6.3.x VIB uninstall, and future NSX 6.3.0 to NSX 6.3.x upgrades. Upgrading from NSX versions earlier than 6.3.0 to NSX 6.3.x still requires that you reboot hosts to complete the upgrade. Upgrading NSX 6.3.x on vSphere 5.5 still requires that you reboot hosts to complete the upgrade.
  • NSX 6.3.0 also checks for NSX readiness before taking a host out of maintenance mode. This ensures that DRS only moves workloads to a host where NSX is ready. This prevents loss of networking for some workload VMs.
  • OVF Parameters now comma-separated: The following OVF parameters have changed from being space separated to comma separated:
    • DNS Server list (vsm_dns1_0)
    • Domain Search List (vsm_domain_0)
    • NTP Server List (vsm_ntp_0)

[NSX] Curso básico sobre virtualização de redes

A VMware está oferecendo um fcurso gratuito de três horas para mostrar os conceitos básicos sobre virtualização de rede, abaixo segue conteúdo e link para o curso:

 

Summary:
Add to myEnrollments
Course Description
Self-Paced (3 Hours)
Overview:
The VMware Network Virtualization Fundamentals eLearning course provides you with a fundamental understanding of virtual networking and the VMware NSX product, including the business challenges these products are intended to solve.

Objectives: There are three modules in this course.

At the end of module one you will be able to:
•  Define data center networking and discuss the challenges encountered without network virtualization.

At the end of module two you will be able to:
•  Describe the VMware NSX Virtualization Platform and how its features and components benefit the data center.

At the end of module three you will be able to:
•  Identify real life use cases where NSX can either solve or enhance current data center network operations and/or limitations.

Outline: Module 1 provides you with an introduction to the limitations of a current data center with server virtualization, but without network virtualization.

At the end of module one you will be able to:

•  Identify the use and benefits of standard and distributed switches

•  Examine how a physical network is realized compared to a virtual network

•  Assess the current Software-Defined Data Center’s network challenges

Module 2 provides an introduction to the benefits of the NSX Network Virtualization Platform.

At the end of module two you will be able to do the following:

•  Describe VMware NSX and its components at a high level
•  Outline the implications of network and security services
•  Describe on-demand usage deployment of VMware NSX
•  Describe various VMware NSX usage scenarios

Module 3 provides an introduction to NSX deployment, NSX architecture and network services.

At the end of module three you will be able to:

•  Identify the NSX architecture

•  Determine how to deploy VMware NSX

•  Examine the benefits VMware NSX brings to the modern data center

•  Describe NSX network services

[NSX] Material sobre NSX em português!!!

Achei essas páginas sobre NSX em português e gostaria de compartilhar com vocês:

O conteúdo foi feito pela Agility Networks:

ÍNDICE

  1. Introdução e componentes
  2. Requisitos para instalação
  3. Instalação do NSX Manager
  4. Gerenciamento de permissões de usuários
  5. Preparação de clusters para a virtualização de rede com NSX
  6. Conceitos gerais de VXLAN, Logical Switches e NSX Controllers
  7. Instalação de controllers NSX
  8. Criação de Segment ID Pool no NSX Manager
  9. Criação de logical switches
  10. NSX Edge como Distributed Logical Router
  11. Distributed Firewall
  12. NSX Edge como Services Gateway
  13. Configuração de Roteamento Dinâmico OSPF no NSX
  14. Configuração de Balanceamento One-Arm com o NSX
  15. Configuração de Balanceamento Inline com o NSX
  16. Proteção do servidor web com regras no balanceador e SSL
  17. NAT e regras do Edge Firewall

Espero que aproveitem!

 

Instalação do vSphere Replication 6

Neste post vou falar sobre uma ferramenta “gratuita” da vmware, digo que é gratuita entre aspas pois é necessário que voce tenha a licença do vsphere no seu ambiente.

Basta ter algumas dessas licenças no seu ambiente:
■ vSphere Essentials Plus
■ vSphere Standard
■ vSphere Enterprise
■ vSphere Enterprise Plus

Então porque pagar para a Zerto e Veeam se podemos utilizar essa ferramenta da VMWARE que já está inclusiva no pacote ?

Para quem não sabe com o Replication da VMWARE conseguimos ter uma solução de disastre/recover. Com uma facilidade grande de implementação e operação. Como não sou vendendor e não estou aqui para vender. Segue um paper da VMWARE que mostra tudo que ele tem a oferecer: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/vsphere/vmw-vsphr-replication-6-0-white-paper.pdf

 

Atualmente ele está na versão 6.1 e é ela que vamos utilizar em nosso LAB.

Você pode realizar o download clicando neste link: https://my.vmware.com/web/vmware/details?productId=491&downloadGroup=VR610

Antes de realizar a importação do OVF do Replication, devemos criar um pool de IPs que será usado por ele no momento da instalação, então abra o seu vsphere, clique em Networking, clique no seu Datacenter e clica na guia IP Pools.

Agora clique em ADD e preencha os dados solicitados:
replication2

Aqui devemos associar a qual interface será usada este range de IP:

replications3

Após der o OK, deve ficar assim:
replications4

Após realizar o download, você precisa descompactar e ele irá trazer alguns diretórios:
replication1
Vamos acessar o diretorio bin e teremos esses dois OVF, vamos utilizar o OVF10.ovf

Abrimos o vsphere, clicamos em file, Deploy OVF Template e selecionamos o OVF acima citado:

 

Na tela abaixo conseguimos ter um overview do que estamos fazendo o deploy?

 

Contrato:

 

Colocamos um nome da VM:

Setamos quantidade de CPU:

 

 

Selecionamos o datastore:

 

 

Deixamos como Thin para não ocupar mais do que o necessário:

 

Selecionamos o port group:

 

Setamos como queremos que o IP seja configurado:

 

 

Abaixo configuração do appliance, IP, NTP, Password:

Agora deixamos marcado a opção para que o Replication se integre no vcenter:

Após concluir o deploy do OVF, podemos logar no appliance e realizar a integração com o vcenter:

 

Após logar, clicamos em configuration e colocamos as informações do vcenter:

 

Após realizar com sucesso a integração, devemos fazer logoff e realizar o logon. Quando logarmos no vcenter, teremos a opção no menu de vSphere Replication:

 

Para realizar a replicação de alguma VM, vamos ir até vcenter, VM and templates e selecionamos a VM. Clicamos em cima da VM com o botão direito, vamos até All vSphere Replication Actions e clicamos em Configure Replication:

 

 

Agora clicamos em Replication to a vCenter Server:

 

Selecionamos o target de destino:

 

Podemos deixar o auto-assign vSphere Replication Server:

 

Selecionamos o datastore que vai ficar a VM:

 

Podemos deixar do jeito que está:

 

Podemos deixar do jeito que está (recomendo ler https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.replication_admin.doc%2FGUID-84FAF645-1C65-413D-A89B-70DBA0990631.html):

 

 

E é isso! Valeu

Falha ao realizar deploy de um OVF – deviceImage-0.iso was not found

Você está no seu trabalho e alguem interno ou externo, pede para realizar um deploy de um OVF que veio exportado de um outro ambiente virtual, você que conhece esse mundo, pensa: mamão com açucar, e no momento de realizar o deploy se depara com a mensagem:

error
File ds:///vmfs/volumes/uuid/_deviceImage-0.iso was not found

Aí você suspeita que o OVF veio corrompido e pede para realizar de novo a exportação, nessa outra exportação dá a mesma mensagem. Nesta hora voce busca um auxilio no google e cai aqui 😀

O que voce precisa fazer, esse arquivo exportado possui – geralmente tres arquivos – vai depender da quantidade de discos que o servidor tem, neste exemplo temos tres arquivos:

OVF10.ovf – arquivo que vamos alterar
OVF10.mf – manifest file
OVF10.vmdk – o disco do SO

O que precisamos realizar é alteração no arquivo OVF10.ovf em um bloco de notas/notepad++,. Abaixo a linha que precisamos localizar:
linha1before
Está linha: <rasd:ResourceSubType> vmware.cdrom.iso</rasd:ResourceSubType>

 

E a linha como deve ficar:
linha2after
Alterar para: <rasd:ResourceSubType>vmware.cdrom.remotepassthrough</rasd:ResourceSubType>

Por fim, você precisa deletar o arquivo OVF10.mf para realizar o deploy, pois ele irá reclamar que o OVF não é mais íntegro.

 
Abraço!

O melhor site de troubleshooting de NSX

Para você que trabalha com NSX e fica passando horas para realizar troubleshooting, achei esse blog fantastico que mostra várias formas de debugar o nosso querido NSX:

Blog: http://www.yet.org/2014/09/nsxv-troubleshooting/

Fiz uma cópia do post para deixar guardando essa reliquia aqui no blog:

 

NSX vSphere Web UI

Before jumping into the marvelous world of command lines, as a starter, we’ll check the state of the environment from the vSphere Web Client UI standpoint.

Authenticate to your web client, and click on Network & Security > Installation > Management.

You should see a green status for your three controllers.

Next click on Network & Security > Installation > Host Preparation and open up each cluster.

All the nodes are also green.

Now click on Network & Security > Installation > Logical Network Preparation and open up each cluster

Each compute node should have a Virtual Tunnel Endpoint (VTEP) vmkernel interface (vmk3 here) with an IP Address assigned to it.

Don’t worry if you get any errors, this article is meant to help you troubleshoot the root cause.

Transport Network

If VXLAN Connectivity isn’t operational, I mean if a VM on a VXLAN cannot ping another one on the same logical switch the most common reason is a misconfiguration on the transport network. To check that, SSH to a Compute node and type :

ping ++netstack=vxlan -d -s 1572 -I vmk3 1.2.3.4

++netstack=vxlan instruct the ESXi host to use the VXLAN TCP/IP stack.
-d set Don’t Fragment bit on IPv4 packet
-s 1572 set packet size to 1572 to check if MTU is correctly setup up to 1600
-I VXLAN vmkernel interface name
1.2.3.4 Destination ESXi host IP Address

If the ping fails, launch another one without the don’t fragment/size argument set

ping ++netstack=vxlan -I vmk3 1.2.3.4

If this one succeed, it means your MTU isn’t correctly set to at least 1600 on your transport network.

If both fails it’s a VLAN ID or Uplink misconfiguration. Before going any further you have to make sure that these pings works.

If both succeed, but you still don’t have connectivity on the virtual wire, I’ll show you, in the Compute node controller connectivity section, how to investigate that using net-vdl2 -l.

Note: If you don’t know the name of your VXLAN vmkernel you can easily check it, by looking at the configuration of your VDS.

But you’ve also seen that information in the Logical Network Preparation UI above.

Controller

You can get the IP Address of your Controller by clicking on the VM named NSX_Controller_<ID> in the vSphere Web Client.

To investigate controller issues, SSH to one of your controller VM to use the CLI (login: admin, password: the one set at deployment time).

status

# show control-cluster status

Type                Status                                       Since
--------------------------------------------------------------------------------
Join status:        Join complete                                09/14 14:08:46
Majority status:    Connected to cluster majority                09/18 08:45:16
Restart status:     This controller can be safely restarted      09/18 08:45:06
Cluster ID:         b20ddc88-cd62-49ad-b120-572c23108520
Node UUID:          b20ddc88-cd62-49ad-b120-572c23108520

Role                Configured status   Active status
--------------------------------------------------------------------------------
api_provider        enabled             activated
persistence_server  enabled             activated
switch_manager      enabled             activated
logical_manager     enabled             activated
directory_server    enabled             activated

List all the nodes in the cluster.

# show control-cluster startup-nodes

192.168.110.201, 192.168.110.202, 192.168.110.203

List the implemented role on your controller, the Not Configured for api_provider is normal, it’s the NSX-Manager who’s published the NSX-v API.

# show control-cluster roles

                          Listen-IP  Master?    Last-Changed  Count
api_provider         Not configured      Yes  09/18 08:45:17      6
persistence_server              N/A      Yes  09/18 08:45:17      5
switch_manager            127.0.0.1      Yes  09/18 08:45:17      6
logical_manager                 N/A      Yes  09/18 08:45:17      6
directory_server                N/A      Yes  09/18 08:45:17      6

List current connections to your controller.

# show control-cluster connections

role                port            listening open conns
--------------------------------------------------------
api_provider        api/443         Y         1
--------------------------------------------------------
persistence_server  server/2878     Y         2
                    client/2888     Y         3
                    election/3888   Y         0
--------------------------------------------------------
switch_manager      ovsmgmt/6632    Y         0
                    openflow/6633   Y         0
--------------------------------------------------------
system              cluster/7777    Y         2

Get Controller Statistics

# show control-cluster core stats

role                port            listening open conns
--------------------------------------------------------
api_provider        api/443         Y         1
--------------------------------------------------------
persistence_server  server/2878     Y         2
                    client/2888     Y         3
                    election/3888   Y         0
--------------------------------------------------------
switch_manager      ovsmgmt/6632    Y         0
                    openflow/6633   Y         0
--------------------------------------------------------
system              cluster/7777    Y         2

Controller networking

# show network interface

Interface       Address/Netmask     MTU     Admin-Status  Link-Status
breth0          192.168.110.201/24  1500    UP            UP
eth0                                1500    UP            UP

# show network default-gateway
# show network dns-servers

NTP is mandatory, so make sure it’s correctly configured

# show network ntp-servers
# show network ntp-status

To troubleshoot controller networking you can also use

# traceroute <ip_address or dns_name>
# ping <ip address>
# ping interface addr <alternate_src_ip> <ip_address>
# watch network interface breth0 traffic

L2 networking troubleshooting

First make sure to connect on the master controller of the virtual network you want to troubleshoot. You can then use the following commands

# show control-cluster logical-switches vni 5001
VNI      Controller      BUM-Replication ARP-Proxy Connections VTEPs
5001     192.168.110.201 Enabled         Enabled   0           0

As you can see above, you should now connect to 192.168.110.201 to troubleshoot VNI 5001.

Let see what we can use on that controller node to get more information on this virtual wire (VNI = 5001).

If you want to check the managemement TCP connection between Controller and ESXi

# show control-cluster logical-switches connection-table 5001
Host-IP         Port  ID
192.168.110.51  17528 2
192.168.110.52  46026 3
192.168.210.56  42257 4
192.168.210.51  30969 5
192.168.210.57  12127 6
192.168.210.52  30280 7

To see which ESXi instantiate this Logical Switch (LS).

Note: Mac address in the output are the one from the VXLAN vmkernel interface not the one from the Physical uplink.

# show control-cluster logical-switches vtep-table 5001
VNI      IP              Segment         MAC               Connection-ID
5001     192.168.250.54  192.168.250.0   00:50:56:6c:20:30 4
5001     192.168.250.53  192.168.250.0   00:50:56:60:24:53 6
5001     192.168.250.52  192.168.250.0   00:50:56:61:23:00 7
5001     192.168.250.51  192.168.250.0   00:50:56:6b:4b:a4 5
5001     192.168.150.51  192.168.150.0   00:50:56:60:6a:3a 2
5001     192.168.150.52  192.168.150.0   00:50:56:6e:5e:e3 3

List Mac addresses on a Logical Switch.

# show control-cluster logical-switches mac-table 5001
VNI      MAC               VTEP-IP         Connection-ID
5001     00:50:56:ae:9b:be 192.168.250.51  5

Same for ARP table on the Logical Switch

# show control-cluster logical-switches arp-table 5001

List the VNIs on a specific ESXi

# show control-cluster logical-switches joined-vnis <ESXi_MGT_IP>
VNI      Controller      BUM-Replication ARP-Proxy Connections VTEPs
5001     192.168.110.201 Enabled         Enabled   6           6

Shows ESXi VTEP IP/Mac addresses for each joined VNIs

# show control-cluster logical-switches vtep-records <ESXi_MGT_IP>
VNI      IP              Segment         MAC               Connection-ID
5001     192.168.150.51  192.168.150.0   00:50:56:60:6a:3a 2

List all VMs Mac addresses on each VNIs of a specific ESXi with their associated VTEP.

# show control-cluster logical-switches mac-records <ESXi_MGT_IP>

List all VMs IP/Mac on each each VNIs of a specific ESXi

# show control-cluster logical-switches arp-records <ESXi_MGT_IP>

L3 networking troubleshooting

First you can list all of your logical routers

# show control-cluster logical-routers instance all
LR-Id      LR-Name            Hosts[]         Edge-Connection Service-Controller
1460487509 default+edge-1     192.168.110.51                  192.168.110.201
                              192.168.110.52
                              192.168.210.52
                              192.168.210.51
                              192.168.210.57
                              192.168.210.56

You can then use the LR-Id above to get interface details on one instance

# show control-cluster logical-routers interface-summary 1460487509
Interface                        Type   Id           IP[]
570d45550000000b                 vlan   100
570d45550000000c                 vxlan  5004         10.10.10.1/24
570d45550000000a                 vxlan  5000

Use the Interface name to get even more details on VXLAN 5004 LIF for example

# show control-cluster logical-routers interface 1460487509 570d45550000000c
Interface-Name:   570d45550000000c
Logical-Router-Id:1460487509
Id:               5004
Type:             vxlan
IP:               10.10.10.1/24
DVS-UUID:         1cec0e50-029c-a921-b6d8-d0fc73e57969
                  ee660e50-e861-6d04-b4d8-1d462df952bc
Mac:              02:50:56:8e:21:35
Mtu:              1500
Multicast-IP:     0.0.0.1
Designated-IP:
Is-Sedimented:    false
Bridge-Id:
Bridge-Name:

To get the routing table of your logical router

# show control-cluster logical-routers routes 1460487509

Bridging

To get more information on a all bridge instance hosted on your logical router

# show control-cluster logical-routers bridges <lr-id> all
LR-Id       Bridge-Id   Host            Active
1460487509  1           192.168.110.52  true

And now the Mac address on them

# show control-cluster logical-routers bridge-mac <lr-id> all
LR-Id       Bridge-Id   Mac               Vlan-Id Vxlan-Id Port-Id   Source
1460487509  1           00:50:56:ae:9b:be 0       5000     50331650  vxlan

Compute Nodes

In the introduction we’ve seen how to send ping on the transport network. But from a SSH connection to an ESXi node, we can use many more troubleshooting commands. This section will details most of them.

VIBs

When you prepare a compute host for NSX-v, vSphere Installation Bundle (VIBs) are automatically installed by the NSX Manager via ESX Agency Manager (EAM). You can check they were correctly installed (output abridged):

# esxcli software vib get --vibname esx-vxlan
    ...
    Summary: Vxlan and host tool
    Description: This package loads module and configures firewall for vxlan networking.
    ...
    Provides: vxlan = 2.0.0.0-nsx, vdr = 1.0.0.0
    Maintenance Mode Required: False
    ...

# esxcli software vib get --vibname esx-vsip
    ...
    Summary: vsip module
    Description: This package contains DFW and NetX data and control plane components.
    ...
    Provides: vsip = 1.0.0-0
    Maintenance Mode Required: False
    ...

# esxcli software vib get --vibname esx-dvfilter-switch-security
   ...
   Summary: dvfilter-switch-security module
   Description: This package contains dvfilter-switch-security module.
   ...
   Provides: switchSecurity = 0.1.0.0
   Maintenance Mode Required: False
   ...

When you remove a host from an NSX Prepared cluster the VIBs will be automatically removed, but you can remove them from the command line :

# esxcli software vib remove -n esx-vsip
# esxcli software vib remove -n esx-vxlan
# esxcli software vib remove -n esx-dvfilter-switch-security

You’ll then have to reboot your host.

Physical Nics

To list all the host physical interface, to check driver and MTU:

# esxcli network nic list

Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address         MTU  Description
------  -------------  ------  ----  -----  ------  -----------------  ----  ------------------ -------------------------
vmnic0  0000:000:14.0  igb     Up     1000  Full    00:25:90:f4:76:8e  1600  Intel Corporation  Ethernet Connection I354
vmnic1  0000:000:14.1  igb     Up     1000  Full    00:25:90:f4:76:8f  1600  Intel Corporation  Ethernet Connection I354
vmnic2  0000:000:14.2  igb     Up     1000  Full    00:25:90:f4:76:90  1600  Intel Corporation  Ethernet Connection I354
vmnic3  0000:000:14.3  igb     Up     1000  Full    00:25:90:f4:76:91  1600  Intel Corporation  Ethernet Connection I354

To get more details on a Nic

# esxcli network nic get  -n vmnic0

Advertised Auto Negotiation: true
Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Ful  1000baseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 7
Driver Info:
      Bus Info: 0000:00:14.0
      Driver: igb
      Firmware Version: 0.0.0
      Version: 5.2.5
Link Detected: true
Link Status: Up by explicit linkSet
Name: vmnic0
PHYAddress: 0
Pause Autonegotiate: true
Pause RX: false
Pause TX: false
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
Wakeon: MagicPacket(tm)

To check TCP segmentation offload (TSO) and checksum offload (CSO) settings for a Nic

# esxcli network nic tso get -n vmnic0
# esxcli network nic cso get -n vmnic0

If you see some strange behavior while using LACP, you can use the following commands to leave only one interface up to verify if LACP negotiation is reponsible for your issues:

# esxcli network nic down -n vmnic1
# esxcli network nic down -n vmnic2
# esxcli network nic down -n vmnic3

To revert it:

# esxcli network nic up -n vmnic1
# esxcli network nic up -n vmnic2
# esxcli network nic up -n vmnic3

VMs

Get a list of all VMs on the compute node

# esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         1536        1           128               1500

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  VM Network            0        0

DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
Mgmt_Edge_VDS    1536        13          512               1600    vmnic1,vmnic0

  DVPort ID           In Use      Client
  897                 1           vmnic0
  767                 1           vmk2
  639                 1           vmk1
  510                 1           vmk0
  895                 0
  905                 1           vmk3
  907                 1           vmnic1
  506                 1           NSX_Controller_c6aea614-0dc7-40fd-b646-0230608d4709.eth0
  497                 1           dr-4-bridging-0.eth0
  127                 1           br-sv-01a.eth0

VMkernel Port

# esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                              Netmask            Broadcast       MAC Address       MTU     TSO MSS   Enabled Type
vmk0       510                 IPv4      192.168.110.52                          255.255.255.0   192.168.   110.255 00:50:56:09:08:3c 1500    65535     true    STATIC
vmk1       639                 IPv4      10.10.20.52                             255.255.255.0   10.10.20   .255    00:50:56:64:f0:9b 1500    65535     true    STATIC
vmk2       767                 IPv4      10.10.30.52                             255.255.255.0   10.10.30   .255    00:50:56:65:67:8e 1500    65535     true    STATIC
vmk3       905                 IPv4      192.168.150.52                          255.255.255.0   192.168.   150.255 00:50:56:6e:5e:e3 1600    65535     true    STATIC

ARP Table

# esxcli network ip neighbor list
Neighbor         Mac Address        Vmknic    Expiry  State  Type
---------------  -----------------  ------  --------  -----  -------
192.168.110.202  00:50:56:8e:52:25  vmk0     674 sec         Unknown
192.168.110.42   00:50:56:09:45:60  vmk0    1196 sec         Unknown
192.168.110.10   00:50:56:03:00:2a  vmk0    1199 sec         Unknown
192.168.110.203  00:50:56:8e:7a:a4  vmk0     457 sec         Unknown
192.168.110.201  00:50:56:8e:ea:bd  vmk0     792 sec         Unknown
192.168.110.22   00:50:56:09:11:07  vmk0    1146 sec         Unknown
10.10.20.60      00:50:56:27:49:6b  vmk1     506 sec         Unknown

VXLAN

Dump VXLAN configuration

# esxcli network vswitch dvs vmware vxlan list
VDS ID                                           VDS Name        MTU  Segment ID     Gateway IP     Gateway MAC        Network Count  Vmknic Count
-----------------------------------------------  -------------  ----  -------------  -------------  -----------------  -------------  ------------
1c ec 0e 50 02 9c a9 21-b6 d8 d0 fc 73 e5 79 69  Mgmt_Edge_VDS  1600  192.168.150.0  192.168.150.2  00:50:56:27:48:7d              2             1

List VXLAN networks, great to see who’s the master controller for each VXLAN.

# esxcli network vswitch dvs vmware vxlan network list --vds-name=<VDS_NAME>
VXLAN ID  Multicast IP               Control Plane                        Controller Connection  Port Count  MAC Entry Count  ARP Entry Count  MTEP Count
--------  -------------------------  -----------------------------------  ---------------------  ----------  ---------------  ---------------  ----------
    5000  N/A (headend replication)  Enabled (multicast proxy,ARP proxy)  192.168.110.202 (up)            1                1                0           0
    5004  N/A (headend replication)  Enabled (multicast proxy,ARP proxy)  192.168.110.203 (up)            1                0                0           0   

Get more details on a specific VXLAN wire

# esxcli network vswitch dvs vmware vxlan network mac list --vds-name=Mgmt_Edge_VDS --vxlan-id=<VXLAN ID>
Inner MAC          Outer MAC          Outer IP        Flags
-----------------  -----------------  --------------  --------
00:50:56:ae:9b:be  ff:ff:ff:ff:ff:ff  192.168.250.51  00001111

But you can also get specific information on the VXLAN like mac, arp, port, mtep and stats like this

For example to list all the remote Mac addresses, pushed by the controller for a specific VNI:

# esxcli network vswitch dvs vmware vxlan network mac list –-vds-name=<VDS_NAME> --vxlan-id=<VXLAN_ID>

You can get also the remote IP Addresses/Mac Addresses that are still in the local ESXi cache. They timeout after 5’ if no traffic.

# esxcli network vswitch dvs vmware vxlan network arp list --vds-name=<VDS_NAME> --vxlan-id=<VXLAN_ID>

To get a list of remote known ESXi for a VNI. You’ll also see who’s the MTEP on each Transport Network subnet.

# esxcli network vswitch dvs vmware vxlan network vtep list --vds-name=<VDS_NAME> --vxlan-id=<VXLAN_ID>

Note: if you don’t get the vtep argument but he mtep one intead, just run /etc/init.d/hostd restart

# esxcli network vswitch dvs vmware vxlan network port list --vds-name=<VDS_NAME> --vxlan-id=<VXLAN_ID>

# esxcli network vswitch dvs vmware vxlan network stats list --vds-name=<VDS_NAME> --vxlan-id=<VXLAN_ID>    

Controller connectivity

To check Controller connectivity from ESXi (VDL= Virtual Distributed Layer 2)

# net-vdl2 -l
VXLAN Global States:
        Control plane Out-Of-Sync:      No
        UDP port:       8472
VXLAN VDS:      Mgmt_Edge_VDS
        VDS ID: 1c ec 0e 50 02 9c a9 21-b6 d8 d0 fc 73 e5 79 69
        MTU:    1600
        Segment ID:     192.168.150.0
        Gateway IP:     192.168.150.2
        Gateway MAC:    00:50:56:27:48:7d
        Vmknic count:   1
                VXLAN vmknic:   vmk3
                        VDS port ID:    905
                        Switch port ID: 50331656
                        Endpoint ID:    0
                        VLAN ID:        0
                        IP:             192.168.150.52
                        Netmask:        255.255.255.0
                        Segment ID:     192.168.150.0
                        IP acquire timeout:     0
                        Multicast group count:  0
        Network count:  2
                VXLAN network:  5000
                        Multicast IP:   N/A (headend replication)
                        Control plane:  Enabled (multicast proxy,ARP proxy)
                        Controller:     192.168.110.202 (up)
                        MAC entry count:        1
                        ARP entry count:        0
                        Port count:     1
                VXLAN network:  5004
                        Multicast IP:   N/A (headend replication)
                        Control plane:  Enabled (multicast proxy,ARP proxy)
                        Controller:     192.168.110.203 (up)
                        MAC entry count:        0
                        ARP entry count:        0
                        Port count:     1

Or

# esxcli network vswitch dvs vmware vxlan network list -–vds-name <vds name>
VXLAN ID  Multicast IP               Control Plane                        Controller Connection  Port Count  MAC Entry Count  ARP Entry Count  MTEP Count
--------  -------------------------  -----------------------------------  ---------------------  ----------  ---------------  ---------------  ----------
    5000  N/A (headend replication)  Enabled (multicast proxy,ARP proxy)  192.168.110.202 (up)            1                1                0           0
    5004  N/A (headend replication)  Enabled (multicast proxy,ARP proxy)  192.168.110.203 (up)            1                0                0           0

If you see a controller down message above, you can fix it by restarting netcpa like this

# /etc/init.d/netcpad restart

Note: netcpa is a user world agent that communicate thru SSL with the NSX Controller.

To check ESXi controller connections.

# esxcli network ip connection list| grep tcp | grep 1234

tcp         0       0  192.168.110.52:43925  192.168.110.203:1234  ESTABLISHED     44923  newreno  netcpa-worker
tcp         0       0  192.168.110.52:46026  192.168.110.202:1234  ESTABLISHED     46232  newreno  netcpa-worker
tcp         0       0  192.168.110.52:39244  192.168.110.201:1234  ESTABLISHED     44923  newreno  netcpa-worker

As you can see, your compute node is connected to all Controllers.

Logical Router

First get a list of distributed router (VDR) instances

# net-vdr --instance -l

VDR Instance Information :
---------------------------

VDR Instance:               default+edge-1:1460487509
Vdr Name:                   default+edge-1
Vdr Id:                     1460487509
Number of Lifs:             3
Number of Routes:           1
State:                      Enabled
Controller IP:              192.168.110.201
Control Plane Active:       Yes
Control Plane IP:           192.168.110.52
Edge Active:                Yes

Dump all the logical interfaces (LIFs) for a VDR instance

# net-vdr --lif -l default+edge-1

VDR default+edge-1:1460487509 LIF Information :

Name:                570d45550000000c
Mode:                Routing, Distributed, Internal
Id:                  Vxlan:5004
Ip(Mask):            10.10.10.1(255.255.255.0)
Connected Dvs:       Mgmt_Edge_VDS
VXLAN Control Plane: Enabled
VXLAN Multicast IP:  0.0.0.1
State:               Enabled
Flags:               0x2288

Name:                570d45550000000b
Mode:                Bridging, Sedimented, Internal
Id:                  Vlan:100
Bridge Id:           mybridge:1
Ip(Mask):            0.0.0.0(0.0.0.0)
Connected Dvs:       Mgmt_Edge_VDS
Designated Instance: No
DI IP:               192.168.110.51
State:               Enabled
Flags:               0xd4

Name:                570d45550000000a
Mode:                Bridging, Sedimented, Internal
Id:                  Vxlan:5000
Bridge Id:           mybridge:1
Ip(Mask):            0.0.0.0(0.0.0.0)
Connected Dvs:       Mgmt_Edge_VDS
VXLAN Control Plane: Enabled
VXLAN Multicast IP:  0.0.0.1
State:               Enabled
Flags:               0x23d4

Check routing status

# net-vdr -R -l default+edge-1

VDR default+edge-1:1460487509 Route Table
Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]
Legend: [H: Host], [F: Soft Flush] [!: Reject]

Destination      GenMask          Gateway          Flags    Ref Origin   UpTime     Interface
-----------      -------          -------          -----    --- ------   ------     ---------
10.10.10.0       255.255.255.0    0.0.0.0          UCI      1   MANUAL   410777     570d45550000000c

ARP information

# net-vdr --nbr -l default+edge-1

VDR default+edge-1:1460487509 ARP Information :
Legend: [S: Static], [V: Valid], [P: Proxy], [I: Interface]
Legend: [N: Nascent], [L: Local], [D: Deleted]

Network           Mac                  Flags      Expiry     SrcPort    Interface Refcnt
-------           ---                  -----      -------    ---------  --------- ------
10.10.10.1        02:50:56:56:44:52    VI         permanent  0          570d45550000000c 1

Designated instance statistics

# net-vdr --di —stats

VDR Designated Instance Statistics:

    RX Pkts:                        0
    RX Bytes:                       0
    TX Pkts:                        0
    TX Bytes:                       0
    ARP Requests:                   0
    ARP Response:                   0
    ARP Resolved:                   0
    Err RX:                         0
    Err TX:                         0
    Err Message too Small:          0
    Err Message too Big:            0
    Err Invalid Version:            0
    Err Invalid Message Type:       0
    Err Instance Not Found:         0
    Err LIF not DI:                 0
    Err No memory:                  0
    Err ARP not found:              0
    Err Proxy not found:            0
    Err DI Remote:                  0

Bridging

Dump bridge info

# net-vdr --bridge -l <vdrName>

VDR default+edge-1:1460487509 Bridge Information :

Bridge config:
Name:id             mybridge:1
Portset name:
DVS name:           Mgmt_Edge_VDS
Ref count:          2
Number of networks: 2
Number of uplinks:  0

        Network 'vlan-100-type-bridging' config:
        Ref count:          2
        Network type:       1
        VLAN ID:            100
        VXLAN ID:           0
        Ageing time:        300
        Fdb entry hold time:1
        FRP filter enable:  1

                Network port '50331655' config:
                Ref count:          2
                Port ID:            0x3000007
                VLAN ID:            4095
                IOChains installed: 0

        Network 'vxlan-5000-type-bridging' config:
        Ref count:          2
        Network type:       1
        VLAN ID:            0
        VXLAN ID:           5000
        Ageing time:        300
        Fdb entry hold time:1
        FRP filter enable:  1

                Network port '50331655' config:
                Ref count:          2
                Port ID:            0x3000007
                VLAN ID:            4095
                IOChains installed: 0

Lists MAC table, learnt on both VXLAN and VLAN sides

# net-vdr -b --mac default+edge-1

VDR default+edge-1:1460487509 Bridge Information :

Network 'vlan-100-type-bridging' MAC address table:
MAC table on PortID:              0x0
MAC table paging mode:            0
Single MAC address enable:        0
Single MAC address:               00:00:00:00:00:00
MAC table last entry shown:       00:50:56:91:5e:93 VLAN-VXLAN: 100-0 Port: 50331661
total number of MAC addresses:    1
number of MAC addresses returned: 1
MAC addresses:
Destination Address  Address Type  VLAN ID  VXLAN ID  Destination Port  Age
-------------------  ------------  -------  --------  ----------------  ---
00:50:56:91:5e:93    Dynamic           100         0          50331661  0


Network 'vxlan-5000-type-bridging' MAC address table:
MAC table on PortID:              0x0
MAC table paging mode:            0
Single MAC address enable:        0
Single MAC address:               00:00:00:00:00:00
MAC table   last entry shown:       00:50:56:ae:9b:be VLAN-VXLAN: 0-5000 Port: 50331650
total number of MAC addresses:    1
number of MAC addresses returned: 1
MAC addresses:
Destination Address  Address Type  VLAN ID  VXLAN ID  Destination Port  Age
-------------------  ------------  -------  --------  ----------------  ---
00:50:56:ae:9b:be    Dynamic             0      5000          50331650  0

Dump statistics (output not shown)

# net-vdr -b --stats default+edge-1

Distributed Firewall

To investigate the Distributed Firewall Rules applied to a Virtual Machine vNic, first get the VM UUID (vcUuid) for it using (output abridged)

# summarize-dvfilter
..
world 230869 vmm0:br-sv-01a vcUuid:'50 11 97 6c fa 73 a7 5b-a7 0e 36 1f a2 f5 84 38'
..

Now find the filter name for that VM UUID

# vsipioctl getfilters

Filter Name              : nic-230869-eth0-vmware-sfw.2
VM UUID                  : 50 11 97 6c fa 73 a7 5b-a7 0e 36 1f a2 f5 84 38
VNIC Index               : 0
Service Profile          : --NOT SET--

Use the filter name above to list the associated Distributed Firewall Rules

# vsipioctl getrules -f nic-230869-eth0-vmware-sfw.2

To details Addresses Sets

# vsipioctl getaddrsets -f nic-230869-eth0-vmware-sfw.2

Packet Capture

vSphere 5 offers a new command, pktcap-uw to capture packet at different level of the processing.

You can get look at all possibilities

pktcap-uw -h |more

As you can see on the diagram above, we can now capture traffic at the vmnic, vmknic, ‘vnic’ level.

Let see how it works from the outside world to the VM. I’m not going to include the ouput of the command here, I advice you to try on your hosts instead. By the way I also advice to save the output to a file in pcap format with -o ./save.pcap, you’ll then be able to open it from Wireshark.

uplink/vmnic

You can open up the DVUplinks section of your VDS to get the name of your uplink interface. Here we’ll be using vmnic0. So to capture packets received on this uplink, use

pktcap-uw --uplink vmnic0

By default it will only capture received traffic (RX), to capture packets sent on the uplink to the outside world use the --capture argument like this

pktcap-uw --uplink vmnic0 --capture UplinkSnd

We’ll details all the filtering options at the end of this section, but in the meantime you can for example filter out only ICMP packet received on a specific destination by using --proto 0x01 and --destip <ip>

pktcap-uw –uplink vmnic0 –proto 0x01 –dstip <IP>

Or to capture ICMP Packets that are sent on vmnic0 from an IP Address 192.168.25.113

pktcap-uw --uplink vmnic0 --capture UplinkSnd –proto 0x01 --srcip 192.168.25.113

You can also capture ARP packets

pktcap-uw --uplink vmnic0 –ethtype 0x0806 

vmknic – Virtual Adapters

Capture packets reaching vmknic adapter is also possible, just use --vmk argument.

pktcap-uw --vmk vmk0 

Switchport

To capture on a specific switchport, you first have to get the ID of the port. Launch

# esxtop

Type n, to get a list of all the ports with the corresponding attachment. Take note of Port ID of the port you’re interested in and use a --switchport argument like this

pktcap-uw –switchport <port-id> –-proto 0x01 

Traffic direction

For –switchport, –vmk, –uplink, –dvfilter, direction of traffic is specified using --dir 0 for inbound and --dir 1 for outbound but inbound is assumed.

0- Rx (Default)
1- Tx

So don’t be surprised, pktcap-uw doesn’t work like tcpdump and by default only capture the received (RX) traffic. Don’t forget to change that if necessary by specifying --dir 1, it will switch the capture to the Transmit (Tx) direction.

Argument and Filtering Options

-o save.pcap to save capture to a file in pcap format
-c 25 capture only 25 packets
–-vxlan <segment id> to specify VXLAN VNI of flow
--vlan <VLANID> filter for VLAN ID
–-ip <x.x.x.x> filter for SRC or DST
--srcmac <xx:xx:xx:xx:xx> filter for source mac address
--dstmac <xx:xx:xx:xx:xx> filter for source mac address
--srcip <x.x.x.x[/<range>]> filter for source IP
--dstip <x.x.x.x[/<range>]> filter for destination IP
--dstport <DSTPORT> to specify a TCP destination Port
--srcport <SRCPORT> to specify a TCP source Port
--tcpport <PORT> filter for source or destination Port
--proto 0x<IPPROTYPE> filter on hexadecimal protocol id: 0x01 for ICMP, 0x06 for TCP, 0x11 for UDP.list here
--ethtype 0x<ETHTYPE> filter on ethernet type, 0x0806 for ARP

Decoding capture

We’ve shown you how to save the captured packets to a file, to get a quick overview of the kind of traffic passing by, you can decode the pcap using tcpdump like this

tcpdump-uw -r save.pcap

But using Wireshark will give you a better vision of the traffic, with all the details.

Tracing

If you are interested in seeing even more details on the processing of the packet through the ESXi TCP/IP stack, just add --trace argument to see packet traversing the ESXi network stack. Looks for Drop message that indicate something went wrong in the processing.

Drops

When things don’t work as you expect, one really usefull command is

pktcap-uw --capture Drop 

You should see here some errors like VLAN Mismatch or something else that will give you a hint about why traffic isn’t flowing as you would expect.

DVFilter

This command captures packets as seen by the dvfilter (before the filtering happens)

pktcap-uw --capture PreDVFilter --dvfilterName <filter name>

This command captures packets after being subject to the dvfilter.

pktcap-uw --capture PostDVFilter --dvfilterName <filter name>   

Capture point

You can get a list of all possible capture point with -A

pktcap-uw -A
Supported capture points:
    1: Dynamic -- The dynamic inserted runtime capture point.
    2: UplinkRcv -- The function that receives packets from uplink dev
    3: UplinkSnd -- Function to Tx packets on uplink
    4: Vmxnet3Tx -- Function in vnic backend to Tx packets from guest
    5: Vmxnet3Rx -- Function in vnic backend to Rx packets to guest
    6: PortInput -- Port_Input function of any given port
    7: IOChain -- The virtual switch port iochain capture point.
    8: EtherswitchDispath -- Function that receives packets for switch
    9: EtherswitchOutput -- Function that sends out packets, from switch
    10: PortOutput -- Port_Output function of any given port
    11: TcpipDispatch -- Tcpip Dispatch function
    12: PreDVFilter -- The DVFIlter capture point
    13: PostDVFilter -- The DVFilter capture point
    14: Drop -- Dropped Packets capture point
    15: VdrRxLeaf -- The Leaf Rx IOChain for VDR
    16: VdrTxLeaf -- The Leaf Tx IOChain for VDR
    17: VdrRxTerminal -- Terminal Rx IOChain for VDR
    18: VdrTxTerminal -- Terminal Tx IOChain for VDR
    19: PktFree -- Packets freeing point

Let me share with you a little bit more details about some of them.

PortOutput show traffic delivered from the vSwitch to the Guest when used with switch port or to the physical adapter if used with a physical adapter
VdrRxLeaf – Capture packets at the receive leaf I/O chain of a dynamic router in VMware NSX. Use this capture point together with the –lifID option
VdrRxTerminal – Capture packets at the receive terminal I/O chain of a dynamic router in VMware NSX. Use this capture point together with the –lifID option
VdrTxLeaf – Capture packets at the transmit leaf I/O chain of a dynamic router in VMware NSX. Use this capture point together with the –lifID option
VdrTxTerminal – Capture packets at the transmit terminal I/O chain of a dynamic router in VMware NSX. Use this capture point together with the –lifID option
`

CTRL-D

Never press CTRL-D to interupt a running packet capture or you’ll be left with a background process still running. If you’ve done it you can kill it like this

kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)

Then check it was killed

lsof |grep pktcap-uw |awk '{print $1}'| sort -u  

NSX Edge CLI

NSX Edge offers lots of Layer 4 to Layer 7 services, to name a few :

  • VPN
  • SSL-VPN
  • LB
  • FW
  • DHCP Relay

Lets now details the command line interface available from a SSH connection to an NSX Edge

show ip route
show ip ospf neighbor
show ip ospf database

show configuration {ospf|bgp|isis|static-routing}
show configuration {firewall|nat|dhcp|dns}
show configuration {loadbalancer|ipec|sslvpn-plus} 
show interface [IFNAME]
show firewall
show ip {route|ospf|bgp|forwarding}
show arp
show system {cpu|memory|network-stats|storage|uptime}
show service {dhcp|dns|highavailability|ipsec|loadbalancer|sslvpn-plus}
show log {follow|reverse}
show floatable

But one of the most convenient one is the following which enable you to dump all ingress/egress traffic on a specific edge interface

debug packet display interface vNic_0

You’ll then get a live display of what’s flowing on that virtual network interface:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vNic_0, link-type EN10MB (Ethernet), capture size 65535 bytes
22:43:44.190770 IP 172.16.16.3 > 224.0.0.5: OSPFv2, Hello, length 48
22:43:45.914868 IP 172.16.16.1.17773 > 192.168.2.20.8080: Flags [S], seq 1790616868, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 3], length 0
22:43:45.915713 IP 192.168.2.20.8080 > 172.16.16.1.17773: Flags [S.], seq 841084052, ack 1790616869, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
22:43:45.915750 IP 172.16.16.1.17773 > 192.168.2.20.8080: Flags [.], ack 1, win 1825, length 0
22:43:45.915990 IP 172.16.16.1.17773 > 192.168.2.20.8080: Flags [P.], seq 1:101, ack 1, win 1825, length 100
22:43:45.916497 IP 192.168.2.20.8080 > 172.16.16.1.17773: Flags [.], ack 101, win 457, length 0
22:43:45.922654 IP 192.168.2.20.8080 > 172.16.16.1.17773: Flags [P.], seq 4381:4434, ack 101, win 457, length 53
22:43:45.922674 IP 192.168.2.20.8080 > 172.16.16.1.17773: Flags [F.], seq 4434, ack 101, win 457, length 0
22:43:45.922842 IP 172.16.16.1.17773 > 192.168.2.20.8080: Flags [.], ack 1, win 1825, options [nop,nop,sack 1 {4381:4434}], length 0

You can use the same filtering syntax as the one used by tcpdump, for example :

debug packet display interface vNic_0 icmp

If you have multiple words for your filter, just add underscore between them

debug packet display interface vNic_0 dst_port_443

Logs

Controller Logs

Check ESXi connectivity issues from the Controller

show log cloudnet/cloudnet_java-vnet-controller.<start-time-stamp>.log

NSX Manager

SSH (l: admin) to the NSX Manager and use the following command to access logs

nsxmgr-l-01a> show manager log follow

You can switch over to a unix shell using

nsxmgr-l-01a> enable
Password: <your NSX-Mgr pwd>
nsx_manager# st e
Password: <ASK NSX SUPPORT FOR PASSWORD>
[[email protected] ~]#

ESXi Logs

/var/log/esxupdate.log check this file if you have VIB installation issues
/var/log/vmkernel.log Distributed Firewall logs are sent to this file
/var/log/netcpa.log User World Agent logs

EAM Logs

EAM logs should be checked when installation of the VIBs module fails.

/storage/log/vmware/vpx/eam.log on Linux vCenter
ProgramData/VMware/VMware VirtualCenter/Logs/ on Windows vCenter

Advanced troubleshooting tips & tricks

If you want to troubleshoot your User World Agent, you can increase the netcpa log level like this:

Start by stopping the daemon

# /etc/init.d/netcpad stop

Enable write permisions on netcpa’s config file:

# chmod +wt /etc/vmware/netcpa/netcpa.xml

Increase log level:

# vi /etc/vmware/netcpa/netcpa.xml

Change the XML’s /config/log/level value to “verbose”, save and restart netcpad

# /etc/init.d/netcpad start

Todo

Section TBD :

  • Expand NSX Edge CLI
  • Expand Logs Section

Conclusion

This article is a work in progress. I hope you’ll accelerate your troubleshooting session by having a great understanding of all the internals.

Treinamentos gratuitos e online da VMWARE

Abaixo uma relação de cursos gratuitos da VMWARE, tudo feito online e a qualquer horário:

1 Data Center Virtualization Fundamentals [V6]  Show Offerings
Delivery Type: Self-Paced
 2. VMware vSphere: What’s New [V6] Fundamentals  Show Offerings
Delivery Type: Self-Paced
 3. Software-Defined Data Center (SDDC) – Overview  Show Offerings
Delivery Type: Self-Paced
 4. VMware Cloud Fundamentals  Show Offerings
Delivery Type: Self-Paced
 5. VMware Network Virtualization Fundamentals 2016  Show Offerings
Delivery Type: Self-Paced
 6. VMware Network Virtualization Fundamentals  Show Offerings
Delivery Type: Self-Paced
 7. VMware Horizon 7 Fundamentals  Show Offerings
Delivery Type: Self-Paced
 8. Virtual SAN Fundamentals [V6.2]  Show Offerings
Delivery Type: Self-Paced
 9. VMware Workforce Mobility Fundamentals  Show Offerings
Delivery Type: Self-Paced
 10. VMware Virtual SAN Fundamentals [V5.5]  Show Offerings
Delivery Type: Self-Paced
Role: Administrator
 11. VMware vRealize Automation Fundamentals  Show Offerings
Delivery Type: Self-Paced
 12. Horizon (With View) Fundamentals [V6.0]  Show Offerings
Delivery Type: Self-Paced
 13. VMware vCloud Director Fundamentals [V8.x]  Show Offerings
Delivery Type: Self-Paced
 14. vRealize Orchestrator for vRealize Automation [V6.1] Fundamentals  Show Offerings
Delivery Type: Self-Paced
Role: Administrator
 15. Software-Defined Storage Fundamentals  Show Offerings
Delivery Type: Self-Paced
 16. vCloud Air Fundamentals  Show Offerings
Delivery Type: Self-Paced
 17. VCAP Datacenter Design Simulation  Show Offerings
Delivery Type: Self-Paced
 18. VMware vRealize Operations Manager Fundamentals [V6.2]  Show Offerings
Delivery Type: Self-Paced
 19. vSphere 6.0 Product Walkthrough  Show Offerings
Delivery Type: Self-Paced
 20. VMware Site Recovery Manager [V6.1] Fundamentals  Show Offerings
Delivery Type: Self-Paced
 21. How to Navigate the vSphere Web Client  Show Offerings
Delivery Type: Self-Paced
 22. VMware Integrated OpenStack Fundamentals  Show Offerings
Delivery Type: Self-Paced
 23. vRealize Business for Cloud Fundamentals  Show Offerings
Delivery Type: Self-Paced
 24. VMware User Environment Manager Fundamentals [V9]  Show Offerings
Delivery Type: Self-Paced
 25. VMware Workforce Mobility Fundamentals (2016)  Show Offerings
Delivery Type: Self-Paced
 26. VMware vRealize Operations Manager Fundamentals [V6.0]  Show Offerings
Delivery Type: Self-Paced
 27. VMware vSphere: What’s New Fundamentals [V5.5]  Show Offerings
Delivery Type: Self-Paced
Product: VMware vSphere
Roles: Administrator, Architect
 28. Performing Common vCenter Server Tasks in the vSphere Web Client – Part 1  Show Offerings
Delivery Type: Self-Paced
 29. VMware vCloud Automation Center Fundamentals [V6.0]  Show Offerings
Delivery Type: Self-Paced
Product: VMware vCloud Automation Center
Role: Administrator
 30. VMware vRealize Operations For Horizon 6 Fundamentals  Show Offerings
Delivery Type: Self-Paced
31 vSphere Data Protection Advanced [V5.8] Fundamentals  Show Offerings
Delivery Type: Self-Paced
Roles: Administrator, Architect
 32. vRealize Log Insight [V3] Fundamentals  Show Offerings
Delivery Type: Self-Paced
 33. Business Continuity and Disaster Recovery Design [v5.X]  Show Offerings
Delivery Type: Self-Paced
Product: Site Recovery Manager (SRM)
Roles: Administrator, Engineer
 34. Test Course – VMware NSX: Install, Configure, Manage [V6.2] – On Demand Content  Show Offerings
Delivery Type: Self-Paced
 35. VMware ThinApp Fundamentals [V5.X]  Show Offerings
Delivery Type: Self-Paced
Products: VMware ThinApp, VMware View
Roles: Administrator, Architect, Engineer
 36. VMware vCenter Infrastructure Navigator Fundamentals [V2.X]  Show Offerings
Delivery Type: Self-Paced
Roles: Administrator, Architect, Engineer
 37. VMware App Volumes Fundamentals  Show Offerings
Delivery Type: Self-Paced
Role: Administrator
 38. vRealize Operations Manager What’s New [V6.0] Fundamentals  Show Offerings
Delivery Type: Self-Paced
Role: Administrator
 39. EVO:RAIL Fundamentals  Show Offerings
Delivery Type: Self-Paced
 40. VMware User Environment Manager: Fundamentals  Show Offerings
Delivery Type: Self-Paced
 41. How to Search using the vSphere Web Client  Show Offerings
Delivery Type: Self-Paced
 42. Sneak Peek – On Demand Classroom for VMware vSphere: Optimize and Scale [V6]  Show Offerings
Delivery Type: Self-Paced
 43. Performing a Virtual Machine Guest OS install with VMware Tools  Show Offerings
Delivery Type: Self-Paced
 44. How to use Quick Filters in the vSphere Web Client  Show Offerings
Delivery Type: Self-Paced
 45. Sneak Peek – VMware NSX: Install, Configure, Manage [V6.2]  Show Offerings
Delivery Type: Self-Paced
 46. Sneak Peek – On Demand Classroom for VMware vSphere: Install, Configure, Manage [V6]  Show Offerings
Delivery Type: Self-Paced
 47. VMware vCloud Director Fundamentals [V5.1/V5.5]  Show Offerings
Delivery Type: Self-Paced
Product: VMware vCloud Director
 48. Performing Common vCenter Server Tasks in the vSphere Web Client – Part 2  Show Offerings
Delivery Type: Self-Paced
 49. VMware Identity Manager Fundamentals  Show Offerings
Delivery Type: Self-Paced
 50. VMware vCloud Air Fundamentals  Show Offerings
Delivery Type: Self-Paced

Fonte: http://mylearn.vmware.com/portals/www/mL.cfm?menu=topfreecourses