Forum Discussion

Felix888_164906's avatar
Felix888_164906
Icon for Nimbostratus rankNimbostratus
Aug 13, 2014

hairpin forwarding on the Nexus switch Show stopper here

Hello everyone:
   I have the communication problem between F5 and Nexus switches 5596, which already becomes my show stopper now. Basically I am trunking (LACP) between F5 LTM and Nexus with vPC. All virtual servers created in vCMP guest are not able to be accessed. Also I am unable to ping the self IP of the guest from the host or vice versa. By further investigation, it appears that the Nexus switch discards the ARP, because it doesn't know how to retransmit a frame out the same port on which it was received. I can see here there are lots of success cases of F5/Nexus combination networks. Does anyone has the similar situation during the initial network setup for F5?? Your help will be greatly appreciated.
P.S. the Cisco TAC has been opened, but the case is still under investigation. I would also through a question here in case anyone has known this already. Thanks! 

9 Replies

  • I think there are two ways to set-up interface mappings to guests on vCMP platforms but unfortunately I'm don't know enough to advise. Worth looking into.

     

    A few questions on the F5 to Nexus stuff;

     

    • Are you tagging multiple VLANs?
    • Are you tagging the native VLAN on the Nexus (or not)?
    • What is generating the ARP, it's not clear?
    • Why would an ARP request hairpin?
  • Thanks for the reply. To answer your question:
    1. Yes, I have about 20 VLANs, they are all tagged to the trunking port from F5 to Nexus. The Nexus switch is Layer 2.
    2. I tag the native VLAN on the Nexus side, but I don't tag native in F5. Do I need to do it?
    3. 
      When I ping from vCMP from Host (10.0.2.1) to Guest (10.0.2.3), by tcpdump I see the guest replies the arp:
    09:51:55.095627 arp who-has 10.0.2.3 tell 10.0.2.1
    09:51:55.095641 arp reply 10.0.2.3 is-at 00:11:d3:89:04:45
    09:51:56.096234 arp who-has 10.0.2.3 tell 10.0.2.1
    09:51:56.096247 arp reply 10.0.2.3 is-at 00:11:d3:89:04:45
    09:51:57.095798 arp who-has 10.0.2.3 tell 10.0.2.1
    but the host doesn't get the reply.
     Also all virtual server created in the guest cannot be accessed by the servers which connect to the Nexus switch.
     4. This is what Cisco and I assume: when the vCMP response the ping from the host, it sends the arp to the Nexus, Nexus will broadcast to the rest of 47 ports (48 ports switch), based on the Cisco (and F5) it will not retransmit the frame to the port it receives on. So there must be a way to let the Nexus ports doing so called hairpining. Because there is no virtual switch in F5 (like ESX / VMware) all the virtual servers on the guest have to use the Nexus port for internal switching. But the Nexus simply either never response or discard the arp as the flooding.
     This is the Nexus 5596, I setup the etherchannel trunk with LACP and vPC on it.  The switches seem work OK, it is layer 2 configured. 
     Thanks
    
  • I've got something improved, although I am still unable to ping from the host to the guest. But some other servers from the remote are able to ping the virtual servers in the guest. The solution is simply: Just use the self-IP of the guest as the gateway for the real servers in the vCMP guest. Isn't that simply? Previously I used the self-IP of the host as the gateway.
    Now I am facing to problems:
    we have the admin vlan in the vCMP host, say valn 2. This vlan2 needs to see all network and servers for admin purposes. The server in these vlan 2 connecting the Nexus has been configured to use the vCMP host self-IP as the gateway. I found this server cannot access to any real or virtual servers in the guest. Unless I add the static route in the server, basically like this:
    route add (guest vlan network) mask 255.255.255.0 guest-self-ip
    This way this vlan 2 server can ping the servers in the guest. Obviously I can not do this for hundreds servers in the future. Any suggest? there are still issues forwarding the traffic from ther host to the guest. As I am still unable to ping from the host to the guest.
    2. the real servers in the guest is unable to access to the Internet. The servers are able to forward the internal traffic through the self ip from the guest to the host for all vlans except the Internet. But it won't forward the Internet traffic from the guest to the host. I have route setup to route all vlans through the address pool which contains all selfIP of the vCMP host (for routing to the next hop). I also setup the IP forward VS to forward all IP traffic. 
    But the server is unable to ping the Internet. 
    Please help!!
    
  • I figure out most of the problems myself, unfortunately there is almost no help from F5. Choosing the proper gateway fixed all routing problems and apply SNAT fixed the Internet issues. I still can't ping from host to guest but this doesn't stop me from going forward.

     

    • boneyard's avatar
      boneyard
      Icon for MVP rankMVP
      thanks for reporting back. are you sure you should be able to ping from the host to the guest?
  • If you don't mind me saying so I think you've rather rushed your design and done it 'on the fly'. You've got it working but I'm not sure its ideal. I don't know enough about your requirements or vCMP to really comment myself but perhaps it would be a good idea to take a step back and consider if what you've ended up with is actually suitable for your needs?

     

    A few points;

     

    • I don't think you'd put a VMWare host on the same network as the Virtual Machines/Guests running on it, not sure why you'd want to do so in this case with vCMP.
    • Will you have to manage your VM servers through the F5? Is this desirable?
    • Wouldn't it be better to SNAT at a firewall or something else?
    • Do you have a 'routing' VS for that outbound traffic?
    • You've effectively created a 'stub' network, is that what you want?

    You get the idea, just something to consider.

     

  • Hi What Lies Beneath:
    
    Thanks for the reply!
    Yes, I am doing thing rush at the moment, because my schedule is very tight :(
    To answer your question:
    1. VMware shares the same IP ranges with F5 Virtual host and guests. 
    2.I think VMware guest can be managed by F5 by setting up the default gateway of VMware guests all pointing to F5 guests' self IP on the according vlan.
    3. I did the snat at the F5 so now all F5 guest servers are able to access to the Internet and also internal network. I need to do the security policy for the restriction among the vlans. I don't think I need the route domain, because I don't have the overlapping ip addresses.
    4. I did setup a bunches of the routing VS to make all routing works.
    5. I am not quite sure the stub network means at this case, did you mean the F5 guest network? That is the vCMP guest created on the host...
    Anyway please help me as I am in a rush, have to make this work sooner. Sorry for this and thanks again!!!