Forum Discussion

Scott_C_16492's avatar
Scott_C_16492
Icon for Nimbostratus rankNimbostratus
Sep 05, 2008

Error Message

Hi

 

 

I was doing a test last night of taking one of my caching dns servers offline. The majority of dns goes via a LTM via the loadbalanced ip 10.15.10.155 (to .156 and .157).

 

 

During the test, the LTM was spitting out a lot of these errors:

 

 

2008-09-04T16:31:06+01:00 tmm tmm[1541]: 01010201:2: Inet port exhaustion on 10.15.11.243 to 10.15.10.156:53 (proto 17)

 

2008-09-04T16:31:06+01:00 tmm tmm[1541]: 01010201:2: Inet port exhaustion on 10.15.11.243 to 10.15.10.156:53 (proto 17)

 

2008-09-04T16:31:06+01:00 tmm tmm[1541]: 01010201:2: Inet port exhaustion on 10.15.11.243 to 10.15.10.156:53 (proto 17)

 

2008-09-04T16:31:06+01:00 tmm tmm[1541]: 01010201:2: Inet port exhaustion on 10.15.11.243 to 10.15.10.156:53 (proto 17)

 

2008-09-04T16:31:06+01:00 tmm tmm[1541]: 01010201:2: Inet port exhaustion on 10.15.11.243 to 10.15.10.156:53 (proto 17)

 

 

What does that mean, and how can I fix it?

 

 

Scott

8 Replies

  • Jesse_42849's avatar
    Jesse_42849
    Historic F5 Account

     

    Are you SNAT'ing inbound connections to the servers through the VIP to make them appear as a single address? If so, my first thought would be the near-complete utilization of ephemeral ports available for use by an inbound SNAT. This would lead to the SNAT being unable to handle additional connections until some source ports become available. You might not see this with multiple servers in the pool because the inbound connections are split between two servers, allowing each port to be used twice, once per internal server. When reduced to just one active server the inbound ephemeral port would be the only point of uniqueness among all inbound DNS requests and could not be used more than once.

     

     

    If the above scenario is accurate you may be able to handle the issue with the following options: Since it's DNS traffic I'd make sure that the protocol timeout in the UDP profile assigned to the VIP is 2 to 5 seconds. If the problem continues I'd place the existing SNAT address into a SNAT pool with at least one other address, then assign the SNAT pool to this VIP.

     

     

    If you are not SNAT'ing your inbound requests I would need more information about your config.

     

     

     

    --jesse
  • I am geeting following error meny time on LTM 6800 Box, what does "Inet port exhaustion on x.x.x.x to x.x.x.x (proto 6)" mean?

     

     

    Jan 31 13:26:35 tmm tmm[18353]: 01010201:2: Inet port exhaustion on 10.199.178.9 to 200.199.178.105:80 (proto 6)

     

    Jan 31 13:26:39 tmm tmm[18353]: 01010201: repeated 393 times
  • Same thing as Jesse explained above, with the TCP protocol (TCP is proto 6).

     

     

    Denny
  • You config looks correct. What do you see in a tcpdump?

     

     

    tcpdump -nni 0.0 -Xs0 "port 53 and (host CLIENT_IP or host 169.143.48.22 or host 169.143.48.23 or host 169.143.48.24 or host 169.143.48.25)"

     

     

    Aaron
  • Jesse,

     

     

    I faced the same issue , seems to me there was a burst of traffic from a particular server to vip ( SNATed into a single ip ) , the ports ( 65535 ) was exhausted and threw the error and all subsequent connections dropped until the ports were freed.

     

     

    My query is , is there a command to see the table which has the mapping of how many ports are free / being used ?

     

    something like a session table.

     

     

    Cheers,

     

    Roger
  • The b conn command can be used to check the connection table. But it's limited to displaying ~7k requests so I'm not sure how useful this would be in determining the number of open connections.

     

     

    Anyone have ideas on this?

     

     

    Aaron
  • Aaron,

     

     

    how abt this

     

    b conn client protocol tcp

     

    this returned all the connections to the vip ( and showed me the ports that were being used )

     

     

    Cheers,

     

    Roger
  • That's what I was thinking originally, and it will work well up until you have ~7k connections from the SNAT address. But if you're getting anywhere near the point of exhaustion, 'b conn' (with any or no options) won't be able to display all of the current connections from the SNAT address.

     

     

    Aaron