Are you SNAT'ing inbound connections to the servers through the VIP to make them appear as a single address? If so, my first thought would be the near-complete utilization of ephemeral ports available for use by an inbound SNAT. This would lead to the SNAT being unable to handle additional connections until some source ports become available. You might not see this with multiple servers in the pool because the inbound connections are split between two servers, allowing each port to be used twice, once per internal server. When reduced to just one active server the inbound ephemeral port would be the only point of uniqueness among all inbound DNS requests and could not be used more than once.
If the above scenario is accurate you may be able to handle the issue with the following options: Since it's DNS traffic I'd make sure that the protocol timeout in the UDP profile assigned to the VIP is 2 to 5 seconds. If the problem continues I'd place the existing SNAT address into a SNAT pool with at least one other address, then assign the SNAT pool to this VIP.
If you are not SNAT'ing your inbound requests I would need more information about your config.
--jesse