How to make a http switching iRule that can deal with websocket requests?

If you upgrade to at least 11.5 there is "native support" for Web sockets. I assume the you get connection reset when you try the rule above?

The problem is that the initial request from the client is over HTTP but then the connection gets "upgraded" and suddenly the F5 is not able to decipher the traffic from the client.

Since you're basing your pool decisions on host header it means that whenever the browser wants to connect to a resource in a different pool a new connection must be opened anyway, so there would be no critical loss in disabling HTTP after choosing the pool (unless you have other rules, policies etc depending on http profiles).

If you're lucky this might work:

when HTTP_REQUEST { 

 Check requested host header (set to lowercase) 
    switch [string tolower [HTTP::host]] {

        "fsbp.abc.com" { 
            pool pool_fsbp.abc.com_dmz
        } 
        "vwcc.abc.com" { 
            pool pool_vwcc.abc.com_dmz 
        } 
        "afms.abc.com" { 
            pool pool_afms.abc.com_80_dmz 
        }
        "kf1.abc.com" { 
            pool pool_kf1.abc.com_80_dmz 
        } 
        "kf2.abc.com" { 
            pool pool_kf1.abc.com_80_dmz 
        } 
  }
  After choosing the pool, disable the HTTP profile.
  HTTP::disable
}

Hope that helped.

/Patrik