just curious how come server-side active connection is higher than client-side.
if you have qkview which was taken when memory usage was normal, you may compare bigpipe memory show output to see what different is.
anyway, i suggest opening a support case, so support engineer can assist you. there are a number of memory leak issue in the past.
e.g.
sol10995: TCP connections that do not terminate properly may become zombie connections and cause TMM to leak memory
http://support.f5.com/kb/en-us/solutions/public/10000/900/sol10995.html