New Secure option can break cookie persistence in v12

In v12 F5 introduced 2 new options into the cookie persistence profile. Secure and httponly. In a decidedly brilliant move (heavy sarcasm) F5 enabled them by default and didn’t document the change in behavior. As of 7/14/2017 this is missing from all v12 release notes. Hopefully this will be corrected in the near future. Better late than never right? Even as of this date, K83419154 (Overview of Cookie persistence) doesn’t specify when secure became a new option. Although, oddly it does for httponly. F5 also failed to create a known issue for these new options’ potential to break persistence in a certain scenarios. Because there is not a known issue article, it obviously can’t be linked from iHealth’s upgrade advisor.

Admittedly, I haven’t run into a situation where the httponly option will break persistence but the secure option most definitely can. This caught us completely unaware and left an entire data center half broke for almost 2 days after upgrading from 11.5.4 to 12.1.2. Before I get into the details of this just know that if all your HTTP/HTTPS VIPs force SSL, you are in the clear.

If you are not forcing SSL and your setup is what one F5 support engineer refers to as “extremely rare and a corner case” and looks like this:

VIP: 10.10.10.10:80 with default cookie persistence, Pool (test_pool): 10.1.1.10:8080, 10.1.1.11:8080, 10.1.1.12:8080, 10.1.1.13:8080

VIP: 10.10.10.10:443 with default cookie persistence, Pool (test_pool): 10.1.1.10:8080, 10.1.1.11:8080, 10.1.1.12:8080, 10.1.1.13:8080

If you hit the HTTP VIP first, the persistence cookie will look something like this:

Set-Cookie: BIGipServertest_pool=0000000000.00000.0000; path=/; Httponly \r\n

Notice that the secure option is missing. That’s because secure can’t be set on an HTTP connection. Now even if you switch to HTTPS, your browser will still send the BigIP persistence cookie.

BUT, if you hit the HTTPS VIP first, now there is trouble brewing. So, you hit the HTTPS VIP first and get the BigIP persistence cookie again looking something like this:

Set-Cookie: BIGipServertest_pool=0000000000.00000.0000; path=/; Httponly; Secure\r\n

Then you switch to HTTP. According to the RFC, any cookie with the secure option set, can’t be transmitted over an unsecured connection. So, you next request does not contain the persistence cookie and therefor the F5 will make a new load balancing decision and send a new cookie looking like this:

Set-Cookie: BIGipServertest_pool=0000000000.00000.0001; path=/; Httponly \r\n

Here’s where it gets interesting. If you are using IE, that browser will accept the new cookie and continue to persist with that cookie on HTTP or HTTPS since it no longer has the secure flag set. I am pretty sure that accepting of the new cookie is bad behavior. However, if you are using a recent version of Chrome or Firefox, those browsers will NOT accept the new cookie. Instead the browser decides to keep the one with the secure flag. To my thinking this is the way it should be so that sensitive data doesn’t go over an unsecured connection. Now, each request that the browser makes over HTTP will NOT contain a persistence cookie and the F5 will keep making new load balancing decisions and keep trying to set the persistence cookie without the secure flag. When you switch back to HTTPS, Firefox and Chrome will include the original persistence cookie (with the secure option set) in the requests.

In our case this affected shopping carts which created a horrible user experience and lost sales for us and our customers.

If you are in this situation like we are, how do you fix it for 1000+ VIPs that all use the default cookie persistence profile? This is what I did:

Create a new cookie persistence profile. create ltm persistence cookie cookie-secure_not_set secure disabled save sys config
Open up the bigip.conf file in your favorite editor and do a find and replace all then save. Find: /Common/cookie { Replace with: /Common/cookie-secure_not_set {
Load the modified config file. tmsh load sys config

This should probably be done on the standby device just in case something inadvertently gets messed up. You may also need to go into other cookie persistence profiles and change them as well.

I originally had more detail in this but hit Dev Central's character limit so I had to shorten it.