Enabling Maintenance Mode - Can this apply to AFM rules?

A firewall rule wouldn't be the place to put this. If you want to bleed off your connections your best option would be to disable the pool members for the relevant Virtual Servers. We will refuse all new connections, but still allow active connections and connections with existing persistence records. Over time those should also fade. If you want something a bit more drastic you can force the pools offline which only allows current active connections.

If you have a GTM you can use that to force all traffic to a different data center, allowing your system to be worked on that way. Once again it should only impact new connections, allowing traffic to bleed off naturally.

The firewall rules really either allow the traffic through or don't. Most of the actual intelligence lies with the BigIP after you get through the gate.

You can also fail the unit over to its peer. Depending on your traffic, this will either be virtually seamless or very impacting. If you can give me more information on what you are trying to do I can likely be a bit more specific.