Forum Discussion

Greg_Jewett's avatar
Greg_Jewett
Icon for Cirrus rankCirrus
Jul 30, 2024

AS3 w/ certificates and renewals..

So, I found myself in a little bit of a quandary with the use AS3 declarations to deploy our F5 configurations for our services.  

So to create a virtual server with SSL certificate and profiles, and the nine-yards, you need to have as part of your AS3 declaration:  SSL certificate (key and cert), that populate the profile, that then populates the profile section within the virtual server.

So far so good...

Now, the certificate has a TTL (if you will), and needs to be renewed.  

In the past, I had a Python script that goes through the F5 using REST API to find expiring certificates and get new certs and updates the configuration.  That worked just fine, and I have adapted that to be used on our new F5s using partitions/tenants, and it works.

Unfortunately is also breaks the one source of truth (AS3), so if I go make a change to an AS3 declaration to make pool member or other configuration changes, and I then redeploy the AS3 declaration, then the OLD certificate if put back into play (which could be expired) and the service goes down.  Has not happened yet, because this FUBAR situation popped into my head.

..and that is my quandary...

How do I redeploy configurations and have it ignore the certificate and profile stanzas in AS3 declaration - so it  does not redeploy an old and possibly expired certificate?

One may think .. well update the declaration with the updated certificate.  Not as easy as one may think.  I would have to do this for any declaration that I want to modify, not a easy task.. log-into the F5, fetch the new PEMdata for the certificate and key, update the declaration, and then deploy it.  In some AS3 declarations, I am defining multiple environments for a service, and there might be up to 5 different certificate/key pairs that I would need to update prior to redeploying .. ugh!

I am in a pickle.   Thoughts?  

The only solution that I have been able to conjure up in my head is additional automation and scripting that would automatically update the AS3 declarations when a certificate is renewed, which makes sense .. just have no idea how to go about this just yet.

Hoping there are other alternatives?!

AS3
certificate
devops
ssl
  • Greg_Jewett's avatar
    Greg_Jewett
    Aug 07, 2024

    I appreciate the response.   We have already made an extensive investment in AS3, so we are not backing out of that.  For the new setup, we are forgoing the Python script that we used in the past to renew certificates - due to it breaking the source of truth of AS3.

    So I have some ideas, and we do have the Ansible Automation Platform (AAP), so I think we will be writing a playbook that will basically do the following:

    1. Use a home-made python based AAP plugin that will scan all the files in our GitHub repository.  It will return a list of files that have expired certificates within, based on a threshold date. 
    2. Read in each file into a JSON variable, this would be the entire declaration.  I am doing this, so it will be easy to literally pull out the certificate(s) into a separate variable using JSON/YAML references. 
    3. Pull out the Common Name and any SANs from the existing certificate.
    4. Generate a new CSR / private key based on info above.
    5. Using ACME, generate a new certificate.
    6. Insert the certificate and new key into the JSON/YAML structure (again using the references, using an assignment).
    7. Write out the JSON/YAML structure to a file, and upload up to Github, replace what exists (new version, and all).
    8. Do a push and pull request (all automated).
    9. Then run the two plays I have that fetch the declaration from GitHub and deploy it (thus renewing the certificate) -- all using the source of truth.

     

    I am sure you saw that I am uploading the certificate and key to the GitHub repository.  Right now  -- yes -- probably a bad practice, but one we are following until we get everything migrated.   The GitHub repository is private to myself and the service owners (white list access only), which is a stopgap.   During and after migration -- I will be working on pulling the private keys out, and putting references there instead, that I would read and fetch the appropriate private key from a safer storage location, and then insert it into a read in declaration  -- then deploy it, all from memory, and the private key is purged from memory once the playbook ends.

  • Juergen_Mang's avatar
    Juergen_Mang
    Icon for MVP rankMVP
    Aug 07, 2024

    I manage the certificates separately from the AS3 declarations in the /Common partitions.

    An other idea would be to keep only the private key in the /Common partition and include only the certificate in the declaration. I think it is the best if the private key does not float around and is only kept on the F5.

  • Zdenek's avatar
    Zdenek
    Icon for Cirrus rankCirrus
    Aug 07, 2024

    Does it make any sense?

    I would be happy if you shared the solution you chose and how satisfied you are with that. We are staying away of AS3 at the moment, basically waiting how it goes with it and especially with NEXT

    • Greg_Jewett's avatar
      Greg_Jewett
      Icon for Cirrus rankCirrus
      Aug 07, 2024

      I appreciate the response.   We have already made an extensive investment in AS3, so we are not backing out of that.  For the new setup, we are forgoing the Python script that we used in the past to renew certificates - due to it breaking the source of truth of AS3.

      So I have some ideas, and we do have the Ansible Automation Platform (AAP), so I think we will be writing a playbook that will basically do the following:

      1. Use a home-made python based AAP plugin that will scan all the files in our GitHub repository.  It will return a list of files that have expired certificates within, based on a threshold date. 
      2. Read in each file into a JSON variable, this would be the entire declaration.  I am doing this, so it will be easy to literally pull out the certificate(s) into a separate variable using JSON/YAML references. 
      3. Pull out the Common Name and any SANs from the existing certificate.
      4. Generate a new CSR / private key based on info above.
      5. Using ACME, generate a new certificate.
      6. Insert the certificate and new key into the JSON/YAML structure (again using the references, using an assignment).
      7. Write out the JSON/YAML structure to a file, and upload up to Github, replace what exists (new version, and all).
      8. Do a push and pull request (all automated).
      9. Then run the two plays I have that fetch the declaration from GitHub and deploy it (thus renewing the certificate) -- all using the source of truth.

       

      I am sure you saw that I am uploading the certificate and key to the GitHub repository.  Right now  -- yes -- probably a bad practice, but one we are following until we get everything migrated.   The GitHub repository is private to myself and the service owners (white list access only), which is a stopgap.   During and after migration -- I will be working on pulling the private keys out, and putting references there instead, that I would read and fetch the appropriate private key from a safer storage location, and then insert it into a read in declaration  -- then deploy it, all from memory, and the private key is purged from memory once the playbook ends.

      • Zdenek's avatar
        Zdenek
        Icon for Cirrus rankCirrus
        Aug 07, 2024

        Thanks, and I appreciate your answer :)

        Regarding the private key, yeah that is probably optimal way to go if cert/key is to be placed into declaration. We would do it very similar way. But still, even though it is securely stored it has to be read, that means it is sent through the wire...

        Once we deep dive into this, we will most probably go the way to keep certs and keys in Common partition, ideally the private key is not ever sent anywhere. But I realize I might be wrong and I like to see other people's approaches.

         

        Thanks, and good luck with your solution.

        Zdenek

  • Zdenek's avatar
    Zdenek
    Icon for Cirrus rankCirrus
    Jul 31, 2024

    Well you dont have many options to solve it. This is something I was already thinking about, discussed with few F5 guys and here are the options:

    1. add SSL cert and key to every declaration where needed and (re)deploy declaration
      • Not that easy as you wrote, but could be done if you use automation
      • you do NOT want to have SSL private key anywhere in the declaration. SSL private key is not made to be stored just somewhere, you probably know..
    2. store SSL certificate and key to Common and use it in AS3 declaration
      • if you use partitioned environment then your Common partition might be messed up with certificates used by services in the partitions
      • any SSL certificate name change (we for example include some tags and date in the cert name as part of naming standard - automated) needs AS3 declaration update

     

    I also use python to automate SSL cert lifecycle, and this is one of the reasons we have stayed away from AS3.

    We have defined source of truth and our way of automation, but with AS3 we are forced to do it differently. And dont forget that with Bigip NEXT you have no Rest API, but just AS3...

     

    This is pain yes, also to us, would be nice to get more people involved in this thread.