Forum Discussion

asalamatov's avatar
asalamatov
Icon for Nimbostratus rankNimbostratus
Feb 03, 2020

Python SDK UCS download takes long time

Hello guys,

 

Did anybody implement file backups using Python SDK? UCS creation works fine for me, however file transfer takes forever.

I'm using example at https://f5-sdk.readthedocs.io/en/latest/userguide/file_transfers.html

I does create local file on my PC, but takes forever (20-40 mins) to download it. I'm on BIG-IP 13.1.1.4 Build 0.0.4

Looking for real world experience from folks.

5 Replies

  • Python SDK or iControl REST, which is the base of the SDK, is not optimal for downloading a large file.

     

    This is because there is a 1 MB size limitation in downloading or uploading a file. Any file larger than the limit must be chunked (split into small pieces and concatenate later on the receiving end) manually. See also Demystifying iControl REST Part 5: Transferring Files.

     

    Python SDK does the chunking for you (0.5 MB per chunk). Due to the chunking, it will create a numerous calls (including the one that gets the information on the file size) for a single file retrieval. For example, for a 1 GB file, it calls the iControl REST endpoint 1,910 times. That is not exactly a cheap overhead. In my test, it took 12s for scp and 92s for Python for a 1 GB file.

     

    Another factor that may affect file downloading performance is the system load. Since Python SDK interacts with a various components on the box, any resource contention (such as CPU and memory) would slow the job down (I guess 20-40 min in your case is this). Avoiding the busy time would be a good option.

     

    If you need speed and do not need to use iControl REST, scp (secure copy) is a good alternative solution for file downloadin.

  • I have observed delay while downloading via Python SDK to my PC. but all depend on the bandwidth speed.

     

    Try to download UCS/Qkview to same DC where LB reside.

    • asalamatov's avatar
      asalamatov
      Icon for Nimbostratus rankNimbostratus

      Could you share your code? I'm just curious if I'm missing something.

  • I cannot tell why one is faster than the other as I did not see much difference, at least on the iControl REST calls. Both access the same endpoint (/mgmt/shared/file-transfer/ucs-downloads) and do 512 kb chunking. Possibly the F5 SDK performs additional verification or throttling but I am not sure (I haven't read the code). I hope you can get information from the Github.