Boto3 and botocore need upgrading #356

Activity

1. thomas-icomplai commented on Jul 7, 2022

   

   thomas-icomplai

   on Jul 7, 2022

   More actions

   is there a way to upgrade this yourself?
2. mdavis-xyz commented on Jul 8, 2022

   

   mdavis-xyz

   on Jul 8, 2022

   Author

   More actions

   There's probably a way.
   Note that the normal image doesn't include pip. The build one does though.

   Note also that the version of botocore and boto3 in lambda tends to lag behind the latest version of the libraries. So you can't simply upgrade to the latest, you must pin the versions. I was hoping that's something that this project would handle for me, by just copy-pasting the files from the Lambda runtime regularly.
3. thomas-icomplai commented on Jul 8, 2022

   

   thomas-icomplai

   on Jul 8, 2022

   More actions

   I worked around this now with the wonderful blog post from https://www.mandsconsulting.com/lambda-functions-with-newer-version-of-boto3-than-available-by-default/

   effectively just boils down to prioritize the pip installed packages before importing them:

   if "LAMBDA_TASK_ROOT" in os.environ:
       envLambdaTaskRoot = os.environ["LAMBDA_TASK_ROOT"]
       sys.path.insert(0, "/var/lang/lib/python3.9/site-packages")
   
   import boto3
   import botocore
   

   It's a real shame that the runtime does not do this itself before executing the script.
4. Homas commented on Dec 26, 2022

   

   Homas

   on Dec 26, 2022

   More actions

   You can add a layer with updated packages
   On Linux:

   mkdir -p lambda_layer/python
   cd lambda_layer/python
   pip install boto3 -t ./
   pip install botocore -t ./
   cd ..
   zip -r ~/lambda_layer.zip .
   

   after that upload lambda_layer.zip as a layer to your function (it is relatively small so no need to store on S3)
5. mdavis-xyz commented on Dec 26, 2022

   

   mdavis-xyz

   on Dec 26, 2022

   Author

   More actions

   From memory boto3 and botocore and not relatively small. They're like 60MB, right? So that's 1/8th of the size quota gone just to the AWS SDK. If you're trying to also squeeze in some other library like matplotlib or awswrangler, that's going to pose a challenge for real deployments.

   Although this project is for unit testing. so size isn't an issue. the issue is just that the docker container doesn't accurately reflect the environment its emulating. Installing the latest version of boto3 and botocore won't either, since real lambda tends to lag the latest SDK versions.

   Also, you should probably install boto3 and botocore in one single pip command, to ensure the two versions are compatible with each other.