Upload files or folders (even with subfolders) to Amazon S3 in a totally automatized way taking advantage of:
-
Amazon S3 Multipart Upload: The uploaded files are processed transparently in parts improving the throughput and the quick recovery from any network issues.
-
Resilent Retry System: Intelligent S3 Upload has been built to detect any error during the uploading process and to perform any retries whenever is necessary.
-
User Friendly Interface: Just check the demo to see with your own eyes how the upload process is performed.
Installation
Clone the repository
git clone https://github.com/polius/intelligent-s3-upload.git
Install the dependencies
python3 -m pip install boto3 --user
python3 -m pip install requests --user
Setup
Before executing the Intelligent S3 Upload, modify the credentials.json file.
{
"aws_access_key_id": "",
"aws_secret_access_key": "",
"region_name": "",
"bucket_name": "",
"bucket_prefix": "",
"storage_class": "",
"skip_s3_existing_files": true,
"server_side_encryption": true,
"slack_url": ""
}
- aws_access_key_id | aws_secret_access_key: Credentials generated by Amazon IAM.
- region_name: The AWS Region Code where the bucket is located.
- bucket_name: The bucket name created by Amazon S3.
- bucket_path: (Optional) The bucket path to store the uploaded objects.
- storage_class: The type of storage to use for the uploaded object. These are the possible values:
| storage_class |
|---|
| STANDARD |
| REDUCED_REDUNDANCY |
| STANDARD_IA |
| ONEZONE_IA |
| INTELLIGENT_TIERING |
| GLACIER |
| DEEP_ARCHIVE |
| OUTPOSTS |
- skip_s3_existing_files: Skip uploading objects if these already exists in S3. Possible values: [ true | false ]
- server_side_encryption: Enable Server-side encryption using the Amazon S3 key (SSE-S3). Possible values: [ true | false ]
- slack_url: (Optional) Enter a Webhook URL to send a message to Slack when a upload finishes.
AWS Policy
To be able to run the script check that your policy meets the following requirements:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:ListBucketMultipartUploads",
"s3:AbortMultipartUpload",
"s3:ListBucket",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::yourbucket/*",
"arn:aws:s3:::yourbucket"
]
}
]
}
Execution
python3 upload.py --path "{PATH}"
Replace the {PATH} string with the absolute file/folder path.
License
This project is licensed under the MIT license. See the LICENSE file for more info.
1 Dec 13, 2021
428 Jan 8, 2023
71 Jul 19, 2022
1 Nov 6, 2021
0 Dec 18, 2021
4 Dec 10, 2021
240 Jan 6, 2023
219 Oct 3, 2022
59 Nov 27, 2022
3 Feb 28, 2022
39 Jan 01, 2023
26 Nov 30, 2022
3.2k Dec 31, 2022
1 Jan 06, 2022
56 Jan 02, 2023
12 Nov 20, 2022
8 Sep 18, 2021
14.2k Jan 03, 2023
6 Oct 21, 2022
204 Apr 27, 2022
32 Oct 11, 2022
4 Feb 05, 2022
173 Dec 19, 2022
2 Dec 18, 2021
129 Dec 31, 2022
0 Sep 27, 2022
4 Jan 10, 2022
6 Apr 20, 2022
206 Jan 03, 2023