Steps:
- Split file
- Create upload id
- Upload all file parts
- Compose json file with all Key Tags
- Complete upload
- Verify
The below explained multipart upload procedure using s3api should be used only when file cannot be uploaded to S3 using high level aws s3 cp command. To perform the multi-part upload using s3api, first split the file into smaller parts. In this example, I’m going to upload file Cambridge.pdf to the S3 bucket awsmultipart.
I have split my file Cambridge.pdf which is about 86MB in size into 4 pieces. First 3 file parts are 25MB each and 4th piece is about 11MB. The command I issued to create fileparts is split -b 25m Cambridge.pdf piece- where Cambridge.pdf is the filename, 25m is the size of file parts and piece- is the prefix for each split file. The 4 fileparts generated from above command are piece-aa, piece-ab,piece-ac,piece-ad.
Next, create the multipart upload job using command aws s3api create-multipart-upload –bucket awsmultipart –key Cambridge.pdf. Replace awsmultipart with your bucket name and Cambridge.pdf with your original unpartitioned file name. You will see the output as below. Copy the UploadID generated as you will need it to upload each individual partitions to S3.
You can now upload each individual file parts to S3 using the command aws s3api upload-part –bucket awsmultipart –key Cambridge.pdf –part-number 1 –body piece-aa –upload-id youruploadid. In the above command, replace the bucket name, original file name, part number, partitioned file name and upload id with appropriate values. Each file part upload,if successful, will generate an ETag.
Whereas the upload id, bucket and key values remain the same for all subsequent file part uploads, you have to change part-number and body values for each partition upload.
Once all file parts are uploaded, check the status by running the command aws s3api list-parts –bucket awsmultipart –key Cambridge.pdf –upload-id youruploadid .
You will see the output as shown in the below image.
Create a file pieces.json as shown below using the output generated from list-parts command.
Finally, to complete the upload, run the command aws s3api complete-multipart-upload –multipart-upload file://pieces.json –bucket awsmultipart –key Cambridge.pdf –upload-id youruploadid.
Verify the uploaded file size by logging into the Management console.
