Author:
erics, September 9th, 2021
The aws s3 sync command is slow and painful! I needed a more efficient way to sync to large buckets (prod to dev). Finally settled on RClone: https://rclone.org/docs/
|
|
cd {extracted zip dir} ./rclone config cat ~/.config/rclone/rclone.conf |
During rclone config I called remote “s3” ;-}
|
|
./rclone sync -v --progress --fast-list --checksum s3:wyzaerd-demo-prod s3:wyzaerd-demo-dev |
Categories: How-To's, Technology Tags: AWS, Bucket, howto, rclone, rsync, S3, s3s3, s3s3mirror, Sync, tips
| 
No comments
Author:
erics, July 28th, 2021
SUMMARY: Needed to create an AWS IAM Policy to allow a user access to two buckets and their associated CDN’s in WordPress W3 Total Cache
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::wyzaerd-demo-prod" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::wyzaerd-demo-prod/*" }, { "Sid": "VisualEditor2", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::wyzaerd-demo-dev" }, { "Sid": "VisualEditor3", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::wyzaerd-demo-dev/*" }, { "Sid": "VisualEditor4", "Effect": "Allow", "Action": [ "s3:ListAllMyBuckets", "cloudfront:ListDistributions" ], "Resource": "*" } ] } |
Categories: How-To's, Technology Tags: Amazon, AWS, Bucket, CDN, CloudFront, howto, IAM, JSON, Policy, S3, tips, User, W3 Total Cache
| No comments
Author:
erics, October 3rd, 2019
To sync various cloud resources, use the excellent cli tool rclone from https://rclone.org/docs/ For this use case, the need was to sync from Box to an AWS S3 bucket. Install rclone:
|
|
curl https://rclone.org/install.sh | sudo bash |
Configure both S3 and Box – for remote name labels I just used “S3” and “Box”:
Validate Access and Functionality:
|
|
rclone lsd Box: rclone lsd S3: |
Perform […]
Categories: How-To's, Technology Tags: Amazon, AWS, AWS S3, Box, Clone, rclone, rsync, S3, Sync
| No comments
Author:
erics, August 15th, 2013
|
|
yum install python python /usr/bin/easy_install pip pip install boto |
Categories: How-To's, Technology Tags: Amazon, AWS, Boto, EC2, howto, Install, Linux, pip, Python, S3, tips, Yum
| No comments
Author:
erics, June 7th, 2013
Install Procedure Download and extract into /usr/local: http://downloads.sourceforge.net/project/s3tools/s3cmd/1.0.1/s3cmd-1.0.1.tar.gz
|
|
cd /usr/local/bin ln -s /usr/local/s3cmd-1.0.1/s3cmd /usr/local/bin/s3cmd --configure Access Key: {ACCESS_KEY} Secret Key: {SECRET_KEY} Encryption password: {PASSWORD} |
Configuration saved to ‘/root/.s3cfg’ Command Line Examples List all buckets: s3cmd ls Make a new bucket in the default region (US): s3cmd mb s3://{YOUR_BUCKET_HERE} Make a new bucket in a specific region: s3cmd mb –bucket-location={DATACENTER} s3://{YOUR_BUCKET_HERE} As of now the datacenters are: US (default), EU, us-west-1, […]
Categories: How-To's, Technology Tags: Amazon, AWS, howto, Install, rsync, S3, s3cfg, s3cmd, s3sync, s3tools, Sync, tips
| No comments
Author:
erics, May 31st, 2012
1. You need Ruby, rubygems and the right_aws gem. yum install ruby rubygems gem update –system gem install right_aws 2. Replace any values inside curly braces {} below with real values and remove the curly braces. YMMV
|
|
require 'rubygems' require 'right_aws' S3ID = "{S3 ID GOES HERE}" S3KEY = "{S3 KEY GOES HERE}" SRCBUCKET = "{your-s3-bucket-1}" DESTBUCKET = "{your-s3-bucket-2}" s3 = RightAws::S3Interface.new(S3ID, S3KEY) s3.incrementally_list_bucket(SRCBUCKET) do |h| h[:contents].each do |o| puts('Copying ' + o[:key]) s3.copy(SRCBUCKET, o[:key], DESTBUCKET, o[:key]) end end puts('Done.') |
Categories: How-To's, Technology Tags: Amazon, AWS, Bucket, Copy, gem, right_aws, Ruby, rubygems, S3
| No comments
Author:
erics, May 3rd, 2012
I found a fantastic tutorial on using the W3 Total Cache plugin with Amazon’s S3/CDN that I had to share: http://www.makeuseof.com/tag/give-wordpress-blog-lightning-fast-speeds-w3-total-cache/ The basic gist of the article says to do the following: Install and activate the W3 Total Cache plugin Sign up for Amazon Web Services (AWS) Obtain your AWS Access and Secret key pair […]
Categories: How-To's, Technology Tags: Amazon, AWS, Cache, CDN, CloudFront, howto, S3, tips, WordPress
| No comments
Author:
erics, April 2nd, 2012
To be performed as the root user:
|
|
yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap fuse fuse-devel wget http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz tar xvzf s3fs-1.61.tar.gz cd s3fs-1.61/ ./configure --prefix=/usr make make install vim /etc/passwd-s3fs chmod 640 /etc/passwd-s3fs s3fs yourBucketNameHere /mnt |
http://code.google.com/p/s3fs/wiki/FuseOverAmazon The s3fs password file has this format (use this format if you have only one set of credentials): accessKeyId:secretAccessKey If have more than one set of credentials, then you can have default credentials as specified above, but this syntax will be recognized as well: bucketName:accessKeyId:secretAccessKey
Categories: How-To's, Technology Tags: Amazon, Amazon Linux, AWS, fuse, gcc, howto, S3, s3fs, tips, Yum
| 5 comments
