Mount AWS S3 Bucket On Amazon EC2

Introduction:

Amazon S3 (Simple Storage Service) is a highly scalable and cost-effective cloud storage service provided by AWS. It allows users to store and retrieve any amount of data at any time from virtually anywhere on the web. S3FS is a tool that enables you to mount an S3 bucket as a local file system on an Amazon EC2 instance running Linux. In this blog, we will guide you through the steps to mount an AWS S3 bucket on an EC2 Linux instance using S3FS, leveraging IAM roles for secure access.

Let’s explore the versatile use cases where this setup can significantly benefit your AWS infrastructure

Use Cases for Mounting S3 Bucket on EC2:

1. Data Backup and Archiving: Efficiently store and retrieve data from S3 to perform reliable backups and long-term archiving.
2. Big Data and Analytics: Directly access large datasets in S3 for seamless data processing and analytics.
3. Web Hosting and Content Distribution: Serve static content from S3 to host websites and media files efficiently.
4. Log Collection and Analysis: Centrally store logs in S3 for easy log analysis and monitoring.
5. File Sharing and Collaboration: Share and collaborate on files among multiple EC2 instances using S3 as a shared data repository.

Prerequisites:

1. An Amazon EC2 instance running Linux (Amazon Linux, Ubuntu, CentOS, etc.).
2. IAM role attached to the EC2 instance with appropriate permissions to access the S3 bucket.

You can use the default AmazonS3FullAccess policy or either create your custom policy below:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::<your-s3-bucket-name>"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": ["arn:aws:s3:::<your-s3-bucket-name>/*"]
    }
  ]
}


Note: You can change the policy or restrict the permission accordingly 

Step 1: Install S3FS on EC2 Linux Before mounting the S3 bucket, we need to install S3FS on the EC2 instance. Follow these steps to install S3FS:

1. Connect to your EC2 instance using SSH.
2. Update the package manager:

sudo apt-get update -y
sudo apt-get install awscli -y
sudo apt-get install s3fs -y

Step 2: Mount S3 Bucket

Now that S3FS is installed, we can mount the S3 bucket as a local file system. Since we are using an IAM role, there’s no need to configure AWS credentials explicitly.

1. Create a mount point directory for the S3 bucket:

sudo mkdir /mnt/s3-bucket

2. Go to your home directory and run the below commands to create a new directory and generate some sample files

cd /mnt/s3-bucket ;touch test1.txt test2.txt 

3. Now run the below command to sync the local directory with the S3 bucket to check the access

aws s3 sync /mnt/s3-bucket s3://<your-s3-bucket-name>

Step 3: Now you can run the command to mount the S3 bucket as a filesystem

sudo s3fs <your-s3-bucket-name> /mnt/s3-bucket -o iam_role=<iam-role-name> -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 -o use_path_request_style -o url=https://s3-{{aws_region}}.amazonaws.com

Replace <your-s3-bucket-name> with the name of your S3 bucket, and <iam-role-name> with the name of the IAM role attached to the EC2 instance.

Debug issue (optional) :

To debug at any point, add

$ sudo s3fs <your-s3-bucket-name> /mnt/s3-bucket -o iam_role=<iam-role-name> -o use_cache=/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 -o use_path_request_style -o url=https://s3-{{aws_region}}.amazonaws.com -o dbglevel=info -f -o curldbg

1. Verify the mount status:

df -h | grep s3fs

The S3 bucket is now mounted on the EC2 instance at /mnt/s3-bucket.

Step 4: Test uploading files & Verify the files on S3 bucket

Step 5: Make the Mount Persistent (After Reboot)

To make the S3 bucket mount persistent across reboots, we will add an entry to the /etc/fstab file.

1. Open the /etc/fstab file for editing using a text editor:

sudo nano /etc/fstab

2. Add the following line at the end of the file:

s3fs#<your-s3-bucket-name> /mnt/s3-bucket fuse _netdev,iam_role=<iam-role-name>,allow_other 0 0

Replace <your-s3-bucket-name> with the name of your S3 bucket and <iam-role-name> with the name of the IAM role attached to the EC2 instance.

Note: Please include the hashtag #.

If your server attempts to reboot, the command assigned on the Fstab is automatically executed to remount S3fs during system startup. Be careful with the fstab file if you made a mistake with that file, there is a possibility that you will not be able to boot up your server.

Step 6: Test the Persistent Mount

To test the persistent mount, reboot the EC2 instance:

sudo reboot

After the instance reboots, verify that the S3 bucket is still mounted by checking the mount status:

df -h | grep s3fs

The S3 bucket mount should be present and available even after the reboot.

Conclusion:

Mounting an AWS S3 bucket on an Amazon EC2 Linux instance using S3FS with IAM roles offers a versatile and powerful solution for various use cases in the AWS environment. By following the steps in this blog, you can seamlessly integrate S3 with EC2 instances, unlocking a range of benefits from data backup and analytics to web hosting and disaster recovery. The persistent mount ensures that the S3 bucket remains accessible across reboots, making it a reliable and scalable storage solution for your EC2 applications. Embrace the power of S3FS and IAM roles to elevate your AWS infrastructure and harness the full potential of cloud storage and computing.

Thanks for reading this far, and good luck. I appreciate your comments/feedback.

About The Author
Suraj Solanki
DevOps Engineer — II
LinkedIn: https://www.linkedin.com/in/suraj-solanki