EFS | CodeWisdom

What is an EFS?

Elastic File System (EFS) in Amazon Web Services (AWS) is a cloud-based file storage service for applications and workloads that run in the AWS Cloud. It's designed to be easy to use and offers a simple interface that allows you to create and configure file systems quickly. Here are some key features of EFS:

Scalability: EFS automatically scales without needing to provision storage capacity or performance. This means it can grow and shrink automatically as you add and remove files, making it a good choice for applications with fluctuating storage needs.
Highly Available and Durable: EFS is designed to be highly available and durable. It stores data across multiple Availability Zones (AZs) in an AWS Region, ensuring high availability and reliability.
Shared File Access: EFS supports the Network File System (NFS) protocol, which means that multiple EC2 instances and other AWS services can access your file system simultaneously, making it suitable for scenarios where shared file storage is needed.
Performance Modes: EFS offers two performance modes - General Purpose, which is the default and is suitable for a broad range of applications, and Max I/O, which is optimized for applications requiring higher levels of I/O throughput.
Lifecycle Management: EFS can automatically move your files to lower-cost storage classes when they are not accessed for a period of time. This feature helps in optimizing costs according to access patterns.
Security: EFS integrates with AWS Identity and Access Management (IAM) for access control and also supports POSIX permissions. It offers encryption of data at rest and in transit.
Integration with AWS Services: EFS integrates seamlessly with other AWS services like Amazon EC2, AWS Lambda, and container services such as Amazon ECS and EKS, allowing for a wide range of cloud-based applications and use cases.

EFS is often used for use cases such as content management, data analytics, media processing workflows, and home directories, where shared file storage is necessary. It's particularly advantageous in situations where you need a file system that can automatically adjust to fluctuating storage demands without manual intervention.

Choice between using S3 and EFS?

Amazon EFS

Use EFS when:

File System Interface: You need a traditional file system interface and file system semantics (like POSIX). This is essential for applications that are designed for a file system hierarchy.
Shared Access: Your application requires shared access to files, where multiple instances or services need to read and write to the same file system simultaneously.
Low Latency: Your application needs low-latency access to files. EFS is particularly suited for use cases where millisecond-level latencies are important.
Automatically Scaling Storage: You require a storage solution that automatically scales with your usage, without needing to manage the scaling process.
Integration with EC2 and other AWS Services: You need tight integration with AWS compute services like EC2, AWS Lambda, and container services such as ECS and EKS.

Typical use cases for EFS include web serving and content management, data analytics, application testing and development environments, and database backups.

Amazon S3

Use S3 when:

Object Storage: Your application works with data as objects (not files), and you require capabilities like metadata tagging, versioning, and lifecycle policies.
Durability and Scalability: You need high durability (S3 provides 99.999999999% durability) and the ability to store and manage a massive amount of data.
Data Distribution: You need to distribute content with URLs (S3 can host static websites and distribute large files).
Archiving and Backup: For use cases involving data archiving, backup, and disaster recovery where the immediate availability of data is less critical.
Event-Driven Computing: If you are using AWS Lambda for serverless computing, S3 can act as a trigger for Lambda functions.
Cost-Effective Storage: If you need a more cost-effective solution for storing large amounts of data. S3 offers various storage classes for different use cases, such as S3 Standard, S3 Infrequent Access, and S3 Glacier for long-term archival.

Typical use cases for S3 include storing and serving static resources for web applications, data lakes, big data analytics, and backup and disaster recovery solutions.

Summary

Use EFS: When you need a file system for shared, low-latency access, and for applications built for a file system structure.
Use S3: For object storage with massive scalability, for static web hosting, and for cost-effective storage with various access patterns.

Both EFS and S3 are highly reliable and integrate well with other AWS services, but the choice largely depends on whether you need file-level or object-level storage and the specific access patterns of your application.

Real-world usecase

Amazon S3 Use Cases

Website Hosting: A company hosts a static website on AWS. All the website's static assets like HTML, CSS, JavaScript files, and images are stored in an S3 bucket. S3 is ideal for this because it can serve these files directly over the web, handle high levels of traffic, and is cost-effective for storing web content.
Data Lakes and Big Data Analytics: A business uses S3 to store massive datasets collected from various sources. The data is then processed and analyzed using big data processing services like Amazon EMR (Elastic MapReduce) or Redshift. S3 is suitable for this due to its scalability, data durability, and integration with these analytics services.
Backup and Disaster Recovery: A company backs up critical data to S3 because it offers high durability, ensuring data safety. In case of a disaster, the data can be quickly retrieved and restored. S3's versioning and lifecycle policies also help in managing the backups efficiently.
Media Hosting: A media company uses S3 to store and distribute large video files. S3 is used due to its ability to handle large files and high throughput, alongside integration with CDN services like Amazon CloudFront for efficient distribution.

Amazon EFS Use Cases

Shared File Storage for Compute Instances: A company uses a cluster of EC2 instances for processing complex scientific simulations. These instances need shared, fast access to a common dataset. EFS is used here as it provides a common file system that can be mounted across all EC2 instances, allowing for efficient data sharing and high-performance read/write operations.
Content Management Systems: A content management system (CMS) deployed on AWS requires a shared file system to store and manage web content, templates, and media files. EFS is used because it allows multiple servers (like web and application servers) to access and serve the same content.
Software Development and Testing: For software development and testing environments, multiple developers need to access and modify application code and configurations. EFS provides a shared storage solution where files can be consistently and concurrently accessed and updated.
Container Storage: An application deployed using container services like AWS ECS or Kubernetes on AWS requires persistent, shared storage for containers. EFS is suitable as it can be easily integrated as shared file storage for containers, allowing them to read, write, and persist data consistently.

Summary

Use S3: For static content hosting, large-scale data storage, backups, and scenarios needing high durability and scalability. It's ideal for "write once, read many" scenarios.
Use EFS: For applications needing a shared file system, low-latency file access, and consistent performance across multiple EC2 instances or containers. It's suited for "read and write many times" scenarios.

In what scenario that ECS fits better than S3?

Scenario: Collaborative Application Development Environment

Imagine a software development company working on a large-scale application. This project involves multiple teams of developers, testers, and DevOps engineers who need to collaborate efficiently. They require a shared environment where code, libraries, and other development artifacts are stored and can be accessed and modified concurrently.

Requirements of This Scenario:

File Locking and Concurrent Access: Developers need to edit code files simultaneously. A file system supports file locking mechanisms to prevent conflicts, ensuring that only one user at a time can modify a particular file.
Low-Latency Access: The development environment demands low-latency access to files for compiling code, running tests, and performing continuous integration/continuous deployment (CI/CD) tasks. File-system storage provides faster read/write operations compared to object storage.
POSIX-Compliance: Many development tools and applications rely on POSIX-compliant file systems for operations like seeking specific positions in files, renaming files, and directory manipulations. POSIX compliance is not something S3 provides.
Hierarchical File Structure: Developers are accustomed to a hierarchical file structure (directories and subdirectories), which is a natural way to organize project files. File-system storage aligns well with this requirement.

Why S3 is Not Suitable Here:

S3 is an object storage solution, not a file system. It does not support file locking or allow for in-place edits. Each time a file is modified, it needs to be uploaded as a new object.
Accessing data in S3 can have higher latency compared to file-system storage. This could slow down development processes that require frequent read/write operations.
S3 is not POSIX-compliant, which can limit its compatibility with certain development tools or scripts that expect a file system environment.
S3's flat namespace structure might not be ideal for organizing files in the way required by development teams.

In this scenario, a solution like Amazon EFS would be more suitable due to its POSIX-compliant file system, low-latency characteristics, and support for concurrent access with proper file locking mechanisms. It provides an environment similar to traditional on-premises file servers, which is a key requirement for collaborative software development and deployment environments.

EFS Usage Example with Code

Step 1: Create an EFS File System

You can create an EFS file system using the AWS Management Console or the AWS CLI. For example, using the AWS CLI:

aws efs create-file-system --creation-token MyEfsFileSystem

This command creates a new file system. You'll get a File System ID (e.g., fs-12345678) in response.

Step 2: Create Mount Targets

After creating the file system, you need to create mount targets in your VPC:

aws efs create-mount-target --file-system-id fs-12345678 --subnet-id subnet-abcdefgh --security-groups sg-12345678

This command creates a mount target in a specific subnet and security group.

Step 3: Mount the File System on EC2 Instances

Now, you can mount the file system on your EC2 instances. First, ensure that the amazon-efs-utils package is installed on your EC2 instance:

For Amazon Linux or RHEL:

sudo yum install -y amazon-efs-utils

For Ubuntu:

sudo apt-get install -y amazon-efs-utils

Then, you can mount the file system:

sudo mount -t efs fs-12345678:/ /mnt/efs

This command mounts the EFS file system to a directory on your EC2 instance (here /mnt/efs).

Example Use Cases with Code

Shared Data for Web Servers: If you're running multiple web servers in an Auto Scaling group, you can use EFS to store shared assets like images or stylesheets. The mount command (sudo mount -t efs fs-12345678:/ /mnt/efs) would be part of your EC2 instance's initialization script.
Storing Application Logs: You can configure your applications to write logs directly to a directory mounted on EFS. For instance, configure your logging framework to write logs to /mnt/efs/logs.
Persistent Storage for Containers: If you're using AWS ECS or Kubernetes on AWS (EKS), you can mount an EFS file system to your containers for persistent storage. In Kubernetes, you'd use a PersistentVolume (PV) and PersistentVolumeClaim (PVC) referencing your EFS.
Kubernetes PV example:
```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs
  csi:
    driver: efs.csi.aws.com
    volumeHandle: [FileSystemId]::[AccessPointId]
```
This YAML file defines a PersistentVolume that points to your EFS file system.

Remember, these examples assume you've set up your AWS environment, including your VPC, security groups, and IAM roles. Additionally, always ensure your security groups and network ACLs allow the necessary traffic to and from your EFS mount points.