Demystifying Storage System 🧠 ✨

Have you ever wondered how a pen drive stores your favorite songs 🤔 , how an SSD makes your laptop blazing fast, or how Google Photos magically keeps your memories safe🤔🤯? The answer lies in the marvel of Storage Systems 🌟✨.

From the tiniest USB drives to the sprawling data centers powering the internet, storage systems form the foundation of modern computing⚡. They enable everything from basic file storage to advanced cloud computing, data analytics, and even artificial intelligence. Without them, our digital world would simply cease to exist. Curious to know how this invisible backbone works? Let’s dive in!

Storage System: The Essential Pillar of Computing 💪

1. Your Digital Memory: Imagine writing an important report, but every time you turn off your laptop, it vanishes. Without storage, your work, apps, and even your operating system wouldn't survive a reboot.

2. Stream Your Favorites: Your favorite movies and music are at your fingertips thanks to massive storage systems powering platforms like Netflix and Spotify, storing and streaming endless content.

3. AI's Brainpower: Artificial intelligence learns by processing data—without storage, AI couldn't access or analyze the massive datasets needed to power your smart assistants or recommendation engines.

4. The Lifesaver: Lost your phone? Cloud storage ensures your memories, photos, and files are safe, giving you peace of mind and instant recovery.

5. Mission Critical: NASA’s Mars missions depend on robust storage to retain and transmit priceless data from millions of miles away, ensuring progress in space exploration.

Without storage, computing is like a chalkboard wiped clean every second—fascinating but utterly unusable!

Three main Types

Three main types of storage systems power our digital world, Each plays a unique role in modern computing!

1. Block-Based Storage

2. File-Based Storage

3. Object-Based Storage

Block Storage System 💪⚡

Block storage breaks data into fixed-sized chunks (blocks) and stores them separately with unique identifiers. These blocks can be retrieved and assembled quickly, providing high performance and flexibility.

Block storage offers high IOPS (Input/Output Operations Per Second) and low latency, making it perfect for compute-intensive tasks where speed and efficiency are critical. Its direct access nature ensures minimal delays, ideal for scenarios requiring real-time data processing.

Use Cases for Block Storage

1. Databases: High-performance SQL or NoSQL databases.

2. Virtual Machines: Backend storage for VM environments like VMware or Hyper-V.

3. High-Performance Applications: Real-time analytics, transaction processing systems, or enterprise ERP solutions.

4. Containers: Persistent storage for containerized applications in Kubernetes or Docker.

Examples of Block Storage

1. Amazon Elastic Block Store (EBS)

2. Google Persistent Disks

3. Microsoft Azure Managed Disks

4. SAN (Storage Area Network)

In answer to Our Earlier Question Pen Drive and Our SSD are also Examples of Block Storage.

File Storage System 📂🗃️

File storage organizes data in a hierarchical structure of directories and files, similar to how your computer organizes documents. It is user-friendly, making it easy to access and manage data with familiar file paths.

Key Capabilities of File Storage

1. Simplicity: Works seamlessly with applications needing traditional file access protocols (e.g., NFS, SMB).

2. Scalability: Can expand to accommodate growing datasets while retaining the directory structure.

3. Concurrent Access: Enables multiple users or applications to access shared files simultaneously.

Use Cases for File Storage

1. Content Management Systems: Hosting media assets like images, documents, or videos.

2. Shared File Repositories: Collaboration tools and shared drives for teams.

3. Web Servers: Storing and serving static website files like HTML, CSS, and images.

Examples of File Storage

1. Amazon Elastic File System (EFS)

2. Azure Files

3. Google Filestore

4. NAS (Network Attached Storage)

Object Storage System 🌐

Object storage manages data as individual objects with unique identifiers, rather than as files or blocks. It's highly scalable and designed to handle massive amounts of unstructured data, such as media files or backups.

Unique Capabilities of Object Storage

1. Scalability: Unlimited storage capacity, perfect for managing exabytes of data.

2. Metadata Customization: Allows rich metadata tagging for advanced data indexing and retrieval.

3. Global Accessibility: Data can be accessed over the internet via APIs, making it ideal for cloud applications.

Use Cases for Object Storage

1. Cloud Applications: Storing user-generated content like images and videos.

2. Data Backup and Archiving: Cost-effective, long-term storage for backups.

3. Big Data and Analytics: Storing logs or sensor data for analysis.

4. Streaming Services: Hosting media content for streaming platforms.

Examples of Object Storage

1. Amazon Simple Storage Service (S3)

2. Google Cloud Storage

3. Microsoft Azure Blob Storage

4. MinIO

Now, let’s dive in to have some hands-on experience with three types of storage systems in the AWS environment.

EBS(Block Storage)

1. Create a EC2 instance with Default Settings and Connect it as usual and Execute the Following Command to check the list of block devices

    lsblk
   

   

   you could see that it consists of one block storage which is responsible for booting the os and default activities.

2. Now let's create a New volume, attach it to EC2 instance, and use it . To Implement this I am going to use AWS console

   

3. Make Sure Both EC2 and Volume should be in the same Availablity Zone to attach them both

   

4. Now you can log in to the server and list all the block devices

   

5. Since the volume is new you need to format it and create a Filesystem on it

   

6. Now let mount point and mount our volume to the EC2 instance

   

7. Follow these steps to mount your volume permanently

    blkid
   

8. Get you bulkid and write it in /etc/fstab

   

sudo mount -a

Extra tip 💡: If you try EBS volume of io2 you could attach it to the Multiple Instance(Only Nitro-based instance type)

EFS(File Based Storage)

The main use-case of EFS is the ability to connect with multiple instances concurrently.So I am going to create two test Instances and attach it to the EFS.

Extra tip 💡: The Key Flexibility you can get from it is Super Scalable, and if you choose the Regional option, which is the Default one, it creates a File System in all AZ to offer High availability.

1. Create a Security Group that allows necessary ports for our NFS and SSH connection

   

   

2. Let’s Create the EFS

keep the Rest as Default and Create the FIle System.

3. Now Select the File System and Select the Attach to get the mount Point . Let’s select Mount Via IP the IP will be Differ for Differ AZ . For Simplicity, we will be creating Instances in “us-east-1a” alone so let’s copy the mount point of us-est-1a alone.

   

   

4. Now Let’s log in to the Instance and Configure the necessary items to access the EFS

    sudo apt update
    sudo apt install nfs-common
    #Create a mount point 
    sudo mkdir /practicalEFS
    #Edit th /etc/fstab to auto mount instead of manual mount
    sudo nano /etc/fstab
    #Add the line 172.31.41.113:/ /practicalEFS   nfs4    nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport     0 0
    sudo mount -a
    sudo systemctl daemon-reload
    #Check whether the EFS got mounted by
    df -h
   

   

5. let’s create a file in the mounted location

cd /practicalEFS
sudo touch hello.txt

6.Now let's log in to next instance mount the EFS and Check whether the file is present

After mounting you could see

S3 (Object-based Storage)

The Object-based Storage System Offered by AWS is S3 it is one of the most widely used AWS services Let's Dive into the Demo

1. Create an IAM Role and Attach a policy that gives full access to S3

   

   This was the role created by me and I have attached “AmazonS3FullAccess” which was an Amazon-managed policy if you want you could create your custom policy.

2. Now Spin up an EC2 instance and attach the Role to that instance

   

   Attaching the Role is simple Select the Instance —>Actions—>Security—→Modify IAM Role That’s it

3. Create a S3 bucket and upload some example .txt file

   

4. Now let’s Login to our EC2 instance and try to write our text file in S3 bucket

    #For performing you need to Install awscli
    sudo apt-get update
    sudo apt-get upgrade -y
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    sudo apt-get install unzip -y
    unzip awscliv2.zip
    sudo ./aws/install
   

   

    echo "Hi this is Koushal After the test"|aws s3 cp - s3://myobjectstorageexample/Hello.txt
   

   

   That’s it Now we have Practically Explored 3 main Storage System 🌟✨.

The Overview Comparison of Storage System:

That’s it Kudos for today💪✌️. Continue Exploring 🏄‍♂️😎