EFS and SOLR Cloud Backup(1)EFS Set Up for EC2 and Docker

EFS and SOLR Cloud Backup(1)EFS Set Up for EC2 and Docker

EFS Set Up
Amazon Elastic File System EFS provide simple, scalable file storage for EC2 instance.

In the EFS console, we can create a File System, there is a File system ID and then we can set up config in Elastic Beantalk.

In the configuration file .ebextensions/storage-efs-mountfilesystem.config
###################################################################################################
#### Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
####
#### Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file
#### except in compliance with the License. A copy of the License is located at
####
####     http://aws.amazon.com/apache2.0/
####
#### or in the "license" file accompanying this file. This file is distributed on an "AS IS"
#### BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#### License for the specific language governing permissions and limitations under the License.
###################################################################################################

###################################################################################################
#### This configuration file mounts an Amazon EFS file system to a directory named /efs. To mount
#### the file system to a different path, modify the MOUNT_DIRECTORY value in the "option_settings"
#### section.
####
#### The FILE_SYSTEM_ID setting references a resource named "FileSystem", which is created by the
#### storage-efs-createfilesystem.config configuration file. To use this file to mount a
#### file system that you created outside of AWS Elastic Beanstalk, replace the Ref with the
#### resource ID (e.g., fs-e7605f4e):
####
####      FILE_SYSTEM_ID: fs-e7605f4e
####
#### If your environment and file system are in a custom VPC, you must configure the VPC to allow
#### DNS resolution and DNS host names. See this topic in the VPC User Guide for more information:
####    http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html
###################################################################################################

option_settings:
  aws:elasticbeanstalk:application:environment:
    FILE_SYSTEM_ID: 'fs-a5838eec'
    MOUNT_DIRECTORY: '/efs'

##############################################
#### Do not modify values below this line ####
##############################################
    REGION: '`{"Ref": "AWS::Region"}`'

packages:
  yum:
    nfs-utils: []
    jq: []

commands:
  01_mount:
    command: "/tmp/mount-efs.sh"

files:
  "/tmp/mount-efs.sh":
      mode: "000755"
      content : |
        #!/bin/bash

        EFS_REGION=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.REGION')
        EFS_MOUNT_DIR=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.MOUNT_DIRECTORY')
        EFS_FILE_SYSTEM_ID=$(/opt/elasticbeanstalk/bin/get-config environment | jq -r '.FILE_SYSTEM_ID')

        echo "Mounting EFS filesystem ${EFS_DNS_NAME} to directory ${EFS_MOUNT_DIR} ..."

        echo 'Stopping NFS ID Mapper...'
        service rpcidmapd status &> /dev/null
        if [ $? -ne 0 ] ; then
            echo 'rpc.idmapd is already stopped!'
        else
            service rpcidmapd stop
            if [ $? -ne 0 ] ; then
                echo 'ERROR: Failed to stop NFS ID Mapper!'
                exit 1
            fi
        fi

        echo 'Checking if EFS mount directory exists...'
        if [ ! -d ${EFS_MOUNT_DIR} ]; then
            echo "Creating directory ${EFS_MOUNT_DIR} ..."
            mkdir -p ${EFS_MOUNT_DIR}
            if [ $? -ne 0 ]; then
                echo 'ERROR: Directory creation failed!'
                exit 1
            fi
        else
            echo "Directory ${EFS_MOUNT_DIR} already exists!"
        fi

        mountpoint -q ${EFS_MOUNT_DIR}
        if [ $? -ne 0 ]; then
            echo "mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 ${EFS_FILE_SYSTEM_ID}.efs.${EFS_REGION}.amazonaws.com:/ ${EFS_MOUNT_DIR}"
            mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 ${EFS_FILE_SYSTEM_ID}.efs.${EFS_REGION}.amazonaws.com:/ ${EFS_MOUNT_DIR}
            if [ $? -ne 0 ] ; then
                echo 'ERROR: Mount command failed!'
                exit 1
            fi
            chmod 777 ${EFS_MOUNT_DIR}
            runuser -l  ec2-user -c "touch ${EFS_MOUNT_DIR}/it_works"
            if [[ $? -ne 0 ]]; then
                echo 'ERROR: Permission Error!'
                exit 1
            else
                runuser -l  ec2-user -c "rm -f ${EFS_MOUNT_DIR}/it_works"
            fi
        else
            echo "Directory ${EFS_MOUNT_DIR} is already a valid mountpoint!"
        fi

        echo 'EFS mount complete.'

Because we use Docker in the Elastic Beantalk EC2 Host Machine, so we will mount that to our Docker as well.
elasticbeanstalk/Dockerrun.aws.json
{
  "AWSEBDockerrunVersion": 2,
  "volumes": [{
    "name": "solr-home",
    "host": {
      "sourcePath": "/solr"
    }
  },
  {
    "name": "efs-directory",
    "host": {
      "sourcePath": "/efs"
    }
  }],
  "containerDefinitions": [{
      "name": "solr",
      "image": “xxxxxxx.dkr.ecr.us-east-1.amazonaws.com/odt/solr:latest",
      "environment": [{
        "name": "OPERATION",
        "value": "run-solr"
      }],
      "essential": true,
      "memory": 35000,
      "mountPoints": [{
          "sourceVolume": "solr-home",
          "containerPath": "/solr"
        },
        {
          "sourceVolume": "awseb-logs-solr",
          "containerPath": "/opt/solr/server/logs"
        },
        {
          "sourceVolume": "efs-directory",
          "containerPath": "/efs"
        }
      ],
      "portMappings": [{
        "hostPort": 8983,
        "containerPort": 8983
      }]
    },
    {
      "name": "init",
      "image": “xxxxxx.dkr.ecr.us-east-1.amazonaws.com/odt/solr:latest",
      "environment": [{
        "name": "OPERATION",
        "value": "join-cloud"
      }],
      "essential": false,
      "memory": 1000,
      "links": [
        "solr"
      ],
      "mountPoints": [{
        "sourceVolume": "solr-home",
        "containerPath": "/solr"
      }]
    },
    {
      "name": "healthcheck",
      "image": “xxxx.dkr.ecr.us-east-1.amazonaws.com/odt/solr-healthcheck:latest",
      "essential": true,
      "memory": 1000,
      "links": [
        "solr"
      ],
      "portMappings": [{
        "hostPort": 8080,
        "containerPort": 8080
      }]
    }
  ]
}


References:
https://aws.amazon.com/efs/?sc_channel=PS&sc_campaign=acquisition_US&sc_publisher=google&sc_medium=efs_b&sc_content=aws_efs_e&sc_detail=efs%20aws&sc_category=efs&sc_segment=208381315165&sc_matchtype=e&sc_country=US&s_kwcid=AL!4422!3!208381315165!e!!g!!efs%20aws&ef_id=WDe8wQAABKdZQ0gc:20180111034816:s