EFS and SOLR Cloud Backup(3)Cron Backup and Archive

EFS and SOLR Cloud Backup(3)Cron Backup and Archive

Try to clean the directories older than X days
>find /home/ec2-user/users/carl/ -mtime +3 -exec sudo rm -fr {} \;

Try to clean the directory under directory, only keep X
>sudo rm -fr `ls -t /Users/carl/company/code/solr-alljobs/crontask/test | awk ‘NR>3'`

Try to make a Docker and Task Definition in ECS and Start a scheduled Task there.

I first create a Docker Image and Task Definition
Dockerfile
#Start a Clean Machine to Execute Shell Script

#Prepre the OS
FROM            centos:7
MAINTAINER      Carl Luo <[e-mail]>

#Start the Application
RUN     mkdir -p /app/
ADD     start.sh /app/
WORKDIR /app
CMD    [ "./start.sh" ]

Makefile
IMAGE=odt/cron-task
TAG=1.6
NAME=cron-task
REPOSITORY=xxxxxx.dkr.ecr.us-east-1.amazonaws.com

push-local:
    docker push  $(REPOSITORY)/$(IMAGE):$(TAG)

docker-context:

build: docker-context
    docker build -t $(REPOSITORY)/$(IMAGE):$(TAG) .

run:
    docker run -d --name $(NAME) $(REPOSITORY)/$(IMAGE):$(TAG)

clean:
    docker stop ${NAME}
    docker rm ${NAME}
   
logs:
    docker logs ${NAME}

publish:
    docker push ${IMAGE}

The start.sh will be the key part, it will call backup and archive the too old backup files.
#!/bin/sh -ex

DATE=`date +%y-%m-%d-%H-%M-%S`
COLLECTION=${COLLECTION:-"allJobs"}
CLUSTER=${CLUSTER:-"alljobstest"}

echo "Start to check the backup and archive ${DATE}"

echo "list the backup directory"
ls /efs

echo "start the backup process"
curl "http://${CLUSTER}.us-east-1.elasticbeanstalk.com/solr/admin/collections?action=BACKUP&name=allJobsBackup${DATE}&collection=${COLLECTION}&location=/efs&async=${DATE}"
echo "backup request is sent"

echo "start to archive data 3 days ago"
#find /efs/ -mtime +3 -exec sudo rm -fr {} \;
sudo rm -fr `ls -t /efs/ | awk 'NR>3'`

I build and push this Docker image to the ECS Repositories
From the Task Definition, some settings are as follow:
Task Definition Name: Cron-Task
Task Role: None
Network Mode: Bridget
Compatibilities: EC2
Volumes: efs = /efs
Container Name:
     CronTask: Image: xxxxx.dkr.ecr.us-east-1.amazonaws.com/odt/cron-task:1.6
     Mount Points:
                            Container Path: /efs
                            Source Volume: efs

Log Configuration
    Log driver: awslogs
    awslogs-group: Feed2GLogging
    awslogs-region: us-east-1
    awslogs-stream-prefix: crontask

Creat a Cron Service
Go to the cluster page, choose [Scheduled Tasks]—> [Create]
You can choose, Run at Fixed interval, I put 2 hours there for testing.
You can choose Cron expression as well, for example
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
0 23 * * ? *  (every 11pm)

Put the task Definition string : Cron-Task:10

That is it.

References:
https://stackoverflow.com/questions/13489398/delete-files-older-than-10-days-using-shell-script-in-unix
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled_tasks.html
https://lucene.apache.org/solr/guide/7_2/making-and-restoring-backups.html#delete-snapshot-api