Regis Memory and Cleanup

Regis Memory and Cleanup

Get total key
> dbSize
(integer) 73857708

Get random key
> randomkey
"job_12_02d5b44de7a4af119504abbed239b6f4"

Check the type of the key
> type job_budget_4313410226
hash

Check the expiration time
> ttl job_12_02d5b44de7a4af119504abbed239b6f4
(integer) -1

dump command can dump the value of that key
> dump job_12_02d5b44de7a4af119504abbed239b6f4
"\x04;\x06paused\xc0\x00\x05title\x1cLease Purchase Truck Drivers\x0blast_update\xc2\xc9\xce\xa2X\x05phone\x00\x0cdaily_capped\xc0\x00\teducation\xc0\x02\tsrcseg_id\xc0\x0c\x10excluded_company\xc0\x01\x0eminor_category\xc2\xfa\x9c\t\x00\rquality_score\xc0\x1e\x13top_spot_industries\x00\x06budget\x00\x0ejob_id_mapping\n4310204121\x04tags\x00\bindustry\xc0\x14\x14major_category_score\xc0\x01\x0bis_jobboard\xc0\x01\tsource_id\xc0\x0c\x06is_add\xc0\x00\nsearchtags\x00\x06posted\xc2\xc9\xce\xa2X\x03url\xc3@\x80@\x88\x1fhttp://www.linkup.com/joblisting\x02s/? \x0c\x1f=02d5b44de7a4af119504abbed239b6f\x0c4&utm_source= /\bs2careers`\x17\tmedium=cpc`\x0e\acampaign\xe0\x02(\x01rs\x0bcustomer_id\xc0\x03\x06cities\nDecatur,AL\x0bcampaign_id\xc0\x0c\x05email\x00\areg_cpc\xc0\x00\x06action\x04save\x0faffiliate_group\xc0\x00\acreated\xc2\xc9\xce\xa2X\x06digest 7aefe0914bb80f90e65d4a3f7710ff70\nsegment_id\xc1\xfa\x01\rqq_multiplier\xc0\x00\tj2c_apply\xc0\x00\nbudgetType\bLIFETIME\x06status\aEXPIRED\x0cboost_factor\x0f3.0931627568599\ajobtype\xc0\x01\x03cpc\xc0\x1e\adeduped\x00\x13is_top_spot_preview\xc0\x00\bzipcodes\xc2\x12\x8b\x00\x00\bcpc_high\xc0\x01\nsrccamp_id\xc0\x0c\rjob_reference 02d5b44de7a4af119504abbed239b6f4\acompany\x12Super Service, LLC\x0fmobile_friendly\xc0\x01\x0cdedup_digest 978c2722410a9299ec35229babeeaf37\ncompany_id\xc2R_\x01\x00\bcategory\x00\tstate_ids\xc0\x02\x11quality_sensitive\x00\x14minor_category_score\x030.3\x0cindustry_ids\xc0\x14\x06salary\xc0\x1e\rtop_spot_type\xc0\x00\apool_id\xc0\x0c\x0emajor_category\xc2\xf0\x9c\t\x00\nexperience\xc0\x02\a\x00\xa0\xa0\xbc9\xfd\x84\xa3\""

The tool in here can get the size of one key
https://github.com/sripathikrishnan/redis-rdb-tools

> git clone https://github.com/sripathikrishnan/redis-rdb-tools

> cd redis-rdb-tools/

> sudo python3 setup.py install

> redis-memory-for-key -s test.amazonaws.com -p 6379 job_12_02d5b44de7a4af119504abbed239b6f4
Key job_12_02d5b44de7a4af119504abbed239b6f4
Bytes 5716.0
Type hash
Encoding hashtable
Number of Elements 59
Length of Largest Element 136

Pick up Keys to scan
> scan 77594624 count 10
1) "68157440"
2)  1) "idmapping_4320118968"
    2) "job_2465_CB-UnDNml2EZMrepChV7fgIJIrUQWcmv"
    3) "job_dedupdigest_f95fa0aea4c4fc1038a191611e69bd7a"
    4) "idmapping_4301743608"
    5) "job_2465_DkljWeZhGWJ4KpkY023TsSJTq8wRM-xL"
    6) "job_budget_4318069707"
    7) "job_dedupdigest_d1c8e5ba31218058d0ab96c110f5024f"
    "job_budget_4314514161"
    9) "job_dedupdigest_4856f59b52ce150c22b92cdd253f3e46"
   10) "job_1624_787_416056211"

Scan with Pattern
> scan 0 count 10 match job_*
1) "92274688"
2) 1) "job_budget_4314725990"
   2) "job_2465_a1r2Sr6OfBfWgcoEGlDlWumX3Ckx4t3S"
   3) "job_4130_2124_951ecb1c50c572cdbcaf0ee261ecd04f_228_i"
   4) "job_1627_90433loc459690"
   5) "job_1052_8WMXY5c-IUs-T2Y2aPch6xVnhUYPXIEM"
   6) "job_budget_4316788589"
   7) "job_dedupdigest_6782069fbe5c5e51b742a26b48fe8156"

Finally I write some scripts to measure the memory usage and write a script to scan and clean the Redis.

The major party will be these
if($index % 50 == 0){
     $logger->info("processed {$index} / {$count} ...");
}
//fetch key
$key = $redisClient->getRandomKey();
//memory information
$memory = $this->getMemoryUsage($key);
//find the pattern
$pattern = $this->matchPattern($key);
//add the report
if($pattern == '/job_.*/'){
     //find job status
     $status = $redisClient->getJobStatus($key);
     if(empty($status)){
          $logger-warning("job {$key} does not have status....");
     }
     $reports[$pattern . $status]['memory'] = $reports[$pattern . $status]['memory'] + $memory['Bytes'];
     $reports[$pattern . $status]['type'] = $memory['Type'];
     $reports[$pattern . $status]['count'] = $reports[$pattern . $status]['count'] + 1;
}else{
     $reports[$pattern]['memory'] = $reports[$pattern]['memory'] + $memory['Bytes'];
     $reports[$pattern]['type'] = $memory['Type'];
     $reports[$pattern]['count'] = $reports[$pattern]['count'] + 1;
}

$reports['all']['memory'] = $reports['all']['memory'] + $memory['Bytes'];
$reports['all']['count'] = $reports['all']['count'] + 1;

…snip...

How to fetch memory information
public function getMemoryUsage($key)
{
     $redisClient = $this->ioc->getService("redisClient");
     exec('redis-memory-for-key -s prod-jobs.pnuura.ng.0001.use1.cache.amazonaws.com -p 6379 ' . $key , $outputs);
     $result = array();
     foreach ($outputs as $item){
          $values = preg_split('/\s+/', $item);
          $result[$values[0]] = $values[1];
     }
     return $result;
}

I scan the keys and clean the related pattern data
    public function scanAndCleanExpireJobs($test = true){
        $logger = $this->ioc->getService("logger");

        $count = 0;
        $pattern = 'job_*';

        foreach (new Iterator\Keyspace($this->client, $pattern) as $key) {
            //$logger->info("key is " . $key);
            if(preg_match("/job_dedupdigest_.*/", $key) || preg_match("/job_budget_.*/", $key)){
                //ingore
                continue;
            }
            //find job status
            $status = $this->client->hget($key, 'status');
            if("EXPIRED" == $status){
                if($test){
                    $logger->info("key is " . $key . " status " . $status);
                }else{
                    //delete the expire job
                    $logger->info("deleting the expire job " . $key . " counting " . $count);
                    $this->client->del($key);
                    $count++;
                }
            }
        }
    }

Reference:
http://stackoverflow.com/questions/7638542/redis-show-database-size-size-for-keys
https://github.com/snmaynard/redis-audit/blob/master/redis-audit.rb
http://ylw6006.blog.51cto.com/470441/1281548
http://blog.csdn.net/jiangguilong2000/article/details/9313603
http://www.10tiao.com/html/256/201609/2247483748/1.html
https://github.com/antirez/redis-sampler
https://github.com/sripathikrishnan/redis-rdb-tools