`
sillycat
  • 浏览: 2542920 次
  • 性别: Icon_minigender_1
  • 来自: 成都
社区版块
存档分类
最新评论

Regis Memory and Cleanup

 
阅读更多
Regis Memory and Cleanup

Get total key
> dbSize
(integer) 73857708

Get random key
> randomkey
"job_12_02d5b44de7a4af119504abbed239b6f4"

Check the type of the key
> type job_budget_4313410226
hash

Check the expiration time
> ttl job_12_02d5b44de7a4af119504abbed239b6f4
(integer) -1

dump command can dump the value of that key
> dump job_12_02d5b44de7a4af119504abbed239b6f4
"\x04;\x06paused\xc0\x00\x05title\x1cLease Purchase Truck Drivers\x0blast_update\xc2\xc9\xce\xa2X\x05phone\x00\x0cdaily_capped\xc0\x00\teducation\xc0\x02\tsrcseg_id\xc0\x0c\x10excluded_company\xc0\x01\x0eminor_category\xc2\xfa\x9c\t\x00\rquality_score\xc0\x1e\x13top_spot_industries\x00\x06budget\x00\x0ejob_id_mapping\n4310204121\x04tags\x00\bindustry\xc0\x14\x14major_category_score\xc0\x01\x0bis_jobboard\xc0\x01\tsource_id\xc0\x0c\x06is_add\xc0\x00\nsearchtags\x00\x06posted\xc2\xc9\xce\xa2X\x03url\xc3@\x80@\x88\x1fhttp://www.linkup.com/joblisting\x02s/? \x0c\x1f=02d5b44de7a4af119504abbed239b6f\x0c4&utm_source= /\bs2careers`\x17\tmedium=cpc`\x0e\acampaign\xe0\x02(\x01rs\x0bcustomer_id\xc0\x03\x06cities\nDecatur,AL\x0bcampaign_id\xc0\x0c\x05email\x00\areg_cpc\xc0\x00\x06action\x04save\x0faffiliate_group\xc0\x00\acreated\xc2\xc9\xce\xa2X\x06digest 7aefe0914bb80f90e65d4a3f7710ff70\nsegment_id\xc1\xfa\x01\rqq_multiplier\xc0\x00\tj2c_apply\xc0\x00\nbudgetType\bLIFETIME\x06status\aEXPIRED\x0cboost_factor\x0f3.0931627568599\ajobtype\xc0\x01\x03cpc\xc0\x1e\adeduped\x00\x13is_top_spot_preview\xc0\x00\bzipcodes\xc2\x12\x8b\x00\x00\bcpc_high\xc0\x01\nsrccamp_id\xc0\x0c\rjob_reference 02d5b44de7a4af119504abbed239b6f4\acompany\x12Super Service, LLC\x0fmobile_friendly\xc0\x01\x0cdedup_digest 978c2722410a9299ec35229babeeaf37\ncompany_id\xc2R_\x01\x00\bcategory\x00\tstate_ids\xc0\x02\x11quality_sensitive\x00\x14minor_category_score\x030.3\x0cindustry_ids\xc0\x14\x06salary\xc0\x1e\rtop_spot_type\xc0\x00\apool_id\xc0\x0c\x0emajor_category\xc2\xf0\x9c\t\x00\nexperience\xc0\x02\a\x00\xa0\xa0\xbc9\xfd\x84\xa3\""

The tool in here can get the size of one key
https://github.com/sripathikrishnan/redis-rdb-tools

> git clone https://github.com/sripathikrishnan/redis-rdb-tools

> cd redis-rdb-tools/

> sudo python3 setup.py install

> redis-memory-for-key -s test.amazonaws.com -p 6379 job_12_02d5b44de7a4af119504abbed239b6f4
Key job_12_02d5b44de7a4af119504abbed239b6f4
Bytes 5716.0
Type hash
Encoding hashtable
Number of Elements 59
Length of Largest Element 136

Pick up Keys to scan
> scan 77594624 count 10
1) "68157440"
2)  1) "idmapping_4320118968"
    2) "job_2465_CB-UnDNml2EZMrepChV7fgIJIrUQWcmv"
    3) "job_dedupdigest_f95fa0aea4c4fc1038a191611e69bd7a"
    4) "idmapping_4301743608"
    5) "job_2465_DkljWeZhGWJ4KpkY023TsSJTq8wRM-xL"
    6) "job_budget_4318069707"
    7) "job_dedupdigest_d1c8e5ba31218058d0ab96c110f5024f"
    "job_budget_4314514161"
    9) "job_dedupdigest_4856f59b52ce150c22b92cdd253f3e46"
   10) "job_1624_787_416056211"

Scan with Pattern
> scan 0 count 10 match job_*
1) "92274688"
2) 1) "job_budget_4314725990"
   2) "job_2465_a1r2Sr6OfBfWgcoEGlDlWumX3Ckx4t3S"
   3) "job_4130_2124_951ecb1c50c572cdbcaf0ee261ecd04f_228_i"
   4) "job_1627_90433loc459690"
   5) "job_1052_8WMXY5c-IUs-T2Y2aPch6xVnhUYPXIEM"
   6) "job_budget_4316788589"
   7) "job_dedupdigest_6782069fbe5c5e51b742a26b48fe8156"

Finally I write some scripts to measure the memory usage and write a script to scan and clean the Redis.

The major party will be these
if($index % 50 == 0){
     $logger->info("processed {$index} / {$count} ...");
}
//fetch key
$key = $redisClient->getRandomKey();
//memory information
$memory = $this->getMemoryUsage($key);
//find the pattern
$pattern = $this->matchPattern($key);
//add the report
if($pattern == '/job_.*/'){
     //find job status
     $status = $redisClient->getJobStatus($key);
     if(empty($status)){
          $logger-warning("job {$key} does not have status....");
     }
     $reports[$pattern . $status]['memory'] = $reports[$pattern . $status]['memory'] + $memory['Bytes'];
     $reports[$pattern . $status]['type'] = $memory['Type'];
     $reports[$pattern . $status]['count'] = $reports[$pattern . $status]['count'] + 1;
}else{
     $reports[$pattern]['memory'] = $reports[$pattern]['memory'] + $memory['Bytes'];
     $reports[$pattern]['type'] = $memory['Type'];
     $reports[$pattern]['count'] = $reports[$pattern]['count'] + 1;
}

$reports['all']['memory'] = $reports['all']['memory'] + $memory['Bytes'];
$reports['all']['count'] = $reports['all']['count'] + 1;

…snip...

How to fetch memory information
public function getMemoryUsage($key)
{
     $redisClient = $this->ioc->getService("redisClient");
     exec('redis-memory-for-key -s prod-jobs.pnuura.ng.0001.use1.cache.amazonaws.com -p 6379 ' . $key , $outputs);
     $result = array();
     foreach ($outputs as $item){
          $values = preg_split('/\s+/', $item);
          $result[$values[0]] = $values[1];
     }
     return $result;
}

I scan the keys and clean the related pattern data
    public function scanAndCleanExpireJobs($test = true){
        $logger = $this->ioc->getService("logger");

        $count = 0;
        $pattern = 'job_*';

        foreach (new Iterator\Keyspace($this->client, $pattern) as $key) {
            //$logger->info("key is " . $key);
            if(preg_match("/job_dedupdigest_.*/", $key) || preg_match("/job_budget_.*/", $key)){
                //ingore
                continue;
            }
            //find job status
            $status = $this->client->hget($key, 'status');
            if("EXPIRED" == $status){
                if($test){
                    $logger->info("key is " . $key . " status " . $status);
                }else{
                    //delete the expire job
                    $logger->info("deleting the expire job " . $key . " counting " . $count);
                    $this->client->del($key);
                    $count++;
                }
            }
        }
    }

Reference:
http://stackoverflow.com/questions/7638542/redis-show-database-size-size-for-keys
https://github.com/snmaynard/redis-audit/blob/master/redis-audit.rb
http://ylw6006.blog.51cto.com/470441/1281548
http://blog.csdn.net/jiangguilong2000/article/details/9313603
http://www.10tiao.com/html/256/201609/2247483748/1.html
https://github.com/antirez/redis-sampler
https://github.com/sripathikrishnan/redis-rdb-tools
分享到:
评论

相关推荐

    ReGIS:将HPGL转换为ReGIS和ReGIS图像存储库-开源

    将HPGL绘图仪输出文件转换为DEC ReGIS文件,适合在支持DEC ReGIS的终端和终端仿真器上显示。 这也是可免费使用的ReGIS贡献文件存储库。 如果您有Regis文件可以给我发电子邮件。 安装zip文件包含许多HPGL文件和一个...

    Prova-Regis-Cardoso:里吉斯·卡多佐(Regis Cardoso)品尝

    很抱歉,根据您提供的信息,"Prova-Regis-Cardoso"似乎与IT行业的主要知识点不直接相关。标题和描述更像是与食品品尝或个人评价有关的内容,这超出了我的专业领域——IT和编程。标签为空,也没有提供任何额外的IT上...

    Regis Jesuit High School FRC Team #3729-开源

    【标题】"Regis Jesuit High School FRC Team #3729-开源"涉及到的是一个高中学校的机器人竞赛团队——Regis Jesuit High School的FIRST Robotics Team 3729,该团队专注于开源软件的开发。他们从2011年的新人赛季...

    Android代码-Console

    An Android console view, which allows you to log text using static calls, to easily debug your application, whilst avoiding memory leaks. Usage Include Console anywhere in your layout: then ...

    regis-practicum-denver-airbnb:这是Regis大学MSDS计划的实践项目

    丹佛Airbnb数据分析项目这是瑞吉斯大学MSDS计划的实践项目。 原始数据集可从获得。 丹佛市的天际线照片,来自埃克森·托德(Erick Todd),来自Pexels介绍在度假方面,Airbnb已成为酒店的代名词。...

    Regis University Explorer-crx插件

    语言:English 禁用RU(里吉斯大学)WorldClass超时警告,并按需隐藏RU标语。 单击按钮,从讨论或内容页面中删除横幅,以实现最大的可视性。 再次单击以显示横幅。 当您专注于其他任务时,会自动禁用重复注销提示,以...

    EditPlus width code

    **EditPlus:强大的文本编辑器** EditPlus是一款广泛使用的文本编辑器,尤其受到程序员和Web开发者的喜爱。它以其高效、轻便和丰富的功能集而闻名。作为一个专业的IT工具,EditPlus提供了许多高级特性,使其在众多...

    edu_regis

    该项目是通过。 可用脚本 在项目目录中,可以运行: npm start 在开发模式下运行应用程序。 打开在浏览器中查看。 如果进行编辑,页面将重新加载。 您还将在控制台中看到任何棉绒错误。... 在交互式监视模式下启动...

    Behavior problems of children as perceived by teachers, mental health professionals, and children

    Regis College University of Mississippi Benedictine College The objective of this study was to examine current teacher and mental health professional ratings regarding the relative serious

    REGSim:补给估算和地下水模拟

    REGSim 补给量估算和地下水模拟是一种灵活的工具,旨在通过使用简约地下水模型估算补给量并模拟地下水位。 该工具包含两个步骤: 仿真与优化方法 不确定度和敏感性分析 它包括使用达西定律估算地下水侧向通量,...

    ez-chat:实时聊天Web应用程序

    聊天室 实时聊天Web应用程序 更新 ez-chat2在这里,试用或去。 如何托管本地聊天服务器: 要求:npm,express(npm模块),socket.io(npm...5. Open another tab and also go to "http://localhost:3000/" to regis

    webpush-demo:简单的演示,展示了使用nodejs进行Web-push的实现示例

    npm installnode ./generate-vapid.js # copy these keys to server.js and index.htmlnode ./server.js扎克(Dakia) Wygeneruj klucze VAPID node ./vapidkey-generator.js Rejestracja服务人员Następujepowejś...

    struts2校验框架使用心得.txt

    // getter and setter methods... @Override public void validate() { // 验证逻辑 if (username == null || username.trim().length() == 0) { addFieldError("username", "用户名不能为空!"); } if ...

    xmljava系统源码-Java-Security:Java安全文档

    xml java系统源码 Java-Security Java Security Documents ##author:genxor Auditing and Bypassing Security Manager policies.docx Java Deserialization Vulnerabilities - The Forgotten ...Regis

    vs2015程序出现无法连接到已配置的开发 Web 服务器问题

    在使用Visual Studio 2015 (VS2015) 开发Web应用程序时,有时会遇到一个常见的问题,即“无法连接到已配置的开发Web服务器”。这个问题可能让开发者感到困扰,尤其是在尝试了多种网上的解决方案却无果的情况下。...

    教师、心理健康专家和儿童所感知的儿童行为问题

    Regis College University of Mississippi Benedictine College The objective of this study was to examine current teacher and mental health professional ratings regarding the relative serious

    2D3DRegistration:医学图像的2D3D配准算法

    在医疗成像领域,2D3D配准是一种关键技术,用于将二维(2D)的医学影像(如X光、CT扫描或MRI切片)与三维(3D)的影像数据(如3D CT或MRI体积)进行精确对齐。这种配准过程对于多种临床应用至关重要,比如介入手术、...

Global site tag (gtag.js) - Google Analytics