- 浏览: 2539378 次
- 性别:
- 来自: 成都
文章分类
最新评论
-
nation:
你好,在部署Mesos+Spark的运行环境时,出现一个现象, ...
Spark(4)Deal with Mesos -
sillycat:
AMAZON Relatedhttps://www.godad ...
AMAZON API Gateway(2)Client Side SSL with NGINX -
sillycat:
sudo usermod -aG docker ec2-use ...
Docker and VirtualBox(1)Set up Shared Disk for Virtual Box -
sillycat:
Every Half an Hour30 * * * * /u ...
Build Home NAS(3)Data Redundancy -
sillycat:
3 List the Cron Job I Have>c ...
Build Home NAS(3)Data Redundancy
Regis Memory and Cleanup
Get total key
> dbSize
(integer) 73857708
Get random key
> randomkey
"job_12_02d5b44de7a4af119504abbed239b6f4"
Check the type of the key
> type job_budget_4313410226
hash
Check the expiration time
> ttl job_12_02d5b44de7a4af119504abbed239b6f4
(integer) -1
dump command can dump the value of that key
> dump job_12_02d5b44de7a4af119504abbed239b6f4
"\x04;\x06paused\xc0\x00\x05title\x1cLease Purchase Truck Drivers\x0blast_update\xc2\xc9\xce\xa2X\x05phone\x00\x0cdaily_capped\xc0\x00\teducation\xc0\x02\tsrcseg_id\xc0\x0c\x10excluded_company\xc0\x01\x0eminor_category\xc2\xfa\x9c\t\x00\rquality_score\xc0\x1e\x13top_spot_industries\x00\x06budget\x00\x0ejob_id_mapping\n4310204121\x04tags\x00\bindustry\xc0\x14\x14major_category_score\xc0\x01\x0bis_jobboard\xc0\x01\tsource_id\xc0\x0c\x06is_add\xc0\x00\nsearchtags\x00\x06posted\xc2\xc9\xce\xa2X\x03url\xc3@\x80@\x88\x1fhttp://www.linkup.com/joblisting\x02s/? \x0c\x1f=02d5b44de7a4af119504abbed239b6f\x0c4&utm_source= /\bs2careers`\x17\tmedium=cpc`\x0e\acampaign\xe0\x02(\x01rs\x0bcustomer_id\xc0\x03\x06cities\nDecatur,AL\x0bcampaign_id\xc0\x0c\x05email\x00\areg_cpc\xc0\x00\x06action\x04save\x0faffiliate_group\xc0\x00\acreated\xc2\xc9\xce\xa2X\x06digest 7aefe0914bb80f90e65d4a3f7710ff70\nsegment_id\xc1\xfa\x01\rqq_multiplier\xc0\x00\tj2c_apply\xc0\x00\nbudgetType\bLIFETIME\x06status\aEXPIRED\x0cboost_factor\x0f3.0931627568599\ajobtype\xc0\x01\x03cpc\xc0\x1e\adeduped\x00\x13is_top_spot_preview\xc0\x00\bzipcodes\xc2\x12\x8b\x00\x00\bcpc_high\xc0\x01\nsrccamp_id\xc0\x0c\rjob_reference 02d5b44de7a4af119504abbed239b6f4\acompany\x12Super Service, LLC\x0fmobile_friendly\xc0\x01\x0cdedup_digest 978c2722410a9299ec35229babeeaf37\ncompany_id\xc2R_\x01\x00\bcategory\x00\tstate_ids\xc0\x02\x11quality_sensitive\x00\x14minor_category_score\x030.3\x0cindustry_ids\xc0\x14\x06salary\xc0\x1e\rtop_spot_type\xc0\x00\apool_id\xc0\x0c\x0emajor_category\xc2\xf0\x9c\t\x00\nexperience\xc0\x02\a\x00\xa0\xa0\xbc9\xfd\x84\xa3\""
The tool in here can get the size of one key
https://github.com/sripathikrishnan/redis-rdb-tools
> git clone https://github.com/sripathikrishnan/redis-rdb-tools
> cd redis-rdb-tools/
> sudo python3 setup.py install
> redis-memory-for-key -s test.amazonaws.com -p 6379 job_12_02d5b44de7a4af119504abbed239b6f4
Key job_12_02d5b44de7a4af119504abbed239b6f4
Bytes 5716.0
Type hash
Encoding hashtable
Number of Elements 59
Length of Largest Element 136
Pick up Keys to scan
> scan 77594624 count 10
1) "68157440"
2) 1) "idmapping_4320118968"
2) "job_2465_CB-UnDNml2EZMrepChV7fgIJIrUQWcmv"
3) "job_dedupdigest_f95fa0aea4c4fc1038a191611e69bd7a"
4) "idmapping_4301743608"
5) "job_2465_DkljWeZhGWJ4KpkY023TsSJTq8wRM-xL"
6) "job_budget_4318069707"
7) "job_dedupdigest_d1c8e5ba31218058d0ab96c110f5024f"
"job_budget_4314514161"
9) "job_dedupdigest_4856f59b52ce150c22b92cdd253f3e46"
10) "job_1624_787_416056211"
Scan with Pattern
> scan 0 count 10 match job_*
1) "92274688"
2) 1) "job_budget_4314725990"
2) "job_2465_a1r2Sr6OfBfWgcoEGlDlWumX3Ckx4t3S"
3) "job_4130_2124_951ecb1c50c572cdbcaf0ee261ecd04f_228_i"
4) "job_1627_90433loc459690"
5) "job_1052_8WMXY5c-IUs-T2Y2aPch6xVnhUYPXIEM"
6) "job_budget_4316788589"
7) "job_dedupdigest_6782069fbe5c5e51b742a26b48fe8156"
Finally I write some scripts to measure the memory usage and write a script to scan and clean the Redis.
The major party will be these
if($index % 50 == 0){
$logger->info("processed {$index} / {$count} ...");
}
//fetch key
$key = $redisClient->getRandomKey();
//memory information
$memory = $this->getMemoryUsage($key);
//find the pattern
$pattern = $this->matchPattern($key);
//add the report
if($pattern == '/job_.*/'){
//find job status
$status = $redisClient->getJobStatus($key);
if(empty($status)){
$logger-warning("job {$key} does not have status....");
}
$reports[$pattern . $status]['memory'] = $reports[$pattern . $status]['memory'] + $memory['Bytes'];
$reports[$pattern . $status]['type'] = $memory['Type'];
$reports[$pattern . $status]['count'] = $reports[$pattern . $status]['count'] + 1;
}else{
$reports[$pattern]['memory'] = $reports[$pattern]['memory'] + $memory['Bytes'];
$reports[$pattern]['type'] = $memory['Type'];
$reports[$pattern]['count'] = $reports[$pattern]['count'] + 1;
}
$reports['all']['memory'] = $reports['all']['memory'] + $memory['Bytes'];
$reports['all']['count'] = $reports['all']['count'] + 1;
…snip...
How to fetch memory information
public function getMemoryUsage($key)
{
$redisClient = $this->ioc->getService("redisClient");
exec('redis-memory-for-key -s prod-jobs.pnuura.ng.0001.use1.cache.amazonaws.com -p 6379 ' . $key , $outputs);
$result = array();
foreach ($outputs as $item){
$values = preg_split('/\s+/', $item);
$result[$values[0]] = $values[1];
}
return $result;
}
I scan the keys and clean the related pattern data
public function scanAndCleanExpireJobs($test = true){
$logger = $this->ioc->getService("logger");
$count = 0;
$pattern = 'job_*';
foreach (new Iterator\Keyspace($this->client, $pattern) as $key) {
//$logger->info("key is " . $key);
if(preg_match("/job_dedupdigest_.*/", $key) || preg_match("/job_budget_.*/", $key)){
//ingore
continue;
}
//find job status
$status = $this->client->hget($key, 'status');
if("EXPIRED" == $status){
if($test){
$logger->info("key is " . $key . " status " . $status);
}else{
//delete the expire job
$logger->info("deleting the expire job " . $key . " counting " . $count);
$this->client->del($key);
$count++;
}
}
}
}
Reference:
http://stackoverflow.com/questions/7638542/redis-show-database-size-size-for-keys
https://github.com/snmaynard/redis-audit/blob/master/redis-audit.rb
http://ylw6006.blog.51cto.com/470441/1281548
http://blog.csdn.net/jiangguilong2000/article/details/9313603
http://www.10tiao.com/html/256/201609/2247483748/1.html
https://github.com/antirez/redis-sampler
https://github.com/sripathikrishnan/redis-rdb-tools
Get total key
> dbSize
(integer) 73857708
Get random key
> randomkey
"job_12_02d5b44de7a4af119504abbed239b6f4"
Check the type of the key
> type job_budget_4313410226
hash
Check the expiration time
> ttl job_12_02d5b44de7a4af119504abbed239b6f4
(integer) -1
dump command can dump the value of that key
> dump job_12_02d5b44de7a4af119504abbed239b6f4
"\x04;\x06paused\xc0\x00\x05title\x1cLease Purchase Truck Drivers\x0blast_update\xc2\xc9\xce\xa2X\x05phone\x00\x0cdaily_capped\xc0\x00\teducation\xc0\x02\tsrcseg_id\xc0\x0c\x10excluded_company\xc0\x01\x0eminor_category\xc2\xfa\x9c\t\x00\rquality_score\xc0\x1e\x13top_spot_industries\x00\x06budget\x00\x0ejob_id_mapping\n4310204121\x04tags\x00\bindustry\xc0\x14\x14major_category_score\xc0\x01\x0bis_jobboard\xc0\x01\tsource_id\xc0\x0c\x06is_add\xc0\x00\nsearchtags\x00\x06posted\xc2\xc9\xce\xa2X\x03url\xc3@\x80@\x88\x1fhttp://www.linkup.com/joblisting\x02s/? \x0c\x1f=02d5b44de7a4af119504abbed239b6f\x0c4&utm_source= /\bs2careers`\x17\tmedium=cpc`\x0e\acampaign\xe0\x02(\x01rs\x0bcustomer_id\xc0\x03\x06cities\nDecatur,AL\x0bcampaign_id\xc0\x0c\x05email\x00\areg_cpc\xc0\x00\x06action\x04save\x0faffiliate_group\xc0\x00\acreated\xc2\xc9\xce\xa2X\x06digest 7aefe0914bb80f90e65d4a3f7710ff70\nsegment_id\xc1\xfa\x01\rqq_multiplier\xc0\x00\tj2c_apply\xc0\x00\nbudgetType\bLIFETIME\x06status\aEXPIRED\x0cboost_factor\x0f3.0931627568599\ajobtype\xc0\x01\x03cpc\xc0\x1e\adeduped\x00\x13is_top_spot_preview\xc0\x00\bzipcodes\xc2\x12\x8b\x00\x00\bcpc_high\xc0\x01\nsrccamp_id\xc0\x0c\rjob_reference 02d5b44de7a4af119504abbed239b6f4\acompany\x12Super Service, LLC\x0fmobile_friendly\xc0\x01\x0cdedup_digest 978c2722410a9299ec35229babeeaf37\ncompany_id\xc2R_\x01\x00\bcategory\x00\tstate_ids\xc0\x02\x11quality_sensitive\x00\x14minor_category_score\x030.3\x0cindustry_ids\xc0\x14\x06salary\xc0\x1e\rtop_spot_type\xc0\x00\apool_id\xc0\x0c\x0emajor_category\xc2\xf0\x9c\t\x00\nexperience\xc0\x02\a\x00\xa0\xa0\xbc9\xfd\x84\xa3\""
The tool in here can get the size of one key
https://github.com/sripathikrishnan/redis-rdb-tools
> git clone https://github.com/sripathikrishnan/redis-rdb-tools
> cd redis-rdb-tools/
> sudo python3 setup.py install
> redis-memory-for-key -s test.amazonaws.com -p 6379 job_12_02d5b44de7a4af119504abbed239b6f4
Key job_12_02d5b44de7a4af119504abbed239b6f4
Bytes 5716.0
Type hash
Encoding hashtable
Number of Elements 59
Length of Largest Element 136
Pick up Keys to scan
> scan 77594624 count 10
1) "68157440"
2) 1) "idmapping_4320118968"
2) "job_2465_CB-UnDNml2EZMrepChV7fgIJIrUQWcmv"
3) "job_dedupdigest_f95fa0aea4c4fc1038a191611e69bd7a"
4) "idmapping_4301743608"
5) "job_2465_DkljWeZhGWJ4KpkY023TsSJTq8wRM-xL"
6) "job_budget_4318069707"
7) "job_dedupdigest_d1c8e5ba31218058d0ab96c110f5024f"
"job_budget_4314514161"
9) "job_dedupdigest_4856f59b52ce150c22b92cdd253f3e46"
10) "job_1624_787_416056211"
Scan with Pattern
> scan 0 count 10 match job_*
1) "92274688"
2) 1) "job_budget_4314725990"
2) "job_2465_a1r2Sr6OfBfWgcoEGlDlWumX3Ckx4t3S"
3) "job_4130_2124_951ecb1c50c572cdbcaf0ee261ecd04f_228_i"
4) "job_1627_90433loc459690"
5) "job_1052_8WMXY5c-IUs-T2Y2aPch6xVnhUYPXIEM"
6) "job_budget_4316788589"
7) "job_dedupdigest_6782069fbe5c5e51b742a26b48fe8156"
Finally I write some scripts to measure the memory usage and write a script to scan and clean the Redis.
The major party will be these
if($index % 50 == 0){
$logger->info("processed {$index} / {$count} ...");
}
//fetch key
$key = $redisClient->getRandomKey();
//memory information
$memory = $this->getMemoryUsage($key);
//find the pattern
$pattern = $this->matchPattern($key);
//add the report
if($pattern == '/job_.*/'){
//find job status
$status = $redisClient->getJobStatus($key);
if(empty($status)){
$logger-warning("job {$key} does not have status....");
}
$reports[$pattern . $status]['memory'] = $reports[$pattern . $status]['memory'] + $memory['Bytes'];
$reports[$pattern . $status]['type'] = $memory['Type'];
$reports[$pattern . $status]['count'] = $reports[$pattern . $status]['count'] + 1;
}else{
$reports[$pattern]['memory'] = $reports[$pattern]['memory'] + $memory['Bytes'];
$reports[$pattern]['type'] = $memory['Type'];
$reports[$pattern]['count'] = $reports[$pattern]['count'] + 1;
}
$reports['all']['memory'] = $reports['all']['memory'] + $memory['Bytes'];
$reports['all']['count'] = $reports['all']['count'] + 1;
…snip...
How to fetch memory information
public function getMemoryUsage($key)
{
$redisClient = $this->ioc->getService("redisClient");
exec('redis-memory-for-key -s prod-jobs.pnuura.ng.0001.use1.cache.amazonaws.com -p 6379 ' . $key , $outputs);
$result = array();
foreach ($outputs as $item){
$values = preg_split('/\s+/', $item);
$result[$values[0]] = $values[1];
}
return $result;
}
I scan the keys and clean the related pattern data
public function scanAndCleanExpireJobs($test = true){
$logger = $this->ioc->getService("logger");
$count = 0;
$pattern = 'job_*';
foreach (new Iterator\Keyspace($this->client, $pattern) as $key) {
//$logger->info("key is " . $key);
if(preg_match("/job_dedupdigest_.*/", $key) || preg_match("/job_budget_.*/", $key)){
//ingore
continue;
}
//find job status
$status = $this->client->hget($key, 'status');
if("EXPIRED" == $status){
if($test){
$logger->info("key is " . $key . " status " . $status);
}else{
//delete the expire job
$logger->info("deleting the expire job " . $key . " counting " . $count);
$this->client->del($key);
$count++;
}
}
}
}
Reference:
http://stackoverflow.com/questions/7638542/redis-show-database-size-size-for-keys
https://github.com/snmaynard/redis-audit/blob/master/redis-audit.rb
http://ylw6006.blog.51cto.com/470441/1281548
http://blog.csdn.net/jiangguilong2000/article/details/9313603
http://www.10tiao.com/html/256/201609/2247483748/1.html
https://github.com/antirez/redis-sampler
https://github.com/sripathikrishnan/redis-rdb-tools
发表评论
-
Update Site will come soon
2021-06-02 04:10 1672I am still keep notes my tech n ... -
Hadoop Docker 2019 Version 3.2.1
2019-12-10 07:39 287Hadoop Docker 2019 Version 3.2. ... -
Nginx and Proxy 2019(1)Nginx Enable Lua and Parse JSON
2019-12-03 04:17 437Nginx and Proxy 2019(1)Nginx En ... -
Data Solution 2019(13)Docker Zeppelin Notebook and Memory Configuration
2019-11-09 07:15 279Data Solution 2019(13)Docker Ze ... -
Data Solution 2019(10)Spark Cluster Solution with Zeppelin
2019-10-29 08:37 243Data Solution 2019(10)Spark Clu ... -
AMAZON Kinesis Firehose 2019(1)Firehose Buffer to S3
2019-10-01 10:15 313AMAZON Kinesis Firehose 2019(1) ... -
Rancher and k8s 2019(3)Clean Installation on CentOS7
2019-09-19 23:25 300Rancher and k8s 2019(3)Clean In ... -
Pacemaker 2019(1)Introduction and Installation on CentOS7
2019-09-11 05:48 333Pacemaker 2019(1)Introduction a ... -
Crontab-UI installation and Introduction
2019-08-30 05:54 441Crontab-UI installation and Int ... -
Spiderkeeper 2019(1)Installation and Introduction
2019-08-29 06:49 492Spiderkeeper 2019(1)Installatio ... -
Supervisor 2019(2)Ubuntu and Multiple Services
2019-08-19 10:53 362Supervisor 2019(2)Ubuntu and Mu ... -
Supervisor 2019(1)CentOS 7
2019-08-19 09:33 321Supervisor 2019(1)CentOS 7 Ins ... -
Redis Cluster 2019(3)Redis Cluster on CentOS
2019-08-17 04:07 365Redis Cluster 2019(3)Redis Clus ... -
MySQL HA Solution 2019(4)MaxScale
2019-08-03 21:19 782MySQL HA Solution 2019(4)MaxSca ... -
Amazon Lambda and Version Limit
2019-08-02 01:42 430Amazon Lambda and Version Limit ... -
MySQL HA Solution 2019(3)MyCat
2019-07-30 11:45 365MySQL HA Solution 2019(3)MyCat ... -
MySQL HA Solution 2019(2)ProxySQL
2019-07-29 09:39 481MySQL HA Solution 2019(2)ProxyS ... -
MySQL HA Solution 2019(1)Master Slave on MySQL 5.7
2019-07-27 22:26 505MySQL HA Solution 2019(1)Master ... -
RabbitMQ Cluster 2019(2)Cluster HA and Proxy
2019-07-11 12:41 455RabbitMQ Cluster 2019(2)Cluster ... -
Zeppelin and Spark Library Dependency
2019-07-09 03:20 454Zeppelin and Spark Library Depe ...
相关推荐
将HPGL绘图仪输出文件转换为DEC ReGIS文件,适合在支持DEC ReGIS的终端和终端仿真器上显示。 这也是可免费使用的ReGIS贡献文件存储库。 如果您有Regis文件可以给我发电子邮件。 安装zip文件包含许多HPGL文件和一个...
很抱歉,根据您提供的信息,"Prova-Regis-Cardoso"似乎与IT行业的主要知识点不直接相关。标题和描述更像是与食品品尝或个人评价有关的内容,这超出了我的专业领域——IT和编程。标签为空,也没有提供任何额外的IT上...
【标题】"Regis Jesuit High School FRC Team #3729-开源"涉及到的是一个高中学校的机器人竞赛团队——Regis Jesuit High School的FIRST Robotics Team 3729,该团队专注于开源软件的开发。他们从2011年的新人赛季...
An Android console view, which allows you to log text using static calls, to easily debug your application, whilst avoiding memory leaks. Usage Include Console anywhere in your layout: then ...
丹佛Airbnb数据分析项目这是瑞吉斯大学MSDS计划的实践项目。 原始数据集可从获得。 丹佛市的天际线照片,来自埃克森·托德(Erick Todd),来自Pexels介绍在度假方面,Airbnb已成为酒店的代名词。...
语言:English 禁用RU(里吉斯大学)WorldClass超时警告,并按需隐藏RU标语。 单击按钮,从讨论或内容页面中删除横幅,以实现最大的可视性。 再次单击以显示横幅。 当您专注于其他任务时,会自动禁用重复注销提示,以...
**EditPlus:强大的文本编辑器** EditPlus是一款广泛使用的文本编辑器,尤其受到程序员和Web开发者的喜爱。它以其高效、轻便和丰富的功能集而闻名。作为一个专业的IT工具,EditPlus提供了许多高级特性,使其在众多...
该项目是通过。 可用脚本 在项目目录中,可以运行: npm start 在开发模式下运行应用程序。 打开在浏览器中查看。 如果进行编辑,页面将重新加载。 您还将在控制台中看到任何棉绒错误。... 在交互式监视模式下启动...
Regis College University of Mississippi Benedictine College The objective of this study was to examine current teacher and mental health professional ratings regarding the relative serious
REGSim 补给量估算和地下水模拟是一种灵活的工具,旨在通过使用简约地下水模型估算补给量并模拟地下水位。 该工具包含两个步骤: 仿真与优化方法 不确定度和敏感性分析 它包括使用达西定律估算地下水侧向通量,...
聊天室 实时聊天Web应用程序 更新 ez-chat2在这里,试用或去。 如何托管本地聊天服务器: 要求:npm,express(npm模块),socket.io(npm...5. Open another tab and also go to "http://localhost:3000/" to regis
npm installnode ./generate-vapid.js # copy these keys to server.js and index.htmlnode ./server.js扎克(Dakia) Wygeneruj klucze VAPID node ./vapidkey-generator.js Rejestracja服务人员Następujepowejś...
// getter and setter methods... @Override public void validate() { // 验证逻辑 if (username == null || username.trim().length() == 0) { addFieldError("username", "用户名不能为空!"); } if ...
xml java系统源码 Java-Security Java Security Documents ##author:genxor Auditing and Bypassing Security Manager policies.docx Java Deserialization Vulnerabilities - The Forgotten ...Regis
在使用Visual Studio 2015 (VS2015) 开发Web应用程序时,有时会遇到一个常见的问题,即“无法连接到已配置的开发Web服务器”。这个问题可能让开发者感到困扰,尤其是在尝试了多种网上的解决方案却无果的情况下。...
Regis College University of Mississippi Benedictine College The objective of this study was to examine current teacher and mental health professional ratings regarding the relative serious
在医疗成像领域,2D3D配准是一种关键技术,用于将二维(2D)的医学影像(如X光、CT扫描或MRI切片)与三维(3D)的影像数据(如3D CT或MRI体积)进行精确对齐。这种配准过程对于多种临床应用至关重要,比如介入手术、...