`
liuxinglanyue
  • 浏览: 565036 次
  • 性别: Icon_minigender_1
  • 来自: 杭州
社区版块
存档分类
最新评论

Redis, from the Ground Up(4)

阅读更多

Redis Virtual Memory

The goal of Redis Virtual Memory (VM) is to swap infrequently-accessed data from RAM to disk, without drastically changing the performance characteristics of the database. This enables a single instance of Redis to support datasets that are larger than main memory.

Virtual Memory is a very important feature of most modern operating systems. However, for efficiency reasons, Redis does not use the OS-supplied VM facilities and instead implements its own system. The rationale is as follows:

  1. A single page, as managed by the OS, is 4 kB.
  2. The value of a single Redis key may touch many different pages, even if the key is small enough to fit in a single page.
  3. For reasons previously discussed, Redis objects can be an order of magnitude larger in RAM than they are when stored on disk. Therefore, if using the OS' Virtual Memory facilities, the OS would need to perform an order of magnitude more I/O versus a custom Redis Virtual Memory implementation.
  4. Hence, by building Virtual Memory into the database server, overall efficiency can be significantly improved.
Limitations

There are a few main limitations of Redis Virtual Memory:

  • All keys must be stored in memory at all times. Values can be swapped to disk, but keys cannot.
  • Values must be swapped in their entirety, even for complex types. For example, if a list has one thousand items, all one thousand items must be resident in main memory before any list-related operation can be performed, including accessing the head of the list or appending a single item to the list’s tail.
Implementation Details

When Virtual Memory is enabled, Redis stores the last time that each object was accessed. Additionally, Redis maintains a swap file that is divided into pages of configurable size, with the page allocation table stored in memory. Each page uses 1 bit of actual RAM.

When Redis is out of memory and there is something to swap, a few random objects from the dataset are sampled. The object with the higher “swappability factor” is the object that will be swapped to disk.

    Swappability = Object.age * Logarithm(Object.used_memory)

Redis maintains a pool of I/O threads that are solely responsible for loading values from disk into RAM.

When a request arrives, the command is read and the list of keys is examined. If any of the keys have been swapped to disk, the client is temporarily suspended while an I/O job is enqueued. Finally, once all keys that are needed by a given client are loaded, then the client resumes execution of the command.

From a configuration perspective, the vm-max-memory setting can be used to set the maximum amount of memory that Redis can use before it swaps to disk.

For more detail, see Redis Virtual Memory: the Story and the Code.

Publish/Subscribe

Redis has native support for publish/subscribe.

In addition to supporting exact matches on channel names, it is also possible to subscribe against a pattern. In this way, subscribers do not need to know the exact name of all channels a priori, thereby increasing the flexibility of this messaging mechanism.

Although pub/sub may seem like an odd fit, Redis' internals are very well suited for this feature. Furthermore, pub/sub brings with it numerous advantages. In particular, this feature is highly convenient in the context of the use cases of a large class of modern web applications, and, with some creativity, can be used as a substitute for not having native scripting support within Redis.

Example

Imagine the scenario where a news-related site needs to update the cached copy of its home page every time that a new article is published.

The background cache worker process subscribes to all channels that begin with ‘new.article.’:

    redis> PSUBSCRIBE new.article.*

The article publishing process creates a new technology article (in this example, this article has ID ‘1021’), adds the article’s ID to the set of all technology articles, and publishes the article’s ID to the ‘new.article.technology’ channel:

    redis> MULTI
    OK
    redis> SET article.technology.1021 "In today's technology news, ..."
    QUEUED
    redis> SADD article.technology 1021
    QUEUED
    redis> PUBLISH new.article.technology 1021
    QUEUED
    redis> EXEC
    1. OK
    2. (integer) 1
    3. (integer) 1

At this point, the background cache worker process will receive a message and know immediately that a new technology article was published, subsequently executing the appropriate callback to re-generate the home page.

Usage Examples

Redis is extremely flexible and highly usable in a number of different scenarios.

I see Redis definitely more as a flexible tool than as a solution specialized to solve a specific problem: his mixed soul of cache, store, and messaging server shows this very well. 

Salvatore Sanfilippo

small sampling of potential applications:

Caching

Caching (particularly for web applications) is likely Redis' most common use case. For details on configuring Redis as an LRU cache, see here.

Interestingly, despite memcached’s dominance in this area, plain key-value stores (i.e. those without support for data types like lists and sets) are at a disadvantage when acting as a web application cache.

For example, the resources returned from requests to web apps are typically composed of lists (lists of posts, lists of comments, lists of friends, etc.). With plain key-value stores, these lists will almost always be stored in single units (“blobs”). This makes very common list-related operations, such as adding an element to a list, getting the first ten items in a list, deleting the last item in a list, etc. very inefficient because the list is stored as a single unit and needs to frequently be serialized and deserialized within the application server. Furthermore, atomic updates of these lists are impossible without implementing some other mutual exclusion system. (Redis, with native support for lists, can perform these operations efficiently and atomically.)

This flexibility enables other cache-related advantages. For example:

One potential use for Redis is as a smarter replacement for memcached. A common challenge with caching systems is de-caching things based on dependencies - if a blog entry tagged with "redis" and "python" has its title updated, the cache for both the entry page and the "redis" and "python" tag pages needs to be cleared. Redis sets could be used to keep track of dependencies and hence take a much more finely grained approach to cache invalidation. 

Simon Willison, Redis Tutorial
Nginx + Redis

This is a more specific type of (web application) caching than described above. Here, responses for certain types of dynamic requests are delivered directly to the requestor via the cache, bypassing the application server entirely. (See here for a more detailed treatment of the subject.)

With the HttpRedis module, the Nginx web server can serve certain requests directly from Redis.

Interprocess Communication

Redis provides a very effective set of primitives for multiple processes on a single machine (or multiple machines connected via a network) to share state and communicate via message passing.

Views

Redis can be used to compute “views” for tables in relational (or other NoSQL) databases that are difficult to query effectively, due to factors such as schema design, index design, data volume, write volume, etc.

For example, given a relational table that is used in an append-only fashion, a daemon could periodically pull down rows that it has not yet processed and “explode” the data into Redis, building out a number of lists, sets, sorted sets, counters, etc. (This is, effectively, hand-rolled index generation.) A reporting script can then perform operations against these data structures to compute all of the desired metrics.

Job Management

Resque (and alternate implementations, like Pyres) leverage Redis' capabilities very extensively.

A number of other job systems/ task queues (e.g. Celery and Octobot) also support Redis.

Locking

Redis can be used to implement a lock service. As described earlier, SETNX is a key element of this locking algorithm.

Designing with Redis

There is no query optimizer. Redis provides extremely fast primitives, but overall query performance is highly dependent on how the user chooses to arrange the data.

The most important things to remember are:

  • The layout of the data should be designed based on how it will be queried.
  • It is the user’s responsibility to manually build indexes.

As a direct consequence, data will almost always be duplicated in several places.

For example, imagine the scenario of using Redis to store a book database. An efficient data layout will include storing the details of each book (title, author, publisher, ISBN, genre, etc.) in a Redis hash.

In order to query the database to answer questions like “what other books did this book’s author write?”, the data layout should also include a number of manually-designed indexes. In this case, sets like the following should be built, each of which contain the ID number of all applicable books:

  • all authors
  • all books by author
  • all publishers
  • all books by publisher
  • all genres
  • all books by genre
  • etc.

In this example, we have duplicated the ID number of each book across multiple disparate data structures. (More generally, we have de-normalized our data to optimize the speed of each query.)

Redis cannot automatically remove all instances of a book from all indexes when the book is deleted. The application developer should keep track of all sets that a book is in (using an additional set) so that clean-up can be performed efficiently.

This type of data duplication is extremely common with non-relational data sets. For most systems, this necessitates running background workers that are responsible for constantly scanning the data set and repairing any inconsistencies that are detected.

Other Resources

Some other fantastic Redis-related resources include:

You should follow me on Twitter here.

分享到:
评论

相关推荐

    windows版redis4

    windows版本的Redis4 Redis 是一个开源(BSD许可)的,内存中的数据结构存储系统,它可以用作数据库、缓存和消息中间件。 它支持多种类型的数据结构,如 字符串(strings), 散列(hashes), 列表(lists), ...

    Redis Desktop Manager 2019.4 win x64

    Redis Desktop Manager 2019.4 win x64 是一款专为Windows用户设计的图形化Redis数据库管理工具,特别适合于对Redis数据库进行可视化操作和管理。它提供了直观的界面,使得数据库的查看、编辑、操作变得更为简单。这...

    Redis-Essentials.pdf

    Redis is the most popular in-memory key-value data store. It is very lightweight and its data types give it an edge over other competitors. If you need an in-memory database or a high-performance ...

    Redis Essentials

    Redis Essentials is a fast-paced guide that teaches the fundamentals on data types, explains how to manage data through commands, and shares experiences from big players in the industry. We start off...

    RedisDesktopManager 2019.4

    **RedisDesktopManager 2019.4:** RedisDesktopManager是一款广受欢迎的开源Redis数据库管理工具,专为简化和优化Redis数据存储的管理工作而设计。2019.4版本是该软件的一个重要更新,提供了对Redis数据库的直观、...

    分布式实现redis锁 出现错误Could not get a resource from the pool

    NULL 博文链接:https://sichen84.iteye.com/blog/2419876

    A SpringBoot project based on Redis from nowcoder..zip

    A SpringBoot project based on Redis from nowcoder.

    基于C#的NewLife.Redis高性能Redis协议封装设计源码

    本项目是基于C#的NewLife.Redis高性能Redis协议封装设计源码,包含119个文件,其中包括95个C#源文件、6个csproj文件、3个YAML文件、2个PNG图片文件、2个pubxml文件、1个Editorconfig文件、1个gitignore文件、1个eddx...

    Redis in Action

    You'll begin by getting Redis set up properly and then exploring the key-value model. Then, you'll dive into real use cases including simple caching, distributed ad targeting, and more. You'll learn ...

    redis-desktop-manager-2020.4.104 mac系统

    RedisDesktopManager 2020.4.104 mac 最新版本。Redis桌面管理器(又名RDM) -是一个可以应用于Windows, Linux和MacOS操作系统的快速开源的Redis数据库管理应用程序。这个工具为你提供了一个易于使用的GUI来访问你的...

    redis配置文件redis.conf

    redis配置文件redis.conf

    redis-5.0.3 redis-5.0.4 redis-5.0.5

    redis-5.0.3 redis-5.0.4 redis-5.0.5

    Windows版 Redis 5.0.14

    4. **Redis Cluster**: - `redis-cluster`标签表示支持Redis的集群功能。Redis Cluster是其分布式解决方案,可以将数据自动分片到多个节点,提供容错性和高可用性。 - 配置Redis Cluster需要至少三个独立的节点,...

    windows版Redis1

    4. `Redis on Windows.docx`、`Windows Service Documentation.docx`、`Redis on Windows Release Notes.docx`:这些文档提供了关于在Windows上安装、配置和管理Redis的详细信息,包括服务的创建、操作指南和版本...

    redis_4.0.10-1_arm64.deb 银河麒麟v4+飞腾

    1、redis_4.0.10-1_arm64.deb 银河麒麟v4+飞腾 安装包 2、自带服务启动 3、目录树 /opt/redis-4.0.10/ ├── bin │ ├── redis-benchmark │ ├── redis-check-aof │ ├── redis-check-rdb │ ├── ...

    Mac Redis Desktop Manager – Redis可视化管理工具 2022.4

    mac 最新redis客户端管理工具 2022.4

    log4net.redis

    《log4net.redis:将日志流导向Redis的实践与解析》 在IT行业中,日志记录是系统监控和故障排查的重要环节。log4net作为.NET Framework中的一个强大的日志框架,广泛应用于各类项目中。然而,随着微服务架构的普及...

    StackExchange.Redis Redis客户端

    最近需要在C#中使用Redis,在Redis的官网找到了ServiceStack.Redis,最后在测试的时候发现这是个坑,4.0已上已经收费,后面只好找到3系列的最终版本,最后测试发现还是有BUG或者是我不会用。没有办法,最好找到了...

    Redis Desktop Manager 2020.4

    Redis Desktop Manager官方版是一款简单快速、跨平台的Redis桌面管理工具,也被称作Redis可视化工具,Redis Desktop Manager官方版是一款开源软件,支持通过SSH Tunnel连接

Global site tag (gtag.js) - Google Analytics