5.3 AMQ消息存储

5.3 The AMQ message store

5.3 AMQ消息存储

 

The AMQ message store, like KahaDB, is a combination of a transactional journal for

reliable persistence (to survive system crashes) and high-performance indexes, which

makes this store the best option when message throughput is the main requirement

for an application. But because it uses two separate files for every index, and there’s

an index per destination, the AMQ message store shouldn’t be used if you intend to

use thousands of queues per broker. Also, recovery can be slow if the ActiveMQ broker

isn’t shut down cleanly. This is because all the indexes need to be rebuilt, which

requires the broker to traverse all its data logs to accurately build the indexes again.

In the next section, we’ll briefly examine the internals of the AMQ message store,

which are similar to the components of KahaDB.

 

AMQ消息存储和KahaDB类似,具有保障可靠持久化的事务日志以及搞性能索引,因而当消息吞吐量是主要需求时,

应用程序使用这种消息存储是最好的选择.但是因为AMQ消息存储中每个索引都使用连个文件,而且为每一个

消息目的地都单独使用索引,所以当每个代理中会产生大量的消息队列是不应该使用AMQ消息存储.

同样,使用这种消息存储代理从非正常关闭中恢复会比较慢.原因是代理恢复时,所有的的索引需要重建,这就

要求代理遍历它的数据日志文件以便正确的重建索引.

 

5.3.1 The AMQ message store internals

5.3.1 深入AMQ消息存储

 

The main components of the AMQ message store are similar to that of the KahaDB

message store, in that there’s a cache, message data logs, and a reference store for

accessing the data logs in order. Figure 5.5 provides a high-level diagram of the AMQ

message store.

 

AMQ消息的存储的主要组成部分和KahaDB消息存储类似,包括缓存,以及用于按顺序存取消息数据日志文件

的引用.图5.5是AMQ消息存储机制图.

 

The diagram provides a view of the three distinct parts of the AMQ message store:

 The data logs—These act as a message journal.

 The cache —This holds messages for fast retrieval in memory after they’ve been written to the data logs.

 The reference store —This holds references to the messages in the journal that are indexed by their message ID.

 

It’s important to understand the file-based directory structure used by the ActiveMQ

message store. This will help with the configuration and also with problem identification

when using ActiveMQ.

 

如图所示,AMQ消息存储包含一下三个独立部分:

数据日志文件 -- 该文件保存消息引用,这些消息已根据消息ID建立了索引

缓存 -- 在消息被写入数据日志文件后,缓存是在内存中保存这些消息以便快速读取这些消息

引用存储 -- 引用存储是掉数据日志中消息的引用,这些消息已经根据消息ID建立了索引

 

5.3.2 The AMQ message store directory structure

5.3.2 AMQ消息存储的目录结构

 

When you start ActiveMQ with the AMQ message store configured, a directory will

automatically be created in which the persistent messages are held. The AMQ message

store directory contains subdirectories for all the brokers that are running on the

machine. For this reason it’s strongly recommended that each broker use a unique

name. In the default configuration for ActiveMQ, the broker name is localhost, which

needs to changed to something unique. This directory structure is represented in

figure 5.6—the AMQ store directory structure.

 

当你启动一个配置了AMQ的ActiveMQ代理,一个用于保存持久化消息的目录会被自动创建.

AMQ消息存储目录包含了一些子目录,每一个子目录对应一个在本机上运行的代理.因此,建议

给不同的代理取不同的名称.ActiveMQ的默认配置中,broker的name属性为localhost,这个需要

修改成一个唯一值.图5.6所示为AMQ消息存储的目录结构.

 

The following directories and files can be found inside the data directory of an

ActiveMQ broker:

下面的目录和文件可以在ActiveMQ代理的data目录中找到:

 A lock file—  Ensures that only one broker can access this data at any given time.

                The lock is commonly used for hot standby purposes where more than one broker

                with the same name will exist on the same system.

lock文件 -- 保证任何时刻只有一个代理可以存取这个数据.lock通常用于双击人备份环境,此时系统中可能

              存在名称相同的代理.

              

  A temp-storage directory— Used for storing nonpersistent messages that can no

            longer be stored in broker memory. These messages are typically awaiting delivery

            to a slow consumer.

  temp文件存储目录 -- 用于存储不能在代理的内存中驻留的非持久化信息.这些消正在等待被发送到速度较慢的消息消费者.

  

 The kr-store — The directory structure used by the reference (index) part of the

          AMQ message store. It uses the Kaha reference store by default (Kaha is part of

          the ActiveMQ core library) to index and store references to messages in the

          data logs. There are two distinct parts to the kr-store:

        * The data directory—Contains the indexes and collections used to reference

          the messages held in the data logs. This data directory is deleted and rebuilt

          as part of recovery, if the broker hasn’t shut down cleanly. You can force

          recovery by manually deleting this directory before starting the broker.

        * The state directory—Holds information about durable topic consumers. The

          journal itself doesn’t hold information about consumers, so when it’s

          recovered it has to retrieve information about the durable subscribers first

          to accurately rebuild its database.

   kr-store目录 -- 改目录用于AMQ消息存储的引用(索引)部分.默认时使用Kaha引用存储(Kaha是ActiveMQ核心库的一部分),以建立数据

                   日志文件中消息索引的引用.kr-store包含一下两个独立部分:

 The journal directory —  Contains the data files for the data logs, and a data-control

                          file that holds some meta information. The data files are reference counted, so

                          when all the contained messages are delivered, a data file can be deleted or

                          archived.

   数据日志目录 -- 数据文件(保存数据日志)和数据控制文件(保存一些元数据信息).数据文件会被引用计数,因此当文件中包含的所有消息

               都被分发了时,改数据文件会被删除或归档.

                          

                          

 The archive directory —  Exists only if archiving is enabled. Its default location can

                            be found next to the journal. It makes sense to use a separate partition or disk.

                            The archive is used to store data logs from the journal directory, which are

                            moved here instead of being deleted. This makes it possible to replay messages

                            from the archive at a later point. To replay messages, move the archived data logs

                            (or a subset) to a new journal directory and start a new broker pointed to the

                            location of this directory. It’ll automatically replay the data logs in the journal.

   归档目录 --  只有启用归档时才产生归档目录.默认情况下,归档目录和数据日志目录同级(即和journal目录在一起).

                归档目录应该放到其他系统分区或者磁盘上(不要与journal放在一起).归档文件用来保存journal中不再使用的

                数据日志文件(如果不归档的话,这些数据日志文件本来应该会被删除).这样就可以在以后重播已经处理过的消息.

                要重播消息,可以移动归档后的数据日志文件(或该文件的一部分)到一个新的journal目录,然后启动一个指向

                这个新journal目录的代理,然后这个新启动的代理会自动重播这个journal中的消息.

                

                            

Now that the basics of the AMQ message store have been covered, the next step is to

review its configuration.

MQ的基础内已经介绍完毕,接下来看看如何配置AMQ消息存储.

 

5.3.3 Configuring the AMQ message store

5.3.3 配置AMQ消息存储

 

The AMQ store configuration allows the user to change its basic behaviour around

indexing, checkpoint intervals, and the size of the journal data files. These items and

many more can be customized through the use of properties. The key properties for

the AMQ store are shown in table 5.2.

 

通过配置AMQ消息存储,用户可以修改它的基本行为,包括索引建立,执行checkpoint的时间间隔和日志数据文件大小.

这些条目还有其他很多条目都可以通过属性自定义.AMQ消息存储的主要属性如表5.2所示:

 

Table 5.2 Configuration properties for the AMQ message store

表5.2 AMQ消息存储的可配置属性

 

Property name         Default value           Description

属性名称              默认值                   描述

 

directory           activemq-data           The directory path used by the AMQ message store.

directory           activemq-data           AMQ消息存储使用的目录

 

useNIO              true                    NIO provides faster write-through to the systems disks.

useNIO              true                    JDK的NIO可以更快的写入文件到系统磁盘

 

syncOnWrite         false                   Syncs every write to disk.

syncOnWrite         false                   是否同步所有的磁盘写操作

 

syncOnTransaction   true                    Syncs every transaction to disk.

syncOnTransaction   true                    是否同步所有的事务写操作

 

maxFileLength       32mb                    The maximum size of the message journal data files before a new one is used.

maxFileLength       32mb                    日志数据文件的最大尺寸,超过这个值将创建一个新文件

 

persistentIndex     true                    Persistent indexes are used. If false, an in-memory HashMap is used.

persistentIndex     true                    是否使用持久化索引.如果设置为false,将使用内存中的HashMap作为替代.

 

maxCheckpointMessageAddSize   4kb           The maximum memory used for a transaction before writing to disk.

maxCheckpointMessageAddSize   4kb           单个事务可使用的最大内存大小,超过这个值数据会被写入磁盘

 

cleanupInterval     3000(ms)                Time before checking which journal data files are still required.

cleanupInterval     3000(ms)                进行日志数据文件是否需要的检查操作 的时间间隔(不需要的数据日志文件会被删除或归档(配置了归档时))

 

checkpointInterval  20000(ms)               Time before moving cached message IDs to the reference store indexes.

checkpointInterval  20000(ms)               checkpoint的时间间隔,缓存的消息ID在内存中存在的时间,超过这个时间缓存被写入索引文件

 

indexBinSize        1024                    The initial number of hash bins to use for indexes.

indexBinSize        1024                    用于索引的哈希容器的初始化大小

 

indexMaxBinSize     16384                   The maximum number of hash bins to use.

indexMaxBinSize     16384                   哈希容器容量的最大值

 

directoryArchive    archive                 The directory path used by the AMQ message store to place archived journal files.

directoryArchive    archive                 归档目录,AMQ消息存储用来存放归档文件的地方

 

archiveDataLogs     false                   If true, journal files are moved to the archive instead of being deleted.

archiveDataLogs     false                   是否启用归档.若启用,不用的日志文件会被移动到规定目录,而不是删除掉.

 

recoverReferenceStore       true                Recovers the reference store if the broker isn’t shut down cleanly; 

                                                this errs on the side of extreme caution.

recoverReferenceStore       true                如果代理非正常关闭,是否恢复索引存储. 这个选项需要谨慎配置                                      

forceRecoverReferenceStore  false               Forces a recovery of the reference store.

 

Here’s an example of using the properties from table 5.2 in an ActiveMQ XML configuration

file:

下面是使用表5.2中列出的属性,配置ActiveMQ的XML代码示例:

 

<?xml version="1.0" encoding="UTF-8"?>

<beans>

  <broker xmlns="http://activemq.apache.org/schema/core">

    <persistenceAdapter>

      <amqPersistenceAdapter

      directory="target/Broker2-data/activemq-data"

      syncOnWrite="true"

      indexPageSize="16kb"

      indexMaxBinSize="100"

      maxFileLength="10mb" />

    </persistenceAdapter>

  </broker>

</beans>

 

This is but a small example of a customized configuration for the AMQ store using the

available properties.

这是使用属性配置AMQ消息存储的小例子.

 

The AMQ store, like the KahaDB store, enables users to get up and running

quickly, as there are no external dependencies on other databases. But when you want

to run an ActiveMQ broker and use an already established relational database, you

need to use a JDBC message store.

 

与KahaDB一样,AMQ消息存储也允许用户快速使用,因为这种消息存储方式不需要

依赖外部的其他数据库.但是,假如你想使用已有的关系数据库来运行ActiveMQ代理,你需要

使用JDBC消息存储.