5.2 KahaDB消息存储

5.2 The KahaDB message store

5.2 KahaDB消息存储

 

The recommended message store for general-purpose messages since ActiveMQ version

5.3 is KahaDB. This is a file-based message store that combines a transactional journal,

for reliable message storage and recovery, with good performance and scalability.

 

ActiveMQ从5.3版本以后,推荐使用KahaDB作为通用的消息存储方式.KahaDB是一种基于文件的存储,具有

卓越的性能和扩展性,具有事务日志功能,能够可靠的今昔消息的存储和恢复.

 

The KahaDB store is a file-based, transactional store that’s been tuned and designed

for the fast storage of messages. The aim of the KahaDB store is to be easy to use and as

fast as possible. Its use of a file-based message database means there’s no prerequisite

for a third-party database. This message store enables ActiveMQ to be downloaded and

running in literally minutes. In addition, the structure of the KahaDB store has been

streamlined especially for the requirements of a message broker.

 

KahaDB存储是一种基于文件,具有事务存储功能,为消息的快速存储而设计并做了优化.

KahaDB存储的目标是易于使用并且做到尽可能的快速.使用基于文件的消息数据库表明

KahaDB不需要使用第三方数据库.这种消息存储使得能在数分钟之内完成ActiveMQ的下载和运行.

并且,为满足消息代理的需求,KahaDB的存储结构已经做了特殊的优化.

 

The KahaDB message store uses a transactional log for its indexes and only uses

one index file for all its destinations. It’s been used in production environments with

10,000 active connections, each connection having a separate queue. The configurability

of the KahaDB store means that it can be tuned for most usage scenarios, from

high throughput applications (for example, trading platforms), to storing large

amounts of messages (for example, GPS tracking).

 

KahaDB消息存储使用一个事物日志文件作为索引,并且所有的destination只使用一个索引文件.

KahaDB已可用于这样的生产环境:具有10000个活动连接,并且每一个连接拥有一个独立的队列.

KahaDB的可配置性说明,KahaDB可以针对从要求高吞吐量(比如交易平台)到处理海量消息(比如GPS跟踪)

等不同的应用场景进行调优.

 

To enable the KahaDB store for ActiveMQ, you need to configure the

<persistenceAdapter> element in the activemq.xml configuration file. Here’s a minimal

configuration for the KahaDB message store:

 

为了在ActiveMQ启用KahaDB存储,你需要在配置文件activemq.xml中配置<persistenceAdapter> 

元素.下面是启用KahaDB存储的一个最小化配置例子:

 

<broker brokerName="broker" persistent="true" useShutdownHook="false">

  ...

  <persistenceAdapter>

    <kahaDB directory="activemq-data" journalMaxFileLength="16mb"/>

  </persistenceAdapter>

  ...

</broker>

 

If you want to embed an ActiveMQ broker inside an application, the message store can

also be configured programmatically. Here’s an example of a programmatic configuration

for KahaDB:

 

如果你打算在应用程序内部启动一个嵌入式的ActiveMQ代理,也可以使用程序代码来配置消息存储.

下面是一个使用程序代码配置KahaDB的例子:

 

public class EmbeddedBrokerUsingAMQStoreExample 

{

  BrokerService createEmbeddedBroker() throws Exception 

  {

    BrokerService broker = new BrokerService();

    File dataFileDir = new File("target/amq-in-action/kahadb");

    KahaDBStore kaha = new KahaDBStore();

    kaha.setDirectory(dataFileDir);

    // Using a bigger journal file

    kaha.setJournalMaxFileLength(1024*100);

    // small batch means more frequent and smaller writes

    kaha.setIndexWriteBatchSize(100);

    // do the index write in a separate thread

    kaha.setEnableIndexWriteAsync(true);

    broker.setPersistenceAdapter(kaha);

    //create a transport connector

    broker.addConnector("tcp://localhost:61616");

    //start the broker

    broker.start();

    return broker;

  }

}

 

 

Although the example seems small, it’s enough to create an ActiveMQ broker using

the KahaDB message store and listen for ActiveMQ clients connecting over TCP. For

more information about embedding ActiveMQ, see chapter 8.

 

尽管上面的实例代码看起来比较小巧,但是代码已足够创建一个ActiveMQ代理并使用KahaDB作为

消息存储同时监听通过TCP协议连接的ActiveMQ客户端.更多关于嵌入式ActiveMQ的内容请参阅

第8章内容.

 

In order to better understand its use and configuration, it’s important to examine

the internals of the KahaDB message store.

 

为了更好的理解和配置KahaDB消息存储,能深入KahaDB消息存储的内部一探究竟变得十分

重要.

 

5.2.1 The KahaDB message store internals

5.2.1 KahaDB消息存储内部探秘

 

The KahaDB message store is the fastest of all the provided message store implementations.

Its speed is the result of the combination of a fast transactional journal comprised

of data log files, the highly optimized indexing of message IDs, and inmemory

message caching. Figure 5.3 provides a high-level diagram of the KahaDB

message store.

 

在所有以提供的消息存储实现中,KahaDB是最快的.KahaDB的高速度源自一种包含数据日志文件的

高速事务日志文件和根据消息ID高度优化的索引以及基于内存的消息缓存.图5.3显示了KahaDB

消息存储概览.

 

The diagram provides a view of the three distinct parts of the KahaDB message

store including the following:

 

图5.3展示了KahaDB消息存储的三个独立的构件如下所示:

 

The data logs act as a message journal, which consists of a rolling

log of messages and commands (such as transactional boundaries and message deletions)

stored in data files of a certain length. When the maximum

length of the currently used data file has been reached,

a new data file is created. All the messages in a data file are reference

counted, so that once every message in that data file is no longer required, 

the data file can be removed or archived. In the data logs, messages are only

appended to the end of the current data file, so storage is fast.

 

数据日志类似消息日记,用固定长度的文件保存滚动消息和命令(比如事务边界和消息删除指令).

当数据日志文件的大小达到最大值时,一个新的数据文件会被自动创建.日志文件中的所有消息

都会被引用计数,这样一旦文件中所有的消息都不在需要了,这个数据文件会被删除或者归档.

在数据日志中,消息只会被附加到当前文件的尾部,所以存储非常快.

 

The cache holds messages temporarily if there are active consumer(s) for the messages. 

If there are active consumers, messages are dispatched at the same time they’re

scheduled to be stored. If messages are acknowledged in time, they don’t need

to be written to disk.

 

当有活动的消息消费者时,消息会临时放到缓存中.如果有活动的消费者,消息在被存储同时会被分发.

如果消息被处理并及时确认过,就没有必要将消息写入磁盘了.

 

The BTree indexes hold references to the messages in the data logs that are

indexed by their message ID. The indexes maintain the FIFO data structure for

queues and the durable subscriber pointers to their topic messages. The redo

log is used only if the ActiveMQ broker hasn’t shut down cleanly, and are used

to insure the integrity of the BTree index is maintained.

 

BTree索引文件保存数据日志文件中的消息引用,这些消息已根据消息ID建立了索引.消息索引

为消息队列维护一个先进先出的数据结构,同时维护由持久的消息订阅者指向其订阅的的主题中消息

的指针.重做日志文件仅用当ActiveMQ没有正常关闭时,保证BTree索引仍然完整.

 

The KahaDB uses different files on disk for its data logs and indexes, so in the next

section we’ll show a typical KahaDB directory structure.

 

KahaDB为数据日志和索引使用不同的磁盘文件,所以在下面的章节中我们将看到典型的KahaDB

目录结构.

 

5.2.2 The KahaDB message store directory structure

5.2.2 KahaDB消息存储的目录结构

 

When you start an ActiveMQ broker configured to use a KahaDB store, a directory will

automatically be created in which the persistent messages are stored. This directory

structure is shown in figure 5.4.

当你启动了配置KahaDB存储的ActiveMQ代理是,消息存储持久化文件目录会自动创建,该目录结构

如图5.4所示:

 

Inside of the KahaDB directory, the following directory and file structures can be

found:

KahaDB的目录结构中的目录和文件如下:

 

 db log files—KahaDB stores messages into data log files named db-<Number>.log of

a predefined size. When a data log is full, a new one will be created, and the log

number incremented. When there are no more references to any of the messages

in the data log file, it’ll be deleted or archived.

db log 文件 --KahaDB将消息存储到数据日志文件中,该文件命名方式为db-<编号>.log,文件大小事先已经定义好.

当数据日志文件达到预定的大小,一个新的文件会自动创建,同时文件名中的编号自动加1.如果没有任何引用指向

数据日志文件中的消息,那么该日志文件将被删除或者归档.

 

archive directory—This exists only if archiving is enabled. The archive is used to store data logs

that are no longer needed by KahaDB, making it possible to replay messages from the archived data 

logs at a later point. If archiving isn’t enabled(the default), data logs that are no longer in 

use are deleted from the file system.

归档目录 -- 归档目录仅在启用归档时才会出现.归档文件用于存储那些不再被KahaDB使用的数据日志文件.

使用归档文件可以保存消息以便以后重新检视消息变为可能.如果没有启用归档功能(默认不启用归档),不在使用的数据

日志文件会从文件系统中删除.

 

db.data —This file contains the persistent BTree indexes to the messages held in the message data logs.

db.data -- 该文件保存了数据日志文件中消息的持久化BTree索引.

 

db.redo —This is the redo file, used for recovering the BTree indexes if the KahaDB 

message store starts after a hard stop.

db.redo -- 这是重做日志文件,用于KahaDB从一次非正常关闭后重启时恢复BTree索引.

 

Now that we’ve covered the basics of the KahaDB store, the next step is to review its

configuration.

现在我们已经了解了KahaDB的基础知识,下一步让我们回顾下KahaDB的配置.

 

5.2.3 Configuring the KahaDB message store

5.2.3 配置KahaDB消息存储

 

The KahaDB message store can be configured in the activemq.xml file. Its configuration

options control the different tuning parameters, as described in table 5.1.

可以在activemq.xml文件中配置KahaDB消息存储.不同的配置选择控制控制不同的功能参数,如表

5.1所示:

 

Table 5.1 Configuration options available for the KahaDB message store

表5.1 KahaDB消息存储的配置选项

 

Property name               Default value            Description

属性名                        默认值                     描述

directory                 activemq-data           Directory path used by KahaDB

directory                 activemq-data           KahaDB使用的目录

 

indexWriteBatchSize           1000        Number of index pages to write in a batch to disk

indexWriteBatchSize           1000        一次写入磁盘的属性页数量

                              

indexCacheSize                10000       Number of index pages cached in memory

indexCacheSize                10000       内存中缓存的所引用数量

                              

enableIndexWriteAsync         false       If set, will asynchronously write indexes

enableIndexWriteAsync         false       如果设置为true,索引将以异步方式写入

                              

journalMaxFileLength          32mb        A hint to set the maximum size of each of the message data logs

journalMaxFileLength          32mb        设置每个消息数据日志文件的最大尺寸

                              

enableJournalDiskSyncs        true        Ensures every nontransactional journal write is followed 

                                          by a disk sync (JMS durability requirement)

enableJournalDiskSyncs        true        保证每个非事务日志写操作后进行磁盘同步(JMS持久化要求)                                                  

 

cleanupInterval               30000       Time (ms) before checking for and discarding/moving message 

                                          data logs that are no longer used

cleanupInterval               30000       检查 丢弃/移动数据日志文件中是不不再使用的消息 间隔时间(单位 毫秒)

 

checkpointInterval            5000        Time (ms) before checkpointing the journal

checkpointInterval            5000        执行checkpoint的时间间隔

 

ignoreMissingJournalfiles     false       If enabled, will ignore a missing message log file

ignoreMissingJournalfiles     false       如果启用,则忽视丢失消息日志(译注:在消息丢失时不记录日志)

 

checkForCorruptJournalFiles   false       If enabled, on startup will validate that the message data logs haven’t been corrupted.

checkForCorruptJournalFiles   false       如果启用,在启动是会检查消息数据日志文件是否被占用

 

checksumJournalFiles          false       If enabled, will provide a checksum foreach message data log

checksumJournalFiles          false       如果启用,将为每一个消息数据日志问价开启检查消息总数服务

 

archiveDataLogs               false       If enabled, will move a message data log to the archive directory instead of deleting it

archiveDataLogs               false       如果启用,不用的消息数据文件会被移动到归档目录 而不是删除掉

 

directoryArchive              null        Defines the directory to move data logs to when all the messages they contain have been consumed

directoryArchive              null        存放规定文件的目录

 

databaseLockedWaitDelay       10000       Time (ms) before trying to acquire the database lock (used by shared master/slave)

databaseLockedWaitDelay       10000       尝试获取数据库锁的等待时间(用于共享主/从数据库)

 

maxAsyncJobs                  10000       Maximum number of asynchronous messages that will be queued awaiting 

                                          storage (should be the same as the number of concurrent MessageProducers)

maxAsyncJobs                  10000       排队等待存储的异步消息的最大数量(应该和并发的消息生产者数量相同)                                               

 

concurrentStoreAndDispatchTransactions  true      Enables the dispatching of messages to interested clients to happen concurrently with transaction storage

concurrentStoreAndDispatchTransactions  true      允许在存储事务时分发消息到对其感兴趣的客户端

 

concurrentStoreAndDispatchTopics        true      Enables the dispatching of topic messages to interested clients to happen concurrently with message storage

concurrentStoreAndDispatchTopics        true      允许在存储消息时,分发主题中的消息给对其感兴趣的客户端

 

concurrentStoreAndDispatchQueues        true      Enables the dispatching of queue messages to interested clients to happen concurrently with message storage

concurrentStoreAndDispatchQueues        true      允许在存储消息是,分发队列中的消息给对其感兴趣的客户端

 

ActiveMQ provides a pluggable API for message stores, and there are three additional

implementations to KahaDB that are shipped with ActiveMQ:

 The AMQ message store—A file-based message store designed for performance

 The JDBC message store—A message store based on JDBC

 The Memory message store—A memory-based message store

 

ActiveMQ为消息存储提供了一种可插拔式的API.ActiveMQ版本中包含一下三种KahaDB消息存储实现:

AMQ 消息存储 -- 基于文件的高性能消息存储

JDBC 消息存储 -- 基于JDBC的消息存储

Memory 消息存储 -- 基于内存的消息存储

 

We’ll look at the use cases and configuration for these additional message stores in the

next three sections. We’ll start with the AMQ message store, which like the KahaDB

message store is a file-based implementation. It predates KahaDB, but because of its

performance characteristics, it can make sense to use the AMQ store instead of

KahaDB, provided the number of persistent destinations is relatively low.

 

在接下来的三个章节中,我们将看到这三个消息存储的应用实例和配置方法.我们将从AMQ消息存储开始,

AMQ消息存储和KahaDB类似是基于文件的存储实现,但鉴于改存储方式的高性能特性,可以使用AMQ消息存储

替代KahaDB,但AMQ消息存储可提供的持久化目的地的数量比较少.