How to Eliminate Mnesia Overload Events

原文地址：http://streamhacker.com/2008/12/10/how-to-eliminate-mnesia-overload-events/

If you’re using mnesia disc_copies tables and doing a lot of writes all at once, you’ve probably run into the following message

=ERROR REPORT==== 10-Dec-2008::18:07:19 ===
Mnesia(node@host): ** WARNING ** Mnesia is overloaded: {dump_log, write_threshold}

This warning event can get really annoying, especially when they start happening every second. But you can eliminate them, or at least drastically reduce their occurance.
Synchronous Writes

The first thing to do is make sure to use sync_transaction or sync_dirty. Doing synchronous writes will slow down your writes in a good way, since the functions won’t return until your record(s) have been written to the transaction log. The alternative, which is the default, is to do asynchronous writes, which can fill transaction log far faster than it gets dumped, causing the above error report.
Mnesia Application Configuration

If synchronous writes aren’t enough, the next trick is to modify 2 obscure configuration parameters. The mnesia_overload event generally occurs when the transaction log needs to be dumped, but the previous transaction log dump hasn’t finished yet. Tweaking these parameters will make the transaction log dump less often, and the disc_copies tables dump to disk more often. NOTE: these parameters must be set before mnesia is started; changing them at runtime has no effect. You can set them thru the command line or in a config file.
dc_dump_limit

This variable controls how often disc_copies tables are dumped from memory. The default value is 4, which means if the size of the log is greater than the size of table / 4, then a dump occurs. To make table dumps happen more often, increase the value. I’ve found setting this to 40 works well for my purposes.
dump_log_write_threshold

This variable defines the maximum number of writes to the transaction log before a new dump is performed. The default value is 100, so a new transaction log dump is performed after every 100 writes. If you’re doing hundreds or thousands of writes in a short period of time, then there’s no way mnesia can keep up. I set this value to 50000, which is a huge increase, but I have enough RAM to handle it. If you’re worried that this high value means the transaction log will rarely get dumped when there’s very few writes occuring, there’s also a dump_log_time_threshold configuration variable, which by default dumps the log every 3 minutes.
How it Works

I might be wrong on the theory since I didn’t actually write or design mnesia, but here’s my understanding of what’s happening. Each mnesia activity is recorded to a single transaction log. This transaction log then gets dumped to table logs, which in turn are dumped to the table file on disk. By increasing the dump_log_write_threshold, transaction log dumps happen much less often, giving each dump more time to complete before the next dump is triggered. And increasing dc_dump_limit helps ensure that the table log is also dumped to disk before the next transaction dump occurs.

评论

3 楼 mryufeng 2010-01-26  

warning是没啥问题 为啥warning是个大问题, 你是没想明白 warning的后果...

2 楼 20.Shadow 2010-01-26  

e, 看来我错了.........
会crash掉的.........

错误如下:
eheap_alloc : cannot allocate 298930300 bytes of memory (of type "heap").

This appication has xxx.................

我的代码如下(应该都是尾递归规则把...):
(db:getMax ---什么的可以无视, 自己写的)
调用:hugeWrite(dist, "XXX", 5000000), 大概到377W的时候就崩溃了...
崩溃以后dist.DCD还只有100M左右...

hugeWrite(Table, Name, MaxTimes) ->
	Fun = fun() ->
			db:getMax(Table, 0)
	end,
	{ok, MaxID} = db:doTrans(Fun),
	hugeWrite(0, MaxTimes, Table, MaxID + 1, Name).

hugeWrite(Index, Max, Table, CurrID, Name) ->
	case Index < Max of
		true ->
			Fun = fun() ->
				db:write({Table, CurrID, Name})
			end,
			db:doTrans(Fun),
			io:format("ID = ~p, Name = ~p~n", [CurrID, Name]),
			hugeWrite(Index + 1, Max, Table, CurrID + 1, Name);
		false ->
			io:format("-------- Over ----------~n")
	end.

1 楼 20.Shadow 2010-01-26  

好文.
刚遇到这问题, 想测试下Mnesia对于单表超过2GB容量的情况如何处理,
所以foreach了很多次write()... (还是同步的)
结果就发生了这个overloaded的问题.

不过顺便问下, 这个应该没有关系把, 按照文章意思来说忽略这个warnnig应该也不会造成错误的把?