MongoDB : Using a Large Number of Collections

( From: http://api.mongodb.org/wiki/current/Using a Large Number of Collections.html )

MongoDB : Using a Large Number of Collections

This page last changed on Dec 01, 2010 by eliot.

A technique one can use with MongoDB in certain situations is to have several collections to store information instead of a single collection.  By doing this, certain repeating data no longer needs to be stored in every object, and an index on that key may be eliminated.  More importantly for performance (depending on the problem), the data is then clustered by the grouping specified.

For example, suppose we are logging objects/documents to the database, and want to have M logs: perhaps a dev log, a debug log, an ops log, etc.  We could store them all in one collection 'logs' containing objects like:

{ log : 'dev', ts : ..., info : ... }

However, if the number of logs is not too high, it might be better to have a collection per log. We could have a 'logs.dev' collection, a 'logs.debug' collection, 'logs.ops', etc.:

// logs.dev: { ts : ..., info : ... }

Of course, this only makes sense if we do not need to query for items from multiple logs at the same time.

Generally, having a large number of collections has no significant performance penalty, and results in very good performance.

Limits

By default MongoDB has a limit of approximately 24,000 namespaces per database.  Each collection counts as a namespace, as does each index.  Thus if every collection had one index, we can create up to 12,000 collections.  Use the --nssize parameter to set a higher limit.

Be aware that there is a certain minimum overhead per collection -- a few KB.  Further, any index will require at least 8KB of data space as the b-tree page size is 8KB. Certain operations can get slow if there are a lot of collections and the meta data gets paged out.

--nssize

If more collections are required, run mongod with the --nssize parameter specified.  This will make the <database>.ns file larger and support more collections.  Note that --nssize sets the size used for newly created .ns files -- if you have an existing database and wish to resize, after running the db with --nssize, run the db.repairDatabase() command from the shell to adjust the size.

Maximum .ns file size is 2GB.

Comments:

How does the amount of namespace exactly compare to the nssize (file size)? Is there a way to calculate this size or does the size depend on the stored data? What is exactly stored in a *.ns file?