Can one mnesia table fragment be further fragmente

Login : Register
Can one mnesia table fragment be further fragmented?
View: New views
5 Messages — Rating Filter:   Alert me 


Can one mnesia table fragment be further fragmented?
Click to flag this post

by devdoer bird Jun 18, 2008; 12:33am :: Rate this Message: - Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message
Hi:
I spilit one mnesia table into 3 fragments which distributed on 3 machines  named A,B and C,but now the the machine "A" is very busy,
so I decide split the fragment on the machine "A" into small fragments.Does mnesia support this?
Thanks.

_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions


Re: Can one mnesia table fragment be further fragmented?
Click to flag this post

by Scott Lystig Fritchie Jun 19, 2008; 08:21am :: Rate this Message: - Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message
devdoer bird <devdoer2@...> wrote:

dd> I spilit one mnesia table into 3 fragments which distributed on 3
dd> machines named A,B and C,but now the the machine "A" is very busy,

dd> so I decide split the fragment on the machine "A" into small
dd> fragments.Does mnesia support this?

No, that isn't possible with Mnesia, not as you wish/state above.

If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and
'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc.
You can choose to put only 1 fragment on node A, 4 fragments on node B,
and 4 fragments on node C.  Assuming that access to the table is evenly
distributed over all fragments, then node A should be getting only 1/9
of the load, and B & C will get 4/9 of the load.

-Scott
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions


Re: Can one mnesia table fragment be further fragmented?
Click to flag this post

by devdoer bird Jun 19, 2008; 10:03am :: Rate this Message: - Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message
Thanks.

Thant means ,I need fragment table many times not for load balance but for decreasing one specific node's (the node is not as powerful as others ) load.

Eg. Node A has 2 fragments,Node B has 2 fragments. Each fragment has equal records(say 100 records each fragment).

Now the Node B is heavily loaded.If I adding one or two   fragments in Node A , the load of Node B won't be

decreased ,for the records from Node A will be splited and moved to the new fragments,the records on Node A is 50+50+50+50=200 ,the total number of  Node B's record  is still 200.

In order to decreas the load on Node B,I have to calcaulate carefully how many new fragments should be added.

If there's a way for mnesia to specify which node's fragment should be splited when addin a new fragment,the problems above will be solved.Take the example above ,when  adding a new fragment,I can tell mnesia to split Node B's fragment.

2008/6/19, Scott Lystig Fritchie <fritchie@...>:

    devdoer bird <devdoer2@...> wrote:

    dd> I spilit one mnesia table into 3 fragments which distributed on 3
    dd> machines named A,B and C,but now the the machine "A" is very busy,

    dd> so I decide split the fragment on the machine "A" into small
    dd> fragments.Does mnesia support this?

    No, that isn't possible with Mnesia, not as you wish/state above.

    If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and
    'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc.
    You can choose to put only 1 fragment on node A, 4 fragments on node B,
    and 4 fragments on node C.  Assuming that access to the table is evenly
    distributed over all fragments, then node A should be getting only 1/9
    of the load, and B & C will get 4/9 of the load.

    -Scott



_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions


Re: Can one mnesia table fragment be further fragmented?
Click to flag this post

by Paul Mineiro Jun 19, 2008; 03:14pm :: Rate this Message: - Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message
Re: picking what gets split, not with the default frag_hash.  You can see
what it's doing in mnesia_frag_hash.erl ... it basically splits the
fragments in order.  This means that unless there is a power-of-two number
of fragments they will be of different magnitudes.  You could leverage
that for your load balancing by placing bigger fragments on less noded
nodes, etc.  In fact, maybe this is why you are seeing
an uneven distribution to begin with?

-- p

p.z. If you are feeling your wheaties, you can make your own frag hash,
and specify the module in the frag_properties when you create the table.

http://www.erlang.org/doc/apps/mnesia/Mnesia_chap5.html#5.3

Don't be tempted to just use phash2 or something like that; in order to go
from N fragments to M fragments you have to pass through all the
intermediate stages N + 1, ..., M - 1, M ; you do not want to rehash all
the data in all the buckets at each step.

On Thu, 19 Jun 2008, devdoer bird wrote:

> Thanks.
>
> Thant means ,I need fragment table many times not for load balance but for
> decreasing one specific node's (the node is not as powerful as others )
> load.
>
> Eg. Node A has 2 fragments,Node B has 2 fragments. Each fragment has equal
> records(say 100 records each fragment).
>
> Now the Node B is heavily loaded.If I adding one or two   fragments in Node
> A , the load of Node B won't be
>
> decreased ,for the records from Node A will be splited and moved to the new
> fragments,the records on Node A is 50+50+50+50=200 ,the total number of
> Node B's record  is still 200.
>
> In order to decreas the load on Node B,I have to calcaulate carefully how
> many new fragments should be added.
>
> If there's a way for mnesia to specify which node's fragment should be
> splited when addin a new fragment,the problems above will be solved.Take the
> example above ,when  adding a new fragment,I can tell mnesia to split Node
> B's fragment.
>
> 2008/6/19, Scott Lystig Fritchie <fritchie@...>:
> >
> > devdoer bird <devdoer2@...> wrote:
> >
> > dd> I spilit one mnesia table into 3 fragments which distributed on 3
> > dd> machines named A,B and C,but now the the machine "A" is very busy,
> >
> > dd> so I decide split the fragment on the machine "A" into small
> > dd> fragments.Does mnesia support this?
> >
> > No, that isn't possible with Mnesia, not as you wish/state above.
> >
> > If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and
> > 'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc.
> > You can choose to put only 1 fragment on node A, 4 fragments on node B,
> > and 4 fragments on node C.  Assuming that access to the table is evenly
> > distributed over all fragments, then node A should be getting only 1/9
> > of the load, and B & C will get 4/9 of the load.
> >
> > -Scott
> >
>
... [show rest of quote]

In an artificial world, only extremists live naturally.

        -- Paul Graham
_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions


Re: Can one mnesia table fragment be further fragmented?
Click to flag this post

by devdoer bird Jun 20, 2008; 01:23pm :: Rate this Message: - Use ratings to moderate (?)

Reply | Reply to Author | Print | View Threaded | Show Only this Message
Thanks ,Paul. I decide to replace the mnesia_frag_hash with  my own hash module   .With this customized hash module,I hope I can explicitly pass a fragment number to make the fragment splited.
I decide to use a mnesia table or other persitent-able  data structure in erlang which can globally seen  by all nodes to store the mapping relations between key value range  and fragment(Eg. [1..999]-> frag 1,[1000-1999]->frag2), so when I need to split a specifed fragment ,I can rejust the mapping relation table.

Here's my plan,any suggestions are welcome.


2008/6/19, Paul Mineiro <paul-trapexit@...>:

    Re: picking what gets split, not with the default frag_hash.  You can see
    what it's doing in mnesia_frag_hash.erl ... it basically splits the
    fragments in order.  This means that unless there is a power-of-two number
    of fragments they will be of different magnitudes.  You could leverage
    that for your load balancing by placing bigger fragments on less noded
    nodes, etc.  In fact, maybe this is why you are seeing
    an uneven distribution to begin with?

    -- p

    p.z. If you are feeling your wheaties, you can make your own frag hash,
    and specify the module in the frag_properties when you create the table.

    http://www.erlang.org/doc/apps/mnesia/Mnesia_chap5.html#5.3

    Don't be tempted to just use phash2 or something like that; in order to go
    from N fragments to M fragments you have to pass through all the
    intermediate stages N + 1, ..., M - 1, M ; you do not want to rehash all
    the data in all the buckets at each step.

    On Thu, 19 Jun 2008, devdoer bird wrote:

    > Thanks.
    >
    > Thant means ,I need fragment table many times not for load balance but for
    > decreasing one specific node's (the node is not as powerful as others )
    > load.
    >
    > Eg. Node A has 2 fragments,Node B has 2 fragments. Each fragment has equal
    > records(say 100 records each fragment).
    >
    > Now the Node B is heavily loaded.If I adding one or two   fragments in Node
    > A , the load of Node B won't be
    >
    > decreased ,for the records from Node A will be splited and moved to the new
    > fragments,the records on Node A is 50+50+50+50=200 ,the total number of
    > Node B's record  is still 200.
    >
    > In order to decreas the load on Node B,I have to calcaulate carefully how
    > many new fragments should be added.
    >
    > If there's a way for mnesia to specify which node's fragment should be
    > splited when addin a new fragment,the problems above will be solved.Take the
    > example above ,when  adding a new fragment,I can tell mnesia to split Node
    > B's fragment.
    >
    > 2008/6/19, Scott Lystig Fritchie <fritchie@...>:
    > >
    > > devdoer bird <devdoer2@...> wrote:
    > >
    > > dd> I spilit one mnesia table into 3 fragments which distributed on 3
    > > dd> machines named A,B and C,but now the the machine "A" is very busy,
    > >
    > > dd> so I decide split the fragment on the machine "A" into small
    > > dd> fragments.Does mnesia support this?
    > >
    > > No, that isn't possible with Mnesia, not as you wish/state above.
    > >
    > > If you have a table 'tab' split into fragments 'tab', 'tab_frag2', and
    > > 'tab_frag3', you can add more fragments: 'tab_frag4', 'tab_frag5', etc.
    > > You can choose to put only 1 fragment on node A, 4 fragments on node B,
    > > and 4 fragments on node C.  Assuming that access to the table is evenly
    > > distributed over all fragments, then node A should be getting only 1/9
    > > of the load, and B & C will get 4/9 of the load.
    > >
    > > -Scott
    > >
    >

    In an artificial world, only extremists live naturally.

           -- Paul Graham
    _______________________________________________
    erlang-questions mailing list
    erlang-questions@...
    http://www.erlang.org/mailman/listinfo/erlang-questions



_______________________________________________
erlang-questions mailing list
erlang-questions@...
http://www.erlang.org/mailman/listinfo/erlang-questions