Quick questions on segments and page map buckets

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Quick questions on segments and page map buckets

John Wilson
Hi,

Two quick questions:


   1. The design documentation here,
   https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood,
   states that the default segment count is equal to the number of logical
   cores available in the underlying machine. However, the segments array in
   PageMemory indicates that the maximum number of segments is: 1 << SEG_BITS.
   Since SEG_BITS = 4, the max # segments is 16. Did I miss something here?
   2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
   allocated segment sequentially in a bump-the-pointer strategy where the
   first 8 bytes of a segment hold a pointer to the index of the last
   allocated page. If this is true, then I don't understand the point of
   having a page map buckets. Why not use a simple arithmetic index * pageSize
   to get the offset of a page?

Thanks.
John
Reply | Threaded
Open this post in threaded view
|

Re: Quick questions on segments and page map buckets

Eduard Shangareev
Hi,

1. It looks weird, yeah. Need to ask Sergey, who has changed it last time.

2. Because we could reuse memory. For example, after cache destroy or
something like that.

On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <[hidden email]>
wrote:

> Hi,
>
> Two quick questions:
>
>
>    1. The design documentation here,
>    https://cwiki.apache.org/confluence/display/IGNITE/
> Ignite+Durable+Memory+-+under+the+hood,
>    states that the default segment count is equal to the number of logical
>    cores available in the underlying machine. However, the segments array
> in
>    PageMemory indicates that the maximum number of segments is: 1 <<
> SEG_BITS.
>    Since SEG_BITS = 4, the max # segments is 16. Did I miss something here?
>    2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
>    allocated segment sequentially in a bump-the-pointer strategy where the
>    first 8 bytes of a segment hold a pointer to the index of the last
>    allocated page. If this is true, then I don't understand the point of
>    having a page map buckets. Why not use a simple arithmetic index *
> pageSize
>    to get the offset of a page?
>
> Thanks.
> John
>
Reply | Threaded
Open this post in threaded view
|

Re: Quick questions on segments and page map buckets

Dmitriy Pavlov
Hi Ed,

Which Sergey do you mean? I know a number of contributors.

Sincerely,


вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <[hidden email]
>:

> Hi,
>
> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last time.
>
> 2. Because we could reuse memory. For example, after cache destroy or
> something like that.
>
> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <[hidden email]>
> wrote:
>
> > Hi,
> >
> > Two quick questions:
> >
> >
> >    1. The design documentation here,
> >    https://cwiki.apache.org/confluence/display/IGNITE/
> > Ignite+Durable+Memory+-+under+the+hood,
> >    states that the default segment count is equal to the number of
> logical
> >    cores available in the underlying machine. However, the segments array
> > in
> >    PageMemory indicates that the maximum number of segments is: 1 <<
> > SEG_BITS.
> >    Since SEG_BITS = 4, the max # segments is 16. Did I miss something
> here?
> >    2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
> >    allocated segment sequentially in a bump-the-pointer strategy where
> the
> >    first 8 bytes of a segment hold a pointer to the index of the last
> >    allocated page. If this is true, then I don't understand the point of
> >    having a page map buckets. Why not use a simple arithmetic index *
> > pageSize
> >    to get the offset of a page?
> >
> > Thanks.
> > John
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Quick questions on segments and page map buckets

Dmitriy Pavlov
John, is this question still actual?

пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <[hidden email]>:

> Hi Ed,
>
> Which Sergey do you mean? I know a number of contributors.
>
> Sincerely,
>
>
> вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> [hidden email]>:
>
>> Hi,
>>
>> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last time.
>>
>> 2. Because we could reuse memory. For example, after cache destroy or
>> something like that.
>>
>> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <[hidden email]>
>> wrote:
>>
>> > Hi,
>> >
>> > Two quick questions:
>> >
>> >
>> >    1. The design documentation here,
>> >    https://cwiki.apache.org/confluence/display/IGNITE/
>> > Ignite+Durable+Memory+-+under+the+hood,
>> >    states that the default segment count is equal to the number of
>> logical
>> >    cores available in the underlying machine. However, the segments
>> array
>> > in
>> >    PageMemory indicates that the maximum number of segments is: 1 <<
>> > SEG_BITS.
>> >    Since SEG_BITS = 4, the max # segments is 16. Did I miss something
>> here?
>> >    2. Reading the code in PageMemoryNoStoreImp, it looks like pages are
>> >    allocated segment sequentially in a bump-the-pointer strategy where
>> the
>> >    first 8 bytes of a segment hold a pointer to the index of the last
>> >    allocated page. If this is true, then I don't understand the point of
>> >    having a page map buckets. Why not use a simple arithmetic index *
>> > pageSize
>> >    to get the offset of a page?
>> >
>> > Thanks.
>> > John
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Quick questions on segments and page map buckets

Eduard Shangareev
Dmitry,

I mean Chugunov, but I have questioned him. His answer was the A. Goncharuk
should now the answer.

On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <[hidden email]>
wrote:

> John, is this question still actual?
>
> пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <[hidden email]>:
>
> > Hi Ed,
> >
> > Which Sergey do you mean? I know a number of contributors.
> >
> > Sincerely,
> >
> >
> > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > [hidden email]>:
> >
> >> Hi,
> >>
> >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> time.
> >>
> >> 2. Because we could reuse memory. For example, after cache destroy or
> >> something like that.
> >>
> >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <[hidden email]>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > Two quick questions:
> >> >
> >> >
> >> >    1. The design documentation here,
> >> >    https://cwiki.apache.org/confluence/display/IGNITE/
> >> > Ignite+Durable+Memory+-+under+the+hood,
> >> >    states that the default segment count is equal to the number of
> >> logical
> >> >    cores available in the underlying machine. However, the segments
> >> array
> >> > in
> >> >    PageMemory indicates that the maximum number of segments is: 1 <<
> >> > SEG_BITS.
> >> >    Since SEG_BITS = 4, the max # segments is 16. Did I miss something
> >> here?
> >> >    2. Reading the code in PageMemoryNoStoreImp, it looks like pages
> are
> >> >    allocated segment sequentially in a bump-the-pointer strategy where
> >> the
> >> >    first 8 bytes of a segment hold a pointer to the index of the last
> >> >    allocated page. If this is true, then I don't understand the point
> of
> >> >    having a page map buckets. Why not use a simple arithmetic index *
> >> > pageSize
> >> >    to get the offset of a page?
> >> >
> >> > Thanks.
> >> > John
> >> >
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Quick questions on segments and page map buckets

Dmitriy Pavlov
Hi, I've double checked code regarding question 1:

PageMemoryImpl#segments segments count (len) comes from sizes array len.
Sizes array come to page memory from
GridCacheDatabaseSharedManager.calculateFragmentSizes(). This method which
gets availableProcessors from runtime (CPU count). I didn't find any shift
to SEG_BITS.

It can be chaged by DataStorageConfiguration.setConcurrencyLevel() but by
default is the same with #CPU.

Which is why I mentioned segments count = CPU count by default in wiki.

Recenly some member of community proposed update to offical doc about
segments count. So now it states #segments=#CPU.

So I guess for 8 CPU we will have 8 segments.

Please correct me if I missing something.

Sincerely,
Dmitriy Pavlov

ср, 27 июн. 2018 г. в 15:30, Eduard Shangareev <[hidden email]
>:

> Dmitry,
>
> I mean Chugunov, but I have questioned him. His answer was the A. Goncharuk
> should now the answer.
>
> On Wed, Jun 27, 2018 at 2:58 PM, Dmitry Pavlov <[hidden email]>
> wrote:
>
> > John, is this question still actual?
> >
> > пт, 22 июн. 2018 г. в 15:18, Dmitry Pavlov <[hidden email]>:
> >
> > > Hi Ed,
> > >
> > > Which Sergey do you mean? I know a number of contributors.
> > >
> > > Sincerely,
> > >
> > >
> > > вт, 19 июн. 2018 г. в 22:15, Eduard Shangareev <
> > > [hidden email]>:
> > >
> > >> Hi,
> > >>
> > >> 1. It looks weird, yeah. Need to ask Sergey, who has changed it last
> > time.
> > >>
> > >> 2. Because we could reuse memory. For example, after cache destroy or
> > >> something like that.
> > >>
> > >> On Tue, Jun 19, 2018 at 9:58 PM, John Wilson <[hidden email]
> >
> > >> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > Two quick questions:
> > >> >
> > >> >
> > >> >    1. The design documentation here,
> > >> >    https://cwiki.apache.org/confluence/display/IGNITE/
> > >> > Ignite+Durable+Memory+-+under+the+hood,
> > >> >    states that the default segment count is equal to the number of
> > >> logical
> > >> >    cores available in the underlying machine. However, the segments
> > >> array
> > >> > in
> > >> >    PageMemory indicates that the maximum number of segments is: 1 <<
> > >> > SEG_BITS.
> > >> >    Since SEG_BITS = 4, the max # segments is 16. Did I miss
> something
> > >> here?
> > >> >    2. Reading the code in PageMemoryNoStoreImp, it looks like pages
> > are
> > >> >    allocated segment sequentially in a bump-the-pointer strategy
> where
> > >> the
> > >> >    first 8 bytes of a segment hold a pointer to the index of the
> last
> > >> >    allocated page. If this is true, then I don't understand the
> point
> > of
> > >> >    having a page map buckets. Why not use a simple arithmetic index
> *
> > >> > pageSize
> > >> >    to get the offset of a page?
> > >> >
> > >> > Thanks.
> > >> > John
> > >> >
> > >>
> > >
> >
>