Return the correct value for sysconf(_SC_PAGESIZE)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Return the correct value for sysconf(_SC_PAGESIZE)

Erik Bray
Greetings,

Currently sysconf(_SC_PAGESIZE) returns the value of
wincap.allocation_granularity()--a change I *think* had to have been
made by mistake in
https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=commit;f=winsup/cygwin/sysconf.cc;h=177dc6c7f6d0608ef6540fd997d9b444e324cae2

There's no obvious reason, anyways, that this value should be returned
and not the actual page size.

Thanks,

Erik

(P.S. Took me about a half-dozen tries to get a message through to
this ML due to a very picky mailer-daemon--I think it doesn't like
gmail aliases--hopefully this does the
trick :)

0001-Return-wincap.page_size-for-sysconf-_SC_PAGESIZE-not.patch (944 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Corinna Vinschen-2
On Nov 15 14:51, Erik Bray wrote:
> Greetings,
>
> Currently sysconf(_SC_PAGESIZE) returns the value of
> wincap.allocation_granularity()--a change I *think* had to have been
> made by mistake in
> https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=commit;f=winsup/cygwin/sysconf.cc;h=177dc6c7f6d0608ef6540fd997d9b444e324cae2
>
> There's no obvious reason, anyways, that this value should be returned
> and not the actual page size.

That's no accident, but a deliberate decision.  Originally we used the
page size at this point, but that's long ago.  This has been discussed
on the cygwin-developers mailing list years ago.  The problem is the
POSIX assumption that the allocation granularity equals the page size.
The only working solution which does not break assumptions is to return
the allocation granularity as page size.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Erik Bray
On Tue, Nov 15, 2016 at 3:58 PM, Corinna Vinschen
<[hidden email]> wrote:

> On Nov 15 14:51, Erik Bray wrote:
>> Greetings,
>>
>> Currently sysconf(_SC_PAGESIZE) returns the value of
>> wincap.allocation_granularity()--a change I *think* had to have been
>> made by mistake in
>> https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=commit;f=winsup/cygwin/sysconf.cc;h=177dc6c7f6d0608ef6540fd997d9b444e324cae2
>>
>> There's no obvious reason, anyways, that this value should be returned
>> and not the actual page size.
>
> That's no accident, but a deliberate decision.  Originally we used the
> page size at this point, but that's long ago.  This has been discussed
> on the cygwin-developers mailing list years ago.  The problem is the
> POSIX assumption that the allocation granularity equals the page size.
> The only working solution which does not break assumptions is to return
> the allocation granularity as page size.

Okay, sorry for suggesting otherwise.  When I looked at that commit it
seemed like a there was a lot of mass renaming going on, so I thought
it might have been an accident.  I didn't see the threads where this
was discussed.

I see the reason for the change now, but the fact remains
sysconf(_SC_PAGESIZE) cannot, then, be relied on to make any
memory-related calculations from page counts.  Is there a different
(posix-compliant) way to get the actual page size, or at least maybe
it could be somewhere in /proc?

Sorry otherwise for the noise.

Thanks,
Erik
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Corinna Vinschen-2
On Nov 15 16:47, Erik Bray wrote:

> On Tue, Nov 15, 2016 at 3:58 PM, Corinna Vinschen
> <[hidden email]> wrote:
> > On Nov 15 14:51, Erik Bray wrote:
> >> Greetings,
> >>
> >> Currently sysconf(_SC_PAGESIZE) returns the value of
> >> wincap.allocation_granularity()--a change I *think* had to have been
> >> made by mistake in
> >> https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=commit;f=winsup/cygwin/sysconf.cc;h=177dc6c7f6d0608ef6540fd997d9b444e324cae2
> >>
> >> There's no obvious reason, anyways, that this value should be returned
> >> and not the actual page size.
> >
> > That's no accident, but a deliberate decision.  Originally we used the
> > page size at this point, but that's long ago.  This has been discussed
> > on the cygwin-developers mailing list years ago.  The problem is the
> > POSIX assumption that the allocation granularity equals the page size.
> > The only working solution which does not break assumptions is to return
> > the allocation granularity as page size.
>
> Okay, sorry for suggesting otherwise.  When I looked at that commit it
> seemed like a there was a lot of mass renaming going on, so I thought
> it might have been an accident.  I didn't see the threads where this
> was discussed.
>
> I see the reason for the change now, but the fact remains
> sysconf(_SC_PAGESIZE) cannot, then, be relied on to make any
> memory-related calculations from page counts.
What do you mean?  If you call sysconf(_SC_PHYS_PAGES) or
sysconf(_SC_AVPHYS_PAGES) you get the number of pages in _SC_PAGESIZE
chunks so all is good.

> Is there a different
> (posix-compliant) way to get the actual page size, or at least maybe
> it could be somewhere in /proc?

There is no good reason to use the non-POSIXy page size.  It doesn't
help you in the least for any pagesize-related functionality.  Mmap
as well as malloc and friends only work with _SC_PAGESIZE sized pages.

It sounds as if you're looking for a solution for which there's no
problem...


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Erik Bray
On Tue, Nov 15, 2016 at 5:19 PM, Corinna Vinschen
<[hidden email]> wrote:

> On Nov 15 16:47, Erik Bray wrote:
>> On Tue, Nov 15, 2016 at 3:58 PM, Corinna Vinschen
>> <[hidden email]> wrote:
>> > On Nov 15 14:51, Erik Bray wrote:
>> >> Greetings,
>> >>
>> >> Currently sysconf(_SC_PAGESIZE) returns the value of
>> >> wincap.allocation_granularity()--a change I *think* had to have been
>> >> made by mistake in
>> >> https://cygwin.com/git/gitweb.cgi?p=newlib-cygwin.git;a=commit;f=winsup/cygwin/sysconf.cc;h=177dc6c7f6d0608ef6540fd997d9b444e324cae2
>> >>
>> >> There's no obvious reason, anyways, that this value should be returned
>> >> and not the actual page size.
>> >
>> > That's no accident, but a deliberate decision.  Originally we used the
>> > page size at this point, but that's long ago.  This has been discussed
>> > on the cygwin-developers mailing list years ago.  The problem is the
>> > POSIX assumption that the allocation granularity equals the page size.
>> > The only working solution which does not break assumptions is to return
>> > the allocation granularity as page size.
>>
>> Okay, sorry for suggesting otherwise.  When I looked at that commit it
>> seemed like a there was a lot of mass renaming going on, so I thought
>> it might have been an accident.  I didn't see the threads where this
>> was discussed.
>>
>> I see the reason for the change now, but the fact remains
>> sysconf(_SC_PAGESIZE) cannot, then, be relied on to make any
>> memory-related calculations from page counts.
>
> What do you mean?  If you call sysconf(_SC_PHYS_PAGES) or
> sysconf(_SC_AVPHYS_PAGES) you get the number of pages in _SC_PAGESIZE
> chunks so all is good.
>
>> Is there a different
>> (posix-compliant) way to get the actual page size, or at least maybe
>> it could be somewhere in /proc?
>
> There is no good reason to use the non-POSIXy page size.  It doesn't
> help you in the least for any pagesize-related functionality.  Mmap
> as well as malloc and friends only work with _SC_PAGESIZE sized pages.
>
> It sounds as if you're looking for a solution for which there's no
> problem...


FWIW the background here is that I'm working on porting psutil [1] to
Cygwin, and trying to accomplish as much as *possible* through the
POSIX interfaces without having to fall back on the Windows API.  It's
actually a great exercise in what is and isn't possible with Cygwin :)

In this case I was trying to compute process memory usage from
/proc/<pid>/statm which gives values in page counts, so I need the
page size (the actual page size) to compute the values in bytes.

Erik


[1] https://github.com/giampaolo/psutil
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Eric Blake (cygwin)-2
On 11/16/2016 07:56 AM, Erik Bray wrote:

>> There is no good reason to use the non-POSIXy page size.  It doesn't
>> help you in the least for any pagesize-related functionality.  Mmap
>> as well as malloc and friends only work with _SC_PAGESIZE sized pages.
>>
>> It sounds as if you're looking for a solution for which there's no
>> problem...
>
>
> FWIW the background here is that I'm working on porting psutil [1] to
> Cygwin, and trying to accomplish as much as *possible* through the
> POSIX interfaces without having to fall back on the Windows API.  It's
> actually a great exercise in what is and isn't possible with Cygwin :)
>
> In this case I was trying to compute process memory usage from
> /proc/<pid>/statm which gives values in page counts, so I need the
> page size (the actual page size) to compute the values in bytes.
If /proc/<pid>/statm is reporting memory in multiples that are not the
POSIX _SC_PAGESIZE, that is a bug in the statm file emulation that
should be fixed there.

--
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


signature.asc (617 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Erik Bray
On Wed, Nov 16, 2016 at 3:01 PM, Eric Blake <[hidden email]> wrote:

> On 11/16/2016 07:56 AM, Erik Bray wrote:
>
>>> There is no good reason to use the non-POSIXy page size.  It doesn't
>>> help you in the least for any pagesize-related functionality.  Mmap
>>> as well as malloc and friends only work with _SC_PAGESIZE sized pages.
>>>
>>> It sounds as if you're looking for a solution for which there's no
>>> problem...
>>
>>
>> FWIW the background here is that I'm working on porting psutil [1] to
>> Cygwin, and trying to accomplish as much as *possible* through the
>> POSIX interfaces without having to fall back on the Windows API.  It's
>> actually a great exercise in what is and isn't possible with Cygwin :)
>>
>> In this case I was trying to compute process memory usage from
>> /proc/<pid>/statm which gives values in page counts, so I need the
>> page size (the actual page size) to compute the values in bytes.
>
> If /proc/<pid>/statm is reporting memory in multiples that are not the
> POSIX _SC_PAGESIZE, that is a bug in the statm file emulation that
> should be fixed there.
So then something like that attached (untested) patch should work,
though it made statm inconsistent with what is reported in
/proc/<pid>/status which reports memory in multiples of page size.  So
that has to be patched as well.

This of course leads to memory reporting that is inconsistent with
what Windows says.  But I guess if that's "normal" in Cygwin (for the
reasons stated by Corinna) then it's an acceptable trade-off?

Thanks,
Erik

0001-statm-should-report-memory-as-multiples-of-allocatio.patch (1K) Download Attachment
0002-Use-allocation-granularity-as-the-page_size-in-proc-.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Corinna Vinschen-2
On Nov 16 15:51, Erik Bray wrote:

> On Wed, Nov 16, 2016 at 3:01 PM, Eric Blake <[hidden email]> wrote:
> > On 11/16/2016 07:56 AM, Erik Bray wrote:
> >
> >>> There is no good reason to use the non-POSIXy page size.  It doesn't
> >>> help you in the least for any pagesize-related functionality.  Mmap
> >>> as well as malloc and friends only work with _SC_PAGESIZE sized pages.
> >>>
> >>> It sounds as if you're looking for a solution for which there's no
> >>> problem...
> >>
> >>
> >> FWIW the background here is that I'm working on porting psutil [1] to
> >> Cygwin, and trying to accomplish as much as *possible* through the
> >> POSIX interfaces without having to fall back on the Windows API.  It's
> >> actually a great exercise in what is and isn't possible with Cygwin :)
> >>
> >> In this case I was trying to compute process memory usage from
> >> /proc/<pid>/statm which gives values in page counts, so I need the
> >> page size (the actual page size) to compute the values in bytes.
> >
> > If /proc/<pid>/statm is reporting memory in multiples that are not the
> > POSIX _SC_PAGESIZE, that is a bug in the statm file emulation that
> > should be fixed there.
>
> So then something like that attached (untested) patch should work,
> though it made statm inconsistent with what is reported in
> /proc/<pid>/status which reports memory in multiples of page size.  So
> that has to be patched as well.
Patches applied as obvious, thank you.

> This of course leads to memory reporting that is inconsistent with
> what Windows says.  But I guess if that's "normal" in Cygwin (for the
> reasons stated by Corinna) then it's an acceptable trade-off?

It is.  From the POSIX POV we have a 64K page size, from the Windows
POV we have a bastard of 4K pages which can only be allocated in
chunks of 16.  The latter simply doesn't fly well with POSIX
assumptions.

Since native Windows tools don't see the Cygwin internal assumptions,
it doesn't matter, unless you mix Windows and Cygwin memory functions.
Which, honestly, should only be done by Cygwin itself.


Thanks,
Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Return the correct value for sysconf(_SC_PAGESIZE)

Erik Bray
On Thu, Nov 17, 2016 at 10:54 AM, Corinna Vinschen
<[hidden email]> wrote:

> On Nov 16 15:51, Erik Bray wrote:
>> On Wed, Nov 16, 2016 at 3:01 PM, Eric Blake <[hidden email]> wrote:
>> > On 11/16/2016 07:56 AM, Erik Bray wrote:
>> >
>> >>> There is no good reason to use the non-POSIXy page size.  It doesn't
>> >>> help you in the least for any pagesize-related functionality.  Mmap
>> >>> as well as malloc and friends only work with _SC_PAGESIZE sized pages.
>> >>>
>> >>> It sounds as if you're looking for a solution for which there's no
>> >>> problem...
>> >>
>> >>
>> >> FWIW the background here is that I'm working on porting psutil [1] to
>> >> Cygwin, and trying to accomplish as much as *possible* through the
>> >> POSIX interfaces without having to fall back on the Windows API.  It's
>> >> actually a great exercise in what is and isn't possible with Cygwin :)
>> >>
>> >> In this case I was trying to compute process memory usage from
>> >> /proc/<pid>/statm which gives values in page counts, so I need the
>> >> page size (the actual page size) to compute the values in bytes.
>> >
>> > If /proc/<pid>/statm is reporting memory in multiples that are not the
>> > POSIX _SC_PAGESIZE, that is a bug in the statm file emulation that
>> > should be fixed there.
>>
>> So then something like that attached (untested) patch should work,
>> though it made statm inconsistent with what is reported in
>> /proc/<pid>/status which reports memory in multiples of page size.  So
>> that has to be patched as well.
>
> Patches applied as obvious, thank you.
>
>> This of course leads to memory reporting that is inconsistent with
>> what Windows says.  But I guess if that's "normal" in Cygwin (for the
>> reasons stated by Corinna) then it's an acceptable trade-off?
>
> It is.  From the POSIX POV we have a 64K page size, from the Windows
> POV we have a bastard of 4K pages which can only be allocated in
> chunks of 16.  The latter simply doesn't fly well with POSIX
> assumptions.
>
> Since native Windows tools don't see the Cygwin internal assumptions,
> it doesn't matter, unless you mix Windows and Cygwin memory functions.
> Which, honestly, should only be done by Cygwin itself.

That works for me.  It makes things a bit tricky in my particular
case, but I admit it's an unusual one, and now that I'm aware of the
issue I can work with it.

Thanks for accepting the patch.

Erik
Loading...