Re: ld: fatal error - cmalloc would have returned NULL

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Rainer Emrich-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mar  1 18:39, Corinna Vinschen wrote:

> On Mar  1 18:26, Rainer Emrich wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Any news on this issue?
>>
>> At the moment it's impossible to build libgcj during bootstrap of gcc!
>>
>> I tried 1.7.7-1 and the snapshot 20110227.
>>
>> Here some diagnostic:
>>
>>   288 117500220 [main] ld 5884 mmap64: 0x600E0000 = mmap()
>>  4283 117504503 [main] ld 5884 mmap64: addr 0, len 65536, prot 3, flags 22, fd
>> - -1, off 0
>>   306 117504809 [main] ld 5884 mmap64: 0x600D0000 = mmap()
>>  4296 117509105 [main] ld 5884 mmap64: addr 0, len 65536, prot 3, flags 22, fd
>> - -1, off 0
>>   290 117509395 [main] ld 5884 mmap64: 0x600C0000 = mmap()
>>  3883 117513278 [main] ld 5884 mmap64: addr 0, len 65536, prot 3, flags 22, fd
>> - -1, off 0
>>   416 117513694 [main] ld 5884 mmap64: 0x600B0000 = mmap()
>>  4503 117518197 [main] ld 5884 mmap64: addr 0, len 65536, prot 3, flags 22, fd
>> - -1, off 0
>
> This looks like a problem in ld.  It uses mmap a lot.  In fact it uses
> mmap so much that the memory allocation (which goes from top to bottom
> addresses) reaches the end of the Cygwin DLL itself and beyond that.
> And it doesn't mmap in a few big chunks, but rather in many small
> chunks.

This seems to be by design.

>
>>   490 117518687 [main] ld 5884 seterrno_from_win_error:
>> /ext/build/netrel/src/cygwin-snapshot-20110227-1/winsup/cygwin/cygheap.cc:145
>> windows error 487
>>    41 117518728 [main] ld 5884 geterrno_from_win_error: windows error 487 ==
>> errno 22
>>    42 117518770 [main] ld 5884 __set_errno: void* creturn(cygheap_types,
>> cygheap_entry*, unsigned int, const char*):265 val 12
>
> Every mmap call needs space on the cygheap, which is a special heap to
> keep inforamtion for later fork or exec calls.  The cygheap resides at
> the end of the Cygwin DLL.  If it's space is insufficient, the Cygwin
> DLL tries to extend it.  But it can't, because the memory slot is already
> taken by one of the many mmap calls.
>
>>    45 117518815 [main] ld 5884 mmap64: 0xFFFFFFFF = mmap()
>>    40 117518855 [main] ld 5884 __set_errno: void* sbrk(int):167 val 12
>>    36 117518891 [main] ld 5884 __set_errno: void __set_ENOMEM():304 val 12
>
> Therefore, the last mmap call fails and returns MAP_FAILED with errno
> set to ENOMEM.
>
>> - --- Process 5884, exception C0000005 at 0042903A
>
> And then ld crashes, because, apparently, it neglects to check the
> return value of mmap.

Yes it's a fault to not check the return value of mmap, but that wouldn't help
here either.

So, the solution for me was to increase the cygheap size. The maximum seems to
be 1792 MBytes. This solves the issue for boostrapping gcc with libjava enabled,
but may fail for even larger libraries.

Cheers

Rainer

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk16DhAACgkQoUhjsh59BL5aEwCgkachczdlzjh8yhDbPHoSYl5C
LMoAn0mH6tNOkvgjbeY1gVc8Ld9jogNr
=/uEs
-----END PGP SIGNATURE-----

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
On Mar 11 12:57, Rainer Emrich wrote:

> On Mar  1 18:39, Corinna Vinschen wrote:
> > And then ld crashes, because, apparently, it neglects to check the
> > return value of mmap.
>
> Yes it's a fault to not check the return value of mmap, but that wouldn't help
> here either.
>
> So, the solution for me was to increase the cygheap size. The maximum seems to
> be 1792 MBytes. This solves the issue for boostrapping gcc with libjava enabled,
> but may fail for even larger libraries.

I don't think you mean to change the size of the cygheap to 1792 Megs,
do you?  This sounds impossible to me.  Keep in mind that you only have
2 Gigs total memory available per application.

The cygheap size is usually 1 MByte, + the number of pages to align the
end of the cygheap section to the next 64K boundary.  In a case like
this you can increase the cygheap to, say, 2 Megs + alignment, but that
should be enough for all cases which fit into memory at all.

Otherwise, ld should use temporary files to store intermediate data.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Rainer Emrich-2
In reply to this post by Rainer Emrich-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> On Mar  11 12:57, Corinna Vinschen wrote:
> On Mar 11 12:57, Rainer Emrich wrote:
>> On Mar  1 18:39, Corinna Vinschen wrote:
>> > And then ld crashes, because, apparently, it neglects to check the
>> > return value of mmap.
>>
>> Yes it's a fault to not check the return value of mmap, but that wouldn't help
>> here either.
>>
>> So, the solution for me was to increase the cygheap size. The maximum seems to
>> be 1792 MBytes. This solves the issue for boostrapping gcc with libjava enabled,
>> but may fail for even larger libraries.
>
> I don't think you mean to change the size of the cygheap to 1792 Megs,
> do you?  This sounds impossible to me.  Keep in mind that you only have
> 2 Gigs total memory available per application.
>
> The cygheap size is usually 1 MByte, + the number of pages to align the
> end of the cygheap section to the next 64K boundary.  In a case like
> this you can increase the cygheap to, say, 2 Megs + alignment, but that
> should be enough for all cases which fit into memory at all.
I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792

>
> Otherwise, ld should use temporary files to store intermediate data.
>
On Linux or *nix this not a problem at all. But to be honest, I have only few
knowledge about ld.

Rainer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk16KVAACgkQoUhjsh59BL6u2QCfTCBnGmLz+G7CvLK4UtsvSmBF
RIYAn0eJaq/gkj26TNETFmywVya/VvyP
=0Y7W
-----END PGP SIGNATURE-----

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
On Mar 11 14:53, Rainer Emrich wrote:

> > On Mar  11 12:57, Corinna Vinschen wrote:
> > On Mar 11 12:57, Rainer Emrich wrote:
> >> So, the solution for me was to increase the cygheap size. The maximum seems to
> >> be 1792 MBytes. This solves the issue for boostrapping gcc with libjava enabled,
> >> but may fail for even larger libraries.
> >
> > I don't think you mean to change the size of the cygheap to 1792 Megs,
> > do you?  This sounds impossible to me.  Keep in mind that you only have
> > 2 Gigs total memory available per application.
> >
> > The cygheap size is usually 1 MByte, + the number of pages to align the
> > end of the cygheap section to the next 64K boundary.  In a case like
> > this you can increase the cygheap to, say, 2 Megs + alignment, but that
> > should be enough for all cases which fit into memory at all.
> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792

But that's the size of the application heap, not the size of the
cygheap.  The cygheap is used by a couple of internal datastructures
of the cygwin DLL itself, while the application heap is used for malloc.

So you raised the size of the application heap, probably not to 1792
Megs, but the next lower allocation possible (cygwin decrements the size
in 1MB steps until the allocation succeeds.

That's weird.  malloc uses mmap, but only for allocations beyond 128K.
Since ld only allocates 64K chunks, it doesn't look like mmap is called
from malloc.  OTOH, if raising the heap size helps, how do the
zillions of mmap calls into this picture?!?

> > Otherwise, ld should use temporary files to store intermediate data.
> >
> On Linux or *nix this not a problem at all. But to be honest, I have only few
> knowledge about ld.

Linux and *nix have an intelligent memory management.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
On Mar 11 15:13, Corinna Vinschen wrote:

> On Mar 11 14:53, Rainer Emrich wrote:
> > I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
> > regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
>
> But that's the size of the application heap, not the size of the
> cygheap.  The cygheap is used by a couple of internal datastructures
> of the cygwin DLL itself, while the application heap is used for malloc.
>
> So you raised the size of the application heap, probably not to 1792
> Megs, but the next lower allocation possible (cygwin decrements the size
> in 1MB steps until the allocation succeeds.
>
> That's weird.  malloc uses mmap, but only for allocations beyond 128K.

Actually mmap is only used if you try to malloc >= 256K.

> Since ld only allocates 64K chunks, it doesn't look like mmap is called
> from malloc.  OTOH, if raising the heap size helps, how do the
> zillions of mmap calls into this picture?!?

Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
Rainer,

On Mar 11 15:23, Corinna Vinschen wrote:

> On Mar 11 15:13, Corinna Vinschen wrote:
> > On Mar 11 14:53, Rainer Emrich wrote:
> > > I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
> > > regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
> >
> > But that's the size of the application heap, not the size of the
> > cygheap.  The cygheap is used by a couple of internal datastructures
> > of the cygwin DLL itself, while the application heap is used for malloc.
> >
> > So you raised the size of the application heap, probably not to 1792
> > Megs, but the next lower allocation possible (cygwin decrements the size
> > in 1MB steps until the allocation succeeds.
> >
> > That's weird.  malloc uses mmap, but only for allocations beyond 128K.
>
> Actually mmap is only used if you try to malloc >= 256K.
>
> > Since ld only allocates 64K chunks, it doesn't look like mmap is called
> > from malloc.  OTOH, if raising the heap size helps, how do the
> > zillions of mmap calls into this picture?!?

I was wondering if I could reduce the pressure on the cygheap by using a
simplified method to allocate the required bookkeeping datastructures.
It passes my homebrew mmap testsuite, but I would be curious if this
might fix your problem.  I have not very much hope, but anyway...

Would you mind if I send you a link to a cygwin DLL for testing by
private email?


Thanks,
Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Rainer Emrich-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Corinna,

Am 11.03.2011 16:07, schrieb Corinna Vinschen:

> Rainer,
>
> On Mar 11 15:23, Corinna Vinschen wrote:
>> On Mar 11 15:13, Corinna Vinschen wrote:
>>> On Mar 11 14:53, Rainer Emrich wrote:
>>>> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
>>>> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
>>>
>>> But that's the size of the application heap, not the size of the
>>> cygheap.  The cygheap is used by a couple of internal datastructures
>>> of the cygwin DLL itself, while the application heap is used for malloc.
>>>
>>> So you raised the size of the application heap, probably not to 1792
>>> Megs, but the next lower allocation possible (cygwin decrements the size
>>> in 1MB steps until the allocation succeeds.
>>>
>>> That's weird.  malloc uses mmap, but only for allocations beyond 128K.
>>
>> Actually mmap is only used if you try to malloc >= 256K.
>>
>>> Since ld only allocates 64K chunks, it doesn't look like mmap is called
>>> from malloc.  OTOH, if raising the heap size helps, how do the
>>> zillions of mmap calls into this picture?!?
>
> I was wondering if I could reduce the pressure on the cygheap by using a
> simplified method to allocate the required bookkeeping datastructures.
> It passes my homebrew mmap testsuite, but I would be curious if this
> might fix your problem.  I have not very much hope, but anyway...
>
> Would you mind if I send you a link to a cygwin DLL for testing by
> private email?
Yes, of course. You're welcome to send me a link. I' m able to test next week.

>
>
> Thanks,
> Corinna
>

Thanks,
Rainer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk16QPQACgkQoUhjsh59BL6YFgCgq3YQIscphjfAk/DMuMNQDkOy
5oYAoINlltB1e4TiPRx6G/q8T9NEb7fd
=e3QZ
-----END PGP SIGNATURE-----

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Dave Korn-9
In reply to this post by Rainer Emrich-2
On 11/03/2011 13:53, Rainer Emrich wrote:

> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792

  I run with this setting all the time, I guess that's why I haven't seen this
problem.  Before I did that (couple of years back) I also used to get crashes
building libjava.

  (I'm sure there is an underlying inefficiency in ld's memory usage, and I
suspect it might relate to comdat handling, 4.6 generating way more of them
than earlier 4.x versions.)

    cheers,
      DaveK


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Christopher Faylor-8
On Fri, Mar 18, 2011 at 05:47:04AM +0000, Dave Korn wrote:
>On 11/03/2011 13:53, Rainer Emrich wrote:
>
>> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
>> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
>
>  I run with this setting all the time, I guess that's why I haven't seen this
>problem.  Before I did that (couple of years back) I also used to get crashes
>building libjava.

That setting should have nothing to do with cmalloc().  cmalloc is for
Cygwin's internal heap which has nothing to do with that setting.

Didn't Corinna already mention this?

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
On Mar 18 02:08, Christopher Faylor wrote:

> On Fri, Mar 18, 2011 at 05:47:04AM +0000, Dave Korn wrote:
> >On 11/03/2011 13:53, Rainer Emrich wrote:
> >
> >> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
> >> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
> >
> >  I run with this setting all the time, I guess that's why I haven't seen this
> >problem.  Before I did that (couple of years back) I also used to get crashes
> >building libjava.
>
> That setting should have nothing to do with cmalloc().  cmalloc is for
> Cygwin's internal heap which has nothing to do with that setting.
>
> Didn't Corinna already mention this?

In this case the bigger heap seems to avoid the aggressive use of small
mmap chunks which in turn disallows to raise the cygheap size.  Without
analyzing the whole situation there's not much else to say or do.  This
is YA case which screams loudly for a testcase...


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
On Mar 18 11:23, Corinna Vinschen wrote:

> On Mar 18 02:08, Christopher Faylor wrote:
> > On Fri, Mar 18, 2011 at 05:47:04AM +0000, Dave Korn wrote:
> > >On 11/03/2011 13:53, Rainer Emrich wrote:
> > >
> > >> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
> > >> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
> > >
> > >  I run with this setting all the time, I guess that's why I haven't seen this
> > >problem.  Before I did that (couple of years back) I also used to get crashes
> > >building libjava.
> >
> > That setting should have nothing to do with cmalloc().  cmalloc is for
> > Cygwin's internal heap which has nothing to do with that setting.
> >
> > Didn't Corinna already mention this?
>
> In this case the bigger heap seems to avoid the aggressive use of small
> mmap chunks which in turn disallows to raise the cygheap size.  Without
> analyzing the whole situation there's not much else to say or do.  This
> is YA case which screams loudly for a testcase...

I created an extensive testcase(*) and it turned out that the current
algorithm to allocate memory on the cygheap is wasting a lot of memory.
Actually it organizes the memory in buckets, each of which cares for a
specific size as a power of 2.  So memory on the cygheap is always
allocated in chunks of 4 byte, 8 byte, 16 byte, 32 byte, etc.  Plus,
every chunk needs extra 8 byte for bookkeeping.

Now, mmap has to keep a per-mmap record on the cygheap to maintain mmaps
across munmap and fork.  The problem is that the size of the mmap record
so far is 64 bytes plus a 4 byte page bitmap per 128K of allocated memory.

So each mmap(64K) requires a 68 byte record.  Due to the way the cygheap
allocation works, this takes actually 128 bytes on the cygheap.  In my
testcase, after about 8500 mmaps, the required size of the mmap records
on the cygheap is 128*8500 = 1088000 bytes.  Now we're out of space on
the cygheap, but it's not possible to raise the size of the cygheap
since the space beyond is taken by the mmaped regions.  Consequentially
mmap fails, even though there's still space left in the processes
address space.

I applied a patch (actually three patches) to Cygwin now, which are
supposed to drop the cygheap usage of mmap.  The size of a mmap record
is now only 48 bytes.  A 64K mmap requires another 4 bytes for the page
bitmap.  Plus 8 bytes bookkeeping for the allocation algorithm.  Total
of 60 bytes.  This happens to fit in a 64 bit chunk on the cygheap.
64*8500 = 544000 bytes, so there's still a lot of space left on the
cygheap.

The result in the below testcase(*) is now that cygheap is exhausted
only after 14826 calls to mmap(64K).

I've create a new snapshot with this change.  Please test the latest
snapshot from http://cygwin.com/snapshots/

If this doesn't fix your issue with ld, then you would have to raise the
size of the cygheap before running ld.  This requires a change to the
Cygwin DLL, but you can do this without rebuilding the DLL.  Just using
objcopy should help to do it.  A cygheap size of 2 Megs should be
sufficient to exhaust the application's address space faster than the
cygheap.


Corinna


(*) #include <stdio.h>
    #include <sys/mman.h>

    int
    main ()
    {
      unsigned long i = 0;
      void *m2 = NULL, *m1;
      while ((m1 = mmap (NULL, 65536, PROT_READ | PROT_WRITE,
                   MAP_PRIVATE | MAP_ANONYMOUS, -1, 0)) != MAP_FAILED)
        {
          m2 = m1;
          ++i;
        }
      if (m2)
        munmap (m2, 65536); /* Otherwise printf fails. */
      printf ("%d\n", i);
      return 0;
    }

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Christopher Faylor-8
On Fri, Mar 18, 2011 at 03:40:48PM +0100, Corinna Vinschen wrote:

>On Mar 18 11:23, Corinna Vinschen wrote:
>> On Mar 18 02:08, Christopher Faylor wrote:
>> > On Fri, Mar 18, 2011 at 05:47:04AM +0000, Dave Korn wrote:
>> > >On 11/03/2011 13:53, Rainer Emrich wrote:
>> > >
>> > >> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
>> > >> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
>> > >
>> > >  I run with this setting all the time, I guess that's why I haven't seen this
>> > >problem.  Before I did that (couple of years back) I also used to get crashes
>> > >building libjava.
>> >
>> > That setting should have nothing to do with cmalloc().  cmalloc is for
>> > Cygwin's internal heap which has nothing to do with that setting.
>> >
>> > Didn't Corinna already mention this?
>>
>> In this case the bigger heap seems to avoid the aggressive use of small
>> mmap chunks which in turn disallows to raise the cygheap size.  Without
>> analyzing the whole situation there's not much else to say or do.  This
>> is YA case which screams loudly for a testcase...
>
>I created an extensive testcase(*) and it turned out that the current
>algorithm to allocate memory on the cygheap is wasting a lot of memory.
>Actually it organizes the memory in buckets, each of which cares for a
>specific size as a power of 2.  So memory on the cygheap is always
>allocated in chunks of 4 byte, 8 byte, 16 byte, 32 byte, etc.  Plus,
>every chunk needs extra 8 byte for bookkeeping.

Right.  That's a fairly classic implementation of malloc which was, in
this case, implemented by DJ Delorie.  I chose that from a few other
implementations because it was, at the time, supposed to be the best
tradeoff between speed and memory usage.  But, back when I implemented
the cygheap it wasn't used as much as it is now, which I guess is
fairly obvious.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
On Mar 18 10:56, Christopher Faylor wrote:

> On Fri, Mar 18, 2011 at 03:40:48PM +0100, Corinna Vinschen wrote:
> >On Mar 18 11:23, Corinna Vinschen wrote:
> >> On Mar 18 02:08, Christopher Faylor wrote:
> >> > On Fri, Mar 18, 2011 at 05:47:04AM +0000, Dave Korn wrote:
> >> > >On 11/03/2011 13:53, Rainer Emrich wrote:
> >> > >
> >> > >> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
> >> > >> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
> >> > >
> >> > >  I run with this setting all the time, I guess that's why I haven't seen this
> >> > >problem.  Before I did that (couple of years back) I also used to get crashes
> >> > >building libjava.
> >> >
> >> > That setting should have nothing to do with cmalloc().  cmalloc is for
> >> > Cygwin's internal heap which has nothing to do with that setting.
> >> >
> >> > Didn't Corinna already mention this?
> >>
> >> In this case the bigger heap seems to avoid the aggressive use of small
> >> mmap chunks which in turn disallows to raise the cygheap size.  Without
> >> analyzing the whole situation there's not much else to say or do.  This
> >> is YA case which screams loudly for a testcase...
> >
> >I created an extensive testcase(*) and it turned out that the current
> >algorithm to allocate memory on the cygheap is wasting a lot of memory.
> >Actually it organizes the memory in buckets, each of which cares for a
> >specific size as a power of 2.  So memory on the cygheap is always
> >allocated in chunks of 4 byte, 8 byte, 16 byte, 32 byte, etc.  Plus,
> >every chunk needs extra 8 byte for bookkeeping.
>
> Right.  That's a fairly classic implementation of malloc which was, in
> this case, implemented by DJ Delorie.  I chose that from a few other
> implementations because it was, at the time, supposed to be the best
> tradeoff between speed and memory usage.  But, back when I implemented
> the cygheap it wasn't used as much as it is now, which I guess is
> fairly obvious.

Maybe we should modify the implementation at one point, but for now I'm
wonderin if we shouldn't just raise the cygheap size to 2 Megs.  It's
still not much memory given today's RAM sizes, and it's only a fraction
of the application's address space.  But it is enough so that ld's
address space will be exhausted before the cygheap is exhausted.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Rainer Emrich-2
In reply to this post by Corinna Vinschen-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mar 18 16:42, Corinna Vnschen wrote:

>On Mar 18 10:56, Christopher Faylor wrote:
>> On Fri, Mar 18, 2011 at 03:40:48PM +0100, Corinna Vinschen wrote:
>> >On Mar 18 11:23, Corinna Vinschen wrote:
>> >> On Mar 18 02:08, Christopher Faylor wrote:
>> >> > On Fri, Mar 18, 2011 at 05:47:04AM +0000, Dave Korn wrote:
>> >> > >On 11/03/2011 13:53, Rainer Emrich wrote:
>> >> > >
>> >> > >> I have to be more clear. I increased the heap_chunk_in_mb to 1792 using:
>> >> > >> regtool -i set /HKLM/Software/Cygwin/heap_chunk_in_mb 1792
>> >> > >
>> >> > >  I run with this setting all the time, I guess that's why I haven't
seen this
>> >> > >problem.  Before I did that (couple of years back) I also used to get
crashes

>> >> > >building libjava.
>> >> >
>> >> > That setting should have nothing to do with cmalloc().  cmalloc is for
>> >> > Cygwin's internal heap which has nothing to do with that setting.
>> >> >
>> >> > Didn't Corinna already mention this?
>> >>
>> >> In this case the bigger heap seems to avoid the aggressive use of small
>> >> mmap chunks which in turn disallows to raise the cygheap size.  Without
>> >> analyzing the whole situation there's not much else to say or do.  This
>> >> is YA case which screams loudly for a testcase...
>> >
>> >I created an extensive testcase(*) and it turned out that the current
>> >algorithm to allocate memory on the cygheap is wasting a lot of memory.
>> >Actually it organizes the memory in buckets, each of which cares for a
>> >specific size as a power of 2.  So memory on the cygheap is always
>> >allocated in chunks of 4 byte, 8 byte, 16 byte, 32 byte, etc.  Plus,
>> >every chunk needs extra 8 byte for bookkeeping.
>>
>> Right.  That's a fairly classic implementation of malloc which was, in
>> this case, implemented by DJ Delorie.  I chose that from a few other
>> implementations because it was, at the time, supposed to be the best
>> tradeoff between speed and memory usage.  But, back when I implemented
>> the cygheap it wasn't used as much as it is now, which I guess is
>> fairly obvious.
>
>Maybe we should modify the implementation at one point, but for now I'm
>wonderin if we shouldn't just raise the cygheap size to 2 Megs.  It's
>still not much memory given today's RAM sizes, and it's only a fraction
>of the application's address space.  But it is enough so that ld's
>address space will be exhausted before the cygheap is exhausted.
>
>
>Corinna
>

I followed the discussion on the list and just tried the snapshot from 18th of
March. This snapshot allows bootstrapping of gcc-4.5.0-1 including libjava.

Rainer
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2JsVwACgkQoUhjsh59BL5r2gCgqw2UhGeDoP0pKPFLeRITuzK0
5pUAniPzPGiM8ucXNawathY7Hk7XckbZ
=COcE
-----END PGP SIGNATURE-----

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: ld: fatal error - cmalloc would have returned NULL

Corinna Vinschen-2
On Mar 23 09:37, Rainer Emrich wrote:

> On Mar 18 16:42, Corinna Vnschen wrote:
> >On Mar 18 10:56, Christopher Faylor wrote:
> >> On Fri, Mar 18, 2011 at 03:40:48PM +0100, Corinna Vinschen wrote:
> >> >On Mar 18 11:23, Corinna Vinschen wrote:
> >> >> In this case the bigger heap seems to avoid the aggressive use of small
> >> >> mmap chunks which in turn disallows to raise the cygheap size.  Without
> >> >> analyzing the whole situation there's not much else to say or do.  This
> >> >> is YA case which screams loudly for a testcase...
> >> >
> >> >I created an extensive testcase(*) and it turned out that the current
> >> >algorithm to allocate memory on the cygheap is wasting a lot of memory.
> >> >Actually it organizes the memory in buckets, each of which cares for a
> >> >specific size as a power of 2.  So memory on the cygheap is always
> >> >allocated in chunks of 4 byte, 8 byte, 16 byte, 32 byte, etc.  Plus,
> >> >every chunk needs extra 8 byte for bookkeeping.
> >>
> >> Right.  That's a fairly classic implementation of malloc which was, in
> >> this case, implemented by DJ Delorie.  I chose that from a few other
> >> implementations because it was, at the time, supposed to be the best
> >> tradeoff between speed and memory usage.  But, back when I implemented
> >> the cygheap it wasn't used as much as it is now, which I guess is
> >> fairly obvious.
> >
> >Maybe we should modify the implementation at one point, but for now I'm
> >wonderin if we shouldn't just raise the cygheap size to 2 Megs.  It's
> >still not much memory given today's RAM sizes, and it's only a fraction
> >of the application's address space.  But it is enough so that ld's
> >address space will be exhausted before the cygheap is exhausted.
>
> I followed the discussion on the list and just tried the snapshot from 18th of
> March. This snapshot allows bootstrapping of gcc-4.5.0-1 including libjava.

Thanks for testing and thanks for this info!


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple