Problem with multiprocessing module from Python

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Problem with multiprocessing module from Python

Jean-Pierre Flori
Dear all,

Thanks a lot once more for the great software.

During my last attempt to build Sage on Cygwin (32), I mainly have
only one problem left: using the multiprocessing module from python is
problematic.
I was using Cygwin 1.7.25 within a Windows XP virtual machine (VirtualBox 4.3).

This is true both with Cygwin's shipped Python 2.7.x and the one I
built for Sage.
Here is a small snippet of code reproducing the problem:
"""
[Blah python stuff]
>>> from multiprocessing import Pool
>>> p = Pool(3)
>>> p.map(anything suitable)
...
ValueError: semaphore or lock released too many times
"""
This might be a problem with Python but I seem to remember succeeding
in this test some time ago with the same version of Python.
I tried Cygwin's GCC (4.7.3) and a custom built FSF GCC 4.8.2 without success.
I also tried to replace Cygwin binaries with version 1.7.0 and a few
others before 1.7.25 without apparent changes.
So I'm a little lost now.

Can anyone reporduce that?
Any help would be appreciated!

I have another quite unrelated question while I'm here:
I recently had trouble because in Sage two DLLs ended up with the same
name and wanted to be loaded at the same time, resulting in a "cannot
reloacte blah.dll at the same address".
Renaming one of the dll (C file and rebuilding) solved the problem.
I assume this is a windows limitations, is that right?

Thanks in advance for your help,
Best,


--
Jean-Pierre Flori

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Larry Hall (Cygwin)
On 10/25/2013 11:08 AM, Jean-Pierre Flori wrote:
> I have another quite unrelated question while I'm here:
> I recently had trouble because in Sage two DLLs ended up with the same
> name and wanted to be loaded at the same time, resulting in a "cannot
> reloacte blah.dll at the same address".
> Renaming one of the dll (C file and rebuilding) solved the problem.
> I assume this is a windows limitations, is that right?

Very likely, yes.

--
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
Le Fri, 25 Oct 2013 15:07:34 -0400, Larry Hall (Cygwin) a écrit :

> On 10/25/2013 11:08 AM, Jean-Pierre Flori wrote:
>> I have another quite unrelated question while I'm here:
>> I recently had trouble because in Sage two DLLs ended up with the same
>> name and wanted to be loaded at the same time, resulting in a "cannot
>> reloacte blah.dll at the same address".
>> Renaming one of the dll (C file and rebuilding) solved the problem.
>> I assume this is a windows limitations, is that right?
>
> Very likely, yes.

Thanks for the quick answer.
JP


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
In reply to this post by Jean-Pierre Flori
Le Fri, 25 Oct 2013 17:08:50 +0200, Jean-Pierre Flori a écrit :

> This is true both with Cygwin's shipped Python 2.7.x and the one I built
> for Sage.
> Here is a small snippet of code reproducing the problem:
> """
> [Blah python stuff]
>>>> from multiprocessing import Pool p = Pool(3)
>>>> p.map(anything suitable)
> ...
> ValueError: semaphore or lock released too many times """
> This might be a problem with Python but I seem to remember succeeding in
> this test some time ago with the same version of Python.
> I tried Cygwin's GCC (4.7.3) and a custom built FSF GCC 4.8.2 without
> success. I also tried to replace Cygwin binaries with version 1.7.0 and
> a few others before 1.7.25 without apparent changes.
> So I'm a little lost now.
>
> Can anyone reporduce that?
> Any help would be appreciated!
>
For your info, I was unable to reproduce such a behavior on Cygwin 32 v
1.7.25 installs running on 64 bits Windows 7 both on real hardware and
VirtualBox 4.3...

JP


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
Le Fri, 25 Oct 2013 19:53:42 +0000, Jean-Pierre Flori a écrit :

> Le Fri, 25 Oct 2013 17:08:50 +0200, Jean-Pierre Flori a écrit :
>> This is true both with Cygwin's shipped Python 2.7.x and the one I
>> built for Sage.
>> Here is a small snippet of code reproducing the problem:
>> """
>> [Blah python stuff]
>>>>> from multiprocessing import Pool p = Pool(3)
>>>>> p.map(anything suitable)
>> ...
>> ValueError: semaphore or lock released too many times """
>> This might be a problem with Python but I seem to remember succeeding
>> in this test some time ago with the same version of Python.
>> I tried Cygwin's GCC (4.7.3) and a custom built FSF GCC 4.8.2 without
>> success. I also tried to replace Cygwin binaries with version 1.7.0 and
>> a few others before 1.7.25 without apparent changes.
>> So I'm a little lost now.
>>
>> Can anyone reporduce that?
>> Any help would be appreciated!
>>
> For your info, I was unable to reproduce such a behavior on Cygwin 32 v
> 1.7.25 installs running on 64 bits Windows 7 both on real hardware and
> VirtualBox 4.3...
>
> JP
I went on with further testing and could reproduce the problem with the
Cygwin shipped Pythons on a 64 bit Windows 7 running under VirtualBox 4.3
with Cygwin 64 and on a 32 bit Windows 7 running on real hardware (intel
Atom).

Best,
JP


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
Le Tue, 29 Oct 2013 19:40:10 +0000, Jean-Pierre Flori a écrit :

>> For your info, I was unable to reproduce such a behavior on Cygwin 32 v
>> 1.7.25 installs running on 64 bits Windows 7 both on real hardware and
>> VirtualBox 4.3...
>>
>> JP
> I went on with further testing and could reproduce the problem with the
> Cygwin shipped Pythons on a 64 bit Windows 7 running under VirtualBox
> 4.3 with Cygwin 64 and on a 32 bit Windows 7 running on real hardware
> (intel Atom).
>
> Best,
> JP
Same problem with Python 2.6.8 from Cygwin on the 32 bit Windows 7 running
on Intel Atom.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Christopher Faylor-8
On Tue, Oct 29, 2013 at 08:41:07PM +0000, Jean-Pierre Flori wrote:

>Le Tue, 29 Oct 2013 19:40:10 +0000, Jean-Pierre Flori a ??crit??:
>>> For your info, I was unable to reproduce such a behavior on Cygwin 32 v
>>> 1.7.25 installs running on 64 bits Windows 7 both on real hardware and
>>> VirtualBox 4.3...
>>>
>> I went on with further testing and could reproduce the problem with the
>> Cygwin shipped Pythons on a 64 bit Windows 7 running under VirtualBox
>> 4.3 with Cygwin 64 and on a 32 bit Windows 7 running on real hardware
>> (intel Atom).
>>
>> Best,
>> JP
>Same problem with Python 2.6.8 from Cygwin on the 32 bit Windows 7 running
>on Intel Atom.

If you want this fixed, the easiest way to get that to happen is to post
a simple test case which reproduces the problem.  That is not the code
snippet that you sent.  A real working example would be required.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a écrit :
> If you want this fixed, the easiest way to get that to happen is to post
> a simple test case which reproduces the problem.  That is not the code
> snippet that you sent.  A real working example would be required.
Sorry about that.

Here you go:
"""
from multiprocessing import Pool

def f(x): return x

p = pool(2)

p.map(f, [1, 2])
"""

Best,
JP


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
In reply to this post by Jean-Pierre Flori
Le Tue, 29 Oct 2013 20:41:07 +0000, Jean-Pierre Flori a écrit :

> Le Tue, 29 Oct 2013 19:40:10 +0000, Jean-Pierre Flori a écrit :
>>> For your info, I was unable to reproduce such a behavior on Cygwin 32
>>> v 1.7.25 installs running on 64 bits Windows 7 both on real hardware
>>> and VirtualBox 4.3...
>>>
>>> JP
>> I went on with further testing and could reproduce the problem with the
>> Cygwin shipped Pythons on a 64 bit Windows 7 running under VirtualBox
>> 4.3 with Cygwin 64 and on a 32 bit Windows 7 running on real hardware
>> (intel Atom).
>>
>> Best,
>> JP
> Same problem with Python 2.6.8 from Cygwin on the 32 bit Windows 7
> running on Intel Atom.
Some additional info on another 64 bit Windows running on a  Corei7:
* no problem with Cygwin 32 up-to-date.
* with Cygwin 64, i first tested with the old 1.7.22 which was installed
and got "ImportError: This platform lacks a functioning sem_open
implementation, therefore, the required synchronization primitives needed
will not function, see issue 3770." when issuing "p = Pool(2)"; then
updated (to 1.7.25 as well as all packages) and got the "ValueError" as
originally reported.

Best,
JP


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
In reply to this post by Jean-Pierre Flori
Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a écrit :

> Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a écrit :
>> If you want this fixed, the easiest way to get that to happen is to
>> post a simple test case which reproduces the problem.  That is not the
>> code snippet that you sent.  A real working example would be required.
> Sorry about that.
>
> Here you go:
> """
> from multiprocessing import Pool
>
> def f(x): return x
>
> p = pool(2)
>
> p.map(f, [1, 2])
> """
And I managed to introduce a typo. The third line should read Pool, so it
is:
"""
from multiprocessing import Pool

def f(x): return x

p = Pool(2)

p.map(f, [1, 2])
"""


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Corinna Vinschen-2
On Oct 29 21:22, Jean-Pierre Flori wrote:

> Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a écrit :
>
> > Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a écrit :
> >> If you want this fixed, the easiest way to get that to happen is to
> >> post a simple test case which reproduces the problem.  That is not the
> >> code snippet that you sent.  A real working example would be required.
> > Sorry about that.
> >
> > Here you go:
> > """
> > from multiprocessing import Pool
> >
> > def f(x): return x
> >
> > p = pool(2)
> >
> > p.map(f, [1, 2])
> > """
> And I managed to introduce a typo. The third line should read Pool, so it
> is:
> """
> from multiprocessing import Pool
>
> def f(x): return x
>
> p = Pool(2)
>
> p.map(f, [1, 2])
> """
Works for me.  I guess.  At least, if I run the script, nothing
happens:

  $ python x.py
  $

Same on 32 and 64 bit Cygwin.


Corinna

--
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Maintainer                 cygwin AT cygwin DOT com
Red Hat

attachment0 (853 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
Le Wed, 30 Oct 2013 10:33:13 +0100, Corinna Vinschen a écrit :

> On Oct 29 21:22, Jean-Pierre Flori wrote:
>> Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a écrit :
>>
>> > Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a écrit :
>> >> If you want this fixed, the easiest way to get that to happen is to
>> >> post a simple test case which reproduces the problem.  That is not
>> >> the code snippet that you sent.  A real working example would be
>> >> required.
>> > Sorry about that.
>> >
>> > Here you go:
>> > """
>> > from multiprocessing import Pool
>> >
>> > def f(x): return x
>> >
>> > p = pool(2)
>> >
>> > p.map(f, [1, 2])
>> > """
>> And I managed to introduce a typo. The third line should read Pool, so
>> it is:
>> """
>> from multiprocessing import Pool
>>
>> def f(x): return x
>>
>> p = Pool(2)
>>
>> p.map(f, [1, 2])
>> """
>
> Works for me.  I guess.  At least, if I run the script, nothing happens:
>
>   $ python x.py $
>
> Same on 32 and 64 bit Cygwin.
>
>
> Corinna

I think I got to the bottom of this.
It seems the new implem of sem_getvalue in cgwin1.dll is the cause, see:
http://cygwin.com/ml/cygwin-patches/2013-q3/msg00006.html
It may also explain the random reproducibility if sval stays uninitialized
or something like that (I did not check it is the case though).

Reverting to 1.7.20 solved the issue on the install I tested.
I guess my previous attempts to downgrade cygwin1.dll were just badly
done:
I just decompressed the cygwin tarball which only overwrote the dll in /
usr/bin but not in /bin.
Is there a canonical way to do that in a proper way?
Putting it in a local directory and pointing setup.exe ot it?

Best,
JP


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Christopher Faylor-8
On Thu, Oct 31, 2013 at 06:27:39PM +0000, Jean-Pierre Flori wrote:

>Le Wed, 30 Oct 2013 10:33:13 +0100, Corinna Vinschen a ??crit??:
>
>> On Oct 29 21:22, Jean-Pierre Flori wrote:
>>> Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a ??crit??:
>>>
>>> > Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a ??crit??:
>>> >> If you want this fixed, the easiest way to get that to happen is to
>>> >> post a simple test case which reproduces the problem.  That is not
>>> >> the code snippet that you sent.  A real working example would be
>>> >> required.
>>> > Sorry about that.
>>> >
>>> > Here you go:
>>> > """
>>> > from multiprocessing import Pool
>>> >
>>> > def f(x): return x
>>> >
>>> > p = pool(2)
>>> >
>>> > p.map(f, [1, 2])
>>> > """
>>> And I managed to introduce a typo. The third line should read Pool, so
>>> it is:
>>> """
>>> from multiprocessing import Pool
>>>
>>> def f(x): return x
>>>
>>> p = Pool(2)
>>>
>>> p.map(f, [1, 2])
>>> """
>>
>> Works for me.  I guess.  At least, if I run the script, nothing happens:
>>
>>   $ python x.py $
>>
>> Same on 32 and 64 bit Cygwin.
>>
>>
>> Corinna
>
>I think I got to the bottom of this.
>It seems the new implem of sem_getvalue in cgwin1.dll is the cause, see:
>http://cygwin.com/ml/cygwin-patches/2013-q3/msg00006.html
>It may also explain the random reproducibility if sval stays uninitialized
>or something like that (I did not check it is the case though).

I doubt that was the problem.  More likely it is something related to
the changes in thread.cc that followed that change.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Larry Hall (Cygwin)
In reply to this post by Jean-Pierre Flori
On 10/31/2013 2:27 PM, Jean-Pierre Flori wrote:
> I just decompressed the cygwin tarball which only overwrote the dll in /
> usr/bin but not in /bin.
> Is there a canonical way to do that in a proper way?

What did you use to do the decompress and extract?  Windows tools don't
understand Cygwin mount points.

> Putting it in a local directory and pointing setup.exe ot it?

Sure, that works and requires no special effort. :-)

--
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Jean-Pierre Flori
In reply to this post by Christopher Faylor-8
Le Thu, 31 Oct 2013 14:41:14 -0400, Christopher Faylor a écrit :

> On Thu, Oct 31, 2013 at 06:27:39PM +0000, Jean-Pierre Flori wrote:
>>Le Wed, 30 Oct 2013 10:33:13 +0100, Corinna Vinschen a ??crit??:
>>
>>> On Oct 29 21:22, Jean-Pierre Flori wrote:
>>>> Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a ??crit??:
>>>>
>>>> > Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a ??crit??:
>>>> >> If you want this fixed, the easiest way to get that to happen is
>>>> >> to post a simple test case which reproduces the problem.  That is
>>>> >> not the code snippet that you sent.  A real working example would
>>>> >> be required.
>>>> > Sorry about that.
>>>> >
>>>> > Here you go:
>>>> > """
>>>> > from multiprocessing import Pool
>>>> >
>>>> > def f(x): return x
>>>> >
>>>> > p = pool(2)
>>>> >
>>>> > p.map(f, [1, 2])
>>>> > """
>>>> And I managed to introduce a typo. The third line should read Pool,
>>>> so it is:
>>>> """
>>>> from multiprocessing import Pool
>>>>
>>>> def f(x): return x
>>>>
>>>> p = Pool(2)
>>>>
>>>> p.map(f, [1, 2])
>>>> """
>>>
>>> Works for me.  I guess.  At least, if I run the script, nothing
>>> happens:
>>>
>>>   $ python x.py $
>>>
>>> Same on 32 and 64 bit Cygwin.
>>>
>>>
>>> Corinna
>>
>>I think I got to the bottom of this.
>>It seems the new implem of sem_getvalue in cgwin1.dll is the cause, see:
>>http://cygwin.com/ml/cygwin-patches/2013-q3/msg00006.html It may also
>>explain the random reproducibility if sval stays uninitialized or
>>something like that (I did not check it is the case though).
>
> I doubt that was the problem.  More likely it is something related to
> the changes in thread.cc that followed that change.
>
> cgf

I must admit that was just a guess.
The thread.cc code changed whereas python code in Modules/
_multiprocessing/semaphore.c did not.
The python multiprocessing module (file semaphore.c) contains:
"""
        if (sem_getvalue(self->handle, &sval) < 0) {
            return PyErr_SetFromErrno(PyExc_OSError);
        } else if (sval >= self->maxvalue) {
            PyErr_SetString(PyExc_ValueError, "semaphore or lock "
                            "released too many times");
            return NULL;
        }
"""
Changing this to
"""
        sval = sem_getvalue(self->handle, &sval);
        if (sem_getvalue(self->handle, &sval) < 0) {
            return PyErr_SetFromErrno(PyExc_OSError);
        } else if (sval >= self->maxvalue) {
            PyErr_SetString(PyExc_ValueError, "semaphore or lock "
                            "released too many times");
            return NULL;
        }
"""
to emulate the previous behavior of sem_getvalue seems to solve my problem
(and was easier than rebuilding cygwin1.dll).

Commit http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/thread.cc?
rev=1.286&content-type=text/x-cvsweb-markup&cvsroot=src
would be exactly what I need.
I'll just be waiting for 1.7.26.

Hopefully this will get sorted out.

Best,
JP


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Larry Hall (Cygwin)
On 10/31/2013 4:47 PM, Jean-Pierre Flori wrote:
<snip>
> Commit http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/thread.cc?
> rev=1.286&content-type=text/x-cvsweb-markup&cvsroot=src
> would be exactly what I need.
> I'll just be waiting for 1.7.26.

If you want to try this, you don't need to wait for 1.7.26.  Just grab a
recent snapshot and give it a go.

<http://cygwin.com/snapshots/>

--
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Christopher Faylor-8
On Thu, Oct 31, 2013 at 06:04:34PM -0400, Larry Hall (Cygwin) wrote:

>On 10/31/2013 4:47 PM, Jean-Pierre Flori wrote:
><snip>
>> Commit http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/thread.cc?
>> rev=1.286&content-type=text/x-cvsweb-markup&cvsroot=src
>> would be exactly what I need.
>> I'll just be waiting for 1.7.26.
>
>If you want to try this, you don't need to wait for 1.7.26.  Just grab a
>recent snapshot and give it a go.
>
><http://cygwin.com/snapshots/>

Yep. You're always welcome to try that but I don't think there have been
any changes in the thread code which would be pertinent here.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Christopher Faylor-8
In reply to this post by Jean-Pierre Flori
On Thu, Oct 31, 2013 at 08:47:58PM +0000, Jean-Pierre Flori wrote:

>Le Thu, 31 Oct 2013 14:41:14 -0400, Christopher Faylor a ??crit??:
>
>> On Thu, Oct 31, 2013 at 06:27:39PM +0000, Jean-Pierre Flori wrote:
>>>Le Wed, 30 Oct 2013 10:33:13 +0100, Corinna Vinschen a ??crit??:
>>>
>>>> On Oct 29 21:22, Jean-Pierre Flori wrote:
>>>>> Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a ??crit??:
>>>>>
>>>>> > Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a ??crit??:
>>>>> >> If you want this fixed, the easiest way to get that to happen is
>>>>> >> to post a simple test case which reproduces the problem.  That is
>>>>> >> not the code snippet that you sent.  A real working example would
>>>>> >> be required.
>>>>> > Sorry about that.
>>>>> >
>>>>> > Here you go:
>>>>> > """
>>>>> > from multiprocessing import Pool
>>>>> >
>>>>> > def f(x): return x
>>>>> >
>>>>> > p = pool(2)
>>>>> >
>>>>> > p.map(f, [1, 2])
>>>>> > """
>>>>> And I managed to introduce a typo. The third line should read Pool,
>>>>> so it is:
>>>>> """
>>>>> from multiprocessing import Pool
>>>>>
>>>>> def f(x): return x
>>>>>
>>>>> p = Pool(2)
>>>>>
>>>>> p.map(f, [1, 2])
>>>>> """
>>>>
>>>> Works for me.  I guess.  At least, if I run the script, nothing
>>>> happens:
>>>>
>>>>   $ python x.py $
>>>>
>>>> Same on 32 and 64 bit Cygwin.
>>>>
>>>>
>>>> Corinna
>>>
>>>I think I got to the bottom of this.
>>>It seems the new implem of sem_getvalue in cgwin1.dll is the cause, see:
>>>http://cygwin.com/ml/cygwin-patches/2013-q3/msg00006.html It may also
>>>explain the random reproducibility if sval stays uninitialized or
>>>something like that (I did not check it is the case though).
>>
>> I doubt that was the problem.  More likely it is something related to
>> the changes in thread.cc that followed that change.
>>
>> cgf
>
>I must admit that was just a guess.
>The thread.cc code changed whereas python code in Modules/
>_multiprocessing/semaphore.c did not.
>The python multiprocessing module (file semaphore.c) contains:
>"""
>        if (sem_getvalue(self->handle, &sval) < 0) {
>            return PyErr_SetFromErrno(PyExc_OSError);
>        } else if (sval >= self->maxvalue) {
>            PyErr_SetString(PyExc_ValueError, "semaphore or lock "
>                            "released too many times");
>            return NULL;
>        }
>"""
>Changing this to
>"""
>        sval = sem_getvalue(self->handle, &sval);
>        if (sem_getvalue(self->handle, &sval) < 0) {
>            return PyErr_SetFromErrno(PyExc_OSError);
>        } else if (sval >= self->maxvalue) {
>            PyErr_SetString(PyExc_ValueError, "semaphore or lock "
>                            "released too many times");
>            return NULL;
>        }
>"""
>to emulate the previous behavior of sem_getvalue seems to solve my problem
>(and was easier than rebuilding cygwin1.dll).

That doesn't emulate the previous behavior.  The return value of sem_getvalue
was changed to 0 or -1 as per POSIX.  If self->maxvalue is > 1 then
you won't see an error but it won't be correct either.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Christopher Faylor-8
In reply to this post by Christopher Faylor-8
On Thu, Oct 31, 2013 at 02:41:14PM -0400, Christopher Faylor wrote:

>On Thu, Oct 31, 2013 at 06:27:39PM +0000, Jean-Pierre Flori wrote:
>>Le Wed, 30 Oct 2013 10:33:13 +0100, Corinna Vinschen a ??crit??:
>>
>>> On Oct 29 21:22, Jean-Pierre Flori wrote:
>>>> Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a ??crit??:
>>>>
>>>> > Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a ??crit??:
>>>> >> If you want this fixed, the easiest way to get that to happen is to
>>>> >> post a simple test case which reproduces the problem.  That is not
>>>> >> the code snippet that you sent.  A real working example would be
>>>> >> required.
>>>> > Sorry about that.
>>>> >
>>>> > Here you go:
>>>> > """
>>>> > from multiprocessing import Pool
>>>> >
>>>> > def f(x): return x
>>>> >
>>>> > p = pool(2)
>>>> >
>>>> > p.map(f, [1, 2])
>>>> > """
>>>> And I managed to introduce a typo. The third line should read Pool, so
>>>> it is:
>>>> """
>>>> from multiprocessing import Pool
>>>>
>>>> def f(x): return x
>>>>
>>>> p = Pool(2)
>>>>
>>>> p.map(f, [1, 2])
>>>> """
>>>
>>> Works for me.  I guess.  At least, if I run the script, nothing happens:
>>>
>>>   $ python x.py $
>>>
>>> Same on 32 and 64 bit Cygwin.
>>>
>>>
>>> Corinna
>>
>>I think I got to the bottom of this.
>>It seems the new implem of sem_getvalue in cgwin1.dll is the cause, see:
>>http://cygwin.com/ml/cygwin-patches/2013-q3/msg00006.html
>>It may also explain the random reproducibility if sval stays uninitialized
>>or something like that (I did not check it is the case though).
>
>I doubt that was the problem.  More likely it is something related to
>the changes in thread.cc that followed that change.

Actually I misread the CVS log.  It's more likely that this is due to changes
that happened just prior to the above change.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Problem with multiprocessing module from Python

Christopher Faylor-8
In reply to this post by Christopher Faylor-8
On Thu, Oct 31, 2013 at 10:59:11PM -0400, Christopher Faylor wrote:

>On Thu, Oct 31, 2013 at 08:47:58PM +0000, Jean-Pierre Flori wrote:
>>Le Thu, 31 Oct 2013 14:41:14 -0400, Christopher Faylor a ??crit??:
>>
>>> On Thu, Oct 31, 2013 at 06:27:39PM +0000, Jean-Pierre Flori wrote:
>>>>Le Wed, 30 Oct 2013 10:33:13 +0100, Corinna Vinschen a ??crit??:
>>>>
>>>>> On Oct 29 21:22, Jean-Pierre Flori wrote:
>>>>>> Le Tue, 29 Oct 2013 21:19:14 +0000, Jean-Pierre Flori a ??crit??:
>>>>>>
>>>>>> > Le Tue, 29 Oct 2013 16:59:35 -0400, Christopher Faylor a ??crit??:
>>>>>> >> If you want this fixed, the easiest way to get that to happen is
>>>>>> >> to post a simple test case which reproduces the problem.  That is
>>>>>> >> not the code snippet that you sent.  A real working example would
>>>>>> >> be required.
>>>>>> > Sorry about that.
>>>>>> >
>>>>>> > Here you go:
>>>>>> > """
>>>>>> > from multiprocessing import Pool
>>>>>> >
>>>>>> > def f(x): return x
>>>>>> >
>>>>>> > p = pool(2)
>>>>>> >
>>>>>> > p.map(f, [1, 2])
>>>>>> > """
>>>>>> And I managed to introduce a typo. The third line should read Pool,
>>>>>> so it is:
>>>>>> """
>>>>>> from multiprocessing import Pool
>>>>>>
>>>>>> def f(x): return x
>>>>>>
>>>>>> p = Pool(2)
>>>>>>
>>>>>> p.map(f, [1, 2])
>>>>>> """
>>>>>
>>>>> Works for me.  I guess.  At least, if I run the script, nothing
>>>>> happens:
>>>>>
>>>>>   $ python x.py $
>>>>>
>>>>> Same on 32 and 64 bit Cygwin.
>>>>>
>>>>>
>>>>> Corinna
>>>>
>>>>I think I got to the bottom of this.
>>>>It seems the new implem of sem_getvalue in cgwin1.dll is the cause, see:
>>>>http://cygwin.com/ml/cygwin-patches/2013-q3/msg00006.html It may also
>>>>explain the random reproducibility if sval stays uninitialized or
>>>>something like that (I did not check it is the case though).
>>>
>>> I doubt that was the problem.  More likely it is something related to
>>> the changes in thread.cc that followed that change.
>>>
>>> cgf
>>
>>I must admit that was just a guess.
>>The thread.cc code changed whereas python code in Modules/
>>_multiprocessing/semaphore.c did not.
>>The python multiprocessing module (file semaphore.c) contains:
>>"""
>>        if (sem_getvalue(self->handle, &sval) < 0) {
>>            return PyErr_SetFromErrno(PyExc_OSError);
>>        } else if (sval >= self->maxvalue) {
>>            PyErr_SetString(PyExc_ValueError, "semaphore or lock "
>>                            "released too many times");
>>            return NULL;
>>        }
>>"""
>>Changing this to
>>"""
>>        sval = sem_getvalue(self->handle, &sval);
>>        if (sem_getvalue(self->handle, &sval) < 0) {
>>            return PyErr_SetFromErrno(PyExc_OSError);
>>        } else if (sval >= self->maxvalue) {
>>            PyErr_SetString(PyExc_ValueError, "semaphore or lock "
>>                            "released too many times");
>>            return NULL;
>>        }
>>"""
>>to emulate the previous behavior of sem_getvalue seems to solve my problem
>>(and was easier than rebuilding cygwin1.dll).
>
>That doesn't emulate the previous behavior.  The return value of sem_getvalue
>was changed to 0 or -1 as per POSIX.  If self->maxvalue is > 1 then
>you won't see an error but it won't be correct either.

Ok.  I was confused by the seemingly contradictory assertions that the
patch from http://cygwin.com/ml/cygwin-patches/2013-q3/msg00006.html was
both the cause and the solution for the problem.  And, I thought my change
to sem_getvalue had gotten into 1.7.25.  It hadn't.

So, it's my night to be wrong.  Larry, however, is not suffering
similarly: He was right.  Snapshots after 9/25 will be the only versions
of the DLL that have the fix for this problem.  It does seem like the
problem was introduced in changes to thread.cc months before the
abovementioned patch ever went in.

http://cygwin.com/snapshots/

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

12