fixing BLODA-caused fork failures

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

fixing BLODA-caused fork failures

Adam Kellas
My company uses Cygwin and we experience fairly frequent fork
failures, believed to be BLODA-related. I say "believed to be" because
in this corporate environment, like many, we cannot uninstall the
virus scanner even long enough to see what happens without it. The
presumed culprit in our case is Microsoft Forefront Endpoint
Protection, by the way.

So we need Cygwin and we're stuck with Forefront, putting us between a
rock and a hard place. It's clear from the documentation and mailing
list that the official stance wrt BLODA is "sorry, can't help you" and
I understand and accept that. I'm looking for the answer to a related
question: are BLODA-caused fork failures a logically unsolvable
problem due to the way Windows works or is it just a matter of round
tuits? In other words, if we were (hypothetically) able to pay someone
to make MS Forefront and Cygwin play nicely together, would that have
a chance of success? And would the Cygwin maintainers allow such work
into the code base or consider it an unfortunate precedent?

Thanks,
AK

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: fixing BLODA-caused fork failures

Ryan Johnson-10
On 03/10/2013 4:55 PM, Adam Kellas wrote:

> My company uses Cygwin and we experience fairly frequent fork
> failures, believed to be BLODA-related. I say "believed to be" because
> in this corporate environment, like many, we cannot uninstall the
> virus scanner even long enough to see what happens without it. The
> presumed culprit in our case is Microsoft Forefront Endpoint
> Protection, by the way.
>
> So we need Cygwin and we're stuck with Forefront, putting us between a
> rock and a hard place. It's clear from the documentation and mailing
> list that the official stance wrt BLODA is "sorry, can't help you" and
> I understand and accept that. I'm looking for the answer to a related
> question: are BLODA-caused fork failures a logically unsolvable
> problem due to the way Windows works or is it just a matter of round
> tuits? In other words, if we were (hypothetically) able to pay someone
> to make MS Forefront and Cygwin play nicely together, would that have
> a chance of success? And would the Cygwin maintainers allow such work
> into the code base or consider it an unfortunate precedent?
I can't comment on actual code changes, but having played around quite a
bit in the fork code, I can at least speak a little to the technical part.

BLODA commonly causes two classes of symptoms:

1. Fork failures due to injecting dlls at inopportune locations or
otherwise messing with address space layouts in a way that violates
posix semantics (memory-mapping a file where some cygwin dll should have
gone, for example). Cygwin can do nothing, because the damage is
invariably done before cygwin1.dll loads. The problem can only be fixed
if the BLODA were to be more circumspect about where it injects things,
which is unlikely because Windows processes normally care very little
(if at all) about address space layouts.

2. Various file-related errors due to the BLODA touching/locking files
at inopportune times (e.g. right when a cygwin process tries to delete
the file, leading to an "unable to delete" error). This is simple
Windows locking at play, which Cygwin actually respects (unlike many
Windows programs). Again, there's little cygwin can do to address the
problem on its side, because the file really is locked...

... all of which is the long way of explaining why the official stance
is "sorry, you're SOL" ...

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: fixing BLODA-caused fork failures

Larry Hall (Cygwin)
On 10/3/2013 5:50 PM, Ryan Johnson wrote:

> On 03/10/2013 4:55 PM, Adam Kellas wrote:
>> My company uses Cygwin and we experience fairly frequent fork
>> failures, believed to be BLODA-related. I say "believed to be" because
>> in this corporate environment, like many, we cannot uninstall the
>> virus scanner even long enough to see what happens without it. The
>> presumed culprit in our case is Microsoft Forefront Endpoint
>> Protection, by the way.
>>
>> So we need Cygwin and we're stuck with Forefront, putting us between a
>> rock and a hard place. It's clear from the documentation and mailing
>> list that the official stance wrt BLODA is "sorry, can't help you" and
>> I understand and accept that. I'm looking for the answer to a related
>> question: are BLODA-caused fork failures a logically unsolvable
>> problem due to the way Windows works or is it just a matter of round
>> tuits? In other words, if we were (hypothetically) able to pay someone
>> to make MS Forefront and Cygwin play nicely together, would that have
>> a chance of success? And would the Cygwin maintainers allow such work
>> into the code base or consider it an unfortunate precedent?
> I can't comment on actual code changes, but having played around quite a bit
> in the fork code, I can at least speak a little to the technical part.
>
> BLODA commonly causes two classes of symptoms:
>
> 1. Fork failures due to injecting dlls at inopportune locations or otherwise
> messing with address space layouts in a way that violates posix semantics
> (memory-mapping a file where some cygwin dll should have gone, for example).
> Cygwin can do nothing, because the damage is invariably done before
> cygwin1.dll loads. The problem can only be fixed if the BLODA were to be
> more circumspect about where it injects things, which is unlikely because
> Windows processes normally care very little (if at all) about address space
> layouts.
>
> 2. Various file-related errors due to the BLODA touching/locking files at
> inopportune times (e.g. right when a cygwin process tries to delete the
> file, leading to an "unable to delete" error). This is simple Windows
> locking at play, which Cygwin actually respects (unlike many Windows
> programs). Again, there's little cygwin can do to address the problem on its
> side, because the file really is locked...
>
> ... all of which is the long way of explaining why the official stance is
> "sorry, you're SOL" ...

Nothing I say here should be misconstrued to in any way contradict what Ryan
just said.  Its a very good summary of the difficulties caused by BLODA.
But as a point of practicality, 64-bit Cygwin can help with some cases of
DLL address space collisions.  So if you haven't experimented with
64-bit Cygwin in your environment, it may be worth your time.


--
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: fixing BLODA-caused fork failures

Christopher Faylor-8
In reply to this post by Adam Kellas
On Thu, Oct 03, 2013 at 04:55:19PM -0400, Adam Kellas wrote:

>My company uses Cygwin and we experience fairly frequent fork
>failures, believed to be BLODA-related. I say "believed to be" because
>in this corporate environment, like many, we cannot uninstall the
>virus scanner even long enough to see what happens without it. The
>presumed culprit in our case is Microsoft Forefront Endpoint
>Protection, by the way.
>
>So we need Cygwin and we're stuck with Forefront, putting us between a
>rock and a hard place. It's clear from the documentation and mailing
>list that the official stance wrt BLODA is "sorry, can't help you" and
>I understand and accept that. I'm looking for the answer to a related
>question: are BLODA-caused fork failures a logically unsolvable
>problem due to the way Windows works or is it just a matter of round
>tuits? In other words, if we were (hypothetically) able to pay someone
>to make MS Forefront and Cygwin play nicely together, would that have
>a chance of success? And would the Cygwin maintainers allow such work
>into the code base or consider it an unfortunate precedent?

It wouldn't be without precedent.  I've done work for companies before
and the work has made it into the code base.  But I can't predict if it
is even possible to make things work better in this case.

Contact me at "me at-sign cgf period cx" and we can discuss the
possibility of at least investigating what's involved.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: fixing BLODA-caused fork failures

Achim Gratz
In reply to this post by Adam Kellas
Am 03.10.2013 22:55, schrieb Adam Kellas:
> My company uses Cygwin and we experience fairly frequent fork
> failures, believed to be BLODA-related. I say "believed to be" because
> in this corporate environment, like many, we cannot uninstall the
> virus scanner even long enough to see what happens without it.

In my experience the culprit is most often the "real-time" or
"behavioral" component of the anti virus program.  Uninstalling the
virus program is a bad idea and it will certainly draw the ire of
corporate IT, but you may persuade them to exclude the whole Cygwin
installation tree from these two functions.  In any case I'd try to get
a test machine from IT and try if that helps.

 > So we need Cygwin and we're stuck with Forefront, putting us
 > between a rock and a hard place.

AFAIK Forefront uses the same engine on the client as MSE.  MSE has not
been a problem w.r.t. Cygwin for me, so either the rules used by
Forefront have been sharpened or the fork problems may not be related to
it after all.  Do you use other programs that are based on Cygwin, like
NoMachine NX or other (perhaps in-house) programs that insist on putting
their DLL at fixed address spaces?


Regards,
Achim.



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: fixing BLODA-caused fork failures

David Boyce
In reply to this post by Larry Hall (Cygwin)
> But as a point of practicality, 64-bit Cygwin can help with some cases of
> DLL address space collisions.  So if you haven't experimented with
> 64-bit Cygwin in your environment, it may be worth your time.

This sounds very reasonable but I'm curious: is it to date just a
theory that makes sense or have there been actual reports of 64-bit
Cygwin solving address-space collisions?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: fixing BLODA-caused fork failures

Larry Hall (Cygwin)
On 10/8/2013 3:25 PM, David Boyce wrote:
>> But as a point of practicality, 64-bit Cygwin can help with some cases of
>> DLL address space collisions.  So if you haven't experimented with
>> 64-bit Cygwin in your environment, it may be worth your time.
>
> This sounds very reasonable but I'm curious: is it to date just a
> theory that makes sense or have there been actual reports of 64-bit
> Cygwin solving address-space collisions?

Solving?  Well no.  It won't "solve" address-space collisions.  But the
larger address space means that there's less chance of the O/S picking
the address your DLL wants for another.  No guarentees of course. ;-)

--
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: fixing BLODA-caused fork failures

Ryan Johnson-10
On 08/10/2013 3:34 PM, Larry Hall (Cygwin) wrote:

> On 10/8/2013 3:25 PM, David Boyce wrote:
>>> But as a point of practicality, 64-bit Cygwin can help with some
>>> cases of
>>> DLL address space collisions.  So if you haven't experimented with
>>> 64-bit Cygwin in your environment, it may be worth your time.
>>
>> This sounds very reasonable but I'm curious: is it to date just a
>> theory that makes sense or have there been actual reports of 64-bit
>> Cygwin solving address-space collisions?
>
> Solving?  Well no.  It won't "solve" address-space collisions. But the
> larger address space means that there's less chance of the O/S picking
> the address your DLL wants for another.  No guarentees of course. ;-)
>
One data point: I managed to make it a few weeks from a clean
win64/cygwin64 install to the first BLODA symptoms, even though Windows
Defender was alive and well that whole time. Once it hit, though, there
was no obvoius way out except to disable WD. Rebooting might also help,
by resetting an especially unfriendly ASLR arrangement.

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple