Re: setup -g ???

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: setup -g ???

Ken Brown-6
[Redirected to cygwin-apps from
https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]

On 3/22/2018 6:46 PM, Jon Turney wrote:

> On 14/03/2018 15:26, David Allsopp wrote:
>> [reformatted for top-posting]
>>
>> Lee wrote:
>>>> ---------- Forwarded message ----------
>>>> From: Jon Turney
>>>> Date: Fri, 3 Nov 2017 15:26:27 +0000
>>>> Subject: Re: Problem running the latest python2-2.7.14-1 under AppVeyor
>>>> To: The Cygwin Mailing List <[hidden email]>
>>>>
>>>> On 03/11/2017 14:45, Vadim Zeitlin wrote:
>>>>> Our build has started on AppVeyor, a continuous integration provider,
>>>>> started failing since a couple of days as a makefile command running a
>>>>> Python script started failing with exit code 127 without any more
>>>>> information. This is a strange situation as I can't reproduce the
>>>>> problem locally, but something definitely seems to be wrong with this
>>>>> package on the AppVeyor machine as Python just doesn't seem to be
>>>>> executable, e.g. the output of these commands in our batch file
>>>> driving the build:
>>>>
>>>> Perhaps you need to provide the -g (--upgrade-also) flag to cygwin's
>>>> setup.
>>>>
>>>> Due to setup terribleness, without this flag, it will install the
>>>> requested packages, and any missing dependencies of them, but not
>>>> upgrade any of the dependencies which are already installed to the
>>>> current (and perhaps needed) version...
>>>>
>>>> See also [1].
>>>>
>>>> [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
>>>
>>> Should we still be using the -g (--upgrade-also) flag on setup?
>>
>> I believe so (or at least hope so). I think it's the case that setup
>> should now know to upgrade a dependency if you install a new package
>> which requires a newer version of it, but I hope that's not become the
>> same as setup effectively acting with --upgrade-also every time you
>> run it (that would be a real nuisance, unless the entire Cygwin
>> package universe is going to be recompiled on every new Cygwin release).
>
> This is basically correct.
>
> setup is now capable of being told about dependencies where upgrading an
> already installed package is required, but this information isn't
> currently collected
>
> (For example, some packages now exist (e.g. vim [1]), which were built
> with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost certainly
> use ssp/fortify functions in the cygwin DLL, and so require a cygwin
> package >=2.10.0-1 (technically, the requirement is cygwin API >=0.320),
> but the dependency recorded is only on the cygwin package at any version)
>
> That's something someone could usefully work on, if they were so inclined.
The attached cygport patch attempts to address this by requiring, for
each dependency of a package, a version >= the version installed at the
time the package was built.  It treats only dependencies found by
cygport, not those added via REQUIRES in the cygport file.  My thinking
is that the cygport user should be in control of added dependencies;
s/he can add version numbers if desired.

mksetupini would need to be updated to parse versioned dependencies.
(Or maybe it's expecting a different syntax; I didn't check.)

Ken

0001-__pkg_dist-record-package-versions-in-requires.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: setup -g ???

Ken Brown-6
On 4/2/2018 1:03 PM, Ken Brown wrote:

> [Redirected to cygwin-apps from
> https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]
>
> On 3/22/2018 6:46 PM, Jon Turney wrote:
>> On 14/03/2018 15:26, David Allsopp wrote:
>>> [reformatted for top-posting]
>>>
>>> Lee wrote:
>>>>> ---------- Forwarded message ----------
>>>>> From: Jon Turney
>>>>> Date: Fri, 3 Nov 2017 15:26:27 +0000
>>>>> Subject: Re: Problem running the latest python2-2.7.14-1 under
>>>>> AppVeyor
>>>>> To: The Cygwin Mailing List <[hidden email]>
>>>>>
>>>>> On 03/11/2017 14:45, Vadim Zeitlin wrote:
>>>>>> Our build has started on AppVeyor, a continuous integration provider,
>>>>>> started failing since a couple of days as a makefile command
>>>>>> running a
>>>>>> Python script started failing with exit code 127 without any more
>>>>>> information. This is a strange situation as I can't reproduce the
>>>>>> problem locally, but something definitely seems to be wrong with this
>>>>>> package on the AppVeyor machine as Python just doesn't seem to be
>>>>>> executable, e.g. the output of these commands in our batch file
>>>>> driving the build:
>>>>>
>>>>> Perhaps you need to provide the -g (--upgrade-also) flag to cygwin's
>>>>> setup.
>>>>>
>>>>> Due to setup terribleness, without this flag, it will install the
>>>>> requested packages, and any missing dependencies of them, but not
>>>>> upgrade any of the dependencies which are already installed to the
>>>>> current (and perhaps needed) version...
>>>>>
>>>>> See also [1].
>>>>>
>>>>> [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
>>>>
>>>> Should we still be using the -g (--upgrade-also) flag on setup?
>>>
>>> I believe so (or at least hope so). I think it's the case that setup
>>> should now know to upgrade a dependency if you install a new package
>>> which requires a newer version of it, but I hope that's not become
>>> the same as setup effectively acting with --upgrade-also every time
>>> you run it (that would be a real nuisance, unless the entire Cygwin
>>> package universe is going to be recompiled on every new Cygwin release).
>>
>> This is basically correct.
>>
>> setup is now capable of being told about dependencies where upgrading
>> an already installed package is required, but this information isn't
>> currently collected
>>
>> (For example, some packages now exist (e.g. vim [1]), which were built
>> with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost
>> certainly use ssp/fortify functions in the cygwin DLL, and so require
>> a cygwin package >=2.10.0-1 (technically, the requirement is cygwin
>> API >=0.320), but the dependency recorded is only on the cygwin
>> package at any version)
>>
>> That's something someone could usefully work on, if they were so
>> inclined.
>
> The attached cygport patch attempts to address this by requiring, for
> each dependency of a package, a version >= the version installed at the
> time the package was built.  It treats only dependencies found by
> cygport, not those added via REQUIRES in the cygport file.  My thinking
> is that the cygport user should be in control of added dependencies;
> s/he can add version numbers if desired.
>
> mksetupini would need to be updated to parse versioned dependencies. (Or
> maybe it's expecting a different syntax; I didn't check.)

I finally got around to checking this, and it seems that mksetupini
already recognizes dependencies with version relations, provided that
the hint file uses "depends:" instead of "requires:".

Unfortunately, changing cygport to use "depends:" instead of "requires:"
might confuse maintainers, given that cygport files currently use
"DEPEND" for build dependencies.

An alternative would be for cygport to start using commas rather than
spaces to separate dependencies.  mksetupini could then parse
"requires:" the same way it now parses "depends:" if it finds commas in
the "requires:" list.

Ken
Reply | Threaded
Open this post in threaded view
|

Re: setup -g ???

Ken Brown-6
On 4/23/2018 4:45 PM, Ken Brown wrote:

> On 4/2/2018 1:03 PM, Ken Brown wrote:
>> [Redirected to cygwin-apps from
>> https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]
>>
>> On 3/22/2018 6:46 PM, Jon Turney wrote:
>>> On 14/03/2018 15:26, David Allsopp wrote:
>>>> [reformatted for top-posting]
>>>>
>>>> Lee wrote:
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Jon Turney
>>>>>> Date: Fri, 3 Nov 2017 15:26:27 +0000
>>>>>> Subject: Re: Problem running the latest python2-2.7.14-1 under
>>>>>> AppVeyor
>>>>>> To: The Cygwin Mailing List <[hidden email]>
>>>>>>
>>>>>> On 03/11/2017 14:45, Vadim Zeitlin wrote:
>>>>>>> Our build has started on AppVeyor, a continuous integration
>>>>>>> provider,
>>>>>>> started failing since a couple of days as a makefile command
>>>>>>> running a
>>>>>>> Python script started failing with exit code 127 without any more
>>>>>>> information. This is a strange situation as I can't reproduce the
>>>>>>> problem locally, but something definitely seems to be wrong with
>>>>>>> this
>>>>>>> package on the AppVeyor machine as Python just doesn't seem to be
>>>>>>> executable, e.g. the output of these commands in our batch file
>>>>>> driving the build:
>>>>>>
>>>>>> Perhaps you need to provide the -g (--upgrade-also) flag to cygwin's
>>>>>> setup.
>>>>>>
>>>>>> Due to setup terribleness, without this flag, it will install the
>>>>>> requested packages, and any missing dependencies of them, but not
>>>>>> upgrade any of the dependencies which are already installed to the
>>>>>> current (and perhaps needed) version...
>>>>>>
>>>>>> See also [1].
>>>>>>
>>>>>> [1] https://sourceware.org/ml/cygwin/2017-03/msg00365.html
>>>>>
>>>>> Should we still be using the -g (--upgrade-also) flag on setup?
>>>>
>>>> I believe so (or at least hope so). I think it's the case that setup
>>>> should now know to upgrade a dependency if you install a new package
>>>> which requires a newer version of it, but I hope that's not become
>>>> the same as setup effectively acting with --upgrade-also every time
>>>> you run it (that would be a real nuisance, unless the entire Cygwin
>>>> package universe is going to be recompiled on every new Cygwin
>>>> release).
>>>
>>> This is basically correct.
>>>
>>> setup is now capable of being told about dependencies where upgrading
>>> an already installed package is required, but this information isn't
>>> currently collected
>>>
>>> (For example, some packages now exist (e.g. vim [1]), which were
>>> built with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost
>>> certainly use ssp/fortify functions in the cygwin DLL, and so require
>>> a cygwin package >=2.10.0-1 (technically, the requirement is cygwin
>>> API >=0.320), but the dependency recorded is only on the cygwin
>>> package at any version)
>>>
>>> That's something someone could usefully work on, if they were so
>>> inclined.
>>
>> The attached cygport patch attempts to address this by requiring, for
>> each dependency of a package, a version >= the version installed at
>> the time the package was built.  It treats only dependencies found by
>> cygport, not those added via REQUIRES in the cygport file.  My
>> thinking is that the cygport user should be in control of added
>> dependencies; s/he can add version numbers if desired.
>>
>> mksetupini would need to be updated to parse versioned dependencies.
>> (Or maybe it's expecting a different syntax; I didn't check.)
>
> I finally got around to checking this, and it seems that mksetupini
> already recognizes dependencies with version relations, provided that
> the hint file uses "depends:" instead of "requires:".

I've just sent a patch series that does this, among other things.

Ken
Reply | Threaded
Open this post in threaded view
|

collecting information for versioned dependencies (was Re: setup -g ???)

Jon TURNEY
In reply to this post by Ken Brown-6
On 02/04/2018 18:03, Ken Brown wrote:
> [Redirected to cygwin-apps from
> https://cygwin.com/ml/cygwin/2018-03/msg00365.html.]
>
> On 3/22/2018 6:46 PM, Jon Turney wrote:
[...]

>>
>> This is basically correct.
>>
>> setup is now capable of being told about dependencies where upgrading
>> an already installed package is required, but this information isn't
>> currently collected
>>
>> (For example, some packages now exist (e.g. vim [1]), which were built
>> with gcc 6.4.0-5 and cygport 0.31.0-1.  These packages almost
>> certainly use ssp/fortify functions in the cygwin DLL, and so require
>> a cygwin package >=2.10.0-1 (technically, the requirement is cygwin
>> API >=0.320), but the dependency recorded is only on the cygwin
>> package at any version)

(The example given is kind of incomplete, in that we could define an
additional provides: to record that API version.)

>> That's something someone could usefully work on, if they were so
>> inclined.
>
> The attached cygport patch attempts to address this by requiring, for
> each dependency of a package, a version >= the version installed at the
> time the package was built.  It treats only dependencies found by

Thanks.  This is an interesting approach to automatically collecting
this information I hadn't considered.

Problems are that (i) it makes package builds less reproducible (as
these dependencies will depend on the currently installed version of
them, which might well change over time), and (ii) it will tend of
over-estimate the version required.

I don't know how these problems are solved in other distros, but that's
probably worth looking at...

> cygport, not those added via REQUIRES in the cygport file.  My thinking
> is that the cygport user should be in control of added dependencies;
> s/he can add version numbers if desired.
>
> mksetupini would need to be updated to parse versioned dependencies. (Or
> maybe it's expecting a different syntax; I didn't check.)
Reply | Threaded
Open this post in threaded view
|

Re: collecting information for versioned dependencies (was Re: setup -g ???)

ASSI
Jon Turney writes:
>> The attached cygport patch attempts to address this by requiring,
>> for each dependency of a package, a version >= the version installed
>> at the time the package was built.  It treats only dependencies
>> found by
>
> Thanks.  This is an interesting approach to automatically collecting
> this information I hadn't considered.

Recording that information is a good thing.  Making these a (hard)
dependency is probably not very helpful.

> Problems are that (i) it makes package builds less reproducible (as
> these dependencies will depend on the currently installed version of
> them, which might well change over time), and (ii) it will tend of
> over-estimate the version required.

Yes.  I'd rather not go there.

> I don't know how these problems are solved in other distros, but
> that's probably worth looking at...

I can only (barely) tell how openSUSE Tumbleweed does it.  The first
part is that the dependencies are using capabilities, not versions
(i.e. required API level) and those requirements are part of the spec
files.  The second part is that the package dependencies are organized
in "rings" so that outer rings can only start building when the inner
ones are finished.  The third part is that the build is fully automated
via OBS, so if anything changes that somebody else depends on, those
builds are automatically triggered.  There is some extensive scripted
and sometimes manual intervention to prevent certain known false
positives from re-building the whole distro.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada