scallywag / cygport not pulling lzip

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

scallywag / cygport not pulling lzip

cygwin-apps mailing list
Hi Jon,

it seems that cygport is not pulling the decompressor
that is supposed to recognise:

https://ci.appveyor.com/project/cygwin/scallywag/builds/37134000/job/0m9h56ptrwwyg3hc

 >> Unpacking source flex-2.6.4.tar.lz
tar (child): lzip: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
*** ERROR: tar xf flex-2.6.4.tar.lz failed



Regards
Marco

Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Achim Gratz
Marco Atzeri via Cygwin-apps writes:

> it seems that cygport is not pulling the decompressor
> that is supposed to recognise:
>
> https://ci.appveyor.com/project/cygwin/scallywag/builds/37134000/job/0m9h56ptrwwyg3hc
>
>>> Unpacking source flex-2.6.4.tar.lz
> tar (child): lzip: Cannot exec: No such file or directory
> tar (child): Error is not recoverable: exiting now
> tar: Child returned status 2
> tar: Error is not recoverable: exiting now
> *** ERROR: tar xf flex-2.6.4.tar.lz failed

Since tar is a Base package it doesn't require lzip (which would
effectively make it a Base install).  You have to add lzip to
BUILD_REQUIRES or switch to the GZip archive.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

cygwin-apps mailing list
On 08.01.2021 14:23, ASSI wrote:

> Marco Atzeri via Cygwin-apps writes:
>> it seems that cygport is not pulling the decompressor
>> that is supposed to recognise:
>>
>> https://ci.appveyor.com/project/cygwin/scallywag/builds/37134000/job/0m9h56ptrwwyg3hc
>>
>>>> Unpacking source flex-2.6.4.tar.lz
>> tar (child): lzip: Cannot exec: No such file or directory
>> tar (child): Error is not recoverable: exiting now
>> tar: Child returned status 2
>> tar: Error is not recoverable: exiting now
>> *** ERROR: tar xf flex-2.6.4.tar.lz failed
>
> Since tar is a Base package it doesn't require lzip (which would
> effectively make it a Base install).  You have to add lzip to
> BUILD_REQUIRES or switch to the GZip archive.
>
>
> Regards,
> Achim.
>

Hi Achim,

this is an upstream source package.

IMHO cygport should pull tar and all the decompressor that
is supposed to manage.

  $ grep  SRC_URI NEWS
...
         * SRC_URI supports .tar.lz archives.
         * SRC_URI accepts .tar.xz archives.
         * SRC_URI accepts .tar.lzo archives.
...

Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Jon TURNEY
On 08/01/2021 13:58, Marco Atzeri via Cygwin-apps wrote:

> On 08.01.2021 14:23, ASSI wrote:
>> Marco Atzeri via Cygwin-apps writes:
>>> it seems that cygport is not pulling the decompressor
>>> that is supposed to recognise:
>>>
>>> https://ci.appveyor.com/project/cygwin/scallywag/builds/37134000/job/0m9h56ptrwwyg3hc 
>>>
>>>
>>>>> Unpacking source flex-2.6.4.tar.lz
>>> tar (child): lzip: Cannot exec: No such file or directory
>>> tar (child): Error is not recoverable: exiting now
>>> tar: Child returned status 2
>>> tar: Error is not recoverable: exiting now
>>> *** ERROR: tar xf flex-2.6.4.tar.lz failed
>>
>> Since tar is a Base package it doesn't require lzip (which would
>> effectively make it a Base install).  You have to add lzip to
>> BUILD_REQUIRES or switch to the GZip archive.
>>
>>
>> Regards,
>> Achim.
>>
>
> Hi Achim,
>
> this is an upstream source package.
>
> IMHO cygport should pull tar and all the decompressor that
> is supposed to manage.
>
>   $ grep  SRC_URI NEWS
> ...
>          * SRC_URI supports .tar.lz archives.
>          * SRC_URI accepts .tar.xz archives.
>          * SRC_URI accepts .tar.lzo archives.
> ...

Yeah, perhaps lzip should be a dependency of cygport.

For the moment, I've applied a tweak to scallywag to ensure lzip gets
installed.

Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Brian Inglis
On 2021-01-08 07:21, Jon Turney wrote:

> On 08/01/2021 13:58, Marco Atzeri via Cygwin-apps wrote:
>> On 08.01.2021 14:23, ASSI wrote:
>>> Marco Atzeri via Cygwin-apps writes:
>>>> it seems that cygport is not pulling the decompressor
>>>> that is supposed to recognise:
>>>>
>>>> https://ci.appveyor.com/project/cygwin/scallywag/builds/37134000/job/0m9h56ptrwwyg3hc 
>>>>
>>>>
>>>>>> Unpacking source flex-2.6.4.tar.lz
>>>> tar (child): lzip: Cannot exec: No such file or directory
>>>> tar (child): Error is not recoverable: exiting now
>>>> tar: Child returned status 2
>>>> tar: Error is not recoverable: exiting now
>>>> *** ERROR: tar xf flex-2.6.4.tar.lz failed
>>>
>>> Since tar is a Base package it doesn't require lzip (which would
>>> effectively make it a Base install).  You have to add lzip to
>>> BUILD_REQUIRES or switch to the GZip archive.
>>>
>>>
>>> Regards,
>>> Achim.
>>>
>>
>> Hi Achim,
>>
>> this is an upstream source package.
>>
>> IMHO cygport should pull tar and all the decompressor that
>> is supposed to manage.
>>
>>   $ grep  SRC_URI NEWS
>> ...
>>          * SRC_URI supports .tar.lz archives.
>>          * SRC_URI accepts .tar.xz archives.
>>          * SRC_URI accepts .tar.lzo archives.
>> ...
>
> Yeah, perhaps lzip should be a dependency of cygport.

...and zstd as Achim is working on that.
I do not think we should encourage use of lower compression ratio packages in
cygport and calm, and their libraries in setup.
It appears zstd gives up compression to gain faster decompression with less
memory which may be advantageous for loading a kernel, but less optimal for MBs
of base packages.

[Download size will remain significant in regions with poor or unreliable
infrastructure, and that may include large rural populations and remote areas in
developed countries, where achievable transfer rates and/or download limits are
low, base data charges or overages are expensive e.g. Canada telecomm costs are
higher than most other countries, has many areas with limited services, and
remote communities and indigenous reservations may still have only metered
landline/microwave service AKA long distance dialup at best, and many northern
and western areas of Canada and the US still have little or no GSM mobile
coverage, perhaps only legacy systems https://www.gsma.com/coverage/ ].

> For the moment, I've applied a tweak to scallywag to ensure lzip gets installed.

...and zstd also please.

Perhaps consideration should also be given to setting appropriate compression
defaults in cygport for all supported package compressors e.g. GZIP=-9,
XZ_OPT=-7 which does not increase the memory required from the default -6 but
improves compression, ZSTD_CLEVEL=19 and ZSTD_NBTHREADS=`nproc` perhaps?

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Achim Gratz
Brian Inglis writes:
>> Yeah, perhaps lzip should be a dependency of cygport.

Since it promises to enable it's use, yes.  I wouldn't mind to add a
dependency to tar also.

> ...and zstd as Achim is working on that.

That dependency has already been added to tar, so there is no need to do
anything special.

[as you could hav seen by looking at any of the recent CI logs]

> I do not think we should encourage use of lower compression ratio
> packages in cygport and calm, and their libraries in setup.
> It appears zstd gives up compression to gain faster decompression with
> less memory which may be advantageous for loading a kernel, but less
> optimal for MBs of base packages.

That difference is in the order of 3% over the full distribution at the
highest compression setting and roughly 5% is you use a compression
setting that is about as slow as xz.  The installation time on the other
hand is dramatically shorter with zstd.

> Perhaps consideration should also be given to setting appropriate
> compression defaults in cygport for all supported package compressors
> e.g. GZIP=-9, XZ_OPT=-7 which does not increase the memory required
> from the default -6 but improves compression, ZSTD_CLEVEL=19 and
> ZSTD_NBTHREADS=`nproc` perhaps?

It is relatively useless to have zstd run with multiple threads in
compression mode since a lot of what takes time there is inherently
single threaded.  It already does use an I/O thread to keep the data
pipes fed unless you specifically forbid that.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Brian Inglis
On 2021-01-08 10:13, Achim Gratz wrote:
> Brian Inglis writes:
>>> Yeah, perhaps lzip should be a dependency of cygport.
>
> Since it promises to enable it's use, yes.  I wouldn't mind to add a
> dependency to tar also.

I would support that for technical reasons alone, although hoped for uptake of
lzip across GNU or by other projects has been limited to the package itself and
others by the same developer, as its goals of providing archive redundancy and
recoverability have been obviated by reliable network links and larger storage
sizes.

>> ...and zstd as Achim is working on that.
>
> That dependency has already been added to tar, so there is no need to do
> anything special.
>
> [as you could hav seen by looking at any of the recent CI logs]
>
>> I do not think we should encourage use of lower compression ratio
>> packages in cygport and calm, and their libraries in setup.
>> It appears zstd gives up compression to gain faster decompression with
>> less memory which may be advantageous for loading a kernel, but less
>> optimal for MBs of base packages.
>
> That difference is in the order of 3% over the full distribution at the
> highest compression setting and roughly 5% is you use a compression
> setting that is about as slow as xz.  The installation time on the other
> hand is dramatically shorter with zstd.

Do we know what the frequency weighted difference is on bandwidth of packages
actually downloaded?
I am more concerned with mirror providers (and also the lack of them) especially
those with limited resources, and those in marginal locations and circumstances,
for whom download time and charges may override other considerations, and
perhaps prevent them (or many) from accessing or taking full advantage of
available software.
I doubt the unarchiving time difference is more than a blip in the total time
required to *download* *AND* install any package, greatly outweighed by the
download time difference, unless you are on a big pipe to a nearby mirror.
[I take into consideration that it used to be cheaper to dialup NIST ACTS
Colorado to set the time, than anywhere in Canada, and downloads did not become
reasonably affordable until I could get network access from a local dialup.] ;^>

>> Perhaps consideration should also be given to setting appropriate
>> compression defaults in cygport for all supported package compressors
>> e.g. GZIP=-9, XZ_OPT=-7 which does not increase the memory required
>> from the default -6 but improves compression, ZSTD_CLEVEL=19 and
>> ZSTD_NBTHREADS=`nproc` perhaps?
>
> It is relatively useless to have zstd run with multiple threads in
> compression mode since a lot of what takes time there is inherently
> single threaded.  It already does use an I/O thread to keep the data
> pipes fed unless you specifically forbid that.

That is valid consideration indeed as suggested.

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Achim Gratz
Brian Inglis writes:
> Do we know what the frequency weighted difference is on bandwidth of
> packages actually downloaded?

Not that I know of, as everything goes through mirrors.  But I happen to
have a complete Cygwin mirror on disk at the moment plus another one
that only has the packages for my install and that's a fairly large
installation, but without a desktop environment:

30G     /mnt/mirror/cygwin
149G    /mnt/fullmirror/cygwin

So you can probably assume that only about 20% of the files are
frequently accessed (likely significantly less since most folks would
not install the debuginfo or source packages that are included in the
above figure).

> I am more concerned with mirror providers (and also the lack of them)
> especially those with limited resources, and those in marginal
> locations and circumstances, for whom download time and charges may
> override other considerations, and perhaps prevent them (or many) from
> accessing or taking full advantage of available software.

We could save way more space than that by de-duplicating the noarch
parts into their own archives as I have already demonstrated before.
The last time I did that I was cutting out around 30GiB IIRC.

> I doubt the unarchiving time difference is more than a blip in the
> total time required to *download* *AND* install any package, greatly
> outweighed by the download time difference, unless you are on a big
> pipe to a nearby mirror.

It is not, with a typical VDSL connection I'd be able to download faster
than I can install on a more typical machine, I need only about 5MiB/s
to saturate the filesystem for small files and around 20…40MiB/s for
large ones (to an NVMe drive, a spinning disk or some of the slower SSD
can't sustain that).  But that point is somewhat moot since setup will
always mirror to disk first and that's not easy to change since we read
the file twice: once for the SHA512 check (which can use up to around
300MiB/s input bandwidth somewhat higher in peaks and then the actual
installation).

I have a large base of internal installations that I feed from a
(single) local repo and some of those machines are behind rather slow
links (not quite modem speed, but still slow by todays standards) and
using zstd still makes quite the difference there.  The more typical
install time was reduced by a bit under 50% for both slow and fast
connections, so I no longer recommend that folks reserve time
specifically for the Cygwin installation as I had done before.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf Blofeld V1.15B11:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Achim Gratz
In reply to this post by Achim Gratz
Achim Gratz writes:
> Since it promises to enable it's use, yes.  I wouldn't mind to add a
> dependency to tar also.

Since lzip is not a very large package and tar just released a new
version… should I add the dependency or should I keep things as they
are?


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

DIY Stuff:
http://Synth.Stromeko.net/DIY.html
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Brian Inglis
In reply to this post by Achim Gratz
On 2021-01-08 12:11, Achim Gratz wrote:

> Brian Inglis writes:
>> Do we know what the frequency weighted difference is on bandwidth of
>> packages actually downloaded?
>
> Not that I know of, as everything goes through mirrors.  But I happen to
> have a complete Cygwin mirror on disk at the moment plus another one
> that only has the packages for my install and that's a fairly large
> installation, but without a desktop environment:
>
> 30G     /mnt/mirror/cygwin
> 149G    /mnt/fullmirror/cygwin
>
> So you can probably assume that only about 20% of the files are
> frequently accessed (likely significantly less since most folks would
> not install the debuginfo or source packages that are included in the
> above figure).
>
>> I am more concerned with mirror providers (and also the lack of them)
>> especially those with limited resources, and those in marginal
>> locations and circumstances, for whom download time and charges may
>> override other considerations, and perhaps prevent them (or many) from
>> accessing or taking full advantage of available software.
>
> We could save way more space than that by de-duplicating the noarch
> parts into their own archives as I have already demonstrated before.
> The last time I did that I was cutting out around 30GiB IIRC.
>
>> I doubt the unarchiving time difference is more than a blip in the
>> total time required to *download* *AND* install any package, greatly
>> outweighed by the download time difference, unless you are on a big
>> pipe to a nearby mirror.
>
> It is not, with a typical VDSL connection I'd be able to download faster
> than I can install on a more typical machine, I need only about 5MiB/s
> to saturate the filesystem for small files and around 20…40MiB/s for
> large ones (to an NVMe drive, a spinning disk or some of the slower SSD
> can't sustain that).  But that point is somewhat moot since setup will
> always mirror to disk first and that's not easy to change since we read
> the file twice: once for the SHA512 check (which can use up to around
> 300MiB/s input bandwidth somewhat higher in peaks and then the actual
> installation).

Some setup phase stats from my own most recent upgrade of about 130MB of
downloads, and stats since 2013 (nearly 8 years) since I last cleared setup.log:

$ cyg-setup-phase-times.awk /var/log/setup.log.full
sv 00:04:26 dl 00:01:28 pr 00:00:02 ui 00:00:36 ex 00:03:35 pi 00:07:12
$ cyg-setup-phase-times.awk /var/log/setup.log
sv 04:32:46 dl 02:31:27 pr 00:16:08 ui 00:51:10 ex 06:06:49 pi 00:00:41

phases are:

sv - solve formerly Adding required packages - high times are interaction delays
dl - download
pr - preremove
ui - uninstall
ex - extract
pi - postinstall

so your comments about extracts are validated, taking about 3 times as long as
downloads even on a currently 2MByte/s medium speed cable modem link to a nearby
(7.5km direct, 11km drive, 15 hop 10ms round trip) university campus mirror in
recent years.

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Brian Inglis
On 2021-01-11 15:05, Brian Inglis wrote:

> On 2021-01-08 12:11, Achim Gratz wrote:
>> Brian Inglis writes:
>>> Do we know what the frequency weighted difference is on bandwidth of
>>> packages actually downloaded?
>>
>> Not that I know of, as everything goes through mirrors.  But I happen to
>> have a complete Cygwin mirror on disk at the moment plus another one
>> that only has the packages for my install and that's a fairly large
>> installation, but without a desktop environment:
>>
>> 30G     /mnt/mirror/cygwin
>> 149G    /mnt/fullmirror/cygwin

Currently mirrors.html states 160GB total.

>> So you can probably assume that only about 20% of the files are
>> frequently accessed (likely significantly less since most folks would
>> not install the debuginfo or source packages that are included in the
>> above figure).
>>
>>> I am more concerned with mirror providers (and also the lack of them)
>>> especially those with limited resources, and those in marginal
>>> locations and circumstances, for whom download time and charges may
>>> override other considerations, and perhaps prevent them (or many) from
>>> accessing or taking full advantage of available software.
>>
>> We could save way more space than that by de-duplicating the noarch
>> parts into their own archives as I have already demonstrated before.
>> The last time I did that I was cutting out around 30GiB IIRC.

Rsync delta download data transfer quantities are still significant on metered
or limited accounts with high overage charges: mirror storage cost is trivial by
comparison.
Parents are being hit with $Ks monthly overage charges run up by their kids
downloading and running games on multiple devices, or cut off after hitting
their limit if they block overages, which prevents access to school or work
until the next month capacity allowance kicks in!

>>> I doubt the unarchiving time difference is more than a blip in the
>>> total time required to *download* *AND* install any package, greatly
>>> outweighed by the download time difference, unless you are on a big
>>> pipe to a nearby mirror.
>>
>> It is not, with a typical VDSL connection I'd be able to download faster
>> than I can install on a more typical machine, I need only about 5MiB/s
>> to saturate the filesystem for small files and around 20…40MiB/s for
>> large ones (to an NVMe drive, a spinning disk or some of the slower SSD
>> can't sustain that).  But that point is somewhat moot since setup will
>> always mirror to disk first and that's not easy to change since we read
>> the file twice: once for the SHA512 check (which can use up to around
>> 300MiB/s input bandwidth somewhat higher in peaks and then the actual
>> installation).
>
> Some setup phase stats from my own most recent upgrade of about 130MB of
> downloads, and stats since 2013 (nearly 8 years) since I last cleared setup.log:

Added average and standard deviation per run for multiple runs, and total times
for all runs, including another set with data only since the solver was added
early 2018; trivial runs with no download etc. phase times are not counted, but
their solve times may be included in stats:

$ cyg-setup-phase-times.awk /var/log/setup.log.full
sv 00:04:26 dl 00:01:28 pr 00:00:02 ui 00:00:36 ex 00:03:35 pi 00:07:12 tot
00:17:19 runs 1
$ cyg-setup-phase-times.awk /var/log/setup.log.solve
sv 00:04:06 dl 00:00:33 pr 00:00:04 ui 00:00:20 ex 00:01:16 pi 00:12:05 avg
00:18:27 runs 66
sv 00:06:11 dl 00:00:53 pr 00:00:17 ui 00:00:25 ex 00:02:05 pi 00:11:08 dev
00:12:57 runs 66
sv 04:30:50 dl 00:37:07 pr 00:04:57 ui 00:22:03 ex 01:24:25 pi 13:18:32 tot
20:17:54 runs 66
$ cyg-setup-phase-times.awk /var/log/setup.log
sv 00:00:45 dl 00:00:25 pr 00:00:02 ui 00:00:08 ex 00:01:01 pi 00:04:01 avg
00:06:24 runs 358
sv 00:03:06 dl 00:00:57 pr 00:00:09 ui 00:00:15 ex 00:05:29 pi 00:06:25 dev
00:09:03 runs 358
sv 04:32:46 dl 02:31:27 pr 00:16:08 ui 00:51:10 ex 06:04:40 pi 23:58:16 tot  1
14:14:27 runs 358

> phases are:
>
> sv - solve formerly Adding required packages - high times are interaction delays
> dl - download
> pr - preremove
> ui - uninstall
> ex - extract
> pi - postinstall
>
> so your comments about extracts are validated, taking nearly 3 times as long as
> downloads on a currently 2MByte/s medium speed cable modem link to a nearby
> (7.5km direct, 11km drive, 15 hop 10ms round trip) university campus mirror in
> recent years.

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
Reply | Threaded
Open this post in threaded view
|

Re: scallywag / cygport not pulling lzip

Achim Gratz
Brian Inglis writes:
> Currently mirrors.html states 160GB total.

I'm only mirroring the stuff that setup might actually use.  The other
mirror, as I said is only what setup actually uses, plus one previous
version plus the sources of that.

> Rsync delta download data transfer quantities are still significant on
> metered or limited accounts with high overage charges: mirror storage
> cost is trivial by comparison.

Storage doesn't cost anything apparently so I'd not worry about it.
I've been using a metered connection over modem for long enough to know
how it feels to run an 8 hour download to get all the stuff for the next
update.  But these days pretty much anybody expects you to be able to
download another GiB each month and in the grand scheme of things Cygwin
doesn't really matter unless you insist on downloading everything ( and
even then that is only once).

> Parents are being hit with $Ks monthly overage charges run up by their
> kids downloading and running games on multiple devices, or cut off
> after hitting their limit if they block overages, which prevents
> access to school or work until the next month capacity allowance kicks
> in!

That's not something we can help with, really.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada