[RFE] calm needs to know about ZStandard compressed archives

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[RFE] calm needs to know about ZStandard compressed archives

Achim Gratz

While waiting for the Perl bootstrap I've been experimenting with having
cygport produce .tar.zst packages.  Calm (mksetupini) does not recognize
these at the moment, but I've been able to monkey-patch it by adding
'|zst' in the obvious places.  Can this be done officially, please?  I
can't help noting that these really shouldn't be literal strings so that
you only need to change them in a single place if necessary; so perhaps
common_constants.py would be a good place for that definition.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Jon TURNEY
On 25/12/2020 19:59, Achim Gratz wrote:
>
> While waiting for the Perl bootstrap I've been experimenting with having
> cygport produce .tar.zst packages.  Calm (mksetupini) does not recognize
> these at the moment, but I've been able to monkey-patch it by adding
> '|zst' in the obvious places.  Can this be done officially, please?  I
> can't help noting that these really shouldn't be literal strings so that
> you only need to change them in a single place if necessary; so perhaps
> common_constants.py would be a good place for that definition.

Yeah, this list should be done in a less ad-hoc way.

This isn't totally straightforward as calm uses the python tar module to
validate the upload archives, but that doesn't (yet) support zstd
compression.
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Achim Gratz
Jon Turney writes:
> This isn't totally straightforward as calm uses the python tar module
> to validate the upload archives, but that doesn't (yet) support zstd
> compression.

Maybe this would help until upstream gets their act together…

https://pypi.org/project/xtarfile/


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

cygwin-apps mailing list


On 31.12.2020 16:47, Achim Gratz wrote:
> Jon Turney writes:
>> This isn't totally straightforward as calm uses the python tar module
>> to validate the upload archives, but that doesn't (yet) support zstd
>> compression.
>
> Maybe this would help until upstream gets their act together…
>
> https://pypi.org/project/xtarfile/

do you need it as python package ?

>
> Regards,
> Achim.



Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Achim Gratz
Marco Atzeri via Cygwin-apps writes:
> On 31.12.2020 16:47, Achim Gratz wrote:
>> Jon Turney writes:
>>> This isn't totally straightforward as calm uses the python tar module
>>> to validate the upload archives, but that doesn't (yet) support zstd
>>> compression.
>> Maybe this would help until upstream gets their act together…
>> https://pypi.org/project/xtarfile/
>
> do you need it as python package ?

The tricky part is if it's available on sourceware, I'd guess.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

cygwin-apps mailing list
On 31.12.2020 16:52, Achim Gratz wrote:

> Marco Atzeri via Cygwin-apps writes:
>> On 31.12.2020 16:47, Achim Gratz wrote:
>>> Jon Turney writes:
>>>> This isn't totally straightforward as calm uses the python tar module
>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>> compression.
>>> Maybe this would help until upstream gets their act together…
>>> https://pypi.org/project/xtarfile/
>>
>> do you need it as python package ?
>
> The tricky part is if it's available on sourceware, I'd guess.

locate says no, for the time being


Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Jon TURNEY
On 31/12/2020 16:01, Marco Atzeri via Cygwin-apps wrote:

> On 31.12.2020 16:52, Achim Gratz wrote:
>> Marco Atzeri via Cygwin-apps writes:
>>> On 31.12.2020 16:47, Achim Gratz wrote:
>>>> Jon Turney writes:
>>>>> This isn't totally straightforward as calm uses the python tar module
>>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>>> compression.
>>>> Maybe this would help until upstream gets their act together…
>>>> https://pypi.org/project/xtarfile/
>>>
>>> do you need it as python package ?
>>
>> The tricky part is if it's available on sourceware, I'd guess.
>
> locate says no, for the time being

Thanks, but since this is only done in one spot, I think I can just
arrange to pipe ztd archives through 'unzstd' before validating.

Achim,

Can you publish your cygport changes, when you are happy with them, so I
can make some packages to test with?
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Achim Gratz
Jon Turney writes:
> Can you publish your cygport changes, when you are happy with them, so
> I can make some packages to test with?

https://repo.or.cz/cygport/rpm-style.git/shortlog/refs/heads/to-upstream

Then you need to put this into .cygport.conf to actually produce
.tar.zst archives:

CYGPORT_TAR_EXT=".tar.xz"

The default compression level can be changed as follows:

CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner nobody:65534 -J"



Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Brian Inglis
On 2020-12-31 11:33, Achim Gratz wrote:

> Jon Turney writes:
>> Can you publish your cygport changes, when you are happy with them, so
>> I can make some packages to test with?
>
> https://repo.or.cz/cygport/rpm-style.git/shortlog/refs/heads/to-upstream
>
> Then you need to put this into .cygport.conf to actually produce
> .tar.zst archives:
>
> CYGPORT_TAR_EXT=".tar.xz"
   CYGPORT_TAR_EXT=".tar.zst" ???

> The default compression level can be changed as follows:
>
> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner nobody:65534 -J"
   CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner
nobody:65534 --zstd" ???

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Achim Gratz
Brian Inglis writes:
>   CYGPORT_TAR_EXT=".tar.zst" ???

Yes, typo.

>   CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner
>   nobody:65534 --zstd" ???

No.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Brian Inglis
On 2020-12-31 12:18, Achim Gratz wrote:
> Brian Inglis writes:
>> CYGPORT_TAR_EXT=".tar.zst" ???
>
> Yes, typo.

>> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner
>> nobody:65534 --zstd" ???
>
> No.

>>> CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner
>>> nobody:65534 -J"
To which part does "No" apply: you specified -J which is --xz; you could instead
specify -a, --auto-compress to use the archive suffix.

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Achim Gratz
Brian Inglis writes:
> To which part does "No" apply: you specified -J which is --xz; you
> could instead specify -a, --auto-compress to use the archive suffix.

Here's the actual content of my test installation that I've used for the
Perl repository:

--8<---------------cut here---------------start------------->8---
CYGPORT_TAR_CMD="env ZSTD_CLEVEL=19 tar --group nobody:65534 --owner nobody:65534 -a"
CYGPORT_TAR_EXT=".tar.zst"
--8<---------------cut here---------------end--------------->8---


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Jon TURNEY
In reply to this post by Jon TURNEY
On 31/12/2020 17:08, Jon Turney wrote:

> On 31/12/2020 16:01, Marco Atzeri via Cygwin-apps wrote:
>> On 31.12.2020 16:52, Achim Gratz wrote:
>>> Marco Atzeri via Cygwin-apps writes:
>>>> On 31.12.2020 16:47, Achim Gratz wrote:
>>>>> Jon Turney writes:
>>>>>> This isn't totally straightforward as calm uses the python tar module
>>>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>>>> compression.
>>>>> Maybe this would help until upstream gets their act together…
>>>>> https://pypi.org/project/xtarfile/
>>>>
>>>> do you need it as python package ?
>>>
>>> The tricky part is if it's available on sourceware, I'd guess.
>>
>> locate says no, for the time being
>
> Thanks, but since this is only done in one spot, I think I can just
> arrange to pipe ztd archives through 'unzstd' before validating.

Actually, on second look, we need to look inside the archive in a couple
of places (to check if it's empty, and to generate the package file
listings), so using xtarfile seems like the right approach.

I can arrange to install it on sourceware, but it would be nice if
python-xtarfile was packaged so it can be made a prerequisite of the
calm package.
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

cygwin-apps mailing list
On 03.01.2021 16:19, Jon Turney wrote:

> On 31/12/2020 17:08, Jon Turney wrote:
>> On 31/12/2020 16:01, Marco Atzeri via Cygwin-apps wrote:
>>> On 31.12.2020 16:52, Achim Gratz wrote:
>>>> Marco Atzeri via Cygwin-apps writes:
>>>>> On 31.12.2020 16:47, Achim Gratz wrote:
>>>>>> Jon Turney writes:
>>>>>>> This isn't totally straightforward as calm uses the python tar
>>>>>>> module
>>>>>>> to validate the upload archives, but that doesn't (yet) support zstd
>>>>>>> compression.

>
> I can arrange to install it on sourceware, but it would be nice if
> python-xtarfile was packaged so it can be made a prerequisite of the
> calm package.

I will do it, but I will appreciate if package maintainers
can validate the python/pip test release I recently uploaded.
As I am rebuiling/adding python packages using latest test
and going backward is not so nice.

We will also need to modify cygport to stop automatically add
OBSOLETES as most of this packages never existed

eg python3-pytest

 >>> python36-pytest requires: python36 python36-attrs
python36-iniconfig python36-pluggy python36-py python36-packaging
python36-toml
 >>> python36-pytest OBSOLETES: python3-pytest
 >>> python37-pytest requires: python37 python37-attrs
python37-iniconfig python37-pluggy python37-py python37-packaging
python37-toml
 >>> python38-pytest requires: python38 python38-attrs
python38-iniconfig python38-pluggy python38-py python38-packaging
python38-toml

an overall cleaning of obsoletes package may be useful in the
near future. I need to work on it.

regards
Marco

Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Achim Gratz
Marco Atzeri via Cygwin-apps writes:
> We will also need to modify cygport to stop automatically add
> OBSOLETES as most of this packages never existed

I think the right thing to do is to stop having actual packages for
OBSOLETES and just emit the obsoltes: hint instead.

> an overall cleaning of obsoletes package may be useful in the
> near future. I need to work on it.

The mailing list is full of folks that seem to update their
installations only after several years, so while I'm all for removing
the empty packages, the obsolete hints need to stay.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

cygwin-apps mailing list
On 03.01.2021 16:57, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> We will also need to modify cygport to stop automatically add
>> OBSOLETES as most of this packages never existed
>
> I think the right thing to do is to stop having actual packages for
> OBSOLETES and just emit the obsoltes: hint instead.

>> an overall cleaning of obsoletes package may be useful in the
>> near future. I need to work on it.
>
> The mailing list is full of folks that seem to update their
> installations only after several years, so while I'm all for removing
> the empty packages, the obsolete hints need to stay.

I was referring only to the "empty packages" artificially created
of course.

> Regards,
> Achim.
>
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

cygwin-apps mailing list
In reply to this post by Jon TURNEY
On 03.01.2021 16:19, Jon Turney wrote:

>
> I can arrange to install it on sourceware, but it would be nice if
> python-xtarfile was packaged so it can be made a prerequisite of the
> calm package.

I just uploaded test versions of
python3X-xtarfile and python3X-zstandard
that seems are needed for that compression format..

let me know if find any issue.

Regards
Marco

Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Jon TURNEY
In reply to this post by Achim Gratz
On 03/01/2021 15:57, Achim Gratz wrote:
> Marco Atzeri via Cygwin-apps writes:
>> We will also need to modify cygport to stop automatically add
>> OBSOLETES as most of this packages never existed
>
> I think the right thing to do is to stop having actual packages for
> OBSOLETES and just emit the obsoltes: hint instead.

Yeah, this change to cygport should be made and we can turn off support
for setup versions which don't understand the obsoletes: hint.

>> an overall cleaning of obsoletes package may be useful in the
>> near future. I need to work on it.

Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Achim Gratz
Jon Turney writes:
>> I think the right thing to do is to stop having actual packages for
>> OBSOLETES and just emit the obsoltes: hint instead.
>
> Yeah, this change to cygport should be made and we can turn off
> support for setup versions which don't understand the obsoletes: hint.

The obsolete hints are already generated for some time, so at least the
newer packages should have them.  The older packages would need to
retroactively get them (or keep their equally old obsoleteion packages
around still).

So I think this is what's left to do in cygport (untested, on top of my
to-upstream branch):

--8<---------------cut here---------------start------------->8---
From 0491ea2934d83c656ba5dacf7f02c31f02b46a38 Mon Sep 17 00:00:00 2001
Subject: [PATCH] lib/pkg_pkg.cygpart: stop generating packages for obsoletions

---
 lib/pkg_pkg.cygpart | 46 ---------------------------------------------
 1 file changed, 46 deletions(-)

diff --git a/lib/pkg_pkg.cygpart b/lib/pkg_pkg.cygpart
index 2b2f8bc..80003e9 100644
--- a/lib/pkg_pkg.cygpart
+++ b/lib/pkg_pkg.cygpart
@@ -828,32 +828,6 @@ _EOF
  warning "${pkg_hint[${n}]%.hint}.hint is missing";
  fi
 
- for obspkg in ${!pkg_obsoletes_var}
- do
- if [ ${obspkg} = ${PN} ]
- then
- obssubdir= ;
- else
- obssubdir=${obspkg};
- fi
-
- mkdir -p ${distdir}/${PN}/${obssubdir};
- ${CYGPORT_TAR_CMD-tar -J} -cf ${distdir}/${PN}/${obssubdir}/${obspkg}-${PVR}${CYGPORT_TAR_EXT-.tar.xz} --files-from /dev/null
-
- __step "${pkg_name[${n}]} OBSOLETES: ${obspkg}"
-
- cat > ${distdir}/${PN}/${obssubdir}/${obspkg}-${PVR}.hint <<-_EOF
-category: _obsolete
-requires: ${pkg_name[${n}]}
-sdesc: "Obsoleted by ${pkg_name[${n}]}"
-ldesc: "The ${obspkg} package is obsolete.  Selecting this package for
-installation will cause the ${pkg_name[${n}]} package, which replaces this
-one, to be installed instead."
-${obssubdir:+external-source: ${PN}}
-${pkg_tag}
-_EOF
- done
-
  n+=1;
  done
 
@@ -887,26 +861,6 @@ _EOF
  fi
  fi
 
- for obspkg in ${!dbg_obsoletes_var}
- do
- mkdir -p ${distdir}/${PN}/${obspkg};
- ${CYGPORT_TAR_CMD-tar -J} -cf ${distdir}/${PN}/${obspkg}/${obspkg}-${PVR}${CYGPORT_TAR_EXT-.tar.xz} --files-from /dev/null
-
- __step "${PN}-debuginfo OBSOLETES: ${obspkg}"
-
- cat > ${distdir}/${PN}/${obspkg}/${obspkg}-${PVR}.hint <<-_EOF
-category: _obsolete
-requires: ${PN}-debuginfo
-sdesc: "Obsoleted by ${PN}-debuginfo"
-ldesc: "The ${obspkg} package is obsolete.  Selecting this package for
-installation will cause the ${PN}-debuginfo package, which replaces this
-one, to be installed instead."
-external-source: ${PN}
-${pkg_tag}
-_EOF
- done
- fi
-
  # source package hint
  if [ ! -f ${distdir}/${PN}/${PN}-${PVR}-src.hint ]
  then
--
2.29.2
--8<---------------cut here---------------end--------------->8---



Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Reply | Threaded
Open this post in threaded view
|

Re: [RFE] calm needs to know about ZStandard compressed archives

Jon TURNEY
In reply to this post by cygwin-apps mailing list
On 03/01/2021 21:55, Marco Atzeri via Cygwin-apps wrote:

> On 03.01.2021 16:19, Jon Turney wrote:
>
>>
>> I can arrange to install it on sourceware, but it would be nice if
>> python-xtarfile was packaged so it can be made a prerequisite of the
>> calm package.
>
> I just uploaded test versions of
> python3X-xtarfile and python3X-zstandard
> that seems are needed for that compression format..

Thanks.

I uploaded a calm-20210110-1 package, which adds support for .zst
package archives.

This is also deployed on sourceware.
12