Package file lists archive

classic Classic list List threaded Threaded
34 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Package file lists archive

Steven Penny
The MSYS2 project keep an up to date archive with files lists for all packages

http://downloads.sourceforge.net/project/msys2/REPOS/MINGW/x86_64/mingw64.files.tar.gz

This file is not very large considering

    $ gzip -l mingw64.files.tar.gz
             compressed        uncompressed  ratio uncompressed_name
                 982852             9585152  89.7% mingw64.files.tar

I was wondering if something like this is possible with Cygwin. Currently if you
want to search a file that is not installed you must use

    cygcheck --package-query

which I have noticed can be very slow. I would be happy to help with maintaining
a Cygwin archive with file lists for all packages.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Christopher Faylor-8
On Sun, May 25, 2014 at 05:37:45PM -0500, Steven Penny wrote:

>The MSYS2 project keep an up to date archive with files lists for all packages
>
>http://downloads.sourceforge.net/project/msys2/REPOS/MINGW/x86_64/mingw64.files.tar.gz
>
>This file is not very large considering
>
>    $ gzip -l mingw64.files.tar.gz
>             compressed        uncompressed  ratio uncompressed_name
>                 982852             9585152  89.7% mingw64.files.tar
>
>I was wondering if something like this is possible with Cygwin. Currently if you
>want to search a file that is not installed you must use
>
>    cygcheck --package-query
>
>which I have noticed can be very slow. I would be happy to help with maintaining
>a Cygwin archive with file lists for all packages.

Not sure how you'd do that.  I suspect that MSYS is a lot more static
than Cygwin.  I don't think package maintainers wants to notify you
every time they upload a package.

And, frankly, I think it's a bad idea to keep a monolithic list of
packages available for people to download anyway.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
On Mon, May 26, 2014 at 3:12 PM, Christopher Faylor wrote:
> And, frankly, I think it's a bad idea to keep a monolithic list of
> packages available for people to download anyway.

Could you not just require each package includes a file list? This way you could
get the information from any mirror.

This would ease the burden of the cygwin.com site, as it is the only site with
that information currently

http://google.com/search?filter=0&q="etc/postinstall/cygcheck-dep.sh"

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Christopher Faylor-8
On Mon, May 26, 2014 at 10:18:20PM -0500, Steven Penny wrote:
>On Mon, May 26, 2014 at 3:12 PM, Christopher Faylor wrote:
>> And, frankly, I think it's a bad idea to keep a monolithic list of
>> packages available for people to download anyway.
>
>Could you not just require each package includes a file list? This way you could
>get the information from any mirror.

Not interested.

>This would ease the burden of the cygwin.com site, as it is the only site with
>that information currently

It is surviving just fine.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
On Tue, May 27, 2014 at 5:11 PM, Christopher Faylor wrote:
>>This would ease the burden of the cygwin.com site, as it is the only site with
>>that information currently
>
> It is surviving just fine.

In response to this apathy, I have found my own way. Using Google Search API is
in many cases faster than "cygcheck -p". Code is here

http://github.com/transcode-open/apt-cyg/blob/29cd844/apt-cyg#L308-L360

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Warren Young
In reply to this post by Christopher Faylor-8
On 5/27/2014 16:11, Christopher Faylor wrote:
>
>> This would ease the burden of the cygwin.com site, as it is the only site with
>> that information currently
>
> It is surviving just fine.

Speaking of, is the source for package-grep.cgi available somewhere?  I
could quarter the page weight, and probably improve the render time in
browsers considerably, too.  It currently takes about 16 seconds to
completely load in Firefox on my system, most of which is just page
rendering time, due to the huge table used for layout.

(I'd also need either a static copy of its input, or a way to easily
generate it locally.)

The /packages page load time is a considerable fraction of this, at
roughly 2-4 seconds.  It's only returning about 120 kB of text
(compressed) so that comes to roughly half a megabit/sec.  I assume that
means the script is running every time, re-generating the whole page
text anew instead of caching the output, despite the fact that the input
probably changes less often than once a day on average.  Caching should
give a page load time of a few hundred ms, and save some CPU time to boot.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
On Mon, Jun 9, 2014 at 10:17 AM, Warren Young wrote:
> Speaking of, is the source for package-grep.cgi available somewhere?  I

http://cygwin.com/cgi-bin/cvsweb.cgi/htdocs/cgi-bin2?cvsroot=cygwin

Of course because of Cygwin flawless website, this is not indexed by Google and
not even linked on Cygwin.com, so you are a left with "Isla de Muerta".

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
In reply to this post by Steven Penny
On Mon, Jun 9, 2014 at 5:45 AM, Steven Penny wrote:
> Using Google Search API is in many cases faster than "cygcheck -p". Code is
> here

Adding to this, here is the sometimes frustrating experience of "cygcheck -p"

    $ time cygcheck -p bin/aclocal
    cygcheck: unable to contact cygwin.com site, InternetOpenUrl() failed: The
              operation timed out
     (win32 error 12002)

    real    0m31.132s
    user    0m0.016s
    sys     0m0.031s

And here is current iteration of my tool

    $ time apt-cyg searchall bin/aclocal
    getting pages . . .
    getting start 0 . . .
    getting start 8 . . .
    gcc-tools-epoch1-automake
    gcc-tools-epoch2-automake
    automake1.4
    automake1.13
    automake1.12
    automake1.11
    automake1.7
    automake1.14
    automake1.10
    automake1.8
    automake1.9

    real    0m1.295s
    user    0m0.165s
    sys     0m0.182s

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

PolarStorm
Steven Penny wrote
Adding to this, here is the sometimes frustrating experience of "cygcheck -p"
...
And here is current iteration of my tool
...
Thank you very much for that apt-cyg patch!

I don't use cygcheck very often, but when I do it's been mostly a waste
of time. Ever since I started using apt-cyg, I've been a much more happy
Cygwin user. Now, with your patch that is even better.  I'm sure there are
loads of Cygwin users who greatly appreciate your effort.

BR,
Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
In reply to this post by Christopher Faylor-8
On Mon, May 26, 2014 at 3:12 PM, Christopher Faylor wrote:
> And, frankly, I think it's a bad idea to keep a monolithic list of
> packages available for people to download anyway.

I would counter that Debian and others who use "apt-get" already do this,
Example

http://http.debian.net/debian/dists/wheezy/main/Contents-amd64.gz

> This file maps each file available in the Debian GNU/Linux system to
> the package from which it originates.  It includes packages from the
> DIST distribution for the ARCH architecture.
> You can use this list to determine which package contains a specific
> file, or whether or not a specific file is available.  The list is
> updated weekly, each architecture on a different day.

so maybe you could voice your opinion to them as well.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Christopher Faylor-8
On Thu, Jun 12, 2014 at 04:58:25PM -0500, Steven Penny wrote:

>On Mon, May 26, 2014 at 3:12 PM, Christopher Faylor wrote:
>> And, frankly, I think it's a bad idea to keep a monolithic list of
>> packages available for people to download anyway.
>
>I would counter that Debian and others who use "apt-get" already do this,
>Example
>
>http://http.debian.net/debian/dists/wheezy/main/Contents-amd64.gz
>
>> This file maps each file available in the Debian GNU/Linux system to
>> the package from which it originates.  It includes packages from the
>> DIST distribution for the ARCH architecture.
>> You can use this list to determine which package contains a specific
>> file, or whether or not a specific file is available.  The list is
>> updated weekly, each architecture on a different day.
>
>so maybe you could voice your opinion to them as well.

FYI, trolling is obviously off-topic here.

It's nice that you found a solution to your problem, however.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Christopher Faylor-8
In reply to this post by Steven Penny
On Mon, Jun 09, 2014 at 11:17:11AM -0500, Steven Penny wrote:
>On Mon, Jun 9, 2014 at 10:17 AM, Warren Young wrote:
>> Speaking of, is the source for package-grep.cgi available somewhere?  I
>
>http://cygwin.com/cgi-bin/cvsweb.cgi/htdocs/cgi-bin2?cvsroot=cygwin
>
>Of course because of Cygwin flawless website, this is not indexed by Google and
>not even linked on Cygwin.com, so you are a left with "Isla de Muerta".

Ok Mr Penny.  Here's warning 1 for you:

Can the attitude.  I'm sick of it.

If you want to experimentally figure out how many warnings you get,
please go right ahead and continue with this style of communication.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
On Tue, Jun 17, 2014 at 9:29 AM, Christopher Faylor wrote:
> If you want to experimentally figure out how many warnings you get,
> please go right ahead and continue with this style of communication.

Really, I can do without veiled threats. If you are going to ban me or whatever,
go ahead and do it.

I have brought some constructive criticisms of this project with the hope of
starting a discussion around fixing/improving certain issues. Instead I am met
with brick wall attitude and threats of being pushed off the mailing list.

My suggestion has merit: Debian is no niche operating system and "apt-get" is no
niche package manager. To look to them for "how it should be done" or
"one way to do it" is not a mistake in my opinion. However I am met with this

> And, frankly, I think it's a bad idea to keep a monolithic list of
> packages available for people to download anyway.

Which is little more than your opinion with no explanation or citations as to
why it is a bad idea. If it is such a bad idea, why are these major Linux
distros using it?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Christopher Faylor-8
On Tue, Jun 17, 2014 at 11:29:53PM -0500, Steven Penny wrote:

>On Tue, Jun 17, 2014 at 9:29 AM, Christopher Faylor wrote:
>>On Mon, Jun 09, 2014 at 11:17:11AM -0500, Steven Penny wrote:
>>>Of course because of Cygwin flawless website...
>>>
>> If you want to experimentally figure out how many warnings you get,
>> please go right ahead and continue with this style of communication.
>
>Really, I can do without veiled threats. If you are going to ban me or whatever,
>go ahead and do it.
>
>I have brought some constructive criticisms of this project with the hope of
>starting a discussion around fixing/improving certain issues. Instead I am met
>with brick wall attitude and threats of being pushed off the mailing list.

Your idea of "constructive" is apparently flawed.  The comment that you
cut (and I just put back) illustrates what I was talking about, as does
your suggestion that I should contact Debian maintainers because I
disagreed with the idea of maintaining a package file list.

Your truculent insistence that I should knight you as the coreutils/bash
maintainer is another example.

>My suggestion has merit: Debian is no niche operating system and
>"apt-get" is no niche package manager.

You can make any suggestions you want.  Just avoid the snide asides.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
On Wed, Jun 18, 2014 at 1:18 AM, Christopher Faylor wrote:
> Your idea of "constructive" is apparently flawed.

This is an interesting comment, taking in that your response ignores the
original question and focuses on off topic quibbling. I will pose your viewpoint
and my question again

> And, frankly, I think it's a bad idea to keep a monolithic list of
> packages available for people to download anyway.

Do you have an explanation for this opinion, or citations as to why this is a
bad idea, taking into account that major Linux distros are already using this
very method?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Larry Hall (Cygwin)
On 06/18/2014 02:53 AM, Steven Penny wrote:
> On Wed, Jun 18, 2014 at 1:18 AM, Christopher Faylor wrote:
>> Your idea of "constructive" is apparently flawed.
>
> This is an interesting comment, taking in that your response ignores the
> original question and focuses on off topic quibbling. I will pose your viewpoint
> and my question again

I also find it interesting that you dismiss your "off topic quibbling"
since you were the one that was feeding the fire while Chris was
on vacation last week.

> Do you have an explanation for this opinion, or citations as to why this is a
> bad idea, taking into account that major Linux distros are already using this
> very method?

If this was the question you wanted to ask originally, this is all you
needed to add to the thread.

--
Larry

_____________________________________________________________________

A: Yes.
 > Q: Are you sure?
 >> A: Because it reverses the logical flow of conversation.
 >>> Q: Why is top posting annoying in email?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
On Wed, Jun 18, 2014 at 7:28 AM, Larry Hall (Cygwin) wrote:
> If this was the question you wanted to ask originally, this is all you
> needed to add to the thread.

Hey Larry, thanks for your comments. However, did you have anything constructive
to add, like an answer to the question?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Christopher Faylor-8
In reply to this post by Steven Penny
On Wed, Jun 18, 2014 at 01:53:14AM -0500, Steven Penny wrote:
>On Wed, Jun 18, 2014 at 1:18 AM, Christopher Faylor wrote:
>> Your idea of "constructive" is apparently flawed.
>
>This is an interesting comment, taking in that your response ignores the
>original question and focuses on off topic quibbling. I will pose your viewpoint
>and my question again

I was actually staying on-topic with what I wanted to address: namely
the thin ice that you are walking with your posting style.

>>And, frankly, I think it's a bad idea to keep a monolithic list of
>>packages available for people to download anyway.
>
>Do you have an explanation for this opinion, or citations as to why
>this is a bad idea, taking into account that major Linux distros are
>already using this very method?

Frankly, I don't have to justify my opinion.  I understand how the
package generation would have to work and don't see a major benefit to
introducing a package list file given the amount of work and the fact
that I would have to do the work.  I also don't want to spend my time
debating with you about why it is or isn't a good idea.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Christopher Faylor-8
In reply to this post by Steven Penny
On Wed, Jun 18, 2014 at 09:24:13AM -0500, Steven Penny wrote:
>On Wed, Jun 18, 2014 at 7:28 AM, Larry Hall (Cygwin) wrote:
>>If this was the question you wanted to ask originally, this is all you
>>needed to add to the thread.
>
>Hey Larry, thanks for your comments.  However, did you have anything
>constructive to add, like an answer to the question?

Although we are friends, there is no reason to expect Larry to be
able to explain my opinions.

Please move on.  This isn't going to happen.

Or take it to cygwin-talk.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: Package file lists archive

Steven Penny
In June I presented the idea of a "Package file lists archive" and was met with
this

> I understand how the package generation would have to work and don't see a
> major benefit to introducing a package list file given the amount of work and
> the fact that I would have to do the work.

http://cygwin.com/ml/cygwin/2014-06/msg00276.html

However Christopher has since left the project

> In the last several months I've told various key players in the Cygwin project
> that I was thinking about withdrawing from the project. Today is the day that
> this becomes official.

http://cygwin.com/ml/cygwin/2014-07/msg00392.html

I would like to revisit this issue. I would like to see if I can help in any way
to make this happen. My work on "apt-cyg" has been hindered by the issue with
"cygcheck --package-query"

http://github.com/transcode-open/apt-cyg/issues/21

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

12