setup.ini dependency graph?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

setup.ini dependency graph?

Charles Wilson-2
Does anybody have a script or a tool that can parse a setup.ini and
generate a dependency graph?  I'm using pmcyg to create a stripped-down
standalone installation CD and it's too big, so I'm trying to figure out
where the culprit is that's pulling in so much stuff...

--
Chuck

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Ryan Johnson-10
On 25/10/2013 12:12 PM, Charles Wilson wrote:
> Does anybody have a script or a tool that can parse a setup.ini and
> generate a dependency graph?  I'm using pmcyg to create a
> stripped-down standalone installation CD and it's too big, so I'm
> trying to figure out where the culprit is that's pulling in so much
> stuff...
I threw one together a while back when dealing with packages trying to
pull in latex: http://cygwin.com/ml/cygwin/2012-03/msg00242.html

It doesn't build a full graph, but if you name the offending package it
tells you who depends on it.

HTH,
Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

David Stacey
In reply to this post by Charles Wilson-2
On 25/10/13 17:12, Charles Wilson wrote:
> Does anybody have a script or a tool that can parse a setup.ini and
> generate a dependency graph?  I'm using pmcyg to create a
> stripped-down standalone installation CD and it's too big, so I'm
> trying to figure out where the culprit is that's pulling in so much
> stuff...

Oooo - this sounds like fun. I've knocked up some (very bad) perl that
gives you what you need. It generates a graphviz file that you can pipe
to 'dot' to generate the dependency graph in whatever format you
require. Put the perl script and your 'setup.ini' file in the same
directory and type:

     ./graph_setup_ini.pl | dot -Tpdf -osetup.pdf

I did this in Fedora 19; if you want this to work in Cygwin then you'll
need to install graphviz from CygwinPorts.

Your problem here is Big Data: Cygwin has 3041 packages, and any
dependency graph with this number of nodes is going to look a mess. It
also takes a while to process the data. Oh, and some PDF viewers won't
display the output file (LibreOffice Draw was the only tool I have that
managed it). However, if your starting point is a stripped down Cygwin
then you might be OK.

Hope this helps,

Dave.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

graph_setup_ini.pl (608 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

why do psutils & libtool packages need ~25+MB of latex (was Re: setup.ini dependency graph?)

L A Walsh
In reply to this post by Ryan Johnson-10
On 10/25/2013 5:29 PM, Ryan Johnson wrote:
> On 25/10/2013 12:12 PM, Charles Wilson wrote:
>> Does anybody have a script or a tool that can parse a setup.ini and
>> generate a dependency graph?  I'm using pmcyg to create a
>> stripped-down standalone installation CD and it's too big, so I'm
>> trying to figure out where the culprit is that's pulling in so much
>> stuff...
> I threw one together a while back when dealing with packages trying to
> pull in latex: http://cygwin.com/ml/cygwin/2012-03/msg00242.html
---

I keep trying to get rid of latex and it keeps coming back...

I think I figured out why... psutils?   libtool?

Both need latex?  I think I'm must really be missing some features in
psutils -- and doesn't libtool put together binary libraries?  Why would
it need latex in order to run?


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: why do psutils & libtool packages need ~25+MB of latex (was Re: setup.ini dependency graph?)

Ken Brown-6
On 10/27/2013 1:00 AM, Linda Walsh wrote:

> On 10/25/2013 5:29 PM, Ryan Johnson wrote:
>> On 25/10/2013 12:12 PM, Charles Wilson wrote:
>>> Does anybody have a script or a tool that can parse a setup.ini and
>>> generate a dependency graph?  I'm using pmcyg to create a
>>> stripped-down standalone installation CD and it's too big, so I'm
>>> trying to figure out where the culprit is that's pulling in so much
>>> stuff...
>> I threw one together a while back when dealing with packages trying to
>> pull in latex: http://cygwin.com/ml/cygwin/2012-03/msg00242.html
> ---
>
> I keep trying to get rid of latex and it keeps coming back...
>
> I think I figured out why... psutils?   libtool?

If you look at setup.ini, you'll see that psutils is an obsolete
package, which has been replaced by texlive-collection-fontutils.  You
must have once installed psutils for some reason.  If you don't need its
functionality, uninstall it and all the texlive packages that it brought
in.  You may have other obsolete packages installed that are also
bringing in texlive packages.  Just run setup and uncheck the "Hide
obsolete packages" box.

Ken

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Charles Wilson-2
In reply to this post by David Stacey
On 10/26/2013 5:40 AM, David Stacey wrote:
> On 25/10/13 17:12, Charles Wilson wrote:
> Oooo - this sounds like fun. I've knocked up some (very bad) perl that
> gives you what you need. It generates a graphviz file that you can pipe
> to 'dot' to generate the dependency graph in whatever format you
> require. Put the perl script and your 'setup.ini' file in the same
> directory and type:
>
>      ./graph_setup_ini.pl | dot -Tpdf -osetup.pdf

Thanks, that worked well.

> Your problem here is Big Data: Cygwin has 3041 packages, and any
> dependency graph with this number of nodes is going to look a mess. It
> also takes a while to process the data. Oh, and some PDF viewers won't
> display the output file (LibreOffice Draw was the only tool I have that
> managed it). However, if your starting point is a stripped down Cygwin
> then you might be OK.

Yeah; even for my stripped-down version, I need to pre-process the
setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
The ncurses DLLs are also a huge nexus.  (It's probably easier to
exclude those nodes by mucking with the perl, but...)

--
Chuck



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Ryan Johnson-10
On 30/10/2013 8:48 AM, Charles Wilson wrote:

> On 10/26/2013 5:40 AM, David Stacey wrote:
>> On 25/10/13 17:12, Charles Wilson wrote:
>> Oooo - this sounds like fun. I've knocked up some (very bad) perl that
>> gives you what you need. It generates a graphviz file that you can pipe
>> to 'dot' to generate the dependency graph in whatever format you
>> require. Put the perl script and your 'setup.ini' file in the same
>> directory and type:
>>
>>      ./graph_setup_ini.pl | dot -Tpdf -osetup.pdf
>
> Thanks, that worked well.
>
>> Your problem here is Big Data: Cygwin has 3041 packages, and any
>> dependency graph with this number of nodes is going to look a mess. It
>> also takes a while to process the data. Oh, and some PDF viewers won't
>> display the output file (LibreOffice Draw was the only tool I have that
>> managed it). However, if your starting point is a stripped down Cygwin
>> then you might be OK.
>
> Yeah; even for my stripped-down version, I need to pre-process the
> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
> The ncurses DLLs are also a huge nexus.  (It's probably easier to
> exclude those nodes by mucking with the perl, but...)
Quick question: do you have 1+ known-big-unwanted packages and need to
know who's pulling them in, or are you hoping to take some cut of the
graph that gets as many desirable packages as possible given the space
constraints? The graph-building script here is good for the latter, but
I had the impression you were doing the former; if so, my script might
get you to an answer faster by avoiding information overload.

Ryan


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Charles Wilson-2
On 10/30/2013 9:51 AM, Ryan Johnson wrote:

> On 30/10/2013 8:48 AM, Charles Wilson wrote:
>> Yeah; even for my stripped-down version, I need to pre-process the
>> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
>> The ncurses DLLs are also a huge nexus.  (It's probably easier to
>> exclude those nodes by mucking with the perl, but...)
> Quick question: do you have 1+ known-big-unwanted packages and need to
> know who's pulling them in, or are you hoping to take some cut of the
> graph that gets as many desirable packages as possible given the space
> constraints? The graph-building script here is good for the latter, but
> I had the impression you were doing the former; if so, my script might
> get you to an answer faster by avoiding information overload.

A combination of the two, actually.  I've used both David's script and
yours in concert.  In addition, I've modified David's script to color
the nodes based on origination, and to exclude or collapse 'Base' and/or
'required-by-Base' packages.

I've got a few cleanups, and then I'll share the result.  It's already
helped me generate a few re-packaging requests I plan to post over on
cygwin-apps...

--
Chuck



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Christopher Faylor-8
On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:

>On 10/30/2013 9:51 AM, Ryan Johnson wrote:
>> On 30/10/2013 8:48 AM, Charles Wilson wrote:
>>> Yeah; even for my stripped-down version, I need to pre-process the
>>> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
>>> The ncurses DLLs are also a huge nexus.  (It's probably easier to
>>> exclude those nodes by mucking with the perl, but...)
>> Quick question: do you have 1+ known-big-unwanted packages and need to
>> know who's pulling them in, or are you hoping to take some cut of the
>> graph that gets as many desirable packages as possible given the space
>> constraints? The graph-building script here is good for the latter, but
>> I had the impression you were doing the former; if so, my script might
>> get you to an answer faster by avoiding information overload.
>
>A combination of the two, actually.  I've used both David's script and
>yours in concert.  In addition, I've modified David's script to color
>the nodes based on origination, and to exclude or collapse 'Base' and/or
>'required-by-Base' packages.
>
>I've got a few cleanups, and then I'll share the result.  It's already
>helped me generate a few re-packaging requests I plan to post over on
>cygwin-apps...

Is this packagable?  It sounds pretty interesting.

Would it be crazy to generate this and make it available on the cygwin
web site?  Or would the dependency graph generation overload
sourceware.org?

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Ryan Johnson-10
On 04/11/2013 11:00 AM, Christopher Faylor wrote:

> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>> On 10/30/2013 9:51 AM, Ryan Johnson wrote:
>>> On 30/10/2013 8:48 AM, Charles Wilson wrote:
>>>> Yeah; even for my stripped-down version, I need to pre-process the
>>>> setup.ini and remove all mentions of cygwin, libstdc++6, libgcc1, etc.
>>>> The ncurses DLLs are also a huge nexus.  (It's probably easier to
>>>> exclude those nodes by mucking with the perl, but...)
>>> Quick question: do you have 1+ known-big-unwanted packages and need to
>>> know who's pulling them in, or are you hoping to take some cut of the
>>> graph that gets as many desirable packages as possible given the space
>>> constraints? The graph-building script here is good for the latter, but
>>> I had the impression you were doing the former; if so, my script might
>>> get you to an answer faster by avoiding information overload.
>> A combination of the two, actually.  I've used both David's script and
>> yours in concert.  In addition, I've modified David's script to color
>> the nodes based on origination, and to exclude or collapse 'Base' and/or
>> 'required-by-Base' packages.
>>
>> I've got a few cleanups, and then I'll share the result.  It's already
>> helped me generate a few re-packaging requests I plan to post over on
>> cygwin-apps...
> Is this packagable?  It sounds pretty interesting.
>
> Would it be crazy to generate this and make it available on the cygwin
> web site?  Or would the dependency graph generation overload
> sourceware.org?
Throwing graphviz at a full cygwin package dependency graph would make a
pretty effective DoS attack. Smaller graphs are cheaper, but still
consume non-trivial compute. Given how slowly the online regexp package
search goes, I'd hesitate to give users more ways to overload the server...

What about calling out to graphviz from setup.exe (if found in %PATH%),
as a replacement/supplement for the flat list of dependencies it
currently reports? That would put all processing on the client, and
limit the "big data" problem, as the graph only contains packages a user
is currently trying to install.

Alternatively, cygcheck could gain a new -g option to dump subsets of a
dependency graph, extracted from setup.ini, in some appropriate format
like .dot:

`cygcheck -D -g python' would emit the graph of packages that python
depends on
`cygcheck -R -g texlive,xorg-server' would build a braph of packages
that pull in either of texlive or xorg-server (reverse dependencies)
-D -R -g would follow dependencies in both directions, and -g would be
shorthand for -D -g; probably -D or -R by itself implies -g.

The actual work could be done by calling out to a scripting language
that ships with cygwin. Awk would probably be able to, and perl
certainly could.

/daydreaming

Ryan



--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Charles Wilson-2
In reply to this post by Christopher Faylor-8
On 11/4/2013 11:00 AM, Christopher Faylor wrote:
> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>> I've got a few cleanups, and then I'll share the result.  It's already
>> helped me generate a few re-packaging requests I plan to post over on
>> cygwin-apps...
>
> Is this packagable?  It sounds pretty interesting.

Probably. I could put it in cygutils, or standalone (like the new
cygcheck-leaves package is a standalone utility).

> Would it be crazy to generate this and make it available on the cygwin
> web site?  Or would the dependency graph generation overload
> sourceware.org?

The basic processing to generate a .dot file is pretty simple really;
just string comparisons and hash manipulations. But as Ryan already
pointed out, generating the actual graph in whatever format, is probably
compute intensive.

See attached...

The svg files in the tarball correspond to each of the --mode options.
The tool was invoked as:
    setup-ini-graph -i setup.ini -o $dotfile --mode=$mode netpbm
followed by (on linux)
    dot -Tsvg $dotfile > $svgfile

Help output:

> Usage:
>     setup_ini_graph [options] [packagename [packagename ...]]
>
>      Options:
>        --help|-?                brief help message
>        --man                    full documentation
>        --version                show version information
>        --mode STRING            specify the output mode. Possible values:
>                                   strip-base-and-requirements <default>
>                                   strip-base-only
>                                   collapse-base-and-requirements
>                                   collapse-base-only
>                                   show-only-base-and-requirements
>                                   show-base-requirements-only
>                                   show-all
>        --color STRING           set color of specified packages <lightblue>
>        --color-req STRING       set color of package requirements <palegreen>
>        --color-base STRING      set color of Base packages <plum>
>        --color-base-req STRING  set color of Base requirements <pink>
>        --input|-i FILE          specify input setup.ini file <STDIN>
>        --output|-o FILE         specify output .dot file <STDOUT>
>        --verbose                turn on debugging output
>
>     If packagenames are specified, then only the specified packages and
>     their dependencies are analyzed. If no packages are specified, then all
>     packages in the setup.ini are analyzed.
>
> Options:
>     --help  Print a brief help message and exits.
>
>     --man   Prints the manual page and exits.
>
>     --version
>             Prints version information and exits.
>
>     --mode STRING
>             Specify the output mode. The STRING may be one of the following
>             options:
>
>             strip-base-and-requirements
>                 All packages in the Base category, and packages on which
>                 they depend, are completely removed from the output. This is
>                 the default mode.
>
>             strip-base-only
>                 All packages in the Base category are completely removed
>                 from the output.
>
>             collapse-base-and-requirements
>                 All packages in the Base category, and packages on which
>                 they depend, are replaced by a single node 'Base'.
>
>             collapse-base-only
>                 All packages in the Base category are replaced by a single
>                 node 'Base'.
>
>             show-only-base-and-requirements
>                 Ignore all packages that are not in the Base category, or
>                 are not required by packages in the Base category.
>
>             show-base-requirements-only
>                 As above, but the packages which are members of the Base
>                 category are replaced by a single node 'Base'.
>
>             show-all
>                 Chart the entire dependency tree.
>
>     --color STRING
>             Sets the color to be used for nodes that represent the packages
>             listed on the command line (or all packages extracted from
>             setup.ini that do not fall into one of the other categories
>             below). May be an RGB color name such as palegoldenrod, or an
>             RGB triple of the form #DA70D6. Note that in the latter case,
>             the # must be escaped as \#DA70D6 or '#DA70D6'. The default
>             value is lightblue (#ADD8E6).
>
>     --color-req STRING
>             Sets the color to be used for nodes that represent packages that
>             are required by the ones listed on the command line, but which
>             do not fall into one of the other categories below. The default
>             value is palegreen (#98FB98). See --color for more information.
>
>     --color-base STRING
>             Sets the color to be used for nodes that represent packages that
>             are in the Base category. It is also used to color the 'Base'
>             node in the collapse --mode options. The default value is plum
>             (#DDA0DD). See --color for more information.
>
>     --color-base-req STRING
>             Sets the color to be used for nodes that represent packages
>             which are required by packages in the Base category, but are not
>             in the Base category themselves. The default value is pink
>             (#FFC0CB).
>
>     --input FILENAME
>             Use the specified input file. Defaults to <STDIN>.
>
>     --output FILENAME
>             Use the specified output file. Defaults to <STDOUT>.
>
>     --verbose
>             Turn on verbose output (to STDERR).
--
Chuck


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

netpbm-dependency-chart-svg.tar.xz (100K) Download Attachment
setup-ini-graph (22K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

David Stacey
On 04/11/13 17:30, Charles Wilson wrote:

> On 11/4/2013 11:00 AM, Christopher Faylor wrote:
>> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>>> I've got a few cleanups, and then I'll share the result.  It's already
>>> helped me generate a few re-packaging requests I plan to post over on
>>> cygwin-apps...
>>
>> Is this packagable?  It sounds pretty interesting.
>
> Probably. I could put it in cygutils, or standalone (like the new
> cygcheck-leaves package is a standalone utility).
>
>> Would it be crazy to generate this and make it available on the cygwin
>> web site?  Or would the dependency graph generation overload
>> sourceware.org?
>
> The basic processing to generate a .dot file is pretty simple really;
> just string comparisons and hash manipulations. But as Ryan already
> pointed out, generating the actual graph in whatever format, is
> probably compute intensive.
>
> See attached...

My goodness - you have been busy! You've taken some hacky kludgy perl
that I threw together in five minutes flat and turned it into a proper
programme. Tip of the hat to you, sir...

Dave.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: setup.ini dependency graph?

Achim Gratz
In reply to this post by Charles Wilson-2
Charles Wilson writes:

> On 11/4/2013 11:00 AM, Christopher Faylor wrote:
>> On Mon, Nov 04, 2013 at 09:32:10AM -0500, Charles Wilson wrote:
>>> I've got a few cleanups, and then I'll share the result.  It's already
>>> helped me generate a few re-packaging requests I plan to post over on
>>> cygwin-apps...
>>
>> Is this packagable?  It sounds pretty interesting.
>
> Probably. I could put it in cygutils, or standalone (like the new
> cygcheck-leaves package is a standalone utility).

Interesting, indeed.  I've been working on another script that creates
install directories for offline installation (using multiple package
repositories and a config file to control package selection) that I hope
to get polished for public release.  It doesn't explicitly produce a
dependency graph (I do have some optional debug output that could be
used for this), but it wouldn't be difficult to bolt on (I already have
a hash for the forward dependencies).

>> Would it be crazy to generate this and make it available on the cygwin
>> web site?  Or would the dependency graph generation overload
>> sourceware.org?
>
> The basic processing to generate a .dot file is pretty simple really;
> just string comparisons and hash manipulations. But as Ryan already
> pointed out, generating the actual graph in whatever format, is
> probably compute intensive.

It might be easier to keep things a bit more simple: the most common
question is to get the direct dependencies of a package and the direct
dependents (who is requiring this package).  This graph is almost
trivial and could be followed interactively up or down the dependency
chain with little effort.  It wouldn't look as pretty as a full
dependency graph on an 8k display, but it'd be very useful.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple