Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Cygwin list mailing list
I have a folder with a lot of native Windows symbolic links. I want to
copy this folder.

I cannot rsync or cp this folder due to Cygwin being unable to create
symbolic links without also wanting to verify the link target. This
can be demonstrated:

$ ln -s a b
ln: failed to create symbolic link 'b': No such file or directory

If I create a test directory folder_a/ and folder_b/. Inside I will
"touch a" and "ln -s a b".

I cannot rsync this folder:

$ rsync -a folder_a/ folder_b/
rsync: symlink "folder_a/b" -> "a" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1306) [sender=3.2.0dev]

Using "cp -a folder_a/* folder_b/" in this test case DOES work but
this is simply because files were returned in the correct order and
the link could be created.

This can be demonstrated where this works fine:

$ cp -a folder_a/a folder_a/b folder_b/

But this does not:

$ cp -a folder_a/b folder_a/b folder_b/
cp: cannot create symbolic link 'folder_b/b': No such file or directory
cp: warning: source file 'folder_a/b' specified more than once

The order in which files are returned while listing them in a
directory and necessitating their pre-existence while performing a
deep copy is impossible. It's also very normal for symbolic links to
exist which may or may not point to a valid target depending on the
observing path.

Windows does NOT require a link to be valid before creation. This can
be demonstrated with mklink:

C:\mklink b a
symbolic link created for b <<===>> a
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Brian Inglis
On 2021-01-02 21:16, Matt D. via Cygwin wrote:

> I have a folder with a lot of native Windows symbolic links. I want to
> copy this folder.
>
> I cannot rsync or cp this folder due to Cygwin being unable to create
> symbolic links without also wanting to verify the link target. This
> can be demonstrated:
>
> $ ln -s a b
> ln: failed to create symbolic link 'b': No such file or directory
>
> If I create a test directory folder_a/ and folder_b/. Inside I will
> "touch a" and "ln -s a b".
>
> I cannot rsync this folder:
>
> $ rsync -a folder_a/ folder_b/
> rsync: symlink "folder_a/b" -> "a" failed: No such file or directory (2)
> rsync error: some files/attrs were not transferred (see previous
> errors) (code 23) at main.c(1306) [sender=3.2.0dev]
>
> Using "cp -a folder_a/* folder_b/" in this test case DOES work but
> this is simply because files were returned in the correct order and
> the link could be created.
>
> This can be demonstrated where this works fine:
>
> $ cp -a folder_a/a folder_a/b folder_b/
>
> But this does not:
>
> $ cp -a folder_a/b folder_a/b folder_b/
> cp: cannot create symbolic link 'folder_b/b': No such file or directory
> cp: warning: source file 'folder_a/b' specified more than once
>
> The order in which files are returned while listing them in a
> directory and necessitating their pre-existence while performing a
> deep copy is impossible. It's also very normal for symbolic links to
> exist which may or may not point to a valid target depending on the
> observing path.
>
> Windows does NOT require a link to be valid before creation. This can
> be demonstrated with mklink:
>
> C:\mklink b a
> symbolic link created for b <<===>> a

Depending on exactly what you have and what you want to do try:

        $ cp -av
        $ cp -dv
        $ cp -Pv
or
        > robocopy /sl

--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada

This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Cygwin list mailing list
Did you try any of my test cases? This can't and doesn't work for the
reasons I outlined in my previous message:

$ cp -av folder_a/a folder_b/
'folder_a/a' -> 'folder_b/a'
cp: cannot create symbolic link 'folder_b/a': No such file or directory

$ cp -dv folder_a/a folder_b/
'folder_a/a' -> 'folder_b/a'
cp: cannot create symbolic link 'folder_b/a': No such file or directory

$ cp -Pv folder_a/a folder_b/
'folder_a/a' -> 'folder_b/a'
cp: cannot create symbolic link 'folder_b/a': No such file or directory

On Sun, Jan 3, 2021 at 12:00 AM Brian Inglis
<[hidden email]> wrote:

>
> On 2021-01-02 21:16, Matt D. via Cygwin wrote:
> > I have a folder with a lot of native Windows symbolic links. I want to
> > copy this folder.
> >
> > I cannot rsync or cp this folder due to Cygwin being unable to create
> > symbolic links without also wanting to verify the link target. This
> > can be demonstrated:
> >
> > $ ln -s a b
> > ln: failed to create symbolic link 'b': No such file or directory
> >
> > If I create a test directory folder_a/ and folder_b/. Inside I will
> > "touch a" and "ln -s a b".
> >
> > I cannot rsync this folder:
> >
> > $ rsync -a folder_a/ folder_b/
> > rsync: symlink "folder_a/b" -> "a" failed: No such file or directory (2)
> > rsync error: some files/attrs were not transferred (see previous
> > errors) (code 23) at main.c(1306) [sender=3.2.0dev]
> >
> > Using "cp -a folder_a/* folder_b/" in this test case DOES work but
> > this is simply because files were returned in the correct order and
> > the link could be created.
> >
> > This can be demonstrated where this works fine:
> >
> > $ cp -a folder_a/a folder_a/b folder_b/
> >
> > But this does not:
> >
> > $ cp -a folder_a/b folder_a/b folder_b/
> > cp: cannot create symbolic link 'folder_b/b': No such file or directory
> > cp: warning: source file 'folder_a/b' specified more than once
> >
> > The order in which files are returned while listing them in a
> > directory and necessitating their pre-existence while performing a
> > deep copy is impossible. It's also very normal for symbolic links to
> > exist which may or may not point to a valid target depending on the
> > observing path.
> >
> > Windows does NOT require a link to be valid before creation. This can
> > be demonstrated with mklink:
> >
> > C:\mklink b a
> > symbolic link created for b <<===>> a
>
> Depending on exactly what you have and what you want to do try:
>
>         $ cp -av
>         $ cp -dv
>         $ cp -Pv
> or
>         > robocopy /sl
>
> --
> Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
>
> This email may be disturbing to some readers as it contains
> too much technical detail. Reader discretion is advised.
> [Data in binary units and prefixes, physical quantities in SI.]
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Eliot Moss
On 1/4/2021 5:36 AM, Matt D. via Cygwin wrote:
 > Did you try any of my test cases? This can't and doesn't work for the
 > reasons I outlined in my previous message:
 >
 > $ cp -av folder_a/a folder_b/
 > 'folder_a/a' -> 'folder_b/a'
 > cp: cannot create symbolic link 'folder_b/a': No such file or directory
 >
 > $ cp -dv folder_a/a folder_b/
 > 'folder_a/a' -> 'folder_b/a'
 > cp: cannot create symbolic link 'folder_b/a': No such file or directory
 >
 > $ cp -Pv folder_a/a folder_b/
 > 'folder_a/a' -> 'folder_b/a'
 > cp: cannot create symbolic link 'folder_b/a': No such file or directory

So did you mkdir folder_b first?  I don't think cp will create it for you.  I
tried the commands above with folder_b not existing and got the behavior you
indicated, but when I created folder_b first, all three cp commands worked.
This overall behavior does not surprise me.

On the other hand, if I have folder_b non-existing and do (e.g.)

cp -rav folder_a folder_b

then it _does_ create folder_b, and also copies the links.

HTH - Eliot Moss
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Cygwin list mailing list
I think there is a misunderstanding with how to set up your
environment to reproduce my test cases. I did state in the subject
"native symbolic links" but I can see that this can be misinterpreted
and I should have clarified.

I am using symbolic links native to Windows. My CYGWIN environment
variable has been set to "winsymlinks:nativestrict" and my account has
permission to make symbolic links. This is an issue specifically with
Cygwin; I have no problems making links at the windows command line.
Cygwin also does not have a problem making symbolic links-- if the
target already exists. The issue is that I cannot create native
symbolic links with Cygwin for targets that DON'T exist.

The normal behavior for both Windows and Linux is to create the
symbolic link whether the target exists or not. I don't know why
Cygwin fails to do this only for native Windows symbolic links. It
does not have a problem creating links to any target with the default
Cygwin (non-Windows) symbolic links.

On Mon, Jan 4, 2021 at 7:30 AM Eliot Moss <[hidden email]> wrote:

>
> On 1/4/2021 5:36 AM, Matt D. via Cygwin wrote:
>  > Did you try any of my test cases? This can't and doesn't work for the
>  > reasons I outlined in my previous message:
>  >
>  > $ cp -av folder_a/a folder_b/
>  > 'folder_a/a' -> 'folder_b/a'
>  > cp: cannot create symbolic link 'folder_b/a': No such file or directory
>  >
>  > $ cp -dv folder_a/a folder_b/
>  > 'folder_a/a' -> 'folder_b/a'
>  > cp: cannot create symbolic link 'folder_b/a': No such file or directory
>  >
>  > $ cp -Pv folder_a/a folder_b/
>  > 'folder_a/a' -> 'folder_b/a'
>  > cp: cannot create symbolic link 'folder_b/a': No such file or directory
>
> So did you mkdir folder_b first?  I don't think cp will create it for you.  I
> tried the commands above with folder_b not existing and got the behavior you
> indicated, but when I created folder_b first, all three cp commands worked.
> This overall behavior does not surprise me.
>
> On the other hand, if I have folder_b non-existing and do (e.g.)
>
> cp -rav folder_a folder_b
>
> then it _does_ create folder_b, and also copies the links.
>
> HTH - Eliot Moss
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Cygwin list mailing list
On 1/4/2021 10:27 AM, Matt D. via Cygwin ([hidden email]) wrote:

> I am using symbolic links native to Windows. My CYGWIN environment
> variable has been set to "winsymlinks:nativestrict" and my account has
> permission to make symbolic links. This is an issue specifically with
> Cygwin; I have no problems making links at the windows command line.
> Cygwin also does not have a problem making symbolic links-- if the
> target already exists. The issue is that I cannot create native
> symbolic links with Cygwin for targets that DON'T exist.
>
> The normal behavior for both Windows and Linux is to create the
> symbolic link whether the target exists or not. I don't know why
> Cygwin fails to do this only for native Windows symbolic links. It
> does not have a problem creating links to any target with the default
> Cygwin (non-Windows) symbolic links.
Windows native symlinks encode the object type of the target and the
encoded type must match that of the target or the link will not work
when the target exists.

A UNIX symlink does not encode any details of the target.

Cygwin doesn't know what type of native symlink to create if the
target does not exist.

I hope this knowledge helps.

Jeffrey Altman






--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Eliot Moss
In reply to this post by Cygwin list mailing list
On 1/4/2021 10:27 AM, Matt D. via Cygwin wrote:
 > I think there is a misunderstanding with how to set up your
 > environment to reproduce my test cases. I did state in the subject
 > "native symbolic links" but I can see that this can be misinterpreted
 > and I should have clarified.
 >
 > I am using symbolic links native to Windows. My CYGWIN environment
 > variable has been set to "winsymlinks:nativestrict" and my account has
 > permission to make symbolic links. This is an issue specifically with
 > Cygwin; I have no problems making links at the windows command line.
 > Cygwin also does not have a problem making symbolic links-- if the
 > target already exists. The issue is that I cannot create native
 > symbolic links with Cygwin for targets that DON'T exist.
 >
 > The normal behavior for both Windows and Linux is to create the
 > symbolic link whether the target exists or not. I don't know why
 > Cygwin fails to do this only for native Windows symbolic links. It
 > does not have a problem creating links to any target with the default
 > Cygwin (non-Windows) symbolic links.

Ok, I see the behavior now that you are talking about.  You can get it with ln
without any need for cp.  With winsymlinks:nativestrict, if I do:

ln -s foo bar

and foo does not exist, it refuses to create the link.  As you found, it also
refuses to cp it.  However, I _was_ able to mv it.

Regards - Eliot
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Achim Gratz
In reply to this post by Cygwin list mailing list
Matt D. via Cygwin writes:
> The normal behavior for both Windows and Linux is to create the
> symbolic link whether the target exists or not.

That's not the case on Windows or rather it has restrictions that can
trip you up.  On Windows you must specify if the symbolic link points to
a directory or a file in order to create a symbolic link to a
non-existing file.  If that information is not available, it is
obviously safer to not create the symbolic link.


Regards,
Achim.
--
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Cygwin list mailing list
In reply to this post by Cygwin list mailing list
Operations like cp and rsync etc. should still work. The type of
symbolic link to be created can be read from the file attributes of
the one being copied.


On Mon, Jan 4, 2021 at 10:46 AM Jeffrey Altman via Cygwin
<[hidden email]> wrote:

>
> On 1/4/2021 10:27 AM, Matt D. via Cygwin ([hidden email]) wrote:
> > I am using symbolic links native to Windows. My CYGWIN environment
> > variable has been set to "winsymlinks:nativestrict" and my account has
> > permission to make symbolic links. This is an issue specifically with
> > Cygwin; I have no problems making links at the windows command line.
> > Cygwin also does not have a problem making symbolic links-- if the
> > target already exists. The issue is that I cannot create native
> > symbolic links with Cygwin for targets that DON'T exist.
> >
> > The normal behavior for both Windows and Linux is to create the
> > symbolic link whether the target exists or not. I don't know why
> > Cygwin fails to do this only for native Windows symbolic links. It
> > does not have a problem creating links to any target with the default
> > Cygwin (non-Windows) symbolic links.
>
> Windows native symlinks encode the object type of the target and the
> encoded type must match that of the target or the link will not work
> when the target exists.
>
> A UNIX symlink does not encode any details of the target.
>
> Cygwin doesn't know what type of native symlink to create if the
> target does not exist.
>
> I hope this knowledge helps.
>
> Jeffrey Altman
>
>
>
>
>
> --
> Problem reports:      https://cygwin.com/problems.html
> FAQ:                  https://cygwin.com/faq/
> Documentation:        https://cygwin.com/docs.html
> Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Andrey Repin
Greetings, Matt D.!

Please no top posting in this list.

> On Mon, Jan 4, 2021 at 10:46 AM Jeffrey Altman via Cygwin wrote:
>>
>> On 1/4/2021 10:27 AM, Matt D. via Cygwin wrote:
>> > I am using symbolic links native to Windows. My CYGWIN environment
>> > variable has been set to "winsymlinks:nativestrict" and my account has
>> > permission to make symbolic links. This is an issue specifically with
>> > Cygwin; I have no problems making links at the windows command line.
>> > Cygwin also does not have a problem making symbolic links-- if the
>> > target already exists. The issue is that I cannot create native
>> > symbolic links with Cygwin for targets that DON'T exist.
>> >
>> > The normal behavior for both Windows and Linux is to create the
>> > symbolic link whether the target exists or not. I don't know why
>> > Cygwin fails to do this only for native Windows symbolic links. It
>> > does not have a problem creating links to any target with the default
>> > Cygwin (non-Windows) symbolic links.
>>
>> Windows native symlinks encode the object type of the target and the
>> encoded type must match that of the target or the link will not work
>> when the target exists.
>>
>> A UNIX symlink does not encode any details of the target.
>>
>> Cygwin doesn't know what type of native symlink to create if the
>> target does not exist.

> Operations like cp and rsync etc. should still work. The type of
> symbolic link to be created can be read from the file attributes of
> the one being copied.

Again, the properties are read BY SYSTEM at the moment the link is created.
In other words, Windows is unable to create a link to nonexistent
(unreachable) target (not directly, at least).


--
With best regards,
Andrey Repin
Wednesday, January 13, 2021 1:14:52

Sorry for my terrible english...

--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple
Reply | Threaded
Open this post in threaded view
|

Re: Native symbolic link behavior is broken and makes backups using Cygwin command line tools impossible

Cygwin list mailing list
On Tue, Jan 12, 2021 at 5:20 PM Andrey Repin <[hidden email]> wrote:
>
> Again, the properties are read BY SYSTEM at the moment the link is created.
> In other words, Windows is unable to create a link to nonexistent
> (unreachable) target (not directly, at least).

This is false. Links can be created arbitrarily both at the Windows
command line using MKLINK and programmatically with the
CreateSymbolicLink function. The information about what type of link
to make during a copy operation can be read from the link being
copied. The target does not have to exist for a link to be created.
--
Problem reports:      https://cygwin.com/problems.html
FAQ:                  https://cygwin.com/faq/
Documentation:        https://cygwin.com/docs.html
Unsubscribe info:     https://cygwin.com/ml/#unsubscribe-simple