RE: non-persistent storage?

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

RE: non-persistent storage?

Cygwin list mailing list
Eliot Moss sent the following at Thursday, December 12, 2019 2:19 pm

>On 12/12/2019 7:00 AM, Ulli Horlacher wrote:
>
>> I need to store some data (a few kB) non-persistent.
>> On a real UNIX I would use /var/run, because after a shutdown all its
>> content is lost.
>> But on cygwin /var/run is stored on disk.
>>
>> I cannot use an environment variable, because different processes need
>> to read/write the data.
>>
>> /proc is non-persistent (in respect to a reboot), but It is not a
>> generic storage place.
>>
>> What can I use with cygwin instead?
>>
>> Installing third party software is not an option, it must work with a
>> standard Windows (and cygwin).
>
>I would think of temp directories, such as /tmp. They can be cleaned out
>at will on restart, no?

Bash runs the file ~/.bash_logout when it exits.  One could use that to
clear a temp file out of /tmp.

If the temp file was created by mktemp and the name saved in an
environmental variable, each bash shell could have its own file with not
risk that an instance of bash would erase another instance's file.

That assumes that this is consistent with your need for different
processes to read/write data.

Good luck,

- Barry
  Disclaimer: Statements made herein are not made on behalf of NIAID.

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Ulli Horlacher
On Thu 2019-12-12 (21:59), Buchbinder, Barry (NIH/NIAID) [E] via cygwin wrote:

> If the temp file was created by mktemp and the name saved in an
> environmental variable, each bash shell could have its own file with not
> risk that an instance of bash would erase another instance's file.

I need the opposite: all processes must read and write the same data!
And the data must be gone when system shuts down or even if there is a
power failure.

--
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK        
Universitaet Stuttgart         E-Mail: [hidden email]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[hidden email]>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Erik Soderquist-2
On Thu, Dec 12, 2019 at 5:08 PM Ulli Horlacher wrote:

>
> On Thu 2019-12-12 (21:59), Buchbinder, Barry (NIH/NIAID) [E] via cygwin wrote:
>
> > If the temp file was created by mktemp and the name saved in an
> > environmental variable, each bash shell could have its own file with not
> > risk that an instance of bash would erase another instance's file.
>
> I need the opposite: all processes must read and write the same data!
> And the data must be gone when system shuts down or even if there is a
> power failure.

This is very hacky, but I believe you can get the effect you want by
having an admin process auto-start at host startup, identify itself,
and then set parts of its own /proc/ process ID tree as world
read/write.  I think this will give you the "destroyed even at power
failure" impermanence you are l,booking for.  I know it is not in the
normal layouts (like /var/run/ would be), but we are working around
limitation s imposed by Windows.

-- Erik

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Erik Soderquist-2
On Thu, Dec 12, 2019 at 5:45 PM Erik Soderquist wrote:
> This is very hacky, but I believe you can get the effect you want by
> having an admin process auto-start at host startup, identify itself,
> and then set parts of its own /proc/ process ID tree as world
> read/write.  I think this will give you the "destroyed even at power
> failure" impermanence you are l,booking for.  I know it is not in the
> normal layouts (like /var/run/ would be), but we are working around
> limitation s imposed by Windows.

To "move"to /var/run/, the process could also build symlinks to the
location(s) in /proc/ so the other programs/users have a stable
/var/run/ location and only the auto-started process needs to know or
care about the locations in /proc/

-- Erik

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Jürgen Wagner
In reply to this post by Ulli Horlacher
I can see two options:
1. You create a RAM disk and the contents will obviously be gone when
the system reboots or crashes.
2. You encrypt the data with a ephemeral key that will be lost on reboot
(e.g., kept in shared memory). When the system comes up and finds itself
unable to read the data, it will erase it.

That doesn't really have much to do with Cygwin or Windows - you can do
that on any system.

On 12.12.2019 23:07, Ulli Horlacher wrote:
> On Thu 2019-12-12 (21:59), Buchbinder, Barry (NIH/NIAID) [E] via cygwin wrote:
>
>> If the temp file was created by mktemp and the name saved in an
>> environmental variable, each bash shell could have its own file with not
>> risk that an instance of bash would erase another instance's file.
> I need the opposite: all processes must read and write the same data!
> And the data must be gone when system shuts down or even if there is a
> power failure.
>



smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Ulli Horlacher
In reply to this post by Erik Soderquist-2
On Thu 2019-12-12 (17:45), Erik Soderquist wrote:

> This is very hacky, but I believe you can get the effect you want by
> having an admin process auto-start at host startup, identify itself,
> and then set parts of its own /proc/ process ID tree as world
> read/write.

It is not possible to write any data to /proc using standard IO read/write
like on a tmpfs file system.


--
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK        
Universitaet Stuttgart         E-Mail: [hidden email]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<CACoZoo1crFaE0ZHBVZaps+F93MyX1c4MN5-5awJ=[hidden email]>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Ulli Horlacher
In reply to this post by Jürgen Wagner
On Fri 2019-12-13 (00:03), Jürgen Wagner wrote:

> 1. You create a RAM disk

How do I do it with cygwin?


> 2. You encrypt the data with a ephemeral key

This is exactly what I want to share between processes!


--
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK        
Universitaet Stuttgart         E-Mail: [hidden email]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[hidden email]>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Jürgen Wagner
Ok, we're getting closer.

- Run cygserver-config as an administrator.
- Do "net start cygserver"

That will provide shared memory capabilities. Then try this:

# shmtool w myFunnyC8380ufsKey
Shared memory segment exists - opening as client
Done...

# shmtool r
Shared memory segment exists - opening as client
segptr: myFunnyC8380ufsKey

#

On reboot of the host Windows, the shared memory segment will not be
persisted.

Cheers,
--j.




On 13.12.2019 00:21, Ulli Horlacher wrote:

> On Fri 2019-12-13 (00:03), Jürgen Wagner wrote:
>
>> 1. You create a RAM disk
> How do I do it with cygwin?
>
>
>> 2. You encrypt the data with a ephemeral key
> This is exactly what I want to share between processes!
>
>


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Jürgen Wagner
In reply to this post by Ulli Horlacher
And regarding the tmpfs (RAM disk) , you can use /dev/shm.

# date > /dev/shm/key
# cat /dev/shm/key
Fri Dec 13 07:26:03 CET 2019
#

It is not advisable to store larger amounts of data there, but an
encryption key should be fine.

Cheers,
--j.

On 13.12.2019 00:21, Ulli Horlacher wrote:

> On Fri 2019-12-13 (00:03), Jürgen Wagner wrote:
>
>> 1. You create a RAM disk
> How do I do it with cygwin?
>
>
>> 2. You encrypt the data with a ephemeral key
> This is exactly what I want to share between processes!
>
>


smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Ulli Horlacher
On Fri 2019-12-13 (07:27), Jürgen Wagner wrote:
> And regarding the tmpfs (RAM disk) , you can use /dev/shm.
>
> # date > /dev/shm/key
> # cat /dev/shm/key
> Fri Dec 13 07:26:03 CET 2019

/dev/shm ist not a tmpfs on cygwin:

/dev/shm: df -TH /dev/shm
Filesystem     Type  Size  Used Avail Use% Mounted on
C:/cygwin64    ntfs   34G   25G  9.1G  74% /

When I write something into this direcory and reboot, it is still there.


--
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK        
Universitaet Stuttgart         E-Mail: [hidden email]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[hidden email]>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Ulli Horlacher
In reply to this post by Jürgen Wagner
On Fri 2019-12-13 (00:51), Jürgen Wagner wrote:

> - Run cygserver-config as an administrator.

~: cygserver-config
Generating /etc/cygserver.conf file

Warning: The following function requires administrator privileges!

Do you want to install cygserver as service?
(Say "no" if it's already installed as service) (yes/no) yes
/usr/bin/cygserver-config: line 181: cygrunsrv: command not found


> # shmtool w myFunnyC8380ufsKey

shmtool is also missing.
Which package have I to install?


--
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK        
Universitaet Stuttgart         E-Mail: [hidden email]
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<[hidden email]>

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

marco atzeri-4
Am 13.12.2019 um 11:06 schrieb Ulli Horlacher:

> On Fri 2019-12-13 (00:51), Jürgen Wagner wrote:
>
>> - Run cygserver-config as an administrator.
>
> ~: cygserver-config
> Generating /etc/cygserver.conf file
>
> Warning: The following function requires administrator privileges!
>
> Do you want to install cygserver as service?
> (Say "no" if it's already installed as service) (yes/no) yes
> /usr/bin/cygserver-config: line 181: cygrunsrv: command not found
>
>
>> # shmtool w myFunnyC8380ufsKey
>
> shmtool is also missing.
> Which package have I to install?
>
>

$ cygcheck -p shmtool
Found 6 matches for shmtool
...
cygutils-extra-1.4.16-2 - cygutils-extra: A collection of simple
utilities (other tools)

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

RE: non-persistent storage?

Cygwin list mailing list
In reply to this post by Erik Soderquist-2
>On Thu, Dec 12, 2019 at 5:08 PM Ulli Horlacher wrote:
>
>>Erik Soderquist sent the following at Thursday, December 12, 2019 5:45 pm
>>
>> On Thu 2019-12-12 (21:59), Buchbinder, Barry (NIH/NIAID) [E] via cygwin wrote:
>>>
>>> If the temp file was created by mktemp and the name saved in an
>>> environmental variable, each bash shell could have its own file with
>>> not risk that an instance of bash would erase another instance's file.
>>
>> I need the opposite: all processes must read and write the same data!
>> And the data must be gone when system shuts down or even if there is a
>> power failure.
>
> This is very hacky, but I believe you can get the effect you want by
> having an admin process auto-start at host startup, identify itself, and
> then set parts of its own /proc/ process ID tree as world read/write.
> I think this will give you the "destroyed even at power failure"
> impermanence you are looking for. I know it is not in the normal
> layouts (like /var/run/ would be), but we are working around limitations
> imposed by Windows.

One could put a script, batch file, or a link thereto in one's startup
folder that will run on Windows' boot.  The script/batch file would be
written to clean out the tmp files.  Google << windows startup folder >>.
If all processes, regardless of whether they start with the same
shell/subshell, will share the same temp file, one could create the file
upon boot using the same script.

Also, one could make the script that creates the temp file (e.g.,
.bashrc) check the file or an environmental variable so that a subshell
doesn't create a new temp file, if that's how you want it.

- Barry
  Disclaimer: Statements made herein are not made on behalf of NIAID.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Mark Geisert
In reply to this post by Ulli Horlacher
Ulli Horlacher wrote:

> On Fri 2019-12-13 (00:51), Jürgen Wagner wrote:
>
>> - Run cygserver-config as an administrator.
>
> ~: cygserver-config
> Generating /etc/cygserver.conf file
>
> Warning: The following function requires administrator privileges!
>
> Do you want to install cygserver as service?
> (Say "no" if it's already installed as service) (yes/no) yes
> /usr/bin/cygserver-config: line 181: cygrunsrv: command not found

In addition to Marco's reply for shmtool, you'll need the 'cygrunsrv' package.

..mark

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Erik Soderquist-2
In reply to this post by Cygwin list mailing list
On Fri, Dec 13, 2019 at 11:20 AM Buchbinder, Barry (NIH/NIAID) [E] via
cygwin <[hidden email]> wrote:
> One could put a script, batch file, or a link thereto in one's startup
> folder that will run on Windows' boot.

The OP's issue is that the data must be destroyed instantly even in
the event of a power failure.

Use case scenario:  thief breaks in and steals the host, just yanking
power cords/etc out, and then pulls the drive and mounts it on another
system to recover this data; it is sensitive enough that it must be
destroyed immediately by the power failure.

Unfortunately, nothing in Windows is designed for that kind of
security, and Cygwin has to work around a lot of Windows design flaws
to function in general.

I've test all of the suggestions I've seen so far with the exception
of the cygserver and shared memory, and all of the ones I've tested
failed the power failure scenario.  I think if the cygserver/shared
memory suggestion works, it will be the only available option to
prevent the data from default existing on the disc, and due to another
of Window's design flaws, may still be written into the page file,
though it would be much harder to cleanly extract from the page file
than from a normal filesystem file.

-- Erik

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Steven Penny
On Fri, 13 Dec 2019 17:57:23 -0500, Erik Soderquist wrote:
> I've test all of the suggestions I've seen so far with the exception
> of the cygserver and shared memory, and all of the ones I've tested
> failed the power failure scenario.

/dev/clipboard?


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Corinna Vinschen-2
In reply to this post by Ulli Horlacher
On Dec 13 10:34, Ulli Horlacher wrote:

> On Fri 2019-12-13 (07:27), Jürgen Wagner wrote:
> > And regarding the tmpfs (RAM disk) , you can use /dev/shm.
> >
> > # date > /dev/shm/key
> > # cat /dev/shm/key
> > Fri Dec 13 07:26:03 CET 2019
>
> /dev/shm ist not a tmpfs on cygwin:
>
> /dev/shm: df -TH /dev/shm
> Filesystem     Type  Size  Used Avail Use% Mounted on
> C:/cygwin64    ntfs   34G   25G  9.1G  74% /
>
> When I write something into this direcory and reboot, it is still there.
...unless you open the file with the O_TMPFILE flag as outlined
in my other reply.


Corinna

--
Corinna Vinschen
Cygwin Maintainer

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Andrey Repin
In reply to this post by Ulli Horlacher
Greetings, Ulli Horlacher!

> /dev/shm ist not a tmpfs on cygwin:

Cygwin is a userspace DLL. It is not a filesystem driver.

> /dev/shm: df -TH /dev/shm
> Filesystem     Type  Size  Used Avail Use% Mounted on
> C:/cygwin64    ntfs   34G   25G  9.1G  74% /

> When I write something into this direcory and reboot, it is still there.

Use iSCSI RAM disk?
http://woshub.com/create-ram-disk-windows-server/


--
With best regards,
Andrey Repin
Tuesday, December 17, 2019 9:29:29

Sorry for my terrible english...


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Reply | Threaded
Open this post in threaded view
|

Re: non-persistent storage?

Corinna Vinschen-2
On Dec 17 09:32, Andrey Repin wrote:

> Greetings, Ulli Horlacher!
>
> > /dev/shm ist not a tmpfs on cygwin:
>
> Cygwin is a userspace DLL. It is not a filesystem driver.
>
> > /dev/shm: df -TH /dev/shm
> > Filesystem     Type  Size  Used Avail Use% Mounted on
> > C:/cygwin64    ntfs   34G   25G  9.1G  74% /
>
> > When I write something into this direcory and reboot, it is still there.
>
> Use iSCSI RAM disk?
> http://woshub.com/create-ram-disk-windows-server/
Great idea!  Pity this only works on server machines.


Corinna

--
Corinna Vinschen
Cygwin Maintainer

signature.asc (849 bytes) Download Attachment