$Id: README,v 1.2 2013/02/10 10:34:00 sthibaul Exp $

	   Memory based filesystem translators for the Hurd
       Copyright (C) 2000 Farid Hajji <farid.hajji@ob.kamp.net>

   These programs are free software; you can redistribute them and/or
   modify them under the terms of the GNU General Public License as
   published by the Free Software Foundation; either version 2, or (at
   your option) any later version.

   These programs are distributed in the hope that they will be useful, but
   WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
   General Public License for more details.

   You should have received a copy of the GNU General Public License
   along with these programs; if not, write to the Free Software
   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

Memory based filesystems:
-------------------------

Unlike regular filesystems, the contents of memory-based filesystems are
stored entirely in memory (think ramdisk). This is mainly for performance
reasons, but also to provide a testbed for new filesystem implementations
in case no free partition is available.

The programs in the directory provide implementations for memory-based
filesystems in the Hurd.

memfs-1: (filesystems for the lazy and impatient)
--------

A file in the Hurd is nothing more than a port to a translator that
provides the necessary semantics. For this reason, one can basically
create a big file (e.g. with dd), write a filesystem in it and settrans
the corresponding filesystem translator on it:

  dd if=/dev/zero of=/path/to/bigfile bs=<nr_of_bytes> count=1
  mkfs.ext2 /path/to/bigfile
  settrans -ac /path/to/target /hurd/ext2fs /path/to/bigfile

Accessing a file in /path/to/target is a multistage process:

 client -> /hurd/ext2fs[bigfile] -> /hurd/ufs[/path/to] -> storeio[/dev/hd0s1]
        <-                       <-                     <-

memfs-1 is a translator that provides a memory-based file of fixed size.
This file can, just like bigfile, contain a regular filesystem:

  settrans -ac /path/to/memfile /path/to/memfs-1 -s <size_in_bytes>
  mkfs.ext2 /path/to/memfile
  settrans -ac /path/to/target /hurd/ext2fs /path/to/memfile

Again, accessing files in /path/to/target involves many translators,
but fewer than in the previous case:

  client -> /hurd/ext2fs[memfile] -> /path/to/memfs-1
         <-                       <-

memfs-1 allocates an internal buffer on startup (currently via mmap())
and handles every trapped read(), write() and seek() operation by
copying data in/out of this buffer. Physical accesses to a disk are
completely avoided, as long as memfs-1 and the external filesystem
translator are resident in memory (else: accesses to swap disk[s]).

memfs-1 is a simple modification to the multithreaded hello-mt translator
written by the Free Software Foundation, Inc. As such, it is not robust
either, since many trivfs callbacks are not (yet) implemented. For example,
it is not possible to change the size of the memfile. Doing so (e.g. try
this with vi) will result in an EIEIO error.

Using memfs-1:
--------------

### unpack and install
$ tar -zxvf memfs-1.0.tar.gz
$ cd mem-fs
$ make
cc -Wall -g -c memfs-1.c -o memfs-1.o
cc -Wall -g -o memfs-1 memfs-1.o -lpthread -lports -ltrivfs -lfshelp
cc -Wall -g -c test-memfs-1.c -o test-memfs-1.o
cc -Wall -g -o test-memfs-1 test-memfs-1.o

### setup memfs-1 as a translator for xxx
$ make start-memfs-1
settrans -ac xxx ./memfs-1 --size 40 --debug 10

### memfs-1 is really running
$ ps ax | grep memfs
farid      192  - So    0:00.05 ./memfs-1 --size 40 --debug 10

### debug output is in .out:
$ cat .out
./memfs-1 [2000-12-19 19:04:57]: parse_opt
./memfs-1 [2000-12-19 19:04:57]: parse_opt
./memfs-1 [2000-12-19 19:04:57]: parse_opt
./memfs-1 [2000-12-19 19:04:57]: parse_opt

### The memfs-1 translated memory file xxx is there:
$ ls -l xxx
-rw-rw-rw-    1 farid    farid          40 Dec 19 19:04 xxx

### The previous ls(1) triggered the following events:
$ cat .out
./memfs-1 [2000-12-19 19:06:53]: open_hook
./memfs-1 [2000-12-19 19:06:53]: close_hook

### Let's write 10 bytes in xxx at offset 5:
$ ./test-memfs-1 xxx 5
I am now at offset 5
written 10 bytes
I am now at offset 0
got 40 bytes: [.....0123456789.........................]

### This triggered the following events:
$ cat .out
./memfs-1 [2000-12-19 19:08:25]: open_hook
./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_seek(offs=5)
./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_write(offs=-1, data_len=10)
./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_seek(offs=0)
./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_read(offs=-1, amount=40)
./memfs-1 [2000-12-19 19:08:25]: close_hook

### Let's write near the end of the file, effectively overflowing it:
$ ./test-memfs-1 xxx 37
I am now at offset 37
written 3 bytes
I am now at offset 0
got 40 bytes: [.....0123456789......................012]

### Looking at the new events again:
$ cat .out
./memfs-1 [2000-12-19 19:09:33]: open_hook
./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_seek(offs=37)
./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_write(offs=-1, data_len=10)
./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_seek(offs=0)
./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_read(offs=-1, amount=40)
./memfs-1 [2000-12-19 19:09:33]: close_hook

### Writing past the end of the file is silently rejected:
$ ./test-memfs-1 xxx 40
I am now at offset 40
written 0 bytes
I am now at offset 0
got 40 bytes: [.....0123456789......................012]

$ ./test-memfs-1 xxx 500
I am now at offset 40
written 0 bytes
I am now at offset 0
got 40 bytes: [.....0123456789......................012]

$ ./test-memfs-1 xxx -2
I am now at offset 40
written 0 bytes
I am now at offset 0
got 40 bytes: [.....0123456789......................012]

### We don't need the memfs-1 translator anymore:
$ make stop-memfs-1
settrans -fg xxx

### trivfs_goaway() was really called:
$ cat .out
./memfs-1 [2000-12-19 19:10:29]: open_hook
./memfs-1 [2000-12-19 19:10:29]: close_hook
./memfs-1 [2000-12-19 19:10:29]: trivfs_goaway

### translator memfs-1 is away (old time):
$ ls -l xxx
-rw-r--r--    1 farid    farid           0 Dec 19 19:04 xxx

Using memfs-1 as a filesystem translator:
-----------------------------------------

### Let's create a big memory file:
$ settrans -ac xxx ./memfs-1 --size 67108864 --debug 2

### We can distinguish this translator in 'ps'
### but due to the limit in MAXSIZE (see memfs-1.c)
### the file was truncated to 30MB:
$ ps ax | grep memfs
farid      385  - So    0:00.90 ./memfs-1 --size 67108864 --debug 2
$ ls -l xxx
-rw-rw-rw-    1 farid    farid    31457280 Dec 19 19:04 xxx

### Now, create an ext2fs filesystem in xxx:
$ /sbin/mke2fs xxx
mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09
Filesystem label=
OS type: GNU/Hurd
Block size=1024 (log=0)
Fragment size=1024 (log=0)
7680 inodes, 30720 blocks
1536 blocks (5.00%) reserved for the super user
First data block=1
4 block groups
8192 blocks per group, 8192 fragments per group
1920 inodes per group
Superblock backups stored on blocks: 
	8193, 24577

Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done

### The new filesystem can now be mounted as usual:
$ settrans -ac newdir /hurd/ext2fs xxx
$ ls -ld newdir
drwxr-xr-x    3 farid    farid        1024 Dec 19 19:46 newdir
$ /bin/ls -l newdir
total 12
drwxr-xr-x    2 root     root        12288 Dec 19 19:46 lost+found

### Filesystem newdir/ is now usable just like any other filesystem.
### Remove the translators if not needed anymore:

$ settrans -fg newdir
$ settrans -fg xxx

One final note: memfs-1 currently _requires_ a --size parameter.
                Use a --debug value >=5 to get a trace in .out

Suggested future work:
----------------------

  memfs-1: add missing trivfs callbacks.
  memfs-2: like memfs-1, but uses libpager for the internal buffer.
  memfs-3: like memfs-1, but uses libstore (memory or task) for the buffer.
  memfs-4: netfs-based true filesystem with mmap() allocated internal buffers.
  memfs-5: netfs-based true filesystem with libpager backed internal buffers.
  memfs-6: netfs-based true filesystem with libstore (memory/task) for buffers.
  memfs-7: diskfs-based true filesystem, mmap() allocated buffers.
  memfs-8: diskfs-based true filesystem, libpager backed buffers.
  memfs-9: diskfs-based true filesystem, libstore (memory/task) for buffers.

  memfs-{a,b,c,...}: filesystems should export the buffer[s] via mmap() to
                     clients to avoid copyin/copyout. (This can be done by
                     adding a RPC so that clients can send their task control
                     port in a request.)
  shm*() library   : use memfs-{a,b,c...} to implement SysV-ish shared memory.
  Add more filesystem types (minix, vfat, ...) usable with memfs-{1,2,3}.

-- 
Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555
Broicherdorfstr. 83, D-41564 Kaarst, Germany  | farid.hajji@ob.kamp.net
- - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - -
Murphy's Law fails only when you try to demonstrate it, and thus succeeds.
