PFS from INODZ

With the level of confusion and cluelessness we had in the beginning of the project, we never expected the project would pick up such a pace. Guess thats how it is when it comes to linux kernel hacking. Unbelievably interesting work to do. Anyway... coming to the progress of the project, We are still working on the pools. We are done with lets say 85-90% of it. There were features and bugs which took much longer than we expected. Since the last blog update, we added these features successfully:
-Adding block devices to pools
-Removing block devices from pools
-Creating pools via IOCTL commands
-Multiple number of pools
-Names to pools

Adding and removing were pretty straight forward. We were stuck when we needed to write a code to return a block_device when the name of the device was given. We finally used this function called open_by_devnum(). Later when our job with the block_devices is over(exiting the module, deleting disks from pool, etc), we use the function blkdev_put() to close the devices.
Then, for creating pools, we decided to have an initial pool called /dev/pool which would recieve and execute all requests(IOCTL) to create pools.

TO-DO list:
-Writing metadata having information about all pools and constituent block devices to pools.
-Scanning all disks at startup to create pools.
-Minor work such as renaming and deleting pools.
-IOCTL commands to list all pools and their member block devices.

Now with that, would complete the work on the pools. Hopefully in another week. We'll probably put the code for the pool on the blog after that.
Then we have to work on the filesystem. A lot of study has to be done. We have no clear direction yet. We expect to get stuck for a long time. As he said... GOD HELP US WITH THAT.

Pool is the device over which the file system is going to operate on. So, The pool has to redirect the requests that it gets to the original devices below it. I was bit stuck with "how can this communication be achieved ?"................
I was thinking of EXPORTING the transfer functions globally, so that, the pool's job as simple as to call the transfer functions. But is this generic ?
* I've EXPORT them.,
* Recompile the driver
* Also, you cant do this for all the drivers

COMPLETELY NOT GENERIC!

Then, Hari Helped!!!!!

He told me there are several ways by which u can achieve this without EXPORTING. one of the way is submit_bio(). I read in LDD3, "If you want to redirect, you change the bio->bi_bdev, and resubmit the bio".... [ GREAT!!! ]
But how to get the block_device object of a device....??

Hari Helped!!!!!

* Path_lookup the device
* Get the inode of the device from nameidata
* Get the dev_t object fom the inode
* open_by_devnum and get the block_device [GREAT!!!!]

ok... Got the block_device....
I changed the block_device and submited the bio
KERNEL PANICS!!!!!!!
tried tried tried tried tried....... 3 days of trying,
Hari pointed out the bug was there in bio_endio, but i din know what exactly the bug is.
05-01-2008, about 4.30 pm, evrery thing got so clear....

what i thought was,
* submit_bio() returns only after performing the whole I/O operation.
* so after the submit_bio, the bio is a waste
* i killed it after submit_bio

But the thing is,
* submit_bio retuns after "JUST PUTTING THE BIO IN THE REQUEST QUEUE OF THE OTHER DEVICE"
* not knowing this, i was killing the bio (which was still in the request queue)

FINE!
i wrote my bi_end_io function and did all ending operations there.........
TADAAAAAA.....
The code worked. Whatever operations tat i did in the pool, got reflected in ram0... Thankyou hari for all your help.! This project is going awesome..... Lots of learning........

NEXT_STEP : one-to-many mappping...

PFS will have 2 main modules.....
1. Pool manager
2. File system.

* Pool manager is the one tat should replace the volume manager.
The Pool Manager should handle multiple disks as a single logical pool. the size of that logical pool should not be limited. the size should grow when another disk is added to the pool, it should decreased when a disk is removed.! It should provide various functions to the file system tat will be sitting above this.

* the File System - PFS is the one tat interacts with the Pool Manager to get the data written into the hard disk. For this, the FS makes use of the methods provided by the pool manager. This interacts with the upper level layers like system calls etc.

Here is a detailed Description of the PFS...

Current filesystems have this problem : LIMITED SIZE...
u allocate certain size for a file system, format a file system, done.!

say,
in a 50 hard disk, u have 3 logical partitions
MOVIES | PROJECTS | DOCUMENTS..
20GB........20GB...........10GB

* now consider MOVIES partion is full
* there is 15GB free space in PROJECTS
* still u cant add even a single file in MOVIES !!!

Why not make a file system tat can dynamically expand and shrink...
We'll eliminate the term "VOLUMES, PARTITIONS"... We'll make POOLS!!!

Now, how about,
* MOVIES , PROJECTS, DOCUMENTS share the same 50GB.
* there is no seperate size for each of these.
* data can be added into any of these, till the whole 50GB gets full...
* still MOVIES , PROJECTS, DOCUMENTS can be accesed as different file systems
* they can be mounted, and un mounted seperately..

Now,
* u buy a new harddisk (120 GB)
* u plug-in the harddisk...
* boot ur system...
* "a single command" (pool -add /dev/my_new_harddisk already_existing_pool)
* NO-RE-FORMATTING ur harddisk
* [new space 50 + 120 GB]
* data can be added into any of MOVIES , PROJECTS, DOCUMENTS , till the whole 50GB + 120GB gets full...

it is already implemented in,
ZFS......... Open solaris
ZFS-on-FUSE. LINUX
LVM2 can do this (but not as a readymade file system)