Title:  UBC: An Efficient Unified I/O and Memory Caching Subsystem for BSD

Author: Chuck Silvers
Point of contact:
      Chuck Silvers
      chuq@chuq.com
      7588 Lockford Ct.
      Cupertino, CA  95014
      (650) 526-2649 (work)
      (408) 257-7467 (home)

Modern operating systems allow filesystem data to be accessed using
two mechanisms: memory mapping, and I/O system calls such as "read"
and "write."  In traditional unix-like operating systems memory
requests are handled by the virtual memory subsystem and I/O calls are
handled by the I/O subsystem.  In many operating systems these two
subsystems are developed separately and are not tightly integrated.
For example, in the NetBSD operating system the VM and I/O subsystems
each have their own data caching mechanisms that operate
semi-independently of each other.  This lack of integration leads to
inefficient overall system performance and a lack of flexibility.
Thus, to achieve good performance it is important for the virtual
memory and I/O subsystems be highly integrated.

The problems with the existing file data caching system in NetBSD that
we'd like to solve are numerous:
 - files accessed via both read()/write() and memory mappings are
   cached twice, once in the "buffer cache" for read()/write() and
   a second time in the virtual memory system's "page cache" for
   memory-mapped access. some effort is made to keep these two caches
   coherent, but the lack of tight integration makes it possible to
   access stale data.
 - the buffer cache is statically sized at boot time, which effectively
   partitions a machine's RAM into two uses, "cached file data" and
   "everything else".  it is not possible for memory to be moved from
   one catagory to the other, resulting in reduced performance if
   a machine has varying usage patterns.
 - the buffer cache's requirement that buffer cache memory always be
   mapped into the kernel's virtual address space limits the amount of
   memory which can be used to cache file data on machines with large
   amounts of RAM, since these days it's not too hard to have more RAM
   in a machine than will fit in the kernel's virtual address space.

To address theses problem we introduce UBC, a subsystem that bridges the
gap between virtual memory and filesystems by providing a mechanism
for filesystems to easily use VM system resources for caching file data.
UBC allows filesystems to use the VM system's page cache store their data,
taking advantage of features of UVM (NetBSD's VM system) to allow
memory-mapped access to the page-cache pages within the kernel so that
filesystems can easily move data from the kernel's cache to the
application.

This solves the above problems very nicely:
 - file data is stored in only one place, so there's no wastage of memory
   from double-caching and no coherency problems.
 - the page cache is dynamically resized while the system is running based
   on current usage patterns, so memory can be reused for whatever purpose
   is most beneficial at any given time.
 - page-cache data doesn't need to be mapped to retain its identity, so
   there are no limits on how much memory can be used to cache file data
   (other than how much RAM is in the machine).

The implementation of this new design is nearly functionally complete,
performance improvements are the majority of the work remaining to be
done.  To view the work-in-progress, use the CVS source control system to
retrieve the UBC branch from the NetBSD project's source reposity with
the command:
	cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot \
	    co -r chs-ubc2 syssrc/sys
to see just the changes I've made, run:
	cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot \
	    rdiff -rchs-ubc2-base -rchs-ubc2 syssrc/sys

This paper will describe the different approaches I tried in the
course of this work and compare the approaches to those used by other
operating systems.  I'll also show the performance benefits of this
new design for file caching and discuss future related work.