Network Block Device Daemon

The Network Block Device Daemon is obsoleted by Philipp Reisner’s DRBD
Sebastian Tomac version 0.2.1(obsolete) 2000-08-12

 

The Linux Network Block Device Daemon HOWTO

CONTENT

  • Introduction
  • Network Block Device Daemon installation and configuration
    1. Getting the software
    2. Setting up the kernel
    3. Installing the software
    4. Configuring the nbdd server
    5. Configuring the nbd client (/dev/nb)
    6. Setting up automatic server initiation
  • Network Block Device client-server implementation and principles
    1. Operation principles
    2. Log files and binary files
    3. Configuration and programming API
    4. Cluster configuration example
    5. Cluster Troubleshooting
  • Appendix
    • Protocol specification
    • Security
    • Demo configuration script
    • Uninstalling nbdd

INTRODUCTION

Principles of nbdd: The network block device daemon (nbdd) enables
Linux to export a local block device or file to a client nbd client driver
setup. The client will see the exported device as one of its block devices
(/dev/nb). For further information on the client see the nbd client home
page, where you also can find a nbdd server equivalent, running in user
space. Every time the client computer wants to read the network device
(/dev/nb), it sends a request over TCP to the server, which will reply
with the data read or receive data written to the device. This can be used
for server clustering and high availability, by software raid (i.e. mirroring)
over a local block device and network block device. The nbdd server provides
functionality similar to that of commercial UNIX implementation for distributing
access to disks, like VSDs on AIX and CVM in Solaris.

What you need to know: If you only want to get the thing up
and running as fast as possible, you need to read the section “Network
Block Device installation and configuration”. If you want a more deep intrusion
to how the network block device works, as well as an example including
setting up raid mirroring on separate machines, with trouble shooting the
section “Network Block Device kernel client-server implementation and principles”
should be of interest.

Current state: It currently works. Bug reports are welcome.
The nbdd kernel module is distributed as a patch file to the kernel (see
the download
section).

NETWORK BLOCK DEVICE DAEMON INSTALLATION AND CONFIGURATION

  1. Getting the software
  2. Setting up the kernel
  3. Installing the software
  4. Configuring the nbdd server
  5. Configuring the nbd client (/dev/nb)
  6. Setting up automatic server initiation

Getting the software

From Download
download the following packages and patches.

* nbdd-0.2-2.3.39.patch — nbdd kernel patch, it should work with later
versions of the kernel as well.

* nbdd-0.2-0.i386.rpm –nbdd administrative commands

* nbdc-0.2-0.i386.rpm — nbd user land client, required by the client
network block device driver, for initiation. It is a patched version
of Pavel’s nbd client. working as client against the nbdd server.

If you want to recompile the user land binaries you can download the
source package (nbdd-0.2-0.i386.src.rpm) and compile them yourself. Note
that nbdd has been developed on a x86 (i.e. i386) platform, and has not
yet been tested on any other platform.

Setting up the kernel

All actions performed to configure the kernel is done as root. Patch the
kernel with the nbdd patch, available at Download. Note the the patch should work with later kernels as well.

cd /usr/src/linux-2.3.39
patch -p1 /tmp/nbdd-0.2-2.3.39.patch

Compile and install the kernel as usual (se the kernel HOWTO for instruction,
www.linuxdoc.org). Ensure that when you create your .config file (make
xconfig or make config …), select nbd and nbdd module. The menu is located
under devices->block->nbd and nbdd. Remember to; make modules, and; make
modules install. Reboot the new kernel and you are ready to go.

Installing nbdd software

Download the pre compiled binary user part (nbdd-0.2-0.i386.rpm). On RedHat
and SuSe linux install the package as user root, with

 rpm -U nbdd-0.2-0.i386.rpm

The -U makes rpm perform an upgrade, this is equivalent to rpm -i nbdd-0.2-0.i386.rpm,
if the package isn’t already installed. The steps performed in the configuring
the nbd client and nbdd server subsections are performed by a demo script
available in Appendix C. To uninstall the nbdd package see the Appendix
D.

Configuring the nbd server – network block device daemon (nbdd)

  1. Nbdd device and module loading
  2. Nbdd configuration parameters
  3. Nbdd configuration example
Nbdd device and module loading

Load the nbdd server module, on the server machine. In the future the server
will take the port to listen on as argument, for now its static at port
4343.

insmod nbdd

Test that the module is loaded with lsmod.

lsmod
# Module Size Used by
# nbdd 47968 2

Check that the server daemon is running with

ps -ef | grep nbdd
 
# root 595
# 1 0 19:24 ttyp1 00:00:00 [nbdd]
 
#root 596
#1 0 19:24 ttyp1 00:00:00 [nbdd]

Also check that the daemon is listening on the socket

netstat -an | grep 4343
 
#tcp 0
#0 0.0.0.0:4343
#0.0.0.0:* LISTEN
Nbdd configuration parameters

The nbdd command are used for administering the nbdd server. For details
se the man pages. The following table outlines the configurable parameters
and the flag used in the administrative commands.

FLAG ARGUMENT EXAMPLE COMMENT
-l exported device or id -l -d /dev/sdb2 lists exported devices
-c max number of clients connected. -c -x 1000 change maxnumber of connected clients to nbdd server.
-r exported device or id -r -d /dev/sdb2 remove device from export list
-m device to export AND id -m -d /dev/sdb2 -i 0×123456 add local device to export list
–start exported device or id –start -d /dev/sdb2 makes exported device available to the client
–stop exported device or id –stop -d /dev/sdb2 makes exported device unavailable to the client

The connection id ties the exported device to whatever /dev/nbx device
the clients wants to connect to it.

Nbdd configuration example

The following section outlines an configuration, available in Appendix
B as an demo script. All configuration actions are performed as root. The
command following command configures the dummy device (i.e. a file /tmp/nbd_dummy
emulating a device , created with the command “dd if=/dev/zero of=/tmp/nbd_dummy
bs=1024 count=4000″.) for export to any client requesting a connection
with id 0×12345678.

nbdd -m -d /tmp/nbd_dummy -i 0x12345678

with the list options we list the configured devices

nbdd -l -a -s
# Export_device ID

# Status

# /tmp/nbd_dummy 0x12345678 DEFINED

Do not forget to make the export available to the client with the start
argument

nbdd --start -d /tmp/nbd_dummy
nbdd -l -a -s
# Export_device ID
# Status /tmp/nbd_dummy 0x12345678 AVAIL/RUNNING

Configuring the nbd client (/dev/nb)

  1. Installing the nbd software
  2. Nbd device and module loading
  3. Nbd configuration example
Installing the nbd software

The nbd device driver is already part of the main kernel distribution,
so no patching of the kernel is required on the nbd clients behalf, just
ensure that you select nbd module when you are recompiled the kernel, to
get the nbdd driver going. Download
the nbdc user land package and install it with,

rpm -U /tmp/nbdc-0.2-0.i386.rpm

The nbdc is a patched version of Pavel's nbd client. If you want
to compile the package download the source nbd-14.tar.gz. Patch it with
this patch,
"cd nbd; patch -p1 < nbd14-0.2.patch. Configure and make. Two files
are of interest, the binary, nbdc and the man page nbdc.1.

Nbd device and module loading

Load the nbd client module, on the client machine (i.e. the machine using
the nbd device /dev/nb).

insmod nbd

Test that the module is loaded with lsmod. The nbd should be listed
as in the example below.

lsmod
# Module Size Used by 
# nbd 16064 0 (unused)
# nbdd 47968 2

check that the nbd block devices in /dev has been created (if you are
using a standard Linux distribution RedHat, SuSe the nodes are already
there). Nbd uses major 43, minors 0..n (where n is configurable in nbd.h).
If the nodes are not there create these files with mknod. After that, your
ls -l /dev/ should look like:

brw-rw-rw- 1 root root 43, 0 Apr 11 00:28 nb0
brw-rw-rw- 1 root root 43, 1 Apr 11 00:28 nb1
...
Nbd configuration example

The following section outlines an configuration, available in Appendix
C as an demo script. All configuration actions are performed as root. The
command following command configures the device /dev/nb0 for connecting
to a nbdd server running on an host with IP address 127.0.0.1, listening
on port 4343, and exporting a device given the unique id 0x12345678 (i.e.
the device exported in the nbdd example).

nbdc 127.0.0.1 4343 0x12345678 /dev/nb0 &amp;

with the ps command we get the running client devices

ps -ef | grep nbdc
# root 613 1 0 19:45 ttyp1 00:00:00 nbdc 127.0.0.1 4343 0x12345678
0

Check with netstat that you actually have a running connection with

netstat -an | grep 4343
# tcp 0
# 0 0.0.0.0:4343
# 0.0.0.0:* LISTEN
 
# tcp 0 0 127.0.0.1:1034 
# 127.0.0.1:4343 ESTABLISHED
 
# tcp 0 0 127.0.0.1:4343
# 127.0.0.1:1034 ESTABLISHED

Test the device by reading or writing something to it, If the exported
device is the dummy test file, you can test it by creating a file system
on it an mounting the the device.

mke2fs /dev/nb0
mount -t ext2 /dev/nb0 /mnt

If all is running you are pretty much setup.

SETTING UP AUTOMATIC INITIATION

For automatic startup of the nbdd server daemon and export of all devices
on boot, use the control panel or YaST utility to setup initiation. The
nbdd init script should be listed as daemons to start on boot of the machine
(/etc/rc.d/init.d/nbdd). Edit the script to add devices, see example
in the init script included in the rpm.

NETWORK BLOCK DEVICE KERNEL CLIENT-SERVER IMPLEMENTATION AND PRINCIPLES

  1. Operation principles
  2. Log files and binary files
  3. Configuration and programming API
  4. Cluster configuration example
  5. Cluster Troubleshooting

Operation Principle

The nbdd kernel server thread is always running, once loaded, even if no
device is exported. Each exported device from the server is in one of the
following states, UNDEFINED, the exported device is undefined it does not
exist for the nbdd server. DEFINED, the export device is defined to the
nbdd server, but is unavailable to the clients . AVAILABLE the device
is available for the clients for connection; RUNNING, a client is using
the device. The command "nbdd -m" will create an entry in the exported
device table in the nbdd server, making the device DEFINED. "nbdd -r" will
remove an entry in the exported device table. The command "nbdd --start"
will make the device available to the client, i.e. going from state DEFINED
to AVAILABLE. If a client connects to the server the state will change
to RUNNING. The command "nbdd --stop" will bring the device from
AVAILABLE/RUNNING to DEFINED. The attached figure illustrates the principles.

 

Log files and binary files

The nbd client and nbdd server kernel logs in /var/log/messages. All lines
beginning with "nbdd:" are server entries. The user commands prints to
standard output. List of nbd and nbdd files

FILE COMMENT
/lib/modules/nbd.o client kernel module
/lib/modules/nbdd.o server kernel module
/sbin/nbdd nbdd server admin. utility
/sbin/nbdc nbd client initiator.
/etc/rc.d/init.d/nbdd init start and stop script for server
/usr/man/man.1 nbdd and nbdc man files
/var/log/messages kernel (nbd,nbdd) default log file

Configuration and programming API

The nbdd server is configured throe the sysctl system call, that is via
/proc/sys/dev/nbdd files. DO NOT edit the proc nbdd files directly use
the sysctl system call. List of proc files and configuration structures.

PROC_FILE ARGUMENT COMMENT
/exportdev char** nbdd exported devices
./maxconnect int max connected clients
./passwd __u64* passwords for exported devices
./identity __u64* id for exported devices

See the nbdd source code for implementation and examples.

Cluster configuration example

This configuration example only considers the setting up and configuration
of meta devices mirroring over disks located on different physical machines.
Everything above that in terms of applications using the raid mirrored
cluster disks and take over of network interfaces, will be handled separately
in another HOWTO.

.... /*To be continued/

Cluster troubleshooting

.... /To be continued/

Appendix A

Protocol: The protocol is rather simple. A connection is initiated with
a client authentication and identification call, and the server replies
with denial or acc with the size of the exported device (see the nbdc code).
After that we do read write, now inside the kernel. If the driver is asked
to read from block device, it sends packet of following form "request"
(all data are in network byte order):

__u32 magic; must be equal to 0x12560953

__u32 from; position in bytes to read from / write at

__u32 len; number of bytes to be read / written

__u64 handle; handle of operation

__u32 type; 0 = read

1 = write

... in case of write operation, this is

immediately followed len bytes of data

When operation is completed, server responds with packet of following
structure "reply":

__u32 magic; must be equal to

__u64 handle; handle copied from request

__u32 error; 0 = operation completed successfully,

else error code

... in case of read operation with no error,

this is immediately followed len bytes of data

For the security i.e. initiation protocol see Appendix B.

Appendix B

Security sucks. At the moment there is non except correct request and reply
id handle. However the bases for authentication is there with, request
reply challenge challenge-reply mecanism. If some of you kernel people
know how to integrate nbdd with some other kernel authentication (nfs,
ipsec??) send me an e-mail (sebastian.tomac@gmail.com).

Appendix C

The following lines contains a network block device configuration and setup
demo script, which you can create by cutting and pasting the following
lines:

# cut here ----------------------------------------------
#!/bin/sh
 
# dummy demo script running the nbdd server and nbd client on the
same machine.
echo "create dummy export device"
dd if=/dev/zero of=/tmp/nbd_dummy count=4000 bs=1024
echo "load the nbdd and nbd modules"
insmod nbdd; insmod nbd
netstat -an | grep 4343
echo "export and start the device"
nbdd -m -d /tmp/nbd_dummy -i 0x12345678
nbdd --start -i 0x12345678
nbdd -l -a -s
echo "start the nbd client"
nbdc localhost 4343 0x12345678 /dev/nb0 &amp;
echo "create ext2 file system on the dymmy device and mount it at /mnt."
mke2fs /dev/nb0
mount -t ext2 /dev/nb0 /mnt
ls -la /mnt
# end the cut here --------------------------------------

The script should be run by root.

Appendix D

Uninstall the nbdd package with the command,

rpm -e nbdd-0.2-0

and for the userland part of the client, with

rpm -e nbdc-0.2-0