Proxy SSH

June 13, 2015

There was a post on my own blog in February about how to log in directly to lab machines, hopping through Lee in a transparent way. Proxy SSH was described in this helpful article. It consists of running “nc” (netcat) on one machine to pass all of the ssh connection onto another machine.

It depends on public-key authentication and a behind-the-scenes proxy authentication so you don’t have to log in twice. Using the .ssh/config file to automate the details, once setup you don’t even see the proxy.


  • Your login name is pikachu;
  • and created a key pair id_rsa_pikachu/;
  • you wish to proxy through lee to get to shiloh.

Add to lee:~/.ssh/authorized_keys. Since shiloh and lee share your home directory, shiloh now knows this public key as well. In the .ssh/config on your local machine, add:

Host shiloh
ProxyCommand ssh -o StrictHostKeyChecking=no lee nc %h 22
User pikachu
IdentityFile ~/.ssh/id_rsa_pikachu

Host lee
User pikachu
IdentityFile ~/.ssh/id_rsa_pikachu

Then from you local machine log into shiloh with “ssh shiloh”. As a bonus, scp’s such as scp this-file shiloh:that-file will work as well.

Backups and sparse files

July 11, 2014

The subtly of backups is often overlooked. What is meant by a backup depends on the purpose, and no backup scheme can satisfy all requirements.

Some backups serve the purpose of system administrators. The goal of these backups is to provide a mirror-image restore of the file system as it was at certain planned checkpoints in time. This does not require a block-by-block copy, but that it be sufficiently identical.

Backups which serve the purposes of a users only need restore the “heart-and-soul” of the data, and not technical details of the representation or storage. The restored file can differ in unseen ways from the original, and this will be of no consequence to the user.

One example is the sparse file. Most unix systems allow that a file span over character locations in which the file has no content. The classic unix file system uses a tree of pointers to data blocks, and a null pointer signifies a missing data block. If one were to read such a file as a stream, the file system will provide zeros as the contents of the missing blocks. In fact, another viewpoint on sparse files is that no byte of the file is missing, rather blocks of all zeros is represented by a null pointer rather than a pointer to data.

While a user might not care if her sparse file were recovered as a non-sparse file, a system admin might care very much. If sparse files are not recovered as sparse, the recovered files might not fit back onto the medium from which they came. There might be other unforeseen consequences.

The following is a demonstration of sparse files, as well as a demonstration that for purposes of backup which preserves sparse files, cp cannot be used (nor tar, cpio, or scp, for that matter):

$ uname -rs
FreeBSD 8.3-RELEASE-p3
$ dd if=/dev/zero of=sparse.yes bs=1 count=1 seek=128K
1+0 records in
1+0 records out
1 bytes transferred in 0.000039 secs (25732 bytes/sec)
$ cp sparse.yes
$ ls -ls sparse*
130 -rw-r–r– 1 burt user 131073 Jul 11 19:47
18 -rw-r–r– 1 burt user 131073 Jul 11 19:46 sparse.yes

Hill backup schedule

August 17, 2013

The Hill backup system was described in a prior posting. It’s main purpose is to provide roll-back to a known system state in case of complete loss in Ungar. For this reason, it uses dump to provide an block-accurate disk image, including hard links and sparse files and does not modify touch times during backup.

The backups are incremental with the current pattern of a zero level at start of epoch, and four level 4′s at the start of each month and four weekly or semi-weekly level 8′s.

Sunday Monday Tuesday Wednesday Thursday Friday Saturday
11 PM Pickett
3 AM Beauregard Bragg Meade Davis Mcclellan Meade Davis

System backups based on snapshots incidentally serve other purposes, for instance, to recover user files that have been corrupted or deleted. Past snapshots are saved with an accuracy that coarsens with receding time.

It is often unacknowledged that no backup scheme fits all needs. For instance, archive and source-control contrast to snapshots. Snapshots are meant to be mechanical and precise. Archive is permanent, deliberate, and indexed; Source control keeps versions, including deleted files, and possibly renaming and file moves, and has collaboration mechanisms.

Beauregard renovations

August 16, 2013

Beauregard has been updated to Ubuntu 12.04-amd64 LTS , from Debian. Beauregard now provides Mysql, NFS and backup services to the blades.

For backup, the blades are sandboxed, we generally do not snapshot the system; user data should be either on the NFS mounted partition, in the mysql database, or on our subversion server. The blade is a compute engine, treat it as interchangeable and quickly re-installable.

  • NFS Howto:On beauregard:
    mkdir /opt/cluster/nfs/${BLADE} ;\
    echo "/exp/blades/${BLADE} ${BLADEIP}(rw,sync,no_root_squash,no_subtree_check)"\
    >> /etc/exports ;\
    /usr/sbin/exportfs -r

    On {$BLADE}:
    echo "beauregard:/exp/blades/${BLADE} /opt nfs soft,intr 0 0" >> /etc/fstab ; mount -a

    See also: beauregard:/opt/cluster/nfs/Makefile; note symlink from /exp/blades to /opt/cluster/nfs.

  • Mysql Howto:On beauregard:
    create database ${DATABASE};
    grant all privileges on ${DATABASE}.* to '${USER}'@'${BLADEIP}' \
    identified by '${PASSWORD}' with grant option ;

    On ${BLADE} with IP ${BLADEIP}:
    mysql -u ${USER} -p -h
    use ${DATABASE};

Private local numbering

June 22, 2013

The internal IP addresses of the computer science machines are from the address block which is then subnetted by security level to an overall address. Currently there are four security levels – public, sandbox, lab and servers, but this summer we will collapse public and sandbox into a single level.









Online backups

August 19, 2012

Over the past year the department has improved backups with respect to:

  • Accessibility, but having on-line backups to large capacity backup servers;
  • Diversity, locating one of two backup remotely, to reduce site-risk;
  • Security, with end-to-end encryption so that the remote server does not have the data nor keys;
  • Correctness, by making sure the data is in a consistent state throughout the backup.

The problem of backup is more nuanced that it first seems. While most users want from a backup the restoration of a deleted file, the aim of a backup is to restore a machine to the state it was in at a certain point in time. These time points are, for instance, nightly for the past week, weekly for the past month or so, and then several monthly or semi-monthly going back for whatever number of years desired. That this might include the ability to restore a user file is somewhat coincidental.

In default of a separate backup mechanism more appropriate for user files, the response is to segregate user data from system data, and to make sure the periodic backup system has appropriate spacing, retention and technology independence for the purposes of user data.

The accessibility, diversity and security requirements are met using a remote machine, with a standard dump piped first though openssl encryption and then through ssh to the remote machine.

Correctness requires consideration of the particular services. Using filesystem snapshots, dump can now have a consistent view of the filesystem frozen at a point in time. FreeBSD supports snapshots natively in UFS, and Linux supports it universally using LVM. A mysql database can be correctly backed-up using mysqlhotcopy to properly duplicate the files that contain the database, properly locked and flushed to insure consistency. Subversion has svnadmin hotcopy that does the same for subversion.

While this should settle the matter for filesystems and subversion, it does seem possible that an improperly programmed database application might split a transaction, and the hotcopy might occur at the split.

Moving WordPress across servers

August 19, 2012

To move a wordpress to a new server requires that three objects get moved:

  1. The wordpress application;
  2. The database of user content;
  3. and the HTTP configuration.

On a fresh Red Hat derived Linux, the required yum installs are: mysql-server, mysql, httpd, php, php-mysql. WordPress is installed via tar, as a php application there is no build required. To move the wordpress installation tar this and untar it at the destination. Change owner and group to apache. If this is from a fresh install of wordpress, first untar the distribution then overlay wp-config.php and the entire directory wp-content.

To move the database, use mysqlhotcopy to make a copy of the files in, say, /var/lib/mysql/_database_name_ into a suitable location. say /opt/backup/mysql/_database_name_.

bash> mysqlhotcopy -u root -p PASS --allowold wildpages /opt/backup/mysql

Note that you probably want to delete all unapproved comments (spam) before doing the backup, as they can total the largest amount of data in your database. This can be done by mysql as:

mysql> DELETE FROM wp_4_comments WHERE comment_approved = '0';

The cloned directory can be tar’ed and moved, careful to stop the mysql server during the untar. Finally the new owner has to be introduced to the mysql system by:

mysql> GRANT ALL PRIVILEGES ON _database_name_s.* to "_database_name_"@"localhost" \
identified by "mypassword";
mysql> flush privileges;

Modify the apache config, copying in the old config.

Welcome Bragg

August 19, 2012

Bragg is now up and running, thanks to Irina.

Bragg is a Dell R510 running a single Intel E5620 2.4 Ghz, 24G 1333MHz dual ranked RDIMM’s; a PERC H700 Raid, and 600 GB/2 TB mirrored SAS.

It will be taking on the responsibilities of Davis.

We have moved to CentOS, because UM is not longer purchasing a site license with Red Hat, and Fedora, Red Hat’s free release, rev’s very quickly. That keeps us too busy chasing rev’s.

Welcome Hill

June 11, 2012

Welcome our newest server, (CNAME The R510 Dell is located in the physics department server room, and runs Ubuntu. It’s purpose is to hold backups of other cs machines, offsite in case of a room-wide emergency. Running Ubuntu 10.04.4 LTS we are good to go until April 2015. (We missed 12.04 LTS by a day! Else we wouldn’t need a major upgrade until April 2017).

P.S. Mysqld is crashing on on the cloud. Looks like out of memory. Perhaps a micro edition just isn’t enough.

Migration to the cloud

June 11, 2012

In preparation to update meade, the services,, and have been moved to instances on AWS. Only remains on meade. will have a permanent redirect to

Powered by Wordpress and MySQL. Theme by Shlomi Noach,