VMWare madness at Hetzner

Recently we decided to try out hosting at Hetzner (https://www.hetzner.co.za). We’re in need of a new Virtual machine and as we’re cutting down on expensive hosting in data centers their offering was very appealing. The server is cost effective, powerful and they provide a decent amount of Internet data.

The problem

I must clarify, the problems here are not of Hetzners making. Their service is excellent and response times to my queries were very good. The problem was a bit more subtle than that.

In order to install the server Hetzner was supplied with a link to the VMWare ESXi 5.5.0 ISO I needed to install. They downloaded, burned the ISO and inserted it into our new server. No problem so far. I set up the BIOS, installed VMWare and booted into VMWare. Also good.

Here is the issue. After configuring the management network the Intel 82574L network indicated that it was disconnected. No problem, quickly fire off a mail to Hetzner to check network cabling. It’s easy for a techy to make this mistake. However, after some time they contacted me and assured me it was connected.

A handy feature of their hosting is the ability to PXE boot a linux rescue image running Debian or Ubuntu. Very handy, and frustrating, as it meant that the network card is indeed connected and working as expected.

It turns out, the network card, Intel 82574L, is not supported by VMWare. So how does one load a network card driver without internet connectivity?

The solution

Recently we decided to try out hosting at Hetzner (https://www.hetzner.co.za). We’re in need of a new Virtual machine and as we’re cutting down on expensive hosting in data centers their offering was very appealing. The server is cost effective, powerful and they provide a decent amount of Internet data.

The problem

I must clarify, the problems here are not of Hetzners making. Their service is excellent and response times to my queries were very good. The problem was a bit more subtle than that.

In order to install the server Hetzner was supplied with a link to the VMWare ESXi 5.5.0 ISO I needed to install. They downloaded, burned the ISO and inserted it into our new server. No problem so far. I set up the BIOS, installed VMWare and booted into VMWare. Also good.

Here is the issue. After configuring the management network the Intel 82574L network indicated that it was disconnected. No problem, quickly fire off a mail to Hetzner to check network cabling. It’s easy for a techy to make this mistake. However, after some time they contacted me and assured me it was connected.

A handy feature of their hosting is the ability to PXE boot a linux rescue image running Debian or Ubuntu. Very handy, and frustrating, as it meant that the network card is indeed connected and working as expected.

It turns out, the network card, Intel 82574L, is not supported by VMWare. So how does one load a network card driver without internet connectivity?

The solution

Below I copy directly from my crude notes. Keep in mind that Linux does not mount VMFS read / write.

In order to install the driver for the 82575L Intel driver I followed the following steps:

1) Instal VMWare esxi 5.5.0
2) Boot into linux recovery image
3) Execute the following commands on the revovery image
# mount -t vfat  /dev/sda3 /mnt/
# cd /mnt
# wget http://shell.peach.ne.jp/~aoyama/wordpress/download/net-e1000e-2.3.2.x86_64.vib
# cd
# unmount /mnt
# reboot
4) Once VMWare esxi has finished booting switch to console with <ALT> – <F1>
5) Install the driver. It is located in one of the VMFS partitions already mounted. It is a 4 GB partition in my case.
# esxcli software acceptance set –level=CommunitySupported
# cd <directory containing driver>
# cp net-e1000e-2.3.2.x86_64.vib /var/log/vmware/
# esxcli software vib install -v net-e1000e-2.3.2.x86_64.vib
6) Reboot vmware esxi

/dev/sda3 it turns out is a fat32 / vfat partition that is mounted at boot on VMWare. I used the /download/ directory on that partition to store the driver.

Happy days…

 

Securing you Virtual servers comment

I forgot to mention. There is one very big flaw in this process. Nothing prevents someone from stealing your server and investigating the contents and then using Mandos (modified for their needs) to download the decryption password.

Now the above assumes time, and this is where Mandos has some additional security. Mandos will regularly query your client. If the client disappears for a specified period it will disable the key. So setting a timout long enough to allow reboots, but not extended poweroffs add some additional security. And if the client key is disabled, it is very simple to reenable it on the server with the mandos-monitor utility.

Securing your virtual servers

From my previous post it must be obvious that I am sometimes concerned about security. This post is the start of my investigation into securing my servers.

Irrational paranoia and tin foil hats aside, there are valid reasons for wanting to do this.

The problem

1) Protect your data against the man. Now this is not as ridiculous as you might think. We trust out lives to google, dropbox and various companies. Although we are generally well protected we have heard of rumours of data being made available to government agencies. Any big company with lots of useful data will eventually have this problem. But, this is the tin foil hat scenario.

2) Keeping data secure from theft. My email accounts are very important to me. Without email I am not connected to the rest of the world. From getting my CV out to prospective employers to bank statement. My life is stored in email. My on line accounts can be reset via email. If a nefarious character happens apon these details it would be very inconvenient. What about all my files? My work documents, or more important, our company intellectual property. Securing a server in such a way that its data is not available in case of theft is very important.

3) Cool factor. Yes, my data is encrypted, from flash disk to server. It’s awesome, my one friend will appreciate this…

Recall Digital Oceans recent announcement that old data from past droplets might be available to new droplets as the SSDs are not scrubbed. We trust our data to unknown entities, and as in this case there was no malicious intent but potentially valuable data was exposed to the world. Fortunately they are very forthcoming and very quickly let their users in on the potential risk. It just bring home the reality that we are responsible for security, not our service providers.

Requirements

So what do we need to make our lives better? My laptop is encrypted and as such I am reasonable sure the cost of extracting the data is greater than its worth. The password is changed regularly as part of my security regime. But for a server this is not ideal.

So we need the following:

1) Encryption. This is the obvious requirement.

2) No passwords. Now here is the sticky part. How do you decrypt an encrypted partition without a TPM? The password cannot remain on the disk as it is then available to any semi computer literate person. You don’t want to type it in every time. Imagine having to log onto your VSphere client every time the VMWare servers is taken down for maintenance.

Solution

The best compromised I have come across is Mandos (https://wiki.recompile.se/wiki/Mandos).

The diagram above is from the Mandos project web site.

In order to get the password entered during boot, someone, or something needs to type it in. Now there is no reason this cannot be automated. I have a server abroad, and a server locally. It is immensely unlikely that international and local fiends will band together to compromised both servers. So using this newly found confidence the one server is used to serve passwords to the other.

And the awesome part is that this process works both ways. Once again, both servers are very unlikely to bounce at the same time so as long as one is up, the other will be able to get it’s decryption keys from the other.

The how

Now to do this, it is possible that we can just get one server to ssh in to the other and decrypt and mount a remote partition. However, Mandos has made the process very simple and added some security features to boot.

On the client side (device needing the password to decrypt and boot) we install mandos-client conveniently available in most repositories. On the server we install mandos (the server package). Unfortunately, as my machines are geographically inconvenient the ubuntu 12.04 package, version 1.4.something did not have the required networking hooks available so I ended up installing v1.5 from ubuntu 12.10.

The process below assumes an ubuntu installation with full disk encryption configured during the initial install.

Server

# dpkg -i mandos_1.5.5-1_all.deb

Edit the mandos config file to listen on a specific port (/etc/mandos/mandos.conf):

port = <port>

Client

root@hydra:~# dpkg -i mandos-client_1.5.5-1_amd64.deb

root@hydra:~# mandos-keygen –type RSA –force –password

During this process the client will ask for the password that will be used to decrypt and mount the partition. This password will be encrypted and made available to copy to the server as demonstrated below.

Back to the server

The above command will output some text, including a piece that will be familiar GnuPG. This text is copied directly to the server config file (/etc/mandos/clients.conf)

;[foo]
;
;# OpenPGP key fingerprint
;fingerprint =  7788 2722 5BA7 DE53 9C5A  7CFA 59CF F7CD BD9A 5920
;
;# This is base64-encoded binary data.  It will be decoded and sent to
;# the client matching the above fingerprint.  This should, of course,
;# be OpenPGP encrypted data, decryptable only by the client.
;secret =
;        hQIOA6QdEjBs2L/HEAf/TCyrDe5Xnm9esa+Pb/vWF9CUqfn4srzVgSu234
;        REJMVv7lBSrPE2132Lmd2gqF1HeLKDJRSVxJpt6xoWOChGHg+TMyXDxK+N
;        Xl89vGvdU1XfhKkVm9MDLOgT5ECDPysDGHFPDhqHOSu3Kaw2DWMV/iH9vz
;        3Z20erVNbdcvyBnuojcoWO/6yfB5EQO0BXp7kcyy00USA3CjD5FGZdoQGI
;        Tb8A/ar0tVA5crSQmaSotm6KmNLhrFnZ5BxX+TiE+eTUTqSloWRY6VAvqW
;        QHC7OASxK5E6RXPBuFH5IohUA2Qbk5AHt99pYvsIPX88j2rWauOokoiKZo
;        t/9leJ8VxO5l3wf/U64IH8bkPIoWmWZfd/nqh4uwGNbCgKMyT+AnvH7kMJ
;        3i7DivfWl2mKLV0PyPHUNva0VQxX6yYjcOhj1R6fCr/at8/NSLe2OhLchz
;        dC+Ls9h+kvJXgF8Sisv+Wk/1RadPLFmraRlqvJwt6Ww21LpiXqXHV2mIgq
;        WnR98YgSvUi3TJHrUQiNc9YyBzuRo0AjgG2C9qiE3FM+Y28+iQ/sR3+bFs
;        zYuZKVTObqiIslwXu7imO0cvvFRgJF/6u3HNFQ4LUTGhiM3FQmC6NNlF3/
;        vJM2hwRDMcJqDd54Twx90Wh+tYz0z7QMsK4ANXWHHWHR0JchnLWmenzbtW
;        5MHdW9AYsNJZAQSOpirE4Xi31CSlWAi9KV+cUCmWF5zOFy1x23P6PjdaRm
;        4T2zw4dxS5NswXWU0sVEXxjs6PYxuIiCTL7vdpx8QjBkrPWDrAbcMyBr2O
;        QlnHIvPzEArRQLo=

Obviously the above is the commented example in the config file. Paste your own right at the end of the file.

The client in question resides on a VMWare server in a datacenter in Johannesburg. This means, no entropy on the VM. For the client I had some fun assisting the VM with generating the required entropy – I leave that investigation up to the reader.

Server

service mandos restart

Client

At this point you should be able to test whether mandos can return your password to the client.

root@hydra:~# /usr/lib/mandos/plugins.d/mandos-client –connect=<ip address>:<port> –pubkey=/etc/keys/mandos/pubkey.txt –seckey=/etc/keys/mandos/se
ckey.txt ;echo

If it returns the password it worked, otherwise, add –debug to the above command line to get some more information regarding the problem at hand.

The next step is getting Mandos to collect the password automatically at boot. This is more tricky as networking is not yet up. This means somewhere in initramfs the network configuration needs to be specified. Fortunately Mandos comes prepared (this is why I needed to install version 1.5.5 r1).

Client (/etc/mandos/plugin-runner.conf)

–options-for=mandos-client:–connect=<server ip>:<server port>

The above line tells mandos to connect to the server on the specified port (the same port configured on the server above). Next up is networking.

We need to create a script /etc/mandos/network-hooks.d/ethernet

#!/bin/sh

set -e

do_start(){
modprobe e1000 # Substitute for your network module
ip link set dev eth0 up
ip addr add <client ip> dev eth0
ip route add <gateway ip> dev eth0
ip route add default via <gateway IP> dev eth0
}
do_stop(){
ip link set dev eth0 down
}

case “${MODE:-$1}” in
start|stop)
do_”${MODE:-$1}”
;;
files)
;;
modules)
;;
esac

I just copied the bridge script example and stripped what I don’t need. Not very elegant, but it works. Remember to chmod +x /etc/mandos/network-hooks.d/ethernet.

Client

The last step is to install an initramfs with the new parameters.

root@hydra:~# update-initramfs -k all -u

I think that was it. Now the server boots without asking for a password prompt.
As soon as I disable networking, or switch of the Mandos server, the client stops at the password prompt.

ownCloud progress

It turns out Ubuntu does not make it as easy as I expected. I was confronted with this non descriptive error message on the client when I launched and tried to sync.

There wasn’t much more to the error message on the clients. The ubuntu server apache logs sheds some more light on the issue.

[Wed Aug 14 11:58:03 2013] [error] [client w.x.y.z] script ‘/usr/share/owncloud/remote.php’ not found or unable to stat

The remote.php file is actually in another directory, but it still fails even if I link the file. The ownCloud forum has a post that indicates that the ubuntu package is broken and to rather use the downloaded package from their site, so that is what I did…

I followed the instructions on ownCloud doc installation. In summary the following is important.

Make sure all the dependencies are met, apache2, php5 etc. Most of it will likely already be on your system. If you’re not sure, you can take the easy route, apt-get install owncloud, and then remove the package. The dependencies will remain.

# tar -xjf owncloud-x.x.x.tar.bz2

# cp -r owncloud /var/vhosts/

# chown -R www-data:www-data /var/owncloud/ /var/vhosts/owncloud

I decided to host the data and web server in two difference locations in the example above. You can always choose another path or use the defaults.

Also perform the following:

# a2enmod rewrite

# a2enmod headers

You will also need to add a vhost for your site or use the default landing. Make sure AllowOverride is set to All to enable .htaccess.

Thats all there is to it. Point the client to your ownCloud service and voilà. In the background we have my browser opened to my ownCloud server, the local folder with the text file and the open text file.

Cloud storage

I experienced a small bit of frustration recently when Dropbox thought it was a good idea to delete about a third of my about 10 GB of data. Fortunately, with a bit of effort I could perform a restore on the lost data, one file at a time.

Now we use services such as Dropbox because of the convenience, however, if it is no longer convenient, or even safe, then perhaps it is time to investigate alternative diy options. Don’t get me wrong though, I will probably never get rid of Dropbox, however, I will still want to use my own service for items I consider a security risk, such as work documents.

I have been toying with the idea of hosting a backup service for my data, specifically the stuff I usually backup manually and keep in sync between my various desktops and laptop. So why not just spin up something that will replace all the cloud storage I currently use on my own server.

ownCloud

In comes ownCloud promising many features:

  • File sync through a webdav interface,
  • sync contacts, calendars and bookmarks,
  • web access to said services,
  • and an api to build your own apps around ownCloud.

Server

It seems simple enough. For the server on ubuntu it’s as simple as

apt-get install owncloud

This installs the server environment on your server, in my case, a droplet in Amsterdam.

Client

On your workstation their instructions are simple enough to add the ppa to your repository.

# echo ‘deb http://download.opensuse.org/repositories/isv:ownCloud:devel/xUbuntu_12.04/ /’ >> /etc/apt/sources.list.d/owncloud-client.list

 

# wget http://download.opensuse.org/repositories/isv:ownCloud:devel/xUbuntu_12.04/Release.key

 

# apt-key add – < Release.key

# apt-get update

# apt-get install owncloud-client

The above is valid for 12.04 LTS, so just change the 12.04 in the first instruction to your version, e.g. xUbuntu_12.04 becomes xUbuntu_13.04. Now you have both a client and server installed. There are also windows and mobile clients – I’m not that concerned with them for the moment.

Using ownCloud

Its a breeze to use ownCloud as Ubuntu has already done all the work for you. Just point to your domain, http(s)://yourdomain/owncloud

First step is to create your administrator account, but that will fail. The database needs to be configured.

# mysql -u root -p

CREATE DATABASE owncloud;

GRANT ALL ON owncloud.* TO ‘owncloud’@'localhost’ IDENTIFIED BY ‘some_password’;

Under the advanced selection you can enter the db details as configured above. Choose a suitable username and password and Bobs your uncle, you have an ownCloud server.

I’m not sure yet where the data is stored, must be somewhere sensible. I also expect to be able to change the target folder once I add more storage to the system. We’ll explore that in a follow up post.