I hit the Ubuntu documentation for how AoE was set up on a Ubuntu 10.04 virtual machine. I created a small, 10GB test disk with vSphere and attached it to my Ubuntu 10.04 vm. After a reboot, I started up the Disk Utility (System/Administration/Disk Utility),formatted the volume, and then mounted it. I noted the drive letter (sdb), as you will need it in steps 4 and 5. After that I performed the following steps in a command window:

1. sudo apt-get install vblade
2. sudo ip link set eth0 up
3. I then tested to make sure it was working:
   sudo dd if=/dev/zero of=vblade0 coast=1 be=1M
sudo vblade 1 1 eth0 vblade0
4. I exported the storage out:  sudo vbladed 0 1 eth0 /dev/sdb
5. I also edited rc.local to add the above line to startup: sudo nano /etc/rc.local

Next, I began setting up the client side (Windows XP machine).  I downloaded the Starwind Windows AoE Initiator, which required me to register first.  This application was installed on the Windows box, and started up.  I added a new device, chose the adapter, and selected the drive that was listed.  The drive suddenly was recognized by Windows as new and connected.  Very cool.

In XP’s Computer Management, I initialized the new drive and formatted as NTFS.  It took but a few seconds for the 10GB drive to complete.  Everything about this AoE-connected drive made it appear as if it was directly connected to my machine.  The whole process I’ve described to this point took a bit less than twenty minutes, not counting the few minutes to register at Starwind.  Now I wanted to see if it was as fast as local storage.

Testing
I chose a local folder that was 2.62GB in size, with 176 folders within it containing 19,000 files.  This was small enough to not take all night for the tests, but also had enough small files within to make it more challenging that simply a single, large file.  I popped up the system clock (with a second hand), and noted the start time when I pasted the file, and stopped the clock when the transfer completed. I did nothing else on my machine while the tests ran.  The results are below.

Conclusions
While these results are not overly scientific and very specific to my environment, they do give some valuable comparisons.  In my environment, AoE performed no better than an Openfiler or a common Ubuntu (samba) share. It certainly was much slower than local storage but honestly, what isn’t? If AoE had been noticeably faster than OpenFiler, I might have considered finding a use for it. As it stands, it was no faster than any other share on the ESXi box. It did take precious little time and effort to set up and test, so I consider it time well spent. In a large environment, perhaps on a SAN, the results might be different.

What to Back up?
First, though, I needed to figure out exactly what was important to copy off. Obviously the .XBMC folder needed backed up in its entirety as that contained all the XBMC-specific files. In addition, any other system file that was modified needed backed up. I’m not sure what system files were changed automatically, but the following files were changed by me:

/etc/fstab
/etc/rc.local
/sys/block/sda/queue/scheduler

The first thing I needed to do was install SMBFS on XBMC. At a command prompt type: sudo apt-get install smbfs. Respond ‘Y’es to the question, and let it run. It’ll take just a minute or two to complete.

Network drive to backup to
The data would be backed up to a share named XBMC_Backups on my OpenFiler virtual machine. To be able to mount this drive during the backup process (we only want to mount the share for the duration of the script, to minimize needless network traffic), I needed to create an entry in fstab in addition to creating a mount point.

mkdir /mnt/backup
sudo nano /etc/fstab

The entry has noauto so the share is not mounted at boot. I also used OpenFiler’s static IP address, instead of its name, to minimize potential problems for me.

We’ll want to backup a list of folders and individual files, so I created a text file named file_list.txt in /mnt/backup. This will make it easier for me to add files in the future if other important ones are uncovered. Plus it makes the script a little less confusing.

Test the commands manually
I tested the commands via the command line before adding them to the script. I first mounted the backup share:

mount /mnt/backup.

Then I ran the tar command:

tar zvc -T /mnt/backup/file_list.txt --file /mnt/backup/xbmc-`date +\%Y\%m\%d_\%H\%M\%S`.tar.gz

This took a minute or so to churn through all the files, but in the end, a shiny new tar.gz archive showed up on the network.

The script
The script itself is straightforward. I named it ./backup_XBMC_config.sh.

# The backup folder mountpoint is at /mnt/backup.
# Unmount it to ensure it is not already mounted.
umount /mnt/backup 

# Mount the OpenFiler share.
mount /mnt/backup

# Start the tar backup using gzip compression.
tar zvc -T /mnt/backup/file_list.txt --file /mnt/backup/xbmc-`date +\%Y\%m\%d_\%H\%M\%S`.tar.gz 

# Now I delete backups older than 14 days.
/usr/bin/find /mnt/backup/ -depth -mindepth 1 -mtime +14 -name *.gz -delete

# Unmount the share.
umount /mnt/backup

I then gave everyone the ability to execute the file.

sudo chmod 777 ./backup_XBMC_config.sh

I manually tested the job out.

sudo ./backup_XBMC_config.sh

Next I needed to add this job to the crontab.

sudo crontab -e

The job will run at 3am on M, W, Sat so I added the following line to the crontab file:

0 3 * * 1,3,6 ./backup_XBMC_config.sh

And that is it. The job runs three times a week, sending a gzipped file to the OpenFiler backup folder. It also deletes any file older than 14 days with a .gz extension.

In this post I will describe what changes were made to my XBMC Live installation. There are countless sites that describe how to enable TRIM, but I found one especially helpful. This site was cptl.org. Much of this post was based upon information on those pages but specifically directed at the XBMC Live 10.0 standard installation.

PuTTY
You need to get to the command line to make these changes. For me, it was easiest to just SSH into the machine. For this, I use PuTTY.

Once you enter your user id and password, you are at the command line and can do the serious work.

Enabling TRIM Support
There were two options that needed to be added to the SSD’s mount information found in /etc/fstab, “discard” and “noatime”. What the “discard” option does is tell Linux that you wish to enable TRIM support. “Noatime” prevents Linux from updating the last-access time for files and directories, which saves a ton of writes.

sudo nano /etc/fstab

Add "noatime,discard" to the /dev/sda1 entry, then save and exit the file.

Next, you can remount the SSD.

mount -oremount /dev/sda1

Swappiness and noop disk scheduler
The swap file is not really needed on my system. XBMC uses only a small percentage of the available RAM, and I don’t plan to use hibernate due as 1) the machine boots in less than 20 seconds, and 2) it is on 24/7. To keep Linux from swapping to disk except in the most extreme circumstance, I changed the “swappiness” setting to 1 (0-100 is the range) from the default of 60.

There are a variety of disk schedulers Linux can use. I am using a solid-state device, where all seeks take exactly the same time. Consequently, using a scheduler that reorders I/O requests to group by location on the physical drive does not make sense. I changed the default scheduler, cfq, to be “noop”. Noop disk scheduler simply inserts all requests in FIFO order.

sudo nano /etc/rc.local

Add the lines:

echo 1 > /proc/sys/vm/swappiness
echo noop > /sys/block/sda/queue/scheduler

Save and exit the file. This file will get executed each time the machine reboots.

These few tweaks should ensure my solid state drive stays fast for many years.

What I discovered was that the media capabilities of the receivers (at least in the sub-$1k range) left a lot to be desired. The Onkyo I initially gravitated toward had a text-only interface, and would have required way too much scrolling through lists. I dropped down one notch in the product line, eventually deciding on the Onkyo SR-608. This model offered 1080p video-switching, a ton of inputs, and quality sound/video. The difference in costs of the two models would be used to pay for a proper media-serving option.

There are many plug-and-play options out there for serving and/or playing your media: Squeezebox, Popcorn Hour, and Boxee, to name just a few. Although these were popular choices, I quickly decided that a “more flexible” alternative was necessary. By flexible, I meant “build it yourself”.

The Atom processors had spawned Netbooks, and were now being used in “nettops“. There were now small nettop boxes that were being used in home theater applications. I just needed to find one that met my needs. A buddy pointed me to XBMC for the frontend. There is even a live version of XBMC which wraps a Linux-blanket around the app, and makes it a standalone OS product. The XBMC forums often recommended the Zotac ION products. After additional research, the ION/XBMC path seemed the best solution for my situation. Small, very low power, able to play high definition content without problems, and a wealth of threads discussing it. It certainly sounded like a winner.

Hardware
After a little more research, I decided on the Zotac ZBOX HD-ND02. It came without harddrive or RAM. The Muskin 2GB stick of RAM looked nice, and the forums liked pairing solid state devices (SSDs) with the Zotac/XBMC combination. According to them, it made moving through large lists much, much quicker than from a standard, spinning harddrive. This made sense, so I grabbed a 16GB SSD, too (it was a Newegg combo with the Zotac). The 16GB would be plenty as all the media would actually be located elsewhere on the network. Finally, this PC would be headless and not visible, so I’d need a remote to control it. The MCE remotes work with XBMC right out of the box, so I chose the AVS Gear GP-IR01BK.

Next up, I needed to get the XBMC Live onto a bootable USB thumbdrive. The Zotac does not have an internal CD and I’d need a way to install the OS.

XBMC Live
I downloaded XBMC Live 10.0 “Dharma” ISO from the XBMC site. To create the bootable USB, I turned to UNetbootin. This handy little utility can run on my XP machine, and will take an ISO image and quickly generate a bootable USB thumbdrive. I just inserted a spare USB drive, and pointed the application to it and the ISO location. Click OK, and let it perform its magic. When it is complete, don’t click Reboot Now.

The Zotac
Setting up the Zotac was a breeze. I installed the RAM, the SSD, and that was it. The interior of the Zotac was nice and compact, but easy to work in.

I connected the all the cables. Once it was powered on, pressing Del took me to the BIOS. Several settings needed changed and, of course, I wrote none done. Memory serves that ACHI was enabled, hardware graphics was set, and set the initial boot device to USB. 512MB (from the default 256) of memory was assigned to video.

I saved all the BIOS changes, plugged the USB drive in, and rebooted the machine.

Install XBMC Live
The machine quickly posted and the menu came up. I selected to install it to the HD. I kept all the default settings throughout the install, and selected an appropriate username and password. Once done, I removed the USB drive and rebooted. This process was exceedingly simple.

Configuration
You can configure (almost) everything in this release of XBMC from the graphical interface. I understand that previous releases required some tinkering “under the hood”, so it was fortunate I picked this release to work with.

Everything can be tweaked. I stuck with the basics in mine: zipcode for the weather, and adding media sources. Music was one path, movies was a combination of two (MKV one, and .avi storage for the other), and TV shows was another. This would work for me begin with.

Once I was satisfied all the shares were mapped, I powered down the device, unhooked everything, and took it to its final resting place, the main living area.

Testing it all out
The first thing that I noticed was that the screen was missing some vital information. The screen resolution was set at 1280×720 (my TV is 720p) but not all the user interface was visible (in fact, much was off the screen). It took me a few minutes to find the video calibration tool. It is at System -> Settings -> System -> Video Output -> Video Calibration. Be sure to step through all the screens. This fixed my issue.

My initial reaction was how peppy everything was. Moving from screen to screen was as quick as it should have been. Quite nice. Once I went to test a CD, it became apparent there was no audio. A quick Google search showed this is a common problem. To fix this in XBMC Live version 10.0, I simply went to Programs, ran the audio mixer plug-in, and unmuted all the channels. I had already set the audio output to HDMI in the system settings. Suddenly we had wonderful sound.

My wife came into the room and immediately commented upon how nice the system sounded. She was hooked.

Final thoughts
We have been playing with the Zotac for about three weeks and are greatly enjoying it. The sound quality from it though the Onkyo has been amazing. We had been listening to compressed music (eg: MP3s) for so many years on inferior PC 3-ways, it was eye opening to hear CD quality sound once again, and on a quality sound system. The convenience of having all the CDs available from the couch cannot be overstated. The only downside: my wife has trouble deciding what she wants to listen to.

I was initially concerned with only ordering 2GB of RAM. This ended up being a non-issue. The Zotac spends most of its time using less than 300MB of memory.

CPU utilization ended up being less than expected. Idling, it uses just 26% of the capacity, not much more when playing media. It seems the video chip does its job very well, leaving the Atom relatively unstressed.

I plan to spend time in the coming weeks poking around more in the XBMC settings and menus. There is an incredible wealth of options available, but the beauty is that you don’t need to dig in very far to have a fully functioning system. My next few posts will dig a little deeper into XBMC and my setup.

Vortexbox as a virtual machine
Having the Vortexbox virtual machine use my primary machine’s CD drive as input, and have the vm doing all the heavy lifting turned out to work great. I simply kept a stack of CDs to be ripped on the desk and, as each CD was spit out by Vortexbox, the next one in line was slapped into the tray and closed. The length of time to process each CD varied on CD length. The classical discs took the longest, generally in the 20-24 minutes range. The (shorter) rock CDs often took less then 14 minutes. The length of time did not bother me in the least. I knew the output would be the best quality it could be, and that this was the last time I’d have to touch the CD.

On the rare occasion, Vortexbox would spit out a disc just a few seconds after it was fed in. In every case, just closing it again was enough for it to accept the disc. Other than that nit, the ripping/encoding process “just worked”. The virtual machine rarely used much of the two CPUs allocated (a combined maximum of 1750Mhz when encoding), and rarely used more than 100MB of memory. I would 100% recommend it to anyone looking to rip their personal music collection.

Cover art
After Vortexbox rips and encodes each CD, it goes out to the Internet to attempt to find the correct cover art for the album. My experience was that it did a fairly good job but it did make mistakes, and sometimes returned no image at all. Some of the missteps can likely be attributed to the age and my specific CDs. A good portion of my music is from the late 80s through mid-90s, and some of it just is not popular enough these days to have images on Amazon. The rock compilations suffered the most; Vortexbox found almost none of them. This was often due to the fact there just were no hits on Google for these albums. I ended up just scanning the covers myself, and creating 500×500 images and placing them in the correct folders. I did this on about thirty discs (some of which are shown below). It wasn’t the perfect solution but was good enough.

In some instances, the cover art that was found was quite small. Rather than hunt through all the folders manually, I turned to another piece of software to do the searching for me.

BlissHQ
I used BlissHQ running on a Windows 2003 server virtual machine to hunt down cover art that was missing, or was too small (I chose 300×300 as the minimum desirable size). I discovered this software when hunting down methods to efficiently rip my music collection. His excellent three-part discussion of his “Vortexbox as vm” solution was what got me to where I am today. I have communicated with Dan (the developer) a bit during this process and he is a nice guy to deal with, quick to respond and helpful.

BlissHQ was an easy install, and uses a browser as its interface. You can select the settings for a ‘rule’, and then apply this rule to your collection. It can make changes automatically, or allow you to manually make changes (which is what I did). For instance, my rule was that the image sizes had to be at least 300×300. Once you apply (run) the rule, the application will spin through your collection, determining which albums meet your criteria (compliant) or do not (uncompliant). This took a while but when it completed, I could choose to view only the uncompliant albums and go about fixing them. This was done by either 1) clicking on the album name, and finding more artwork, or 2) clicking the little pencil icon you see as you drift over the album. This triggers Bliss to hit the Internet and give you some other artwork options. See one that is better than yours, and you select to use it instead. Most of my replacements ended up coming from Google.

Bliss allows for trial use, but is fee-based after that. It will make changes to 100 items for free for you to test out its functionality. Anything beyond that and you’ll need to purchase “fixes”, either in batches of 1000 or a unlimited license. The unlimited license fee is reasonable and I could see most people opting for that.

I initially had some issues with the product (the browser would just chew up the CPU and the GUI appeared broken) but it turned out IE6 was actually the problem. Dan replied to this post and pointed me in the right direction. I installed Firefox 3.6 and my Bliss experience was completely different. I like it. It displays twelve albums per page, and you can scroll through them fairly quickly. In about twenty minutes I had gone back through my collection and made the couple changes that had been needed. Bliss is a good product and well worth a look.

Filename cleanup and the Various folders
The names of folders generated by Vortexbox during the ripping process resulted in some confusing situations. Classical discs were one example. CDs with Mozart’s music might have Mozart somewhere in the title, but often the title included the orchestra instead. This would have led to confusion and a mess. To make things easier, I created a number of Various folders. Various- Classical, Various- Kid’s Music, Various- Jazz, Various- Soundtracks, etc were created to make hunting down music a little easier. This has proven to be a great help.

Backups
Anyone who knows me knows I preach “backups, backups, backups”, preferably automated. Although I did discuss how I backed the music up in the previous post, it bears repeating.

On the backup machine, I created a folder named Music Backups on a drive that had about 500GB free. Beneath that were placed the flac and mp3 folders. Next, a simple batch job was created on that PC, that would use XCOPY to move any new files from the OpenFiler share to this newly created path. The XCOPY statements I used were:

xcopy "s:\Music\flac\*.*" "L:\Music Backups\flac\" /s /k /d /y
xcopy "s:\Music\mp3\*.*" "L:\Music Backups\mp3\" /s /k /d /y

The batch job was set up as a scheduled task to run each morning at 2:30, a time when I am busy sawing logs.

I still need to create a backup to external media and get that copy moved offsite to be fully protected.

Nearing completion
I still have a batch of classical CDs to rip but for the most part this project is done. It was a long road to get to this point but it is for the better. Technology has improved greatly since I started ripping in 2006. Virtual machines were basically unheard of back then, and now that is how this media is being served. Storage costs are much different as well; 300GB back then was certainly different than today.

The main goal of this project has been met: To be able to listen to all of our CDs (in glorious lossless format) on our main stereo setup, with future expansion to other rooms a trivial process. In addition, we now have a true backup of all our music.

I’ll describe my receiver upgrade and media HTPC setup in the next post.

[Sidenote: I found some great stackable bins by Homz at the local Walmart that hold 30 discs each. These made quick work of organizing the rock discs alphabetically. I now have them all in one location on our 'wall-o-shelves', stacked and neatly labeled. All the classicals found a home in a grapefruit case. Hey, it works.]

Ideally, I wanted to touch each disc only one more time before packing it away for good; I would want it archived in FLAC lossless format as well as a higher-bitrate MP3 for my wife’s player. The FLAC would future-proof me for when I eventually get a high quality media player. I initially figured I would have to rip each one to FLAC manually and then run a script to convert the FLAC to MP3 and place it in a different folder. I did a few cursory web searches, and then put it on the mental back burner for about a week.

Vortexbox
Late last week I happened upon Vortexbox. The product is a standalone operating system , running a modified version of Fedora Release 14. It is free, and is downloaded as a 635+MB ISO. From their site:

“VortexBox is a free, open source (GPL v3), quick-install ISO that turns your unused computer into an easy-to-use music server/jukebox. Once VortexBox has been loaded on an unused PC, it will automatically rip CDs to FLAC and MP3 files, ID3 tag the files , and download the cover art. Vortexbox will then serve the files to network media players such as Logitech Squeezebox, Sonos, or Linn. The music files can also be streamed to a Windows or Mac OSX system.“

What really caught my eye was that it was a “complete OS“. Which meant it could run as a virtual machine on my ESXi box and not consume any resources on my primary machine. A few brief searches later confirmed people were doing this very thing, and that you can redirect the music folders to a NAS or SAN device. In my situation, that would be OpenFiler. The final item that sold it was that I could use the CD drawer on the client machine to rip from, as the VMWare server is on the far side of the room, up on a shelf. Not exactly a convenient location when one plans to rip another 100+ CDs.

I was sold. Now it was time to get it installed and running.

Getting started
I began by downloading the ISO from the Vortexbox site. The file was then uploaded to my ESXi datastore.

I created a new virtual machine with a 10GB boot drive, 512RAM and two vCPUs. I knew that the encoding would benefit from dual processors. I pointed the CD drive to the ISO I had just uploaded, and started up the machine.

The GUI interface is accessed from a web browser. You’ll need to know what IP address the vm was given when it started. As this could vary each time it rebooted, I decided to make the ip address static. I opened the VSphere Vortexbox console and then followed the excellent instructions on the Vortexbox site. Now the ip address is static.

The FLAC and MP3 files were going on OpenFiler, so I created a new folder, Music, on my OpenFiler CommonStorage share, with two folders beneath, flac and mp3 (note the case of the folder names).

Vortexbox has locations it assumes the FLAC and MP3 files will be. I wished to redirect these from the 10GB boot drive to the new OpenFiler shares. To do this, I needed to connect to the Vortexbox console again and edit the fstab file. This will have the share automatically mount at boot. I found how to do this on the Vortexbox forums.

nano /etc/fstab

I then added the following line:

//192.168.0.220/CommonStorage/Music /storage/music cifs guest,_netdev 0 0

This line pointed the default music folder (/storage/music) to my newly created Openfiler share (//192.168.0.220/CommonStorage/Music). I saved and exited the fstab file.

I manually mounted the path:

mount /storage/music

Now the default paths pointed to my OpenFiler shares. This was a much simpler process than I had originally feared.

The OpenFiler shares did need more space allocated to them. I added more space to the OF volume, and bumped guest quotas up, to allocate another 150GB to the shares. This gave me plenty of room for the existing FLAC files, and likely enough space for the rest of my CD collection. If not, I will just repeat the process to make more space available.

Next, I moved the FLAC folders to the S:\CommonStorage\Music\flac\ path. This was projected to take a while so I just started the copy, and left to do other things.

Once the task completed, it was time to get all the cover art. This process is almost too simple to believe. Go to the browser’s GUI interface, and click on the second icon from the top. On the CD ripping status and cover art screen, simply click on the Get Cover Art button and let it do its thing. It did not take very long, and ended up with a cover for almost every album.

Now that the cover art was in place, it was time to let the Squeezebox server scan the flac folder and build its tables. I selected Clear and rescan for changes from the drop down box, and started the scan. This was another good time to walk away to find something else to do for a while.

Finally, it was time to create MP3 mirrors of all the FLAC files. I clicked on the FLAC mirror icon (very cute) to take me to the mirror screen. I kept the bitrate at the standard 190Kbits VBR, and clicked Start mp3 mirroring. This ended up being the longest process. That made sense as it was having to encode over 4,500 songs. Just walk away and let it work. We had some places to go for most of the day, and the process was complete when we returned. All the FLAC files had MP3 copies. Sweet!

Setting up the Backups
Now that everything was setup and working, I needed to ensure it was backed up initially, and going forward. This is a critical step to ensure all my hard work didn’t just go to waste with a hard drive failure. I needed to backup the FLAC and MP3 folders nightly, as well as back up the Vortexbox vm a couple times a week (this would be plenty). The vm wouldn’t have to be backed up due to how easily it was installed, but setting up the backup would be a five minute operation and could potentially save me many times that.

On the backup machine, I created a folder named Music Backups on a drive that had about 500GB free. Beneath that were placed the flac and mp3 folders. Next, a simple batch job was created on that PC, that would use XCOPY to move any new files from the OpenFiler share to this newly created path. The XCOPY statements I used were:

xcopy "s:\Music\flac\*.*" "L:\Music Backups\flac\" /s /k /d /y

xcopy "s:\Music\mp3\*.*" "L:\Music Backups\mp3\" /s /k /d /y

The batch job was set up as a scheduled task to run each morning at 2:30, a time when I am busy sawing logs.

I created a new ghettoVCB job, and another batch file (on the backup machine) to call it. I scheduled this to run later in the night twice a week.

Before I called it a day, I fired off both tasks manually just to get the initial backups started.

VMWare Tools
You will need to install VMWare Tools within Vortexbox to be able to rip CDs from the client CD drive. I found an excellent post that worked perfectly. No reason rehashing it; just go read the post for the details.

How to rip from the client’s drive
I have had time to test rip two CDs. What I had to do to get it to work was change the device type of the CD to Client.

After receiving an error when attempting the initial rip, I had to also change the CD mode to Passthrough IDE (Recommended).

Lastly, I connected the CD drive on the Vortexbox console to my primary machine’s J drive (its CD tray). I put a music CD in the drive, and Vortexbox took off.

It took 10 minutes to rip the disc, and two minutes to encode. It also went out and grabbed the artwork for me at some point in the process. When it was done ripping, Vortexbox opened the CD drawer for the next disc. Very slick and simple.

When you are finished your ripping session, be sure to disconnect the client’s drive from the Vortexbox VMware console. This is rather important.

Ripping Folder Structure
Something I noticed after ripping the first disc was that the folder name was not what I expected. It was in a artist-album/title.flac format, rather than the artist/album/title.flac I was expecting. A quick web search discovered how to solve this issue. According to the Vortexbox forum thread, you need to:

1. Log into your VortexBox as root.

2. Edit /etc/ripit/config, and change

dirtemplate="$suffix/$artist – $album"

to

dirtemplate="$suffix/$artist/$album"

This worked for me, and the second disc was ripped to the correct folder name.

Summary
Vortexbox looks to be the solution to my storage and ripping problems. It was pleasantly simple to install as a virtual machine and get set up. The Vortexbox forums provided some excellent help for the complicated pieces (VMWare Tools install would not have been just a ten minute affair otherwise). The Squeezebox server that comes installed will be fun to play with as I get time. I have also been scoping out potential home receivers and Vortexbox having a DLNA server included looks to be a very important feature.

Give it a try if you have a bit of time. All said, I spent about four hours from start to finish installing, configuring, hunting down solutions, and setting up the backups. Now I can rip the remainder of our music as I have time.

Looking at the poor, unused drive a few weeks back made me wonder if I had room to put it into my Windows backup machine. That headless PC currently has a CD drive, three IDE and one SATA drive. There was room, but I would have to shift things around. The CD got lowered to the bottom slot, and the new addition was placed next to the 1TB Samsung green drive in the external 3.5″ bay. Everything was hooked up, and then it was time to boot it up. I pressed the power button, got an initial hum, then nothing more, save for the power light flashing at me. I flipped the back power switch off, checked all the connections, and then tried again. Still nothing. I reasoned I had a 350 watt power supply in there so that should be plenty. I tried again, and it did boot. Success! I let it finish booting, then VNCed in and restarted the machine. Wouldn’t boot.

I dug out my handy Kill a Watt load meter and connected the machine’s plug to it. The machine idled at 110 watts, significantly less than 350. At initial power up, the machine did spike at 194 watts. I powered it back down, reconnected the fifth drive, and restarted. The machine spiked at 209 watts, an increase of 15 watts for that fifth drive. Once it fully booted, the idle load was 8W higher. Out of curiosity, I hit Newegg for the Rosewill’s specs. It appears that the power supply is rated at a “minimum of 60% efficiency at peak load”. Some quick math (350W * .6 = 210W) showed that I was requesting right at what the supply was rated for.

My thought was that this machine had been rock-solid with the existing configuration for the past four years, and now was hit and miss on boot with a fifth drive. Long-term stability seemed to be uncertain, so I powered it all down, and pulled out the drive.

I briefly considered ordering a new power supply, either a standard one with higher capacity or another 80+% efficiency model my last two machines have used. The frugal side of my personality won out; I could darn near buy another 1TB drive for the ESXi box for the price in would take to replace a perfectly functioning power supply, just to have the 320GB drive off my desk.

Further research showed that power supplies do ‘age’ over time and lose some of their efficiency. Another tidbit I found out there was is a strong correlation between weight of the PSU and the quality of it. The reasoning is that quality components just weight more. I saw that the Rosewill weighed 1.8 lbs, and the Antec used in my quads weighed 3.6 lbs.

I am posting this for the dual-purpose of showing that a entry-level 350W power supply can successfully run an XP1900+ build with four hard drives, but also that that is about its capacity. I fear a fifth drive would induce some eventual failure in another component, and that would not be good. Despite ending back where I started in the process, this was a good exercise to go through.

Update
I’d had need to reboot this machine several times in the last week, and the PC would not start consistently. Just a hum, and then the flashing power light. Apparently the power supply was on its last legs. The Antec Earthwatts 380W Green power supply was ordered and replaced it. I was able to install the old external drive into the case, and the PC now idles at 102 watts. The initial spike of 172 watts is also much lower than the 209 of the Rosewill. This new supply is also quite a bit quieter than the old.

An excellent document on the vSphere cpu scheduler performance can be found on the VMWare site.  Another one, Determining if multiple virtual CPUs are causing performance issues, is worth a read as well.

Power Policy Settings
The Power Policy settings can be reached from the VSphere client;  Configuration -> Power Management -> Properties.

Testing Performance
I used nineteen AVI files of my children playing soccer to feed DVDFlick, and created an AVI file as output. The input files were on the OpenFiler virtual appliance, and the output went to a partition on another local drive. This was to eliminate disk contention from the equations. I set DVDFlick’s encoder priority to Normal, and the tests were run on a Windows 2003 Server virtual machine. DVDFlick will effectively use any amount of CPU cores made available to it, so I ran the test using one through four vCPUs. DVDFlick also allows you to select how many threads to run with, so that was varied during the run as well.

I originally did not plan to not test with more than two vCPUs as my server has only four cores, but decided to give it a go while everything was set up. I would not normally allocate more than two vCPUs to any active vm on my VMWare box as it would negatively affect the other active vms. But this was a test, so all four were included in the results.

I ran each test twice and averaged the two runs until it was apparent that each set varied by no more than a few seconds of each other. At that point, I decided only one run per set was needed. The results gave me some nice data for comparing the three settings.

The Power Draw column indicates the average draw during the encoding phase. It is provided as a relative value for comparisons.

Initial observations
The power usage varied little between the three options when the test was running. The charts also show that going from one to two vCPUs to the virtual machine increased the total by just 10W in each case, and each subsequent vCPU only a couple more watts.

Anytime more threads were used than CPUs allocated, performance suffered. This is not a big surprise, and even DVDFlick cautions against this practice. Performance suffered equally when you went the other direction; running with fewer threads than cores. Obviously, you would want to run with the same number of threads as cores.

Power usage was effectively the same across all the settings when encoding. I would have expected ratios more resembling the idle stats, but that is why I ran these tests.

Encoding times for the Balanced setting were very close to that of High Performance, most within 1%. VMWare states the Balanced setting “reduces energy consumption without sacrificing performance.” From my tests, this statement is 100% true.

The Lower Power encoding times were quite a bit slower than the other two settings. Four cores/four threads took three minutes longer (12.5%) to complete than the other two settings, but this was what VMWare cautions would occur. These results suggest there is no compelling reason to ever use this setting. There was almost no difference in power usage but imparted a huge performance penalty.

Four threads/four vCPUs was a third faster than one/one on Balanced and High Performance settings.

Comparisons to my Desktop
While in the mood to test, I ran two benchmarks on my desktop. That machine runs XP on a Q6600 Quad-core with 4GB installed. I rebooted the machine first and didn’t open anything additional post-boot. The same test bed was used as above.

The first benchmark, from OpenFiler virtual machine to a local drive, was almost exactly the same time as through the Server 2003 vm using either Balanced and Performance settings. I was very pleased, until I saw the results from the second test. Local drive A to local drive B was quite a bit faster on the desktop. The 2003 vm should have been doing almost the same thing. Now I needed to test the vm using input and output partitions on two separate physical drives in the ESXi box, and skip OpenFiler altogether. By eliminating OF from the equation, we’ll see if that was the bottleneck.

Removing OpenFiler as input source
I changed the drives to be on 10GB partitions within the two local drives; a Western Digital WD6400AAKS and a Samsung F3 HD103SJ. I used all four threads on the four vCPU virtual machine. I, again, ran each test twice and averaged the scores. The target drive was swapped for the second set.

By eliminating OpenFiler, we reduced runtime by 3-1/2 minutes! This was another result I was not expecting. I will have to investigate this further in the future.

Conclusion
Using multiple vCPUs for virtual machines that can take advantage of multiple threads can definitely increase performance. Going from one virtual CPU to two in my tests showed the “biggest bang for the buck”. It bears repeating that there were no other virtual machines contending for resources during my testing. If there had been, the outcome might have been affected.

I have vSphere set to the Balanced Power setting. It reduces my power utilization by 15% during all the time it is idle, and does not sacrifice any performance.

These new settings can be reached from the VSphere client;
Configuration -> Power Management -> Properties.

Power Usage at Idle
I began my testing by checking the top three settings at idle. (I decided not to even mess with the custom settings.) The ever useful Kill A Watt load meter was used to measure the server’s power draw. The beauty of these settings are they can be changed on the fly, and the effects seen immediately.

Results
As you can see in the chart below, the Balanced setting reduced the power usage by almost 15%. The Low power setting didn’t appear to make any further savings. In my specific situation, where the machine will be essentially idle 95% of the time, saving 15% is a big deal. I have the server set to Balanced now, and will enjoy the feeling of saving a few cents an hour on the power bill.

My next post will look at power utilization during a test video encoding job, as well as testing how much adding addition vCPUs to the virtual machine affects performance.

My solution
Research pointed me toward the Advanced Software settings in VSphere. Under NFS I changed the following three settings:

I based these values on EMC and NetApp recommendations, except I went crazy with the NFS.HeartbeatTimeout; they only recommend 5. I went with a larger value as my NFS share is much slower than these vendor’s fine products.

These values work as such (cribbed from this post):

• Every “NFS.HeartbeatFrequency” (or 12 seconds) the ESXi server checks to see that the NFS datastore is reachable.
• Those heartbeats expire after “NFS.HeartbeatTimeout” (or 20 seconds), after which another heartbeat is sent.
• If “NFS.HeartbeatMaxFailures” (or 10) heartbeats fail in a row, the datastore is marked as unavailable (“inactive”).

It appeared that ESXi stopped trying to connect to the NSF share before the physical connection was even available. By upping the low defaults, this appears to have corrected the issue. Four reboots and the datastore is still available.