Mike Trello's Blog

My ESXi server has been faithfully serving me for the past five years. As noted previously, it has just 8GB of ram installed (4x2GB). No, you did not misread that; eight gigabytes. This has been fine for my needs as the virtual machines running are not ram-hungry. The 2003 server, vortexbox, Openmediavault and an XubuntuLTS release comfortably run in less then 6GB. Now, however, I have begun to test out Windows 10 and it would be nice to have more memory to devote to the machine(s).

I could have just built a new machine, with a bunch more RAM, but that would have been unnecessary at this time. The server is rarely ever taxed CPU-wise; most of the times it is using much less than 1Ghz in total. Harddrive performance is not an issue as there are five physical drives, and each vm is directed to its own drive. And as there is 10TB of capacity in there… well, available space certainly is not an immediate concern. So there was no compelling reason to upgrade anything other than memory.

I don’t know if you’ve noticed but DDR2 is no longer in demand, it being eclipsed years back by DDR3 (and now that is being overrun by DDR4). The major suppliers have inflated DDR2 to silly levels. So I turned to ebay to see what could be found. A Chinese supplier was selling new Samsung chips for a reasonable price. I ended up paying $51 for 16GB (4x4GB), the maximum my motherboard could support. It arrived in about ten days.

RAM
This should have ended with me having 16GB of ram, and enjoying multiple Win 10 test machines. But that was not to be. Everything went south from here. You can read the details of that story here.

I will be sticking with the existing 8GB until I eventually do a machine upgrade down the road.

I began seeing an error in my vkernel logs in ESXi.

The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? This will also fix this last usable sector as per the new size (7814037168) AlternateLBA (3519069871) LastUsableLBA (3519069838).

I had created the partition using a GParted Live CD, with the harddrive slaved into one of my Windows boxes. Then I removed the drive and added it to my ESXi server case, and created and formatted the drive as VMFS5. It showed the full volume space in vSphere so I assumed it had worked. I noticed, after the drive was filled to a certain degree, I could no longer copy any large files to it. It just wouldn’t work. I could read files just fine, but writing was not possible. It was then that I checked the logs and immediately spotted the recurring error messages.

After a bit of googling, I discovered there was a partedUtil fixGpt function withing ESXi. I SSHed into the host after first jotting down the datastore device location from vSphere (Configuration / Storage / Datastore Details).

I first checked the device from within the console: partedUtil getptbl

Next I ran the fix partedUtil fixGpt. It prompts you to type ‘Fix’. Don’t do like I did the first three times and only type ‘F’. It won’t work.
I then immediately ran the check command again, getting a clean bill of heath this time.

The fixGPT command took only a second to work and did exactly what it was supposed to do. No more errors in the logs, and the entire drive’s capacity can be utilized.

Background
Openfiler had been serving me reliably for about four years but had begun to seriously slow down as my media pool grew. My project of ripping all my DVDs and BluRays to the pool had slowly increased the pool size. The data was spread across three independent drives in the server (I don’t use raid in that box).

I could increase storage as needed, it always worked, it sipped resources. Exactly what you want in storage software. The only problem was it was beginning to slow down. When I did larger transfers, it would start out at a reasonable pace but gradually slow as the transfer continued, usually down to 12-17MBs. On every day processes, this was not very noticeable, but anytime I needed to do a large transfer for any reason (backups to an external for an example), it became painfully apparent.

Now would be a perfect time to test drive another product. I spent a good amount of time on google, investigating options. But what I kept coming back to was a product called Open Media Vault. Strangely enough, when I looked back at my notes from three years ago, it was down to OMV or openfiler. Openfiler got the nod as it was much more mature, and OMV had not been out long enough to prove itself.

Benchmarks of existing infrastructure
I wished to first get an idea of how he existing OSes performed with data transfers. I used Xubuntu14, Win2003 and openfiler vms, plus the Windows backup box and my primary Win 7 machine. I tested with three datasets, and averaged the three runs. The first was a 69GB set of 98 files in 12 folders, the second was a 12GB family video, and the third was a five DVD rip block (25GB).

Test one was my from Windows PC to the various locations.

To Win 2003 VM: 61 MB/s
To openfiler: 23 MB/s always decreasing to 18
To xubuntu 14 LTS: 58 MB/s
To Backup PC: 28 MB /s

I then tested it the other direction from the vms to the Win 7 PC:
From 2003 vm: 70 MB/s
From openfiler: 38 down to 28 MB/s
From Xubuntu: 89 MB/s

Now I had an idea what performance the existing drives could give. Openfiler was significantly behind the rest of the pack on the vmware box.

Open Media Vault (OMV)
I decided to give OMV a spin on the existing storage (to give me a fair comparison to the existing openfiler). I downloaded the ISO image of OMV and installed (per their recommendations) into a virtual machine. I ran through the following steps:

1. Installed the Logical Volume Manager (LVM) plug-in
2. Created three small vmdk drives on the three physical drives (about 300GB in total)
3. Added the drives to the physical volume (PV)
4. created a volume group (VG)
5. added the PVs to the VG
6. Add the VG to the LVM
7. Created a filesystem (ext4) on the LVM
8. Created a shared folder and SMB share on the LVM
9. Created two users to test with and assigned rights

This sounds like a lot of steps (and I might have skipped one or two) but it is logical and goes quickly. Now that I had some storage defined, I could do some testing. But…

In the meantime…
I knew that more storage would be needed on the server. Ideally, I would keep openfiler up and running during the transition. That would cover my backside, in case unforeseen issues popped up, and would allow me more time to get the settings figured out.

So I added more storage to my ESXi server, the Seagate 3TB NAS internal drive. I’m not necessarily recommending this drive as the first one I got was a dud, and it took a while to prove it to Seagate. But, in the end, I have one that is working just fine. (By moving to a drive larger than 2TB required me to upgrade ESXi from 4.1 to 5.1 but that was a trivial process.)

Now back to testing.
I ran many tests to and from the OMV pool. What I saw was a consistent 44MB/s write speed when dealing with a large number of small files, and about 56 MB/s when moving large files. Downloading large files from OMV to my Win 7 machine showed a nice 86 MB/s throughput. A bunch of small files yielded 75 MB/s. And what was nice to see is that the throughput remained quite consistent throughout the entire transfer (plus or minus 3 MB/s). I was suitably impressed and made the decision to move forward with the transition.

Transition
I learned a lot of how OMV worked through the test setup, so I applied the knowledge to setting up the production configuration. I installed and set it up on the new Seagate NAS drive. I ended up created a single volume group and created three shares from that pool.

I could then map the shares to whatever PCs they were needed from.

Some great features
OMV had some very cool features that OF had always lacked. The first was the ability to create users and give very granular rights. In addition, user home folders could be automatically generated (which is great in our house). The second was the integrated recycle bin. Every time you deleted something from the pool (either on purpose or by accident), it moved it to a recycle bin with the user’s folder. Very, very nice!

Fast Forward a Year
All this occurred back in January of 2014. OMV has been serving us very well. We have increased the amount of devices on the network this past year (phones, another two Rokus, another two Nooks), so it often happens that three (or more) devices are accessing data from OMV at the same time. No hiccups, no issues. Incremental backups go so much faster now. I have been very happy with Open Media Vault, and would definitely recommend it.

Roku encoding all our movies
A couple weeks ago we installed a Roku with a Plex channel on it so we can access our home media archive from the bedroom TV (and our mobile devices). I have discovered over the past two weeks that none of our files can be direct-played on the Roku; they all need re-encoded. I have perused the forums and followed a number of procedures to convert a test file to be playable directly, without encoding. All without success. The realization hit me that time was being wasted, to just accept the fact they’ll need encoding on the fly. We watched a number of movies without a single hiccup so I figured it would be fine. The CPU on the virtual machine worked hard but it was keeping up.

ESXi server over taxed
Three nights ago I was backing up a large vm using ghettoVCB. My wife decided to watch a local movie on the Roku. Several times during the movie (the parts when I was actually awake) the picture “garbaged” up for several seconds at a time. It was very noticeable and bothered me, and I knew what was causing it. The plex media server was reading from the openfiler virtual SAN and furiously encoding the movie, while the ghettoVCB job was hitting the same discs backing up 300GB of data to the backup server. The ESXi box was just too stressed and was occasionally hiccuping.

Now I’ll say right now the problem could have been easily solved by just throwing money at it. An ESXi server upgrade (to a single Xeon chip or perhaps dual chips) would have certainly eliminated the CPU bottleneck. But we are not made of money (although a new server would certainly be fun). Another option would be to ignore the problem but there are already times where the kids are streaming to two devices so it is only a matter of time before a third device gets added and everything pixelates again. There had to be another answer.

This problem percolated in my head the next day until a solution presented itself. Sitting right next to the VMware server is my backup machine, an AMD 3.3Ghz tri-core with 8GB of RAM. Aside from backups that occur in the dead of night, it just sits there, patiently idling. The thought was that there was a perfectly good machine for a plex server.

Now, I didn’t want to just move PMS from the virtual machine to the backup PC. I decided to create another PMS instance on the backup server and leave the original instance running on the VM. I renamed the original plex server ‘plexserver_mobile‘ and pointed the Nook Color at it. This instance pointed Music to the MP3 mirror of our FLAC collection, as the NC cannot handle FLAC files nor would they sound any better than MP3 on its microscopic speakers The new PMS instance, which I named ‘plexserver_media‘, instead pointed Music to the FLAC folder. The Roku’s audio goes into my older Onkyo receiver so we could benefit from the higher fidelity FLAC brings. Other than that, everything else was the same. I pointed the Roku to the new instance and it was happy. Now I have more horsepower driving the Roku’s encoding, and have eliminated possible contention from mobile devices. For. Zero. Dollars.

While slower than the XBMC box (with its locally cached thumbnails and database on SSD), the Roku/Plex combination has proven itself to be quite good. The ability to watch our movies (and other content, such as streamed HGTV) in the bedroom is nice. The incremental cost was less than $60 (which includes the long cat5 run used for the Poor Man’s WAP that cured the weak wireless signal), which was a fantastic value.

A couple weeks ago I added a Roku to our bedroom TV setup. I noticed that the wireless signal was listed as merely ‘Fair’ on the Roku, and we routinely toggled between one and two bars on our Nook Color. The router was located in the basement, and the bedroom is on the second floor so it was a fair distance for the signal to travel.

I decided to move the router to the main floor and place it near the top of the arcade cabinet. The router was connected to our Comcast cable modem and the router then connected to the network. There is a gigabit switch behind the arcade so I figured I could just run another cat5e run from the modem up to the new router placement. There was already another line running to that location’s switch so it was easy enough to just follow that cable with this run.

Now the Roku says the signal is ‘Good’ and the Nook routinely shows full bars so the small amount of work was certainly worth it. The only expense was a longer cat5 cable from Monoprice.

I’m always looking for ways to (cheaply) extend the capabilities of my home network. I have an XBMC box connected to our primary TV/speaker system, which is driven by media stored on an OpenFiler vm on my ESXi server. That works wonderfully and is easy enough for my children to navigate. There have been times at night, though, that it would have been nice to watch a movie or show on the bedroom TV instead. Starting it downstairs, and resuming it from the paused point upstairs would be ideal. Right now we just have very basic digital cable there. I’ve been keeping this need simmering in the back of my mind.

A bit less than two weeks ago my buddy made me aware of Plex. Plex is free and consists of two parts: the media server and the client(s). The server is available for a variety of OSs. It sounded very cool and didn’t appear to require much effort to set up. I chose the Windows option which would be installed on an existing Windows 2003 server virtual machine (1 vCPU and 1 GB RAM).

Setting it up
Installing the Plex Media Server was a breeze. After downloading it, it took only a minute to get it installed. The server is driven by a browser interface.

I added the respective locations on OpenFiler to the Movies, Home Movies, Music, and TV Shows sections. Then I let the application spin through the folders and build its listings. It took a little while to complete but it was certainly reasonable given the volume of media we have (our entire music collection and a good chunk of our DVDs take up a bit of space).

In order to kick the tires a bit, I installed the media center (client) on the kids’ Windows 7 machine. The install was quick and the client easily found the server. Response was fine and each local item I played started immediately. Streaming media were almost as responsive. We have high speed Internet, gig Ethernet throughout the house, and all the PCs are hardwired so your personal mileage may vary.

The Plex App for Nook Color
My wife has had a Nook Color since they first came out and loves it. Out of curiosity, I checked to see if Plex was available for it. It was. I paid the required $5 and the app installed. It found the server without issue and in less than a minute was scrolling through the media. It was nice and my wife was pleased with the added Nook functionality.

Plex handles mobile devices differently than local devices (PCs on the local network). It directly streams the media to local devices but the Plex Server transcodes the media before sending to any mobile device. Consequently, the Win 2003 vm ran at 100% when the Nook was being used. Not a big deal when dealing with a virtualized environment. I powered down the vm, added a second vCPU, and restarted. Overall CPU usage stayed more in the 60-70% range, which still gives me a little headroom.

I’ve done some playing on the Nook and overall the Plex experience is quite nice. I needed to change the delay for audio a bit from default but that was no big deal. My wife has watched a couple Shaun the Sheep episodes (brilliant show!) and had no issues. I did discover that FLAC do not play on the Nook. Fortunately, I have Vortexbox mirroring the FLAC folders to MP3s, so it was as simple as changing where the Music category was pointed to in the Plex Server. Now all the music can be played (through the terrible sounding Nook speaker).

Roku
Roku produces streaming media players that can work with Plex. Plex is just added as a channel and then all your local media is available through the player to your TV. The Roku LT fit my requirements and, at just $50 shipped direct, made my wallet happy. Right now our TV is a fifteen year-old 27″ Proscan (CRT) which will eventually be replaced by our nearly six-year-old 42″ LG 720p plasma (great TV). The LT has both composite and HDMI outputs so it will work for both TVs.

The Roku arrived in a few days in its original packaging. I was surprised at exactly how small it was; it really is the size of a hockey puck. Getting it set up was straightforward, and adding the Plex channel was simple. I’ve toyed around a little with it (outside of the Plex channel) and some channels work and some don’t. It’ll take more time to decide what is of value to us and what isn’t; there are just so many channels to choose from.

Plex on the Roku has been a nice experience. It takes about 10-15 seconds at the start to buffer a show but that is reasonable. We have watched a ton of Shaun episodes, plus a movie and the quality was top-notch. In the next couple weeks I plan to move my router from the basement to the middle level so it can be closer to the bedroom, to see if that effects the buffering time any. If anything interesting is discovered, I’ll add an update to this post. We are happy with the Roku/Plex combination. It allows us all our content on the secondary TV now, for just $50. Money well spent.