VMware and NAS: Freenas or Openfiler?

This artice describes my Freenas experiences since march 2009.

From VMware Server to ESXi and a NAS solution (March 2009)

December 2008 after using VMware Server for 3 years I decided that it was time to switch to ESXi. That was a good moment to start experimenting with a NAS. And offcourse because my network is very low budget I was looking for a free open source alternative that could run on simple hardware.By the way, in these three years of using VMware Server to host my website and emailserver at home, I just had two reboots for maintenance reasons. Uptime of the virtual and physical servers was over 300 days 🙂

My setup:
– Two ESX servers: Pentium4 2GB RAM and each one has a 1GB NIC.
– Server used to install FreeNas and OpenFiler: Pentium4, 1GB RAM, 1GB NIC
– Switch 1GB
 

FreeNas

I started with installing FreeNas as I played with this software in the past. When installing I was pleasently surprised by the options that FreeNas offered me. You can boot it from the cd, install it on the harddrive and boot it from there, or install it on an USB drive and then boot it. FreeNas installer also gave me options to upgrade an older installed version on the harddisk or USB device, very nice.

Installing was very simple, booting FreeNas is very fast and configuring is very easy too if you play with it for a while. The webgui is fast and looks professional.

I created an iSCSI LUN and presented it to the ESX's. That worked pretty good. First connected it to ESX1 and then to ESX2. I created a VMFS file system via the ESX2 and then created virtual machines.

Then I ran into a problem. I noticed that the LUN was not available anymore on my ESX1 server, just on my ESX2. I tried to rescan, but I still could not add it anymore. Then I noticed that the ESX1 storage adapter could see the LUN, but I just could not add it anymore as a storage. After some google-time the problem seemed to be that FreeNas can not handle Shared Storage. And that means No VMotion! That was a bummer.

Another issue I ran into was the performance. I have the idea FreeNas is not superfast in my setup at home. The VM's are responding kind of slowish. I also used SMB/CIFS to share files from another disk and this was not fast too compared to my Fedora SAMBA setup that I had been using for  couple of years uptill then.
 

OpenFiler

I decided I had to experiment with another NAS and I wanted to try OpenFiler. Openfiler is a well known open source project and it has shared storage feature. The problem was where to find the time in my life to do this. So yesterday late in the evening I decided to start, and I finished early in the morning.

The installation is easy, not as luxureous as the FreeNas install but good. My monitor could not handle the graphic install so I had to do it via the text gui. No problem. The nice upgrade features that FreeNas has are not available in OpenFiler, and installing on a USB Memory Stick device is not supported too. The OS is bigger then the FreeNas installation. I used a 2GB partition on the harddisk for the system partition and the rest of the space I left free to create a local VMFS. OpenFiler installation uses about 600MB on the disk. After installation I connected my webbrowser using https on port 446 to the webgui, which is not as fast and as cool as FreeNas, but it works fine. Default account is openfiler with password password.

I had my USB 1TB disk that contained a partition with a VMFS filesystem and some VM's. I connected it to OpenFiler, the disk was found but I could not use it. The problem is that OpenFiler expects a disk to be partitioned with Logical Volume Manager. If not then you can not connect to existing filesystems on the disk. This was pretty uncool cause in FreeNas it is really easy to add a disk with an existing partition and use the data that is stored on it. Doesn't matter if it is ETX2, ETX3, NTFS filesystem or that you present it raw to an ESX server when you have a VMFS volume on it.

So I connected the USB 1TB VMFS disk again to a FreeNas server (that I installed on a really old PIII pc), created an iSCISI target and connected it to an ESX. I copied the VM's from it and then deleted the VMFS partition.

Then I connected the 1TB disk to OpenFiler again, used LVM to create a partition and then created an iSCSI LUN. Connected both ESX servers to it..  et voilà: shared storage!

I got OpenFiler running now for one day and I can see that it works fine. And VMotion works! Now I'm able to meet the SLA agreement that I made with myself 😉 It might be interesting to experiment with NFS too to see the performance difference with iSCSI.

There is just one other downside not related to VMware when you compare FreeNas to OpenFiler: with FreeNas I used another 220GB USB drive to share my files that are on a EXT3 filesystem via SMB/CIFS. Another 215GB USB disk I used as a backup for this disk, synchronizing it using RSYNC in FreeNas. That really works great!
As these two disks that I mentioned are not partitioned using Logical Volume Manager, OpenFiler can not find the existing EXT3 filesystems. I have to format one of the disks, use LVM to create a partition, format it to EXT3 and then copy the data. And then do the same for the second disk.
After doing this I have to find out if I can sync the two disks using RSYNC inside OpenFiler, and I'm not sure that this is possible. And then see if the performance is better then Freenas when I use SMB/CIFS to share my files. I'll let you know how this works out when I find the time to try it out.

Conclusions for now

If you want your VMware ESX servers to have shared storage so that you can VMotion then you have to use OpenFiler and not Freenas. OpenFiler works great and is not hard to setup.
FreeNas is more userfriendly and there is more eye for detail in the webgui. Future upgrade situations are handled in the installation phase. FreeNas makes it easy to add disks with existing partitions and filesystems on it when LVM is not used. OpenFiler prefers an empty disk and wants to create partitons using LVM. You can not add a disk with existing LVM's and existing partitions without doing some tricks (which I haven't tried).

Let me know if you find this article usefull, makes me happy 🙂 HaveANiceDay

More then one year later… (august 2010)

Ok, I wrote this article in March 2009. Now it's august 2010 and I have the following situation at home:

• One ESX 4.1 server (instead of two) – new hardware that costed me about €400,- with a Xeon CPU with 4 cores, 8GB Mem and it performs very well and consumes less power
• Six VM's running, mostly linux (no X windows) and also a W2008 (x64) and I still got resources left to experiment
• One of the six VM's is Freenas. I attached a 500GB VMDK to it and this storage contains all of my files (documents, music, etc) And it's my bittorrent server too.
• I used one of the old ESXi servers (desktop pc) on which I installed Freenas, and attached an external USB disk to it. So this is my backup server for my data and I use RSYNC to copy files between the two Freenas instances during the night now and then. And I use VMware's Data Recovery to backup my VM's to the Freenas Backup server.

So as you can see I started to use Freenas more then Openfiler. Why is that? Well, it's it's just a little more userfriendly and easier to set up. You can do everything in the GUI and since I have to be efficient with my time I used freenas. On my webserver I created a "proxy website" so I can connect to and configure my freenas servers from the internet when I'm not at home.
A backup of the Freenas config can be made so in case of a problem a restore is no sweat. Freenas can handle Shared Storage LUN's nowadays so that is cool too. 

Almost two years later… (januari 2011)

Recently for the first time in my life I bought a new TV, and this one is able to play files from a media server. And yes, Freenas has a builtin media server: Fuppes. I configured it and it works! If you don't have a TV with a direct media server connection you can always use a PS3, 360 Xbox and more.

The Media Server feature combined with the torrent feature in Freenas is great. First download a movie and them play it on your TV using the media server.

I only have a problem: subtitles. The .srt files are not recognized by my TV and therefor not used. For a Dutch guy a movie like The Social Network is hard work to understand everything.

By the way: both my Virtual and Physical Freenas system have not crashed yet, nor has VMware ESX 4.1 and all of my other Linux servers running on top of ESX (251 days uptime).

Well, hope you can use this info. Leave a message if you'd like. I'd appreciate it to hear your idea's.

ESXi5 and Freenas iSCSI (September 2011)

I read about the incompatibility of Freenas and ESXi5 iSCSI. On September 2nd the latest Freenas version wat released with fixes for iSCSI.

• Upgraded Freenas 0.7.1 version to 0.7.2.7529 Sabanda. The upgrade failed, but I was warned in the readme file for this.
• Installed Freenas from scratch and restored my old backupped configuration. This worked. So after 15 minutes I could try to connect an iSCSI target to a ESXi5 host. I just needed an ESXi5 host.
• Installed ESXi5 on my ESXi4 platform in a guest and connected it to the iSCSI target. This seemed to work.
• Created a guest with the VMDK on the iSCSI target and linux as a OS, this worked too.

Another job well done by Freenas.