The Tech Outfit - Inside the Brains of our Techs tag:,2008-06-21:/1 2010-01-03T18:44:31Z Our consultants provide in-depth analysis of current technical news, trends, solutions they've personally worked on and more. Movable Type Publishing Platform 4.01 Performance Tuning VMWare Server on Ubuntu Linux tag:blog.techoutfit.net,2010://1.11 2010-01-03T18:38:59Z 2010-01-03T18:44:31Z Donovan Niesen http://www.thetechoutfit.com NOTE:  This article originally appeared on fewt@blog at this url:

http://www.fewt.com/2008/06/performance-tuning-vmware-server-on.html

However, that URL now returns "SITE CLOSED, SORRY".  This article made such an amazing difference in my VMWare setups that I wanted to make sure it was available to all. 

ON WITH THE ARTICLE...


]]>
Here are the example specs for the new VMWare Server:
AMD x2 4200+ 65nm CPU
4GB PC2-6400 DDR2 RAM (Dual Channel)
2x Seagate Barracuda 7200RPM 250GB SATA 3.0Gb drives
ASRock ALiveNF6P-VSTA System Board
580W Power Supply

Operating System: Ubuntu 7.10 Server 64bit

Drives configured as follows:

/dev/sda
1 - Swap (1GB)
2 - md0 (250MB)
3 - md1 (231GB)

/dev/sdb
1 - Swap (1GB)
2 - md0 (250MB)
3 - md1 (231GB)

/dev/hda
1 - ext3 (100GB)

All MD devices are configured as RAID-1 using Linux Software Raid

Device md0 was formatted as ext3 and exists as /boot
Device md1 was added to volume group vg00, then Logical Volume ROOT and mounted on /

LILO was used, and it is written to both sda and sdb (so the system remains bootable if sda fails)

Using Logical Volume Manager (LVM) for / provides flexability in that I can add two more 250GB drives as md2, and grow / to 500GB with a few simple commands (vgextend, lvextend, resize2fs). In addition, using LVM also provides capabilities to mirror or snapshot volumes. Using Software Raid instead of the integrated NVidia MediaShield provides better RAID management flexability, and as both devices are CPU backed since there is no internal RAID processor on the MCP chips there is really no performance gain by using the integrated card.

Tuning:
After building the system, and installing VMWare I started creating Virtual Machines. Something that I immediately noticed was that during even small IO, wait states climbed to 100% CPU and network dropoffs occured, it was causing connections to the web server VM to drop for even static content performance was absolutely terrible. I initially thought that the problem was the Software Raid devices, but I quickly identified that the problem child was actually problem children and that the RAID wasn't one of them. I installed a product called monitorix and started collecting data which was instrumental in identifying the performance bottlenecks.

Virtual Machine configuration:
When creating virtual machines on this platform I elected to us pre-allocated disks. Using pre-allocated disks reduces disk fragmentation, and improves overall performance. Additionally, I always remove the floppy device. Here is an overview of the configurations:

Generic base configuration:

Linux 32bit and 64bit VMs:
384MB Ram 20GB SCSI disk pre-allocated 1 Ethernet device (bridged)

Windows VMs:
256MB Ram 8GB SCSI disk pre-allocated 1 Ethernet device (bridged)

It's recommended by VMWare that Windows VMs be configured to use IDE, however in my experience the Virtual IDE devices use tons more CPU time than the SCSI device. This is due to the emulation level done and lack of I/O threading in VMWare's IDE controller. I have to assume that this is a problem with IDE in general, as it's never been very good at multithreaded I/O (this is one big reason it's never been used for servers). Additionally, I recommend using the LSILogic controller is it supports multithreaded IO while the Buslogic controller doesn't.

One of my virtual machines was destined to become a print server, printing to a USB printer directly connected to the VMWare server. Ubuntu 7.10 doesn't configure USBFS out of the box. This can be corrected by editing a few files:

Add to fstab:
usbfs /proc/bus/usb usbfs auto 0 0

Edit /etc/init.d/mountdevsubfs.sh, and uncomment the following lines:

#mkdir -p /dev/bus/usb/.usbfs
#domount usbfs "" /dev/bus/usb/.usbfs -obusmode=0700,devmode=0600,listmode=0644
#ln -s .usbfs/devices /dev/bus/usb/devices
#mount --rbind /dev/bus/usb /proc/bus/usb

This is done by removing the # from the front of each line. Once this is done, go ahead and run the script.

/etc/init.d/mountdevsubfs.sh start

In the Virtual Machine configuration, I needed to ensure that the printer was always connected on startup so I inserted the following configuration into that Virtual Machine's VMX file:

usb.present = "TRUE"
usb.generic.autoconnect = "FALSE"
usb.autoConnect.device0 = "0x0000:x0000"
usb.autoConnect.device1 = "0x04e8:0x327e"
usb.generic.skipsetconfig = "TRUE"

You can get the IDs for your devices by issuing an lsusb on the VMWare Server, this command will output similar to the following:

Bus 002 Device 002: ID 04e8:327e Samsung Electronics Co., Ltd
Bus 002 Device 001: ID 0000:0000
Bus 001 Device 001: ID 0000:0000

Additionally, I had to blacklist usblp on the VMWare Server so the host didn't connect to the printer making it unavailable to the guest.

echo "blacklist usblp" >>/etc/modprobe.d/blacklist

As you can see above I have used the device ID for the Samsung printer, as well as the USB hub that it's connected to (0000:0000). Now, whenever the Virtual Machine restarts, it scans the USB bus on the host and automatically connects those devices. Printing now "just works" after rebooting. Of course for it to "just work" you also need to configure CUPS or Windows printer shareing, but that is out of the scope of this article ;-).

After I installed monitorix, I identified multiple significant performance bottlenecks. I resolved these bottlenecks by tuning both VMWare and the Linux kernel itself. The difference in performance is astronomical, and it's a very visible improvement in the graphs as seen below.



To resolve the bottlenecks, I made all of the following changes to both VMWare Server, the Virtual Machines and the Linux host itself:

Add each of these to /etc/vmware/config:

mainMem.useNamedFile tells VMWare where to put it's temporary workspace file. This file contains the content of the Virtual Machine memory which is not used. By default it is placed in the directory with the virtual machine, however that can seriously impact performance so we'll turn it off.

mainMem.useNamedFile = FALSE

tmpDirectory is the default path for any temp files. We need to change that to be a shared memory filesystem (in RAM).

tmpDirectory="/dev/shm"

prefvmx.useRecommendedLockedMemSize and prefvmx.minVmMemPct tell VMWare to either use a fixed sized memory chunk or balloon and shrink memory as needed. Since I have 4GB of memory in this "server" I want to make sure that I use a fixed chunk of memory to reduce disk IO.

prefvmx.useRecommendedLockedMemSize="TRUE"
prefvmx.minVmMemPct="100"


To tune each Virtual Machine, I installed VMWare tools and then made the following changes to each VMX file:

Set the time in the Virtual Machine to the hosts time (I use NTP on the host):

tools.syncTime = "TRUE"

When I reboot the host, I want to gracefully stop each VM instead of just powering it off:

autostop = "softpoweroff"

I don't care about collapsing memory into a shared pool, this tells the VM to not share which saves CPU cycles:

mem.ShareScanTotal=0
mem.ShareScanVM=0
mem.ShareScanThreshold=4096
sched.mem.maxmemctl=0
sched.mem.pshare.enable = "FALSE"

This basically performs the same action as the configuration I put in /etc/vmware/config by telling the VM to eliminate the temp files and not to balooning and shrink memory, however it doesn't hurt anything to have it in both locations:

mainMem.useNamedFile = "FALSE"
MemTrimRate = "0"
MemAllowAutoScaleDown = "FALSE"

Additionally, by default Ubuntu writes an access time stamp to every inode that's accessed. This is pretty accessive and known to cause bottlenecks in high I/O scenarios. It doesn't negatively impact the filesystem unless you care about access time stamps, so in each VM and the VMWare host host I add "noatime" as an option to all of my mounted disks in /etc/fstab.

In order for the VMWare configuration to work properly with shared memory, you'll need to increase the default shared memory size for tmpfs to match the amount of memory in your system. This can be done by editing /etc/default/tmpfs:

SHM_SIZE=4G

You can use 'mount -o remount /dev/shm' and 'df -h' to implement and verify the change.

Last, I configure /etc/sysctl.conf on the VMWare Server which configures the kernel to perform better as a Virtual Server by inserting the following configuration:

vm.swappiness = 0
vm.overcommit_memory = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 1000
dev.rtc.max-user-freq = 1024

Lastly, I disable the tickless kernel option in kernel 2.6.22 which further reduces the Virtual Machine I/O constraints by reverting back to using ticks which is better supported by VMWare. This can be done by adding the following option to the kernel options line in /boot/grub/menu.lst or /etc/lilo.conf:

nohz=off

With all of these options configured, the VMWare server now performs wonderfully at under 20% host CPU utilization with 6 Virtual Machines all running various flavors of Windows and Linux. ]]> Physical to Virtual Server on a Budget tag:blog.techoutfit.net,2008://1.9 2008-12-22T11:59:21Z 2009-06-15T18:29:47Z Donovan Niesen http://www.thetechoutfit.com
The Machines

I have created numerous virtual machines from scratch.  I have even converted a fair share of Windows boxes to virtual machines with varying success.  I knew this project was interesting as these were the servers I needed to move:

  • A Windows XP "server" handling some proprietary data collection applications
  • A Debian 3 server running web applications and serving hold music
  • Another Debian 3 server acting as a mail server among many other odd maintenance scripts
  • A FreeBSD 6 server hosting web applications
]]>
The Machines

I have created numerous virtual machines from scratch.  I have even converted a fair share of Windows boxes to virtual machines with varying success.  I knew this project was interesting as these were the servers I needed to move:

  • A Windows XP "server" handling some proprietary data collection applications
  • A Debian 3 server running web applications and serving hold music
  • Another Debian 3 server acting as a mail server among many other odd maintenance scripts
  • A FreeBSD 6 server hosting web applications
The Migration

First, I backed up all data on each box.  When dealing with old hardware that has uptime over 700 days, there's often a surprise when you give it a reboot.

I prepared a VM for the XP box using VMWare Converter which took a number of hours to complete.  Luckily this machine only has one MS Access DB that it stores so it was just a simple task of moving that database once the virtual machine was running.  It should also be noted that most Windows migrations like this are required to re-activate as the hardware changes significantly.

Having never done a Linux or FreeBSD physical to virtual (P2V) conversion, I read a number of sites with various suggestions.  It was suggested I simply clone the box and perform a rescue on it once fired up within the VM.  I initially chose Clonezilla to image the first Linux box.  One suggestion I would make is eyeball your hardware before attempting to clone it.  The first time I booted Clonezilla I didn't realize the server I was converting only had a quad-speed drive in it.  The boot took 15 minutes.  I yanked a 52x from a dead PC and rebooted again in under 3 minutes. 

The image creation seemed to go without a hitch.  I created an empty VM in VMWare server, mounted the ISO and attempted to restore the image.  Clonezilla performed half the restoration and then asked for another file that did not exist.  This seemed to be a common issue with this product so I scrapped it and gave g4l a shot.  It just kept getting worse as g4l would only backup a GB of data and then drop back to the menu screen.

In desperation, I grabbed and Acronis True Image Home boot CD and attempted a backup that way.  It created an image without complaining.  I booted off the same CD in VMWare, recovered the image and was able to boot into my Linux install.  Flawless.  Both Linux servers cloned using this method worked without a hitch.

The FreeBSD box did not fare nearly as well.  While backing up the server unreadable sector errors continued to show up.  I told Acronis to ignore these sectors however the restored image still would not boot.  The BSD box itself did reboot so that was left in production.  It is running simple LAMP applications so they can easily be migrated into a more familiar Linux VM.


]]> Servers on a Budget, Linux Software RAID and VMWare Server 2.0 tag:blog.techoutfit.net,2008://1.8 2008-09-28T14:08:43Z 2008-09-28T17:48:49Z Donovan Niesen http://www.thetechoutfit.com vmware.jpgThis week I was tasked with putting together a virtualization solution to host a number of desktop-turned-servers whose plastic cases are aging from beige to yellow.  As trusty as the little Linux boxes are, when I cringe at the thought of a piece of hardware dying in one of those ancient machines, I know its time to be proactive. 
]]>
Servers on a Budget

The economic climate has already tightened up IT budgets so I had to forego an HP server as much as I hated to do so.  To quote Ferris Bueller, "It is so choice.  If you have the means, I highly recommend picking one up."  So what did I put together instead?  Something not too bad, actually:

At the time of this writing this setup cost under $1,400 before tax/shipping.  SuperMicro makes excellent barebone servers.  If you're rolling your own server and are your own support contract, it's hard to beat the price/features with any other recognizable brand.  The only thing I felt was missing from this configuration was the option for redundant power supplies and SAS.  This was an acceptable trade-off as the servers being migrated are not mission-critical.

The gear arrived this week, I happily assembled the components and fired it up.  POST was successful, all hardware was detected and smiling away.  One note to anyone new to working with rackmount servers, the fans are loud.  If you're not putting them in a server room/closet where the noise won't matter, get tower servers instead.

Linux Software Raid

The SuperMicro board I chose for this solution touts on-board RAID.  While nifty in a feature list the on-board RAID is useless under Linux as it is not supported by the kernel.  Rather then shell out some $600 for a hardware RAID controller, I opted for Linux Software RAID.  There's enough power in this server that the extra overhead for RAID will be barely noticed.

The four SATA hot-swap bays make RAID-5 a perfect candidate.  Without giving it too much thought I popped in an Ubuntu 8.04.1 x64 server CD and started my install.  I gave each drive a partition for RAID and created a RAID-5 array.  Wow, that was easy.

The rest of the base system installed without complaint and finally GRUB tried to install, without success.  Hoping for a fluke I retried the GRUB installation numerous times ignoring the fact that trying the same process repeatedly expecting different results is a symptom of insanity.  After much pondering and research I determined that a Linux software RAID-5 is either too difficult or impossible to boot from.

In searching around the server room for a spare drive, I considered just throwing in another SATA drive for my OS  and then using my RAID-5 as the data partition.  The thing I didn't like about that was not having any redundancy for the OS itself.  I'd like to avoid the enormous single point of failure.  Not finding a drive for my quick fix I let the server be for a day so I could think over my next move.

Redundancy, redundancy... that was my motto for the next couple of days.  While briefly considering buying two more disks to set up a RAID-1 array for my OS I realized that Linux's software RAID is ridiculously flexible.  I can set up a small 10GB partition and use RAID-1 to mirror that across all four of the drives and still create a RAID-5 with the left over space. 

I partitioned out my OS in a RAID-1, created my /vms partition in a RAID-5 and GRUB happily installed onto the mirrored RAID.  Upon rebooting everything was happy, my RAIDs synced up.

NOTE:  This setup is a clever use of software RAID and while it does provide some flexible redundancy, should one of these drives fail, to realize there are two RAIDs to deal with.  Plan your documentation accordingly and practice recovering from failures with this particular configuration.

VMWare Server 2.0

With Ubuntu installed and configured as I wanted it, it was time to download and install VMWare server.  I have setup numerous VMWare 1.0.x configurations and was very comfortable with their feature set.  I was surprised to see that upon downloading VMWare for this setup, 2.0 had been released.  Rather than exciting me this initially annoyed me as I had tested earlier beta versions of 2.0 and was discouraged by the move to their web console from the traditional standalone GUI console applications.  In the particular beta I was using, the console seemed to be a hack of VNC and performance was abysmal. 

Not wanting to completely write off the new version, I reluctantly downloaded and installed, hoping for the best.  The VMWare web access panel seemed to have been polished significantly since my beta experience and was much more responsive and logically laid out.  I built a new VM, selected another Ubuntu server ISO to install and fired it up.  I then opened the dreaded console and was surprised that it was asking me to install a Firefox plug-in.  The plug-in was roughly 18MB so I assumed this was something much better than a simple VNC client.  Upon rebooting Firefox and hitting the console again my fears of the old web console disappeared.  The new console is excellent.  Performance is where I expect it to be over an internet connection and the new portability actually extends the previous generation of VMWare console greatly. 

The console will actually show the CPU and memory usage of each host individually and for the entire host as a whole.  This is fantastic for seeing if you have one hog VM or if you're getting close to the limit for the host. 

I don't have a single complaint about VMWare Server 2.0, in fact I'm now conspiring to upgrade my old 1.0.x installations to make administration easier on myself.  My only nitpick is that the new console advertises VMWare Infrastructure although non-intrusively.  You can even minimize that pane to ignore the ad completely.

Next Steps

Am I done?  Not even close.  Next up:

  • Converting physical machines to virtual machines
  • Monitoring these servers and the RAIDs that support them
  • The backup plan

]]> Will you go over Comcast's 250GB bandwidth cap? tag:blog.techoutfit.net,2008://1.7 2008-09-22T01:03:40Z 2009-06-15T18:19:23Z Donovan Niesen http://www.thetechoutfit.com comcast.gifWe use Comcast's business cable internet service at our office and I quite enjoy the cable service at home.  It really is fast.  Beyond the bandwidth, the ~20ms ping time over VPN to the office is quite nice. 

The high bandwidth made it very quick and easy to download updates, ISO images of popular Linux distributions, stream video content and keep all of my home computers up to date. 

When Ars Tecnica first published an article about this cap I got to thinking about my bandwidth usage.  I know I use a lot of bandwidth but realized I wouldn't be able to know if I was coming near that cap without a proper solution.  After spending a few hours researching and finding some interesting solutions for monitoring bandwidth I came up with an obvious question: Shouldn't Comcast be telling me how much I'm using?

Starting October 1st, Comcast will begin warning their customers who exceed 250GB in a calendar month and on the second instance of exceeded usage disallow them from the service for one year.  Without historical data or even current bandwidth monitoring there will be a number of surprised customers at the end of October. 

If you're lucky enough to be using Comcast this October, take a look at these monitoring tools:

  • For monitoring a single Windows computer, Free Meter is good enough.
  • For monitoring an entire network, do this at the gateway/firewall level.  A Linux box or pfSense running bandwidthd will do very nicely.
Happy downloading.
]]> Google's Chrome, impressions and thoughts tag:blog.techoutfit.net,2008://1.6 2008-09-09T11:32:19Z 2008-09-28T17:45:56Z Donovan Niesen http://www.thetechoutfit.com google-chrome21-50.jpgThere are a ton of reviews on Google's Chrome browser out on the web.  Anybody care for a Tech Outfit perspective?  We take the browser through the paces and compare it against what we use for our day-to-day browsing habits.  Will Chrome be good enough to replace our tried and true Firefox?  Will Google shape the way we look at web pages?

]]> First Impressions

I prefer to operate in a Linux desktop but have recently picked up an Acer Aspire One laptop loaded with Windows XP Home.  Google's Chrome browser is currently written for Windows only with Mac OS X and Linux ports coming eventually.  I downloaded installed and launched the browser.

The first thing that I noticed is that the user interface does not take cues from your overall Windows theme.  A color I will call Google Blue invades the title bar letting you know you're definitely using a Google product. 

Aside from the initial color and UI shock, Chrome interestingly uses the otherwise wasted title bar space for tabbed browsing.   I'm a bit of a screen real-estate crusader which is especially important on the Aspire One's 1280x600 screen so I keep my Firefox crammed into one toolbar and will often jump to full screen mode.  Chrome gets a slight leg up on screen real-estate by utilizing that toolbar. If I could shrink the address bar down a bit there would be a significant difference.

The default page for Chrome is a nifty beast that shows screenshots of your most visited websites, recent bookmarks and a search box for your history.  This has been covered in great detail in other reviews so I will not dig any further on that subject.

I had heard many people speak to the speed of Chrome and didn't expect a vast improvement over my fine-tuned Firefox. I fired up Chrome next to Firefox and compared loading a number of simple and complex pages.  Chrome definitely has a noticeable speed improvement, especially in Javascript-heavy and other media-heavy sites. The browser really feels snappy.

Overall I'm content with the performance of Firefox but one thing I've always noticed is that with a few complex sites open, adding a new tab isn't as instant as I would like.  To compare, I opened up a number of complex news sites in Google and kept opening new tabs while the pages were loading.  The new tabs opened just as fast as the first tab.  Very impressive.

One page I really wanted to compare that I was unable was my Zimbra mail client.  This webmail client makes very heavy use of AJAX and takes a little while to load in Firefox and IE.  Unfortunately it fell back to the Standard HTML version when attempting to login so I was unable to compare.  I imagine this has something to do with the application detecting the browser type and at this point it is unsure if Chrome has the capabilities it needs.

Thoughts

With a very impressive first release is it enough to pull Firefox away from my default browser setting?  No.  Here's why:

  • There are a number of Firefox extensions that I would rather not do without. 
  • Access to my AJAX-rich Zimbra account is a definite show-stopper. 
  • While it makes excellent use of screen real-estate, there is no full-screen mode for those situations where I just want a little more. 
  • I would like to see the browser have a more native-look, although I could overlook this if other features were in place
With those nitpicks in mind, Chrome is a great option for people with slower machines who want a snappier browsing experience.  In my world, I know that I have yet another browser to test websites against (sigh).

]]> College students scrambling for computer support tag:blog.techoutfit.net,2008://1.5 2008-09-08T02:47:48Z 2008-09-28T17:12:14Z Donovan Niesen http://www.thetechoutfit.com
Regardless of the limited visibility, The Tech Outfit has experienced a large number of college students requiring computer assistance.  While we surely appreciated the business we were curious to find out why they were not using the resources on campus. Most students respond that when they need help, they need it today.  On-campus services are often backed up or hard to find.  Unless you know a bright person down the hall who happens to be around, you're scrambling for help. 

With convenient on-site services and a handy Dinkytown location, college students are finding their own way to The Tech Outfit.
]]> One stubborn mail provider tag:blog.techoutfit.net,2008://1.4 2008-09-03T02:20:22Z 2008-09-28T17:43:33Z Donovan Niesen http://www.thetechoutfit.com netsol-logo.gifTwo weeks ago my company received an influx of delayed messages when attempting to email clients whose mail servers were lucky enough to reside at Network Solutions.  After opening up trouble tickets at both our mail provider and theirs', we found that Network Solutions had been a victim of a DDoS attack and took it upon themselves to block the IP range in which our mail server lies. 

The notice from our own provider went like this:

 Apparently Network Solutions was subject to a DDOS (distributed denial of services) attack, and responded rather bluntly by blocking large sections of the Internet, affecting multiple IP transit providers, including AT&T and Cogent, our primary peering partner.  Our provider advised that Network Solutions would remove the block when contacted by the individual delegated IP space owner. We have made that request twice thus far, but remain blocked. Until Network Solutions removes that block, this situation will continue. We apologize for any impact this may have on you.  If you are also a Network Solutions customer, we strongly advise you to open a technical support request with them, and open a ticket with us as well.]]>
Another update from our mail provider states that the IPs were, in fact, spoofed:


As of today, August 28th, our IP range remains blocked by Network Solutions. Network Solutions continues to block portions of the AT&T and Cogent networks. As best we can determine from our contacts with Network Solutions and Cogent, the attack on Network Solutions consisted of a DDOS attack from spoofed IP addresses. What that means is that while the attack appeared to originate from particular IP addresses, those addresses were falsified in the packets. The true origin of the attack is thus very difficult to determine. Network Solutions has blocked the spoofed addresses, not the actual origin of the attack. As the blocked addresses did not in fact originate the attack, there is nothing Cogent or AT&T can do/fix/change. Apparently Network Solutions, however, does not intend to remove the block until they find the true origin of the attack. We do not know how long this could take.

The most frustrating part of this whole ordeal is that our own provider looks like the bad guy when they were essentially denied service to Network Solutions vast customers.
]]>