Windows Insider

VM Within a VM

Yes, it's possible. Greg shows you how.

A training company recently approached me to create some videos on installing and using VMware's flagship ESX product. They wanted me to demonstrate all of the product's features and functionality, including ESX's VMotion capability. That got me thinking about how useful VMotion is to VMware's ESX Server product.

Unless you've had your head in the sand for the last few years, you know that VMotion is VMware's built-in ability to migrate virtual machines (VMs) from one host to another without needing to power them down. This "hot migration" capability is only available in the high-end, high-cost Enterprise Edition of ESX Server.

It's "high-end" because it can migrate machines without losing a beat. That means much less downtime for critical applications. It's "high-cost" because you'll need an expensive Enterprise license to run the ESX Server. You'll also need some form of high-end storage to support VMotion's shared storage requirements.

My immediate response to the training company was to tell them about the hefty hardware I would need to fully demonstrate the software. It just wouldn't be possible on an author's budget. Then I started digging around and discovered some new functionality in the recently released desktop version of VMware Workstation 6.0.

With just a little tweaking, you can get Workstation 6.0 to fully support VMware ESX instances running inside VMware Workstation. A VM within a VM? Absolutely -- and depending on your desktop's configuration, it'll pack enough performance to help you develop your virtualization skills right there at your desktop.

Beefy Workstations
When you're building a VM-within-a-VM environment, you need a powerful desktop host. After all, that host is running virtual machines that are also running virtual machines. They'll require lots of RAM and processor power.

You'll need a desktop with at least a dual core that also supports Intel's Virtualization Technology (VT) processor extensions or newer versions of AMD's equivalent AMD-V extensions. The virtualization extensions are a necessity because they help the processor handle the workload associated with virtualization more efficiently. Your system should also have at least 3GB of RAM to support all the virtual machines you plan to run.

In order to successfully complete your VMotion setup, you'll need at least four simultaneously running virtual machines. Two of those will be ESX servers, the third will serve as the Virtual Center management server, and you will use the fourth as an iSCSI target for shared storage between the hosts.

ESX Server can use SAN-based storage, NFS and iSCSI targets for its shared storage where VMotion-capable virtual machines are housed. iSCSI is an excellent low-cost medium for shared storage as there are numerous open source iSCSI tools available on the Internet.

OpenFiler, which is an easy to use Linux application, is a good choice. You can download it from Once you've installed it within a Linux virtual machine, use OpenFiler's Web interface to create a disk partition, a volume group and a volume for the shared storage. Once you're done, you can also use the same interface to share that volume as an iSCSI target with your ESX servers.

Recursive VMs
Once you've completed the setup process, you can create a fully functional ESX virtual machine by following these steps (or find them online here):

Create a new custom virtual machine. Set its hardware compatibility to "Workstation 5" and check the box marked "ESX Server Compatible."

Select the Red Hat Linux operating system, then provide the machine a name and location.

Give the virtual machine two virtual processors and around 1024MB of RAM. The quantity of RAM assigned to the virtual machine will depend on the RAM available in the host system.

Select the bridged network option and choose an LSI Logic SCSI adapter for the virtual machine.

Create a new virtual SCSI disk. Make sure it's large enough to host the ESX software and any other hosted virtual machines. It's usually a good idea to create these disks as pre-allocated to ensure the best performance.

Once you've created the virtual machine, remove it from the VMware Workstation interface and the favorites list, and close down that Workstation instance. This ensures that the system isn't using the virtual machine's files we need to edit next.

Open the new virtual machine's .VMX file in a text editor and make sure the following lines are present for each connected Ethernet adapter. If there are multiple cards configured for the ESX instance, there will be one copy of each of these lines for each adapter:

ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.connectionType = "bridged"
ethernet0.addressType = "generated"

You'll need to add the following lines in the .VMX file for Intel-based CPUs:

monitor_control.restrict_backdoor = TRUE
monitor_control.vt32 = TRUE

If the processor is AMD-based, replace the line monitor_control.vt32 = TRUE with monitor_control.enable_svm = TRUE.

Reconnect the virtual machine with the Workstation interface, make sure the ESX media is in the CD drive and connected into the virtual machine, and boot the virtual machine.

Final Steps
This will begin the process of installing the ESX operating system into the virtual machine. Once the installation and subsequent reboot is complete, you can tell whether the install was successful if the server boots to the typical ESX screen. Repeat the process for the second ESX server and create the Windows Server virtual machine for the Virtual Center server.

Once all components are configured, connect the ESX servers to the OpenFiler iSCSI data store and create your VM within a VM.

Although this configuration won't be fast enough to run any production virtual machines, it gives you enough performance to learn the product or study for certification exams without consuming valuable server resources. The VMware online community was very helpful in bringing this capability into reality.

About the Author

Greg Shields is a senior partner and principal technologist with Concentrated Technology. He also serves as a contributing editor and columnist for TechNet Magazine and Redmond magazine, and is a highly sought-after and top-ranked speaker for live and recorded events. Greg can be found at numerous IT conferences such as TechEd, MMS and VMworld, among others, and has served as conference chair for 1105 Media’s TechMentor Conference since 2005. Greg has been a multiple recipient of both the Microsoft Most Valuable Professional and VMware vExpert award.

comments powered by Disqus

Reader Comments:

Mon, Oct 19, 2009 Sooraj Bangalore, India

Hi Greg, I have an AMD sempron CPU (2.0ghz)with 2gb RAM . I am able to run 2 win2k3 servers and 2 XP clients.Is that hardware is enough to convert into ESX.If it runs slowly I can adjust,But please let me know it will work or not

Fri, Feb 13, 2009 sunil Anonymous

hello all

i follow the steps o this page.

and i installed esx on vmware6.5

but i am not able to access neither though http nor client software.

http show simple

can't display page

can any body help me

Sat, Sep 20, 2008 Roger Zan Hartsdale, NY

Hi Greg,

Xtravirt published an excellent white paper on running VI3 in a box using VMWare Workstation 6.5, but they left off (either intentially or not) on how one would go about running a 64 bit guest VM within the ESX VMs’ within Workstation 6.5, if it’s even possible. I did setup a system using 8GB of DDR2 RAM on an Intel DP35DP board with an Intel Core 2 Quad Core 2.4Ghz Q6600 CPU, which supports VT, EMT64, and XD, with Windows Server 2003 Enterprise x64 Edition as the host OS and am using a separate HP ProLiant DL360G4p server to run Openfiler to host the VMFS Datastore over iSCSI. Now, I went through the whitepaper and got everything working, but when I tried to create a new x64 VM within the ESX VMs’ to test VMotion capabilities, it wouldn’t turn on and would give me an error about longmode not being enabled and such. After much poking around, I got past that issue by changing the CPUID mask within the 0×80000001 section and now the x64 VM turns on, but when I attempt to install a 64 bit VM, such as Windows Server 2003 Ent x64 or Longhorn x64, it states “Attempting to load an x64 operating system, however this CPU is not compatible with x64 mode. Please install a 32-bit x86 operating system. Setup cannot continue. Press any key to exit.”. Upon further investigation, I suspected it was something with the way VM Workstation was presenting the virtual BIOS to the ESX VMs’ and it indeed looks like the culprit. Going to help–>about in VM Workstation shows that the host OS on my quad core system is Windows Server 2003 “Standard” Edition and not Windows Server 2003 Enterprise x64. Unfortunately, VM Workstation is a 32 bit application that installs itself in to the C:\Program Files (x86) folder on my host system for 32 bit applications instead of the default C:\Program Files for native 64 bit applications. Even though VM Workstation 6.5 can fully host 64 bit VMs’, there is an additional layer I am going through within the ESX VMs’ that is somehow not passing the 64 bit CPU extensions through to the 64 bit guest OS. If anyone knows of a workaround, short of telling me to run x86 based guest OS (LOL..that’s the obvious easy answer), it would be really appreciated as I am looking to do testing of 64 bit systems in a virtual environment. Also, I ran VMWare's cpuid.iso utility within VM Workstation in a new x64 VM and the only option that stated "No" was "Supports 64-bit VMWare", which is puzzling, considering my system is already running a 64 bit host OS. Is VM Workstation the culprit indeed being that it's a 32 bit application and if so, will VMWare ever look into developing a true x64 version of it? I'm open to running other virtual layer software to host the ESX VM, so long as it's coded in native 64 bit and will allow me to achieve what I'm looking to do. I already did try Virtualbox in place of VM Workstation to host the ESX VMs' and didn't get very far so I'm stuck with VM Workstation 6.5.

Thank you.

Wed, Mar 26, 2008 Greg Shields Anonymous

To Gary and Anonymous: I tend to use "bridged" for these and use addresses that are in the subnet here offered up by my internal router. For me those addresses are in the 192.168.0.X range, though yours may be different.

Wed, Mar 19, 2008 Anonymous Anonymous

Hi Greg,

I have the same issue, installed esx server, vm with VC server, trying to get them to talk to each other. Using esx 3.02 and 2.02 vc. My pc is connected to a dsl router, so I guess isp sets IP address, but either way cannot get 2 vms talking to each other. followed xtravirt document, but we are missing something... please advise

Fri, Mar 7, 2008 Gary Bledsoe Chicago, IL

Greg, I've successfully created my two ESX servers and am trying to complete configuration by connecting via browser from an SP VM (also in workstation). I can't seem to create the browser connection to either IP I set on the the servers. Did you do something special in setting IPs on all machines to enable them to see each other? What would you recommend for the IP for the XP machine and should they all be set to Bridged or Local? Thanks for the assistance.

Wed, Jan 2, 2008 Stathy G Touloumis Chicago, IL

I am attempting this with Workstation 6 on a SLED 10.1 (SUSE Enterprise Desktop) platform.

For some reason I am not able to get connectivity to/from the host to/from the esx server. I have tried different types of networking (bridged, host only, etc.) using a static ip. All the network settings appear to be correct. This same ESX guest works fine on a windows platform. Any ideas?

Mon, Nov 5, 2007 Greg Shields Anonymous

To be honest, I'm not sure. I'm not a Mac guy, so I can't talk too much to that platform. That being said, there are specific virtualization extensions that come in Intel and AMD processors that are essentially mandatory for this to work. I don't know if they exist on Mac processors or not. I'd love to see if it were possible. Give 'er a try and drop me an email about it or post it as a comment here. I'd love to find out the answer to your research!

Fri, Nov 2, 2007 Anonymous Anonymous

Instead of running VMWare ESX3 on VMWare Workstation 6, do you think that I could essentially achieve the same goal via VMWare Fusion? Essentially I am trying to build a test lab that would run on a Mac system, ideally I would like to interact with this virtual test lab while running the Mac OS X. Do you think that this would be possible?

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.