VirtualBox: Make your dualboot virtual

This is the first time I make a blogpost like this, but I just managed to pull off exactly what I wanted and because some guessing (both educated guessing and random guessing) was involved I decided to make a post so my experiences could help others in the future.

After I bought my computer, I partitioned off some space on my main disk (the SSD) and installed Ubuntu on it, making my computer a dual boot. However, I continued the tradition that had been standing for years by barely ever booting into it. Now that I wanted to finally start using linux more at home, one of the things I wanted to do was to virtualize both operating systems from the other. Virtualizing Ubuntu from Windows was up first.

Goal: Set up VirtualBox to let me boot the Ubuntu installation on my physical hard disk.
Host System: Windows 7
Client System: Ubuntu 12.04
Processor Manufacturer: AMD (yes, this sort of plays a role)
Special Info: The windows and linux partitions are on the same disk.

Of course, this may be helpful when using different systems, but you may encounter other problems as well as not encounter some of the problems I came across.

Credit where it's due: I scoured the internet for helpful information, but this was the most helpful resource by far: http://cargowire.net/articles/seamlessubuntuwindows

Note up front: sometimes you need to use sudo or su to get administrator access in Ubuntu. I'll leave this for yourself and not mention it in my guide.

Solution:

Done in the past: install Windows and Ubuntu as a dual boot system.

Step 1
We'll start of doing our linux work. In linux, we'll create a VMDK file that specifies the your HD to be used. As such, we install VirtualBox under linux and use the information from the manual to make an image that points to your physical partition with something like this:

-rawdisk /dev/sda -partitions 3 

Problem avoided: Windows doesn't give enough permissions for this command on any disk that is being accessed and you can't do anything about that if your OS is on the same disk.

Step 2.
We'll also need to make a boot disk image from linux. We're not going to need it till later, but to avoid unnecessary rebooting we'll do it already. For now we'll just copy the grub list from your Ubuntu installation. This goes like this:

mkdir iso
mkdir -p iso/boot/grub
cp /usr/lib/grub/i386-pc/stage2_eltorito iso/boot/grub
grub-mkrescue -o grub.iso iso

(note: grub-mkrescue complained about missing something. The key was just to take that name and install it with apt-get as it turned out that one of its dependencies was simply not marked as such by the installation package.)

Problem avoided: The virtual system as we're creating it will not be bootable, so we have created an external way to boot into it.

Next we're going to move to Windows (after a short stop in the bios) so the three created files (<name>.vdmk, <name>-pt.vdmk and grub.iso) need to be made available in Windows. I'll leave it up to you decide to do this.

Step 3.
We reboot into the bios. In the bios we turn on AMD Virtualization (AMD-V). This is an extension to the processor instruction set which allows moving the virtualization closer to the hardware.

Problem avoided: We're going to turn on an option that requires AMD-V later. To prevent rebooting all the time this was the best moment to do this.

Step 4.
We continue in windows. Here we place the files we made in linux in a place we like them to do (like %UserProfile%\VirtualBox VMs\ which is the folder VirtualBox will create/use for the VM). Note that we'll keep both VDMKs while the iso is just temporary, so if you want you keep can keep it in a less permanent location.
Next, we need to edit <name>.vdmk. Open it in your favorite text editor and find the line with something like this:

RW 39999488 FLAT "/dev/sda" 168148992

In Windows we refer to disks with a different vocabulary, so we need to change this to something like:

RW 39999488 FLAT "\\.\PhysicalDrive0" 168148992

Problem avoided: We created a linux compatible file for use in Windows, so we needed to turn it into a Windows compatible file. Luckily for us, all we need is such a trivial change.

Step 5.
It's time to make the VM. First we need to install VirtualBox. This needs to be done in Administrator mode, or else we might get denied access later on. So, run the installer as administrator. Don't ask me why they made it so that this was necessary, all I know is that it is. Secondly, we also need to run virtualbox as administrator. Once we're running, we can simply follow the wizard and create our VM and choose to use an existing HD, and then select <name>.vdmk.

Problem avoided: We avoided permission problems here. They show up rather randomly when not installing it as administrator. Running as an administrator usually isn't necessary, but it is here as otherwise Windows won't allow us to access the raw disk.

Step 6.
Open the VM's settings and go to Storage. Here we add a new disk. We create a new vdi of a static size of 10 MB. Make sure everything is connected on the IDE controller. The 10 MB disk should be the primary master, the vdmk we created the primary slave and the optical drive the secundary master.

Problem avoided: We ought to be able to install a bootloader on the disk we made (<name>.vdmk sets you up with <name>-pt.vdmk as the MBR, and all the partitions on the rest of the disk, only allowing access to the ones we specified in step 1)but for some reason grub2 doesn't play nice with this setup, so we created the other disk to install our bootloader on.

Problem avoided: I found that the vdmk we created only worked on an IDE controller, so we had to make sure it wasn't on a SATA-controller.

Problem avoided: We need to make sure that the disk we wanted to boot from is in the correct place in the boot order, and by making it the master and the raw disk the slave we did this. (We could have done this in other ways, but this is the way I chose.)

Step 7.
Still in the settings, go to System and then to Motherboard. There, make sure that IO APIC is used. This is what we needed AMD-V for (with this setting on, the VM can't start unless AMD-V is on).

Problem avoided: The fact that the setting mentions we can't turn it off after installing the OS is a good hint as to why we need it. Apparently this functionality is used during normal installation of OS. (Yes, I sort of randomly stumbled upon this solution myself...)

Step 8
Still in the settings go back to Storage and mount grub.iso. Now we can close settings and start the VM. It should start the bootloader from the cd. Then, it should start Ubuntu without any further problems. In Ubuntu we can now install grub to the 10 MB disk with something like this:

grub-install /dev/sda

You can now eject the cd and reboot the VM and start it autonomously. Then you might want to do some customizing to remove the Windows installation from the grub list.

Final note: I do believe that due to the way this works when you install something like a new kernel image which should run grub-install anew, this will only happen for the current way you are booted into the OS. I don't believe this could have been avoided by not making the boot hard disk and using the cd permanently, because then it would just mean that if booted into it through the VM it wouldn't update anything (that is used) at all. In the end, you will just have to update grub manually...