I’m super-excited about a presentation I’m giving with Kellyn PotVin-Gormanat RMOUG Training Days next month, called Performance Analytics: Visualizing the Power of the AWR Warehouse. Alright… if you insist… I’ll give a quick plug. Kellyn and the team have been working to build the AWR Warehouse component of Enteprise Manager to store Automatic Workload Repository (AWR) history for the entire enterprise in a single repository… all managed by EM. Pretty slick right? I thought… it would be really slick to see what OBIEE 12c could do against the AWR Warehouse… and well, a presentation was born.
For this presentation, I need an Oracle Database 12c running on Windows. The Windows part is really only a half-truth… I could easily build the Database, EM, AWRW and OBIEE 12c on Linux… thats the standard deployment scenario for most of our customers. But for those OBIEE junkies out there… you know the development of the Metadata Layer is unfortunately still done using a Windows-only utility called the Admin Tool. I want an all-in-one VM I can use to easily show the Metadata Layer during the presentation… and I don’t like the requirement of running two separate VMs during the demo if I can help it.
Oracle Database 12c on Windows… I’m Getting to That
I have a reasonably new Apple MacBook Pro. I’m not bragging… this is actually important, but you have to trust me on that for a moment. So anyway… I use VMware Fusion whenever I can… it was my first, and as they say, you always remember your first. Of course I also use VirtualBox quite a bit as all the Oracle VMs come on ViritualBox, including the fantastic SampleApp and the equally fantastic Big Data Lite VM. For this task though I started with VMware Fusion, and created a Windows 7 VM, downloaded the Oracle Database 12c installation files, and started the installer. I choose the “Create and configure a database” option, and the initial installation of the binaries completed just fine, but then the Database Configuration Assistant kicked off (at Step 8, exactly 83% through the entire process), and then gridlock. No movement. The last thing in the logfile was “Starting Database Creation Assistant…”, and then silence. Crickets. I gave it quite a bit of time to finish… way too much time actually. I helped the kids with homework, folded clothes, watched the newest episode of Mr. Robot. Still… it never moved past 83%.
Through the troubleshooting process, I tried all of the following, in different measures and combinations, with increased dosages of frustration and desperation:
- Installing on Windows Server 2012 instead of Windows 7.0.
- Using VirtualBox instead of VMware Fusion.
- Downloading software from eDelivery versus downloads.oracle.com.
- Using a Windows Built-in Account to own the Oracle Software (more like 11g) instead of using an Oracle Home User (new in 12c).
- Running the software installation alone, followed up by the Database Configuration Assistant, instead of one fell swoop.
- Eliminating the listener.ora file and running in Listener Registration mode (or whatever that feature is called).
- Mucking about with the hosts file, with ORACLE_HOSTNAME, and even (God forbid) installing a Microsoft Loopback Adaptor.
- Foreswearing the DBCA all together, and running an actual CREATE DATABASE command. I was an Oracle DBA back in the day… and this was the only way we did it, so I remembered (sort of) how this works.
None of this worked. I got the exact same result every time. I couldn’t get a database created. However, it wasn’t until I started digging around with fix #8 above that I started to get some hints into the issue.
Clues Left at the Scene of the Crime
So the title of this article is a bit dramatic… I actually know that Oracle Database 12c will run on Windows… because I have personally installed and configured it many times. I’ve done it before in the past on VMware Fusion and VirtualBox, and have done it even recently on Amazon EC2. While building my new Performance Analytics VM (last plug, I promise), I spun up one of the working Amazon EC2 instances for a sanity check and found… nothing. Nothing interesting at all. No Loopback Adaptors, no ORACLE_HOSTNAME, no hosts file magic. Just a perfectly-functioning Oracle Database 12c with nothing exceptional about it. I went back to investigating one of my failing VMs where I had only installed the Oracle Database software, but had not run the DBCA, and finally found a clue. None of the networking tools really worked. I tested out lsnrctl and tnsping on the command line, and though they executed and provided the expected feedback, they never completed, but instead, just hung there.
So instead of focusing on the database configuration, I started to think about the database installation… which up until this point I had taken as a given. I knew there was something not right about those TNS utilities… as if they hadn’t compiled 100% effectively. And I suspected the issue had something to do with virtualization… which was my second clue.
Research from Tim Hall, Laurent Leturgez, Danny Bryant and Mike Dietrich
Back in October, I was trying to get Oracle Database 12c installed on Oracle Linux 7.0, and I stumbled upon what seemed like a virtualization-only issue with Linux as well… this one exposed via the excellent work from Oracle Base Tim Hall. Tim had identified an issue where the Perl installation for Database 12c was hosed and had to be replaced. Tim didn’t really go into the ends and outs of the problem, but he did reference a post from Laurent Leturgez that explained a workaround by downloading a new Perl binary to replace the current one in the Oracle installation. That solved the Linux installation at that time, but now with the Windows issue, I wondered if they weren’t somehow related, specifically because both seemed specific to VMware and VirtualBox. I revisited Laurent’s blog on the issue, and found a recent update on the page referencing a blog post by Danny Bryant that explained the issue being related to newer versions of virtualization software coupled with level 4 cache on newer Intel chips. And the final piece of the puzzle was this blog post from Mike Dietrich at Oracle that tore the case wide open.
As Mike explains… this is not about virtualization software specifically… it just seemed that way. Instead, it has to do with the new cutting-edge Iris Pro processors used by Apple in the newest MacBook Pros. Oracle software isn’t ported to MacOS, so the only way to run it on these brand new Iris Pro chips is to use virtualization software. The links listed above have more detail… but I’ll quote a small piece of Mike’s blog below for context. But note… when he discusses the Perl binary… understand that it’s not just Perl on Linux… but also the compilation of many of the TNS utilities on Windows it would seem, and who knows what else:
The current assumption is that Intel CPUs with Iris Pro graphics are affected. Iris Pro means eDRAM (embedded DRAM) which is reported as 4th level cache in CPUID. We have confirmed that Crystal Well and Broadwell CPUs with Iris Pro are affected. It is likely that the Xeon E3–1200 v4 family is also affected.
It seems to be that there’s a bug in the Perl binary. It links against ancient code from the Intel compiler suite doing optimizations according to the CPU features. Very recent Intel CPUs have 4 cache descriptors.
People who encountered this used Virtual Box VBox 5.0.x — and it passes this information to the guest. This leads to a problem within the Perl code. You won’t see it on VBox 4.3 as this version does not pass the information to the guest.
But actually it seems that this issue is independent of Virtual Box or any other virtualization software. It simply happens in this case as many people use VBox on Macs — and some Macs are equipped with this new CPU model
The reason I hadn’t seen this issue previously using Oracle Database 12c on VMware or VirtualBox is these level 4 cache instructions weren’t passed through in previous versions of the virtualization software, or the MacBook Pro I was using at that time wasn’t outfitted with the Iris Pro chipset. Although Danny and Mike’s blogs referenced above provide the solution that finally worked for me… I’ll include it here as well. In VirtualBox, execute the following command-line calls for each VM that needs to have the Oracle Database installed on it. In this example, “awrw” is the name of my VirtualBox VM:
VBoxManage setextradata awrw “VBoxInternal/CPUM/HostCPUID/Cache/Leaf” “0x4”
VBoxManage setextradata awrw “VBoxInternal/CPUM/HostCPUID/Cache/SubLeaf” “0x4”
VBoxManage setextradata awrw “VBoxInternal/CPUM/HostCPUID/Cache/eax” “0”
VBoxManage setextradata awrw “VBoxInternal/CPUM/HostCPUID/Cache/ebx” “0”
VBoxManage setextradata awrw “VBoxInternal/CPUM/HostCPUID/Cache/ecx” “0”
VBoxManage setextradata awrw “VBoxInternal/CPUM/HostCPUID/Cache/edx” “0”
VBoxManage setextradata awrw “VBoxInternal/CPUM/HostCPUID/Cache/SubLeafMask” “0xffffffff”
These mods will persist through VM clones… so you could execute the instructions once on a Windows VM, and use that VM as the source of all future clones. I haven’t found any comparable solution with VMware… perhaps some one out there knows a person who knows a person who knows a hack.
