Help! The WebLogic Dog Ate My Pages!

OBIEE 11.1.1.9.0 is a big step up from the earlier versions, and there are plenty of great things to appreciate about it. However, one thing that I am not that excited about is an error that crops up from time to time in Enterprise Manager. The mystery error looks a little like this:

The JSP error occurs when navigating into or around the Core Application pages in Enterprise Manager. You may be moving from the Deployment page to the Overview page and “Boom!”, Oracle serves you up with an error that you have never seen, nor does it make much sense when it first comes across your screen.

So… What the heck is an OracleJSP error anyway?

If you’re working in Enterprise Manager, chances are you know that OBIEE “sits” on top of WebLogic, which facilitates admin tasks, queries, networking and all sorts of other fun stuff. WebLogic uses Oracle’s JavaServer Pagestechnology to build the web pages for things like OBIEE, Enterprise Manager and the Admin Server Console. This allows WebLogic to use files to dynamically create and recreate web pages as the system’s architecture changes. In layman’s terms, the error “OracleJSP error:java.io.FileNotFoundException: MDS-00013: No metadata found for metadata object” means that Oracle JSP cannot build the web page because it there is no information. Bizarrely, once this error has arrived, it will persist throughout the entire Core Application suite of pages in Enterprise Manager.

Hang on a second… if Oracle JSP can’t build the page because it doesn’t have information, how did the page get built and rendered in the first place? Interestingly enough there are two My Oracle Support documents (here and here) that address similar errors and neither one lists a definite cause. Peculiar, right? However, I have an idea of what is happening based on the Oracle prescribed solution.

The Solution

The solution to this error is to remove the tmp directories for both the Admin Server and the BI Server (see the how to guide below). My theory is that the .lok files in these locations become corrupted or expire, and without them, JSP can’t build the pages. I also think that this can happen more regularly in an environment where the servers are frequently changing states from running to shutdown (such as a development environment). This is also in line with another error I was having recently with .lok files, the BI Server and Node Manager (which you can read about here). Since JSP will create the files in those tmp directories if they don’t exist, stopping the servers, removing the files and starting the Admin and BI Servers should do the trick.

Wrenching on the Server (The How To Guide)

If you’re really hard core, you can do all of the following via command line. Feel free to use a GUI tool like I did below.

Start by logging into your Admin Console and shutting down the Admin Server and BI Server.

Once the Admin Server is shut down, navigate to the Admin Server’s tmp folder, and delete all of the contents. The path should be: <Middleware_Home>/user_projects/domains/bifoundation_domain/servers/AdminServer/tmp

After you’ve deleted the entire contents of the Admin Server’s tmp folder, navigate back to the servers folder in the bifoundation_domain folder. Then navigate into the bi_server1 folder, and the tmp folder that is within that bi_server1 folder. Delete the contents of that folder as well.

After you have deleted the files in both the Admin Server tmp folder and the BI Server tmp folder, open a command prompt or terminal and navigate to the bifoundation_domain folder. The path should be: <Middleware_Home>/user_projects/domains/bifoundation_domain Once there, run the startWebLogic.sh command (or startWebLogic.cmd if you are on Windows). When the command finishes, you should see an output similar to the below screenshot.

Once your Admin Server is back up, log in the Admin Console and start the BI Server 1 (or all of the managed servers).

Once your servers are back online, navigate to Enterprise Manager and check the Core Application pages that were producing the error before. You should have no issues navigating between the pages now.

Conclusion

Bug? Feature? Fluke? I’m not sure where to place this error (maybe not in the “feature” bucket”), but I am keeping my eyes peeling for other OBIA/OBIEE 11g oddities that may have to do with tmp folders and .lok files. With even Oracle unsure of what is causing these errors, I am guessing that this will be on the usual maintenance “to-do” list until we see a patch.