This post is part 1 of 4 about using the Baseline Validation Tool (BVT) with Oracle Business Intelligence (OBIEE).
Overview of BVT
With the 12c release of OBIEE, Oracle released a new tool called the Baseline Validation Tool or BVT. This tool allows you to run specific tests against your OBIEE reports, dashboards, and BI Publisher objects to quickly find differences between two environments. This might also be called regression testing. BVT is helpful when you are doing an upgrade to OBIEE or when you are doing development and want to verify that you have not inadvertently affected existing reports with your changes.
While BVT comes with the 12c install, it can be used in 11g or 12c environments. You can even use it to compare two environments as I mentioned if you are doing an upgrade. It’s a simple download and configuration before you can start testing your data and environments, even if you are not on 12c yet.
When configuring the BVT tests, you’ll specify the catalog folder that you want BVT to run against in both environments. If your catalog is not large, you can run this against your entire shared folder. However, most instances of OBI that I’ve seen have a lot of reports that are not important or not used anymore. I prefer to create a new folder for BVT with the critical reports that need testing. Another option is to run BVT multiple times against different folders based on the stakeholders that will be helping with validation or who care about those results. The BVT results are not displayed by catalog folder so it can get messy to read if many non-related reports are tested together.
Tests available in BVT
There are six tests available in BVT. You can run one or all of them in one test. My caution: be careful enabling too many tests in one run. It will run each report for each test run which can obviously take a long time depending on the number of reports and their average runtime. By default in the config file, all tests are enabled. The first thing I do when creating a new config file is to mark all tests as false.
<TestPlugin name="com.oracle.biee.bvt.plugin.ui" enabled="false">
Each test has a parameter for CatalogRoot where you can specify which folder you want to run the test on. If you want to run on the entire shared folder, you would specify value=”/shared”.
<Parameter name="CatalogRoot" value="/shared"/>
Catalog
The catalog test compares the metadata of the objects within the tested folder. Most importantly, this checks the permissions of the objects. The following are tested in this plugin test:
- Catalog item targetpath
- Catalog item type
- Catalog item caption
- Catalog item signature
- Some of the catalog ACL attributes that belong to accesscontroltokens, such as account names, account types and permissionMask
Reports
The reports test exports the results of the analysis to compare the data values returned. This test can export to csv, Excel, and/or PDF. If you choose to export to more than one type, BVT will run the report for each export type. Keep this in mind if you have a lot of long running reports. I typically only run this test exporting to csv and not the others.
Logical Query
This test plugin was discontinued in the latest version of BVT (version 17.3.12)
The logical query test will export the logical sql of the report for comparison. This is the sql that you see on the advanced tab when you are editing an analysis. It’s not the physical sql that will be sent to the database.
UI
The UI test will capture a screenshot of the analyses and dashboards to compare pixel by pixel. This test uses either Firefox or Internet Explorer. You can specify the browser in the config or you can allow it to use the computer’s default browser.
The comparison of the screen captures between the two environments uses an algorithm to score the differences between the images with a score between 0–1. A value of 1 means the two images are a pixel-by-pixel exact match. This plug-in’s parameters allow you to specify the threshold percentage between 0–1 that you would like to use as a passing value. The default is 0.95 meaning that there must be a 95% match between the screen captures for the test to pass for that report.
Dashboard
When you run the Dashboard plug-in test, BVT will create XML export files from each dashboard. This export contains the data from the charts and objects on each dashboard page.
Note: We found an issue when running this test in an 11.1.1.7 environment where the export file included 2 decimal points on the numeric values from the dashboards even though the dashboard itself did not show decimals. If you are upgrading from 11g to 12c, this will cause each report to show as failing the test because of a difference between 11.1.1.7 and 12c in exporting decimals as part of numbers. I’m still working to find a resolution to this issue.
BI Publisher
To test BI Publisher reports between environments, this plug-in takes a PDF export of the report and compares the PDF exports. This plug-in only works with Oracle BI Publisher reports, not Answers or Analysis Editor created reports.
If your BI Publisher reports use parameters, you can specify the values you’d like used in your test by creating a URL with the parameter values filled in. For more information about this process, see the BIPub Parameters Input File section in the next post.
VA
New in the 17.3.12 release of Baseline Validation Tool, the VA plug-in will test your Visual Analyzer (VA) or Data Visualization (DV) projects. This plug-in takes screen captures similar to the UI plug-in for reports. The comparison of the screen captures between the two environments uses an algorithm to score the differences between the images with a score between 0–1. A value of 1 means the two images are a pixel-by-pixel exact match. This plug-in’s parameters allow you to specify the threshold percentage between 0–1 that you would like to use as a passing value. The default is 0.95 meaning that there must be a 95% match between the screen captures for the test to pass for that Data Visualization project.
This plug-in uses either Mozilla Firefox or Internet Explorer which you can specify in the configuration file.
12c Upgrade Baseline Testing
One of the best use cases of BVT is when are working on an upgrade project to the latest release such as an upgrade from OBIEE 11g to 12c. To ensure success with BVT in this use case of an upgrade, there are a few things to keep in mind:
- Find a stable environment that you can test against in both versions of 11g and 12c. For example, stop all development in your 11g Dev environment and point it to a stable database instance. Point your 12c environment to that same database instance. If report development is allowed, BVT may show false negatives as it encounters report differences not caused by the upgrade.
- BVT can take a long time to run depending on the size of your catalog. Come up with a testing plan of specific reports and dashboards that will be tested and the priority of testing and resolving issues on the objects in this list. Consider creating a copy of the reports you are testing into a new catalog folder specifically for BVT since it is not easy to run BVT on multiple folders without running on the entire catalog.
- I recommend only running the report, dashboard, and BIP plug-in tests during an upgrade. The catalog test will fail for each report due to ACL differences in out of place upgrades. The UI will fail due to the new look-and-feel of 12c.
- As you resolve differences in your 12c environment, re-run BVT to ensure that the report now passes.
- Copy your BVT comparison results to an Excel file so that you can take notes and show progress as you work through your large catalog of reports. By putting the results into Excel it’ll be easier to share with others also. One downfall, the test results that hyperlink to an export file will not be able to find the export file. A workaround for this is to save the excel file in your BVT folder and keep this BVT folder on a share drive where paths can be maintained and access is shared with other stakeholders.
Finding BVT Help
Your first option for BVT assistance is to look at the BVTGuide.pdf that comes with the BVT tool download in the docs folder. If that doesn’t help, try Oracle Developer Community Forums or Oracle Support.
Another option for basic help is just calling the base bat file to get a list of help commands. (For those interested in the text representation of this, see below the image)
D:\oracle.bi.bvt-17.3.12>bin\obibvt.bat Oracle BI Baseline Validation Tool (17.3.12)
Usage: obibvt
-displaytests Displays all tests in all installed plugins
-createconfig <filename> Creates a default test run configuration XML
-deployment <deploymentname> Deployment Target in the Config XML file that the tests will run against. The -config argument is also required
-config <testconfig.xml> Specify the XML file that describes the test configuration. -deployment or -compareresults argument is also required
-password <password> The password to be used for UserName
-compareresults <folder1> <folder2>
Compare results of two test runs. -config is also requiredExamples: bin\obibvt.bat -displaytests bin\obibvt.bat -createconfig TestConfig.xml bin\obibvt.bat -config TestConfig.xml -deployment PreUpgrade bin\obibvt.bat -config TestConfig.xml -deployment PreUpgrade -password <password> bin\obibvt.bat -compareresults Results/PreUpgrade Results/PostUpgrade -config TestConfig.xml
