Groovy, Web Services and OBIEE 12c

If you’ve followed me for any length of time, you know I’m a fan of the Groovy language. It’s unfettered support for closures has made it a popular choice for writing domain-specific language (DSL) projects such as Gradle and Jenkins, and the healthy dose of syntactic sugar spooned in at runtime means that developers spend less time boiler-plating and more time implementing. I first started writing Groovy when Oracle acquired Sunopis(now Oracle Data Integrator), as Groovy was and is the standard way to script the ODI SDK. I was hooked. If you ever read my work on that other blog, you know I once proposed Groovy as the definitive choice for interacting with JMX MBeans. By the way… it’s a shame their new blog format removed my embedded video of the Matrix Architect… which is the main reason I beefed up the Matrix imagery here.

UPDATE: Thanks Robin Moffatt for “refactoring” my old post and putting The Architect back in.

In OBIEE 12c, JMX Beans are gone, and in it’s place, we’ve inherited extended SOAP support in some cases, but gloriously… new RESTful services for most of the lifecycle management capabilities. Did this cause me to kick Groovy to the curb in favor of groovier languages like JavaScript or Python for OBIEE 12c scripting? Not a chance.

The Matrix Reloaded

I love to hear the scoffs when I say “SOAP” aloud to other developers. It’s easy to forget that web services were invented with SOAP, and without this important link in the chain, we wouldn’t have RESTful APIs, microservices, or any of the new JavaScript-centric projects and platforms so common today. SOAP is an aging standard though, and certainly not the flavor of the month… but the good news for Groovy developers is that groovy-wslitesupports SOAP in the same groovy way it supports REST. We’ll experience Groovy with SOAP with a new service in OBIEE 12c… one giving us the equivalent of “Reload Files and Metadata”.

All the SOAP stuff is documented and completely supported… at least as much as anything is with OBIEE. So lets take a peek at how Groovy makes all of this easier. The SOAP services in OBIEE require that we first get a session id, so we call a separate SOAP method to get the initial session id before we call the method we’re actually interested in. This is easy with groovy-wslite:

@Grab(group='com.github.groovy-wslite', module='groovy-wslite', version='1.1.3')
import wslite.soap.SOAPClient
import wslite.soap.SOAPClientException
def session = new SOAPClient(
  'http://localhost:9502/analytics-ws/saw.dll/wsdl/v8?SoapImpl=nQSessionService'
).send(
  SOAPAction: 'http://localhost:9502/analytics-ws/saw.dll/wsdl/v8?SoapImpl=nQSessionService'
) {
  body {
    logon('xmlns': 'urn://oracle.bi.webservices/v8') {
      name('weblogic')
      password('Admin123')
}
  }
}.envelope
println session

Let’s walk through this. The Grab annotation is a built-in (or in-built for my British friends) way in Groovy to pull down JAR files from Maven repositories… by default using Maven Central. When compiling Groovy we would typically use a build tool such as Gradle to manage dependencies… but Grab is great when doing lightweight scripting jobs. The rest of the code is specific to the web service we are using. The SOAPAction URL specifies the particular web service we are using… obviously the host and port will vary depending on the OBIEE installation.

The body and envelope structure in SOAP is just a standard way to structure a payload for passing to a web service — with the bodydescribing the method we want to call — in this case the logon method which returns the session id we are looking for. This gives us the following output… a simple print of the session id we captured:

iqaa0dofih4625lrsks9ja466q1ii7jhaid8taduha2ldram

So building from this, we need to call the actual reloadMetadata method in a similar way, referencing the session id we just returned using the variable session which I defined in the code above, so consider this code as a continuation of what’s above:

new SOAPClient(
  'http://localhost:9502/analytics-ws/saw.dll/wsdl/v8?SoapImpl=metadataService'
).send(
  SOAPAction: 'http://localhost:9502/analytics-ws/saw.dll/wsdl/v8?SoapImpl=metadataService'
) {
  body {
    reloadMetadata('xmlns': 'urn://oracle.bi.webservices/v8') {
      sessionID(session)
    }
  }
}.envelope

This gives us the ability to call “Reload Files and Metadata” programatically… something that was typically hacked together with JavaScript in an ugly way in OBIEE 11g.

Getting Forensic All Up in Here

To use RESTful web services, we need to know the URL of the service we are interacting with, the different parameters to pass, the content type of those parameters, and the content type of the response payload. But the RESTful services we’ll be using in this post are undocumented in OBIEE 12c, and I would imagine unsupported. So how do we go about about determining all of this?

 

Now seems like a good time for the lawyers to get involved. Although the SOAP stuff is heavily documented, the RESTful stuff is purely caveat emptor. I’ll quote my good friend Robin Moffatt for this… it’s a fine denial of warranty:

None of these Web Services are documented, and they should therefore be assumed to be completely unsupported by Oracle. This article is purely for geek interest. Using undocumented APIs leaves you at risk of the API changing at any time.

But hey: with the simple patch upgrade from 12.2.1.0 to 12.2.1.1… Oracle changed the name of the supported API from data-model-cmd.sh to datamodel.sh, broke it, and now rely on an accidentally exposed functionality for a workaround. Meanwhile, all the RESTful methods I’m describing here were unchanged. So which one put you more at risk…I’m just saying…

 

There are a variety of web service “sniffing” approaches, including projects like Fiddler or Wireshark, or command-line utilities like sysdig. Generally, we would listen to traffic using one of these tools while making a call to the REST API… for instance, using datamodel.sh/datamodel.cmd in 12.2.1.1, and then watch the output from these sniffing tools. While this will certainly work… OBIEE 12c is already logging all REST API calls against bi-lcm, the RESTful web service for all lifecycle management functionality. In the <domain-home>/servers/bi_server1/logs directory, we see the bi-lcm-rest.log.0 file, which has all the calls to bi-lcm there. Let’s take a look at this in action.

The Matrix Uploaded

I’m going to upload a binary repository to OBIEE 12.2.1.1 using the following command:

[oracle@localhost]$ datamodel.sh uploadrpd -I current.rpd -W Admin123 -U weblogic -P Admin123 -SI ssi -S localhost -N 9502 -D
Service Instance: ssi
Operation successful.
RPD upload completed successfully.
[oracle@localhost repository]$

What do we see in the bi-lcm-rest.log.0 file? For one thing… the binary data from the RPD is written to the log file, which muddies the water a bit about what’s there… but we can make out the POST structure, which includes the URL, followed by the multi-part data elements. I’ve cleaned up the results a bit to make it clear:

200 > POST http://localhost:9502/bi-lcm/v1/si/ssi/rpd/uploadrpd
200 > Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
200 > Authorization: Basic d2VibG9naWM6QWRtaW4xMjM=
200 > Connection: keep-alive
200 > Content-Length: 30228
200 > Content-Type: multipart/form-data;boundary=Boundary_1_1813187653_1472059721163
200 > Host: localhost:9502
200 > MIME-Version: 1.0
200 > User-Agent: Jersey/2.22.1 (HttpUrlConnection 1.8.0_101)
--Boundary_1_1813187653_1472059721163
Content-Type: text/plain
Content-Disposition: form-data; name="rpd-password"
Admin123
--Boundary_1_1813187653_1472059721163
Content-Type: application/vnd.oracle.rpd
Content-Disposition: form-data; filename="current.rpd"; modification-date="Wed, 24 Aug 2016 13:22:06 GMT"; size=29708; name="file"

This gives us all we need to construct code to call the API, which is simplistic and easy to follow using Groovy and groovy-wslite:

@Grab(group='com.github.groovy-wslite', module='groovy-wslite', version='1.1.3')
import wslite.http.auth.HTTPBasicAuthorization
import wslite.rest.RESTClient
def client = new RESTClient("http://localhost:9502/")
// basic authorization
client.authorization = new HTTPBasicAuthorization('weblogic', 'Admin123')
def response = client.post(path: '/bi-lcm/v1/si/ssi/rpd/uploadrpd') {
multipart 'file', new File('current.rpd').bytes,
            "application/vnd.oracle.rpd"
multipart "rpd-password", "Admin123".bytes
}
println JsonOutput.prettyPrint(new String(response.data))

Let me walk through a few of the lines. We need to define our client object, providing the address of the OBIEE 12.2.1.1 server, and then use basic HTTP (username and password) authentication against it.

Following the construction and authorization of our client object… we get into the details of the POST method. Notice that, instead of a highly-overloaded API to handle all the different permutations of RESTful methods… we get convenience DSL methods specific to the different method types, in this case the POST. Also, notice we only pass a single explicit parameter to the POST method — the path of the service — while the balance of the parameters are expressed using a closure and DSL content. In the DSL we are able to expressively describe our content to the POST, and the closure structure means we don’t have to worry about API ordering, etc. In this example we need to describe two multi-part REST parameters. Multipart parameters are how we pass content like file data described with content “labels” and specific boundaries as a way to delineate the different parameters… sort of like how we handle attachments in email protocols. This is dead-simple in Groovy: we can pass the binary representation of any data (including that from Files or Strings) by simply using the bytes method as a core part of all Groovy objects, including File objects but also String objects.

Groovy-wslite does something especially nice with the response object… it automatically renders the data attribute — a single attribute in a complex object returned as the payload — as a byte[] datatype. If you see how a cURL call presents this data back… it dumps the binary data which is difficult to render in certain terminals and text editors (this binary data is also what is written to the bi-lcm-rest.log.0). For instance:

[oracle@localhost repository]$ curl -X "POST" "http://localhost:9502/bi-lcm/v1/si/ssi/rpd/downloadrpd" \
> --data-urlencode "target-password=Admin123" --basic --user weblogic:Admin123
d
��2016-08-26 08:21:55.560
 >:HO۟G�1�is3��O�>�:XJ�.9�Gc�{n�0��q`o�+��-L&���?2o���OO��M.o���IY�X�*=��(�9B9|�%��
6���X�����s
                   ��}='^X،VT��]�����QƢ�ÚY�c�9�?��G���^Q��"��=��0O�
                                                                         Kh��w�
�-��e�ʩz�2���*���O��|
�{L�[iCfv�)�v)��'����|�CM'yJ�<C/;�
                                       ��g6p��&�#��:�w�~���L
                                                               �C����Q,��xD.@Z���s��C�h�]�PrJD�\�6Z����0�m� %�$y�A��=Y�C[����
P��|#:�M�T����2����hK�,M�'����V�"��X�q������3�p֠�9��9z
                                                       ��hB�m��ڮU&&p1�9k�S�ﴪ�
[Abbreviated for clarity and sanity...]

With cURL, you could easily redirect this to a file and poof get a binary RPD, but that seems a little… well… messy. With a byte[] datatype, it’s a matrix of numbers stored in nested arrays that we can easily convert to a String to print the payload (as we are doing here, including formatting)… or as we’ll see in the next section… use to easily write binary data to a file.

So when we execute the above code, we get the following JSON result, which I am also formatting using the groovy.json package:

{
    "clazz": [
        "rpd-response"
    ],
    "links": [
        {
            "href": "http://localhost:9502/bi-lcm/v1/si/ssi/rpd/uploadrpd",
            "rel": [
                "self"
            ]
        }
    ],
    "properties": {
        "entry": [
            {
                "key": "si",
                "value": {
                    "type": "string",
                    "value": "ssi"
                }
            },
            {
                "key": "description",
                "value": {
                    "type": "string",
                    "value": "RPD upload completed successfully."
                }
            },
            {
                "key": "desc_code",
                "value": {
                    "type": "string",
                    "value": "DESC_CODE_RPD_UPLOAD_SUCCESSFUL"
                }
            },
            {
                "key": "status",
                "value": {
                    "type": "string",
                    "value": "SUCCESS"
                }
            }
        ]
    },
    "title": "RPD-LCM response, SI=ssi, action=Upload RPD"
}

The Matrix Downloaded

The OBIEE crowd is familiar with the uploadRepository JMX MBean from OBIEE 11g… Fusion Middleware Control interacted with it every time a binary repository was uploaded using the browser. But we also have the downloadRepository MBean in 11g, it’s just not exposed in Fusion Middleware Control anywhere. We have a comparable downloadrpdRESTful method which is exposed using datamodel.sh:

datamodel.sh downloadrpd -O current.rpd -W Admin123 -U weblogic -P Admin123 -SI ssi -S localhost -N 9502

…which produces the following in the log file:

209 > POST http://localhost:9502/bi-lcm/v1/si/ssi/rpd/downloadrpd
209 > Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
209 > Authorization: Basic d2VibG9naWM6QWRtaW4xMjM=
209 > Connection: keep-alive
209 > Content-Length: 24
209 > Content-Type: application/x-www-form-urlencoded
209 > Host: localhost:9502
209 > User-Agent: Jersey/2.22.1 (HttpUrlConnection 1.8.0_101)
target-password=Admin123
Aug 24,2016 16:31:09 oracle.bi.lcm.rest.app.LCMApplication INFO - 209 * Server responded with a response on thread [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'
209 < 200
209 < Content-Length: 30220
209 < Content-Type: application/vnd.oracle.rpd

So with the method parameters and the content types known, we can write the following Groovy code to download a binary repository:

@Grab(group='com.github.groovy-wslite', module='groovy-wslite', version='1.1.3')
import wslite.http.auth.HTTPBasicAuthorization
import wslite.rest.RESTClient
def client = new RESTClient("http://localhost:9502/")
// basic authorization
client.authorization = new HTTPBasicAuthorization('weblogic', 'Admin123')
new File('current.rpd').setBytes(
client.post(path: '/bi-lcm/v1/si/ssi/rpd/downloadrpd') {
     urlenc "target-password": "Admin123"
  }.data
)

Everything is the same until we get to the actual POST portion of the method… where we see the urlenc property. This is a shorthand way of expressing that the target-password parameter is a URL-encoded parameter… meaning it’s encoded in such a way to protect URL-specific literals from getting in the way. Let’s be honest: it’s not necessary to understand this content type except to say… the bi-lcm downloadrpdmethod expects a URL-encoded content type, and this is how we can pass it. We have the entire POST method, plus the return of the data result, wrapped in a setBytes() method, which will write the binary data returned from the POST into a new repository binary file.

Conclusion

I love Groovy… perhaps that is clear. We can mix and match compliant Java and Groovy in the same class file (heck… on the same line) and the Groovy compilier will make sense of it. This is why I would switch every single Java shop to Groovy immediately… especially when using a build tool like Gradle that can compile our code against the JVM regardless of which language it’s written in. We can strongly type, weakly type, do both from one line to the next, all while focusing on delivering an end product instead of reusing the same boiler-plate code time after time.

But closures are the real reason that projects like groovy-wslite are so perfect, and how they make DSL possible. It’s a pleasure to use external projects when the implementation methods are self-describing. It’s the difference between a formal dinner and a casual get-together: there’s a time and a place for each, but we all know which one is more fun.

For more data tips and tricks, check out our blogs or browse the RPA blogs at Medium.