This started due to a problem the customer had in their test-environment.
During big test-scenarios, they needed a large amount of testers to execute different tasks. Several times, they encountered problems with the applications where they did not know if the application worked before they started testing, if the data input was wrong(functional errors) and such. This lead to a huge amount of hours being spent, without being able to test while they waited for AD/AM to find the problem. This was a cost- and time-consuming situation that they would like to avoid. They wanted Synthetic tests to be executed on all applications on the middleware plattform.
Dynatrace on the middleware plattform
We had already installed the monitoring and analysis system Dynatrace in the environment to get full insight into the operations inside the JVM’s
Du to license cost we could not install the DT agents on all JVM’s. The test-environment is built up by Loadbalancer in front, Webservers, Websphere ODR’s, multiple Websphere application servers with 26 Clusters running 157 WebSphere enterprise applications, DB’s, MQ and support/legacy systems. These applications are divided into Connection Layer, Application Layer, Façade Layer, Service Layer and System Layer.
Due to small amount of traffic on the test-system during non-test-period, they could not see if it was working or not in dynatrace, since there where almost no transactions running.
I wanted it to run every 5 minutes, forking all the defined soap-generator and logging the errors.
The functionality of the soap-generator
The Business Transactions
AD/AM had to document the Business Transactions for each application in Dynatrace.
They had to document the classes and methods in use for each application.
We ended up with prox. 30 business transactions (BT) containing prox. 150 classes and 300+ Methods.
The BT’s is using count as Aggregate in the filter for the BT
Which Webservice and which application?
Now I had to find out which webservices was running in front of these applications(classes) in use in the Business Transactions defined. I had to try to find the webservices that triggered most applications and legacy systems, to avoid having to create webservice-requests for each application.
I started by doing drilldown to purepaths in Dynatrace for each BT. This showed me the webservices that triggered these classes. Now I could download the wsdl’s from the webservice and import it to SoapUI. Since I wanted to avoid doing changes in the applications repository, I avoided using methods that executed Update, Create and such. Preferably, I wanted to use DeepPing methods where that was available. I went into the production systems and performed tcpdumps on the JVM’s webcontainer for the http transport port, to get valid data to enter into the soap-request. It would take too long time to get AM to provide me with all this data and I needed to test the requests as well (to find the most efficient request to use). Most of the services was using basic authentication, so I added the possibility for basic authentication to the soap-generator.
The first thing I created was the page for creating the clusters, connected to server with the port for the HttpQueueInboundDefault traffic in the webcontainer of the Websphere applicationservers.
This page is protected with a login prompt.
Then I created the Probe Admin page (with login prompt). Where you could define webservice request to Cluster.
And yes, I did not use to much resources in the design , since I had a lot of other tasks to do as well.
After creating the soap request, you can test it by pressing the green arrow. If it fails and you need to edit it, you can press the pencil symbol or the red X to delete it.
The request executes toward all of the clustermembers defined in the “Admin Servers” page.
Now I also wanted these probes to be available for the AO-team in India, so they can test the webservice during error or after changes in the applications.
This will contain the probe name, service(url), Cluster(endpoint), basic Auth. User and the possibility to execute the probe.
This will give a result to the user based on checking if the webservice was available and searching for the needle in the haystack. If it fails, it the background of the response turns red and shows the error from the webservice.
We also wanted to see the synthetic requests created by my soap-generator, to differentiate between functional and technical errors when a BT turns red. I added my own user-agent info in the header of the soap-requests and created a Business Transaction that looked for that user-agent.
The Final Dashboard in Dynatrace
The overview of the applications (showing both syntetich and usermade requests).
Tagging the requests with a specific User-Agent, gives us the possibility to connect the syntetich transactions to applications and see the static tests that fails.
You can now drill down to the purepaths that fails or take a long time and see what actually fails or where the transaction is using most of the time.
Dynatrace is a really powerful tool which gives you insight in the transactions, classes, methods, database and more. It tags the IP packet, so you can follow the transactions through several systems that has dynatrace agents installed. You can use it in all of the IT Quality Tools Quadrant.
So if I could roll the dice, Dynatrace would have got 5/6…