Software and performance testing by means of open source tools without any license fees sounds tempting. But does this also work under high scalability conditions? Is it applicable for agile working methods? Can SAP GUI and Fiori interactions be mapped?
The FIS-ASP test team did not really doubt that when asking these questions. However, one was aware of the particular software requirements. To come to the point: The test was no disappointment, but quite the opposite.
The task specifications were as follows: The software and performance tests of SAP systems were to be executed with Open Source means. The OpenQa project selected for these tests had originally been developed by SUSE and is today an independently developed Open Source project under GPLv2. Many renowned companies in this sector use OpenQa for automated software tests and integrate it into the Continuous Integration processes of the source code management. This enables fully automated tests of entire Linux distributions.
Familiar technologies considerably reduce the respective efforts
As OpenQa can execute and check GUI entries via image recognition as well as analyze sound samples played, it soon became evident that the tasks were not unsolvable. The tests are made within prepared images representing a completely virtual machine. The virtual machines are generated using QEMU. QEMU with its KVM extension is no unknown product to the FIS-ASP test team since it is also authorized for the virtualization of SAP systems and used in the OpenStack cloud.
It can be concluded that initially planned high efforts for such a project will considerably be reduced if the used technologies are already known.
Software and performance tests by means of defined use cases
The first tests showed that OpenQa cannot only test software in an automated way but also the corresponding performance. Here, FIS-ASP developed various scenarios: The first test included the creation of a predefined number of users in the SAP system. All interactions with SAP were made via SAP GUI for Java, which runs on a small Linux VM. After that, further parallel “tests” were made each representing a user. These can be subdivided into different user groups (standard user, power user, controller, management etc.).
Here, it was only important to consider the company-specific use cases in order to reliably measure the SAP system to be tested under load situations and precisely enter performance data.
Test management controlling
The different tests are defined in JSON files that can easily be processed. They can also be adjusted in a script-controlled way. Once all tests have been executed, another test can be used to again remove the previously created users in SAP.
Tests of browser-based user interfaces, such as Fiori apps, are to be executed in the same way as tests based on SAP GUI. Here, the FIS-ASP testers used Chrome and Firefox as SAP-supported browsers.
Scaling with adequate hardware efforts
The scalability requirement was to be able to execute performance tests with appropriate hardware efforts for a total of over 10,000 users. The hardware scalability was already ensured by OpenQa and Multinode could be configured in the user interface without any problems there. Since, however, every test represents a virtual machine on the OpenQa server, FIS-ASP analyzed some options in the Linux memory management. The best results were achieved with KSM (Kernel Samepage Merging). Here, identical memory pages are stored in the working memory only once and presented to all processes that need these pages.
Further memory tricks such as working memory compression with ZRAM (Virtual Swap Compressed in RAM, aka zSwap or, previously, “compcache”) were rejected again as they would have been comparatively complicated for the CPU. Concerning the CPUs: A high number of kernels is recommended for the workload in the test environment – nothing else could be expected due to the high degree of parallelization.
One comprehensive test environment can replace many individual ones
This is the quintessence: Due to the multitude of different tasks that can be solved with OpenQa, the quantity of the different tools that should be used otherwise will considerably be reduced. The solution can create useful amendments of SAP and surrounding systems and, at the same time, keep the amount of work low due to the high automation degree.
Moreover, working with OpenQa means you always find new use options, such as the execution of standardized reworks after system copies or simple activities like password changes. The integrated control of all steps made in the corresponding tests enables you to control and document the success of the activities.
(Manuel Sammeth)