FIELD OF TECHNOLOGY
[0001] This disclosure relates generally to the technical field of software performance evaluation
and, more specifically to a method and/or a systém for evaluating performance of a software
application through run-time assembly code execution.
BACKGROUND
[0002] Popularity of cloud deployments has brought forth several challenges. One of the
challenges being performance of a software application under different operational workloads.
Cloud computing environments, for example Software as a Service (SaaS) may háve multiple
customers using a common service. Each customer should potentially háve unique workload
characteristics.
[0003] An underlying service may use a set of software programs and/or an architecture for a
typical deployment and/or a specific workload. Any variance from the unique workload
characteristics shall deteriorate the performance of the software application resulting in service
level agreement violations. Often performance enhancements may háve been brought by making
changes to a source code, an architecture and/or a hardware capacity of the systém.
[0004] There may háve several performance evaluation techniques based on a software
development lifecycle. The performance evaluation techniques may include code profiling, stress
tests and/or capacity management.
[0005] Mutation testing has been used extensively to verify correctness of a software
functionality and/or a test suite to be used to test the software application. The source code and/or
an assembly code may be changed to mutate the source code and/or the assembly code. The
mutated code may semantically be different from the originál source code. The mutated code when run may expect to throw an exception, and/or show incorrect results at a point where the
mutated code may háve been introduced.
[0006] Absence of the exception and/or the incorrect results may prove that a part of the source
code of the software application may be redundant and/or the test suite may be inadequate. Thus,
may not cover the mutated part of the source code. In other words, the mutation testing may be
considered as a means to verify completeness and/or correctness of the test suite.
[0007] Software Implemented Fault-injections (SWIFI) may relate to injecting faults in the
source code and/or into a memory address of the software application just before execution. The
SWIFI techniques may be used to test either fault tolerance of the systém, and/or study failure
modes. The SWIFI may operate with the assembly code, where at run-time various instructions
and/or dáta may be manipulated to change randomly and/or in a pre-defined location. Šuch a
manipulation may be a single bit and/or a multi bit flip.
[0008] The source code of a software application written in high level languages šuch as
COBOL, C, C++, JAVA may háve become prevalent. There may háve been various
consequences of providing the source code for testing the software application. Intellectual
property (IP) risk may háve been main concern in organizations. There may háve the source code
changes during the testing of the software application. Bugs may also be added during the testing
of the software application.
SUMMARY
[0009] Software performance evaluation by run-time assembly code execution is disclosed. In
one aspect, a method includes evaluating performance of a software application in a dáta
processing systém. A plurality of program code lines of the software application stored in one or
more computer databases are analyzed by one or more computing devices. The one or more
computing devices access the one or more computer databases through a computer network. One or more equivalent program regions within the plurality of program code lines are identified. One
or more markers in the identified one or more equivalent program regions are inserted. The one or
more markers are stored in the one or more computer databases. The plurality of program code
lines are compiled and assembled by a compiler and an assembler respectively to generate an
executable code. The executable code includes a plurality of instructions. Performance metrics of
the software application is measured recurrently by manipulating the plurality of instructions
based on the one or more equivalent program regions identified by the inserted one or more
markers and executing the executable code.
[0010] The one or more equivalent program regions may include one or more nested control flow
statements and loop statements. The executable code may include at least one of an assembly
code and a byte code. The executable code may be executed by subjecting the executable code to
one or more workloads stored in the one or more computer databases. The plurality of instructions
may be manipulated by changing at least one of a sequence and one or more values of the
plurality of instructions based on the one or more equivalent program regions. An optimal
sequence of the plurality of instructions in the executable code may be determined. The optimal
sequence may be stored in the one or more computer databases.
[0011] In another aspect, a systém for evaluating performance of a software application is
disclosed. The systém includes one or more computer databases, associated through a computer
network. The systém further includes one or more computing devices, associated through the one
or more computer databases. The one or more computing devices analyzes a plurality of program
code lines of the software application stored in the one or more computer database, identifies one
or more equivalent program regions within the plurality of program code lines. The one or more
computing devices further inserts one or more markers in the identified one or more equivalent
program regions and stores the one or more markers in the one or more computer databases. The plurality of program code lines are compiled and assembled by a compiler and an assembler respectively to generate an executable assembly code. The executable code includes a plurality of instructions. A performance measuring unit associated through the one or more computing devices measures performance metrics of the software application by manipulating the plurality of instructions based on the one or more equivalent program regions identified by the inserted one or more markers and executing the executable code.
[0012] In a further aspect a computer program product comprising a non-transitory computer usable médium having a computer readable program code embodied therein for evaluating performance of a software application in a dáta processing systém. The computer program product includes analyzing by one or more computing devices a plurality of program code lines of the software application stored in a one or more computer databases. The one or more computing devices access the one or more computer databases through a computer network. One or more equivalent program regions within the plurality of program code lines are identified. One or more markers in the identified one or more equivalent program regions are inserted. The one or more markers are stored in the one or more computer databases. The plurality of program code lines are compiled and assembled by a compiler and an assembler respectively to generate an executable code. The executable code includes a plurality of instructions. Performance metrics of the software application is measured by manipulating the plurality of instructions based on the one or more equivalent program regions identified by the inserted one or more markers and executing the executable code.
[0013] The methods, systems, and apparatuses disclosed herein may be implemented in any means for achieving various aspects, and may be executed in a form of a mačhine/readable médium embodying a set of instructions that, when executed by a machine, cause the machine to perform any of the operations disclosed herein. Other features will be apparent from the
accompanying drawings and from the detailed description that follows.
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Example embodiments are illustrated by way of example and not limitation in the figures
of the accompanying drawings, in which like references indicate similar elements and in which:
[0015] Figúre 1 illustrates an architectural block diagram of a systém for facilitating software
performance evaluation, according to one embodiment.
[0016] Figúre 2 illustrates a process flow for a method to evaluate performance of a software
application by run-time assembly code execution, according to one embodiment.
[0017] Figúre 3 illustrates a sample program code and a portion of an indicative assembly code
of the sample program code of a software application, according to one embodiment.
[0018] Figúre 4 illustrates changing a sequence of a plurality of instructions in the assembly code
based on the equivalent program regions shown in Figúre 3, according to one embodiment.
[0019] Figúre 5 illustrates performance metrics of the software application described in Figúre
3, according to one embodiment.
[0020] Figúre 6 illustrates a diagrammatic representation of a dáta processing systém capable of
processing a set of instructions to perform any one or more of the methodologies herein,
according to one embodiment.
[0021] Other features of the present embodiments will be apparent from the accompanying
drawings and from the detailed description that follows. DETAILED DESCRIPTION
[0022] Software application performance evaluation by run-time assembly code execution method and systém is disclosed. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments. It will be evident, however to one skilled in the art that the various embodiments
may be practiced without these specific details.
[0023] In various embodiments of the present invention, performance of a software application
may be evaluated with a set of equivalent program regions. Each equivalent program región may
háve samé functional and/or semantic behavior but may differ in performance outcome of the
software application. The performance difference emerging from the set of equivalent program
regions may form a basis for determining most optimal solution for the software application under
a specific workload.
[0024] The equivalent program regions may be executed in an automated and intelligent manner.
The executions may happen without any changes in a source code of the software application.
There are several factors involving access to the source code of the software application. For
example, the source code may not be available, the source code reading may not be permissible
due to intellectual property issues, changing the source code to perform changes may result in
inadvertent bugs to be introduced in the source code. Thus, an assembly level code changes may
be performed for determining the performance of the software application.
[0025] In other embodiments of the present invention, performance evaluation of the software
application may happen by executing the specific workload and measuring of a set of
performance metrics. The set of performance metrics being an application response time, an
application throughput and/or a systém resource utilization. The set of performance metrics may
reflect the performance of the software application for the specific operational workload. With
changes to the workload and/or an operating environment (hardware and/or software) the
performance of the software application may change.
[0026] Figúre 1 illustrates an architectural block diagram of a systém for facilitating software
performance evaluation, according to one embodiment. As shown in Figúre 1, computing devices 100, 102, 104 may be connected to computer databases 108, 110, 112, 114 through a computer network 106. The one or more computer databases 108, 110, 112, 114 may store program code lines of a software application. The program code lines may be in programming languages šuch as C, C++, COBOL, JAVA, Visual C, PHP, XML and/or Pascal. One or more computing devices 100, 102, 104 may access the program code lines of the software application stored in the one or more computer databases 108,110,112,114. The program code lines may be analyzed by the one or more computing devices 100,102,104 and one or more equivalent program regions within the plurality of program code lines may be identified. An identification of the one or more equivalent program regions may be done based on knowledge of functionality and/or architecture of the software application.
[0027] Markers may be defined in the program code lines to mark the identified one or more equivalent program regions for automatic identification during run-time. The markers may provide very exact feedback from known points in the software application. Values and/or properties associated with the markers may create a marker property. The marker property may form run-time conditions for the markers. The markers may be stored in one or more computer databases 108,110,112,114.
[0028] The marker property may be for example, a toggle between 'process per connection' and/or 'shared connection'. A polling frequency of the software application may also be set by the marker. In čase of polling time being too frequent, resources may be wasted, and in čase the polling time being too long the performance may be degraded. To overcome subjectivity of "too frequent" or "too long", the markers may be inserted objectively and/or quantitatively to set arrival rate correctly. The markers may also be inserted in čase statements. The markers may be inserted even in multiple independent filters. Interchanging dáta of the filters may not impact the program code lines of the software application semantically, but may impact the performance.
[0029] In accordance with an example embodiment of the present invention, the program code lines may be in any of a variety of programming languages šuch as C, C++, COBOL, JAVA, Visual C, PHP, XML and/or Pascal, the marker property may be programming language-independent. Thus, the marker property may be samé for two programming languages with different syntax.
[0030] The program code lines may be compiled and assembled by a compiler 116 and an assembler 118 respectively to generate an executable code. The executable code may háve a plurality of instructions. The executable code may include an assembly code and/or a byte code. The markers may be easily identified in the executable code. The one or more equivalent program regions may be identified in the executable code based on the inserted markers. Performance metrics of the software application may be measured through a performance measuring unit 120 by repeating a process of manipulating the plurality of instructions based on the one or more equivalent program regions identified by the inserted one or more markers and executing the executable code. The performance metrics may be stored in the one or more computer databases 108,110,112,114.
[0031] Figúre 2 illustrates a process flow for a method to evaluate performance of a software application by run-time assembly code execution, according to one embodiment. At operation 200, the program code lines of the software application may be received and analyzed. At operation 202, the equivalent program regions within the program code lines may be identified and the markers may be inserted. At operation 204, the program code lines may be compiled and assembled respectively to generate the assembly code. At operation 206, the equivalent program regions may be identified based on the inserted markers. At operation 208, the plurality of instructions in the assembly code may be changed based on the equivalent program regions. At operation 210, one or more workload may be applied to the assembly code. The assembly code
may be executed. The performance metrics of the software application may be measured.
[0032] A combination of one or more workloads may be applied at operation 205 and the
operations 206, 208 and 210 may be repeated to obtain various performance metrics of the
software application. The operations 206, 208 and 210 may be optimized based on various
algorithms. For example, the performance metrics of the software application may be measured
by only executing the plurality of instructions corresponding to the equivalent program regions
with an immediate boundary of the plurality of instructions.
[0033] The various performance metrics may be stored in the one or more computer databases
108,110,112,114. At operation 212, an optimal sequence of the plurality of instructions under a
specific workload may be determined based on the various performance metrics. An accurate
estimation of performance of the software application may be achieved, as the estimation may
rely on the execution of the assembly code.
[0034] For example, during a testing phase of the software application, the equivalent program
regions may be identified in the assembly code based on the markers. The sequence of the
instructions may be changed in the assembly code. The software application may be subjected to
different workloads. The various performance metrics of the software application shall be
measured by testing the software application with the different sequences of instructions in the
assembly code under the different workloads. The optimal sequence of the instructions in the
executable code may be determined. Thus, best performance under a specific sequence of
instructions and a particular workload may be determined.
[0035] Figúre 3 illustrates a sample program code and a portion of an indicative assembly code
of the sample program code of a software application, according to one embodiment. The sample
program code 300 may consist of multiple if statements. The equivalent program región may be identified. The sample program code 300 may be compiled and assembled respectively to generate the assembly code. The assembly code 302 may include the portion of the assembly code of the sample program code. The assembly code of the sample program code may be executed. [0036] Figúre 4 illustrates changing a sequence of a plurality of instructions in the assembly code based on the equivalent program región shown in Figúre 3, according to one embodiment. The sequence of the instructions in the assembly code may be changed from Function 1 - Function 2 -Function 3 as shown in Figúre 3 to Function 2 - Function 1 - Function3. The assembly code of the sample program code may be executed. The performance metrics of the software application may be measured.
[0037] Figúre 5 illustrates performance metrics of the software application described in Figúre 3, according to one embodiment. The performance metrics indicates that the sequence of the instructions in the assembly code Function 2 - Function 1 - Function3 may také a total execution time of 5 minutes and 23 seconds. The sample program code of Figúre 3 may be executed with the various sequences of the plurality of instructions in the assembly code to compare the performance of the software application.
[0038] For example, if a client specifies that the software application shall be used by approximately 5000 users. Out of 5000 users approximately 3000 users shall be updating dáta in the software application and 2000 users shall be inserting dáta in the software application. An appropriate sequence of the plurality of instructions in the assembly code may be determined for best performance of the software application based on the various performance metrics. [0039] In čase of cloud computing environments, especially Software-as-a-Service, multiple clients may access a software service. Each client shall háve a specific workload characteristic and Quality of Service (QoS) requirements. The optimal sequence of the plurality of instructions in the executable code may be determined for each client. Thus, a particular code base for the software application may solve purpose for various clients.
[0040] Another example embodiment of the present invention includes analyzing by the one or more computing devices the plurality of program code lines of the software application stored in the one or more computer databases. Further, the one or more equivalent program regions within the plurality of program code lines may be identified. The one or more markers in the identified one or more equivalent program regions may be inserted and stored in the one or more computer databases. Further, the plurality of program code lines may be compiled and assembled respectively to generate the executable code. The executable code may include the plurality of instructions. Further, the performance metrics of the software application may be measured for the specific workload, by manipulating the plurality of instructions based on the one or more equivalent program regions identified by the inserted one or more markers and executing the executable code, while maintaining semantics of the plurality of program code lines samé. [0041] Figúre 6 is a diagrammatic representation of a dáta processing systém capable of processing a set of instructions to perform any one or more of the methodologies herein, according to an example embodiment. Figúre 6 shows a diagrammatic representation of machine in the example form of a computer systém 600 within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed. In various embodiments, the machine operates as a standalone device and/or may be connected (e.g., networked) to other machines.
[0042] In a networked deployment, the machine may operate in the capacity of a server and/or a client machine in server-client network environment, and or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personál - computer (PC), a tablet PC, a set-top box (STB), a Personál Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch and or bridge, an embedded systém and/or any machine capable of executing a set of instructions (sequential and/or otherwise) that speciŕy actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually and/or jointly execute a set (or multiple sets) of instructions to perform any one and/or more of the methodologies discussed herein.
[0043] The example computer systém 600 includes a processor 602 (e.g., a centrál processing unit (CPU) a graphics processing unit (GPU) and/or both), a main memory 604 and a státie memory 606, which communicate with each other via a bus 608. The computer systém 600 may further include a video display unit 610 (e.g., a liquid erystal displays (LCD) and/or a cathode ray tube (CRT)). The computer systém 600 also includes an alphanumeric input device 612 (e.g., a keyboard), a eursor control device 614 (e.g., a mouse), a disk drive unit 616, a signál generation device 618 (e.g., a speaker) and a network interface device 620.
[0044] The disk drive unit 616 includes a machine-readable médium 622 on which is stored one or more sets of instructions 624 (e.g., software) embodying any one or more of the methodologies and/or funetions deseribed herein. The instructions 624 may also reside, completely and/or at least partially, within the main memory 604 and/or within the processor 602 during execution thereof by the computer systém 600, the main memory 604 and the processor 602 also constituting machine-readable média.
[0045] The instructions 624 may further be transmitted and/or received over a network 106 via the network interface device 620. While the machine-readable médium 622 is shown in an example embodiment to be a single médium, the term "machine-readable médium" should be taken to include a single médium and/or multiple média (e.g., a centralized and/or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "machine-readable médium" shall also be taken to include any médium that is capable of storing, encoding and/or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the various embodiments. The term "machine-readable médium" shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic média, and carrier wave signals. [0046] In addition, it will be appreciated that the various operations, processes, and methods disclosed herein may be embodied in a machine-readable médium and/or a machine accessible médium compatible with a dáta processing systém (e.g., a computer systém), and may be performed in any order. The modules in the figures are shown as distinct and communicating with only a few specific module and not others. The modules may be merged with each other, may perform overlapping functions, and may communicate with other modules not shown to be connected in the Figures. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.