FIELD

[0002] Aspects of the technology described herein relate to software robots that programmatically control one or more computer program(s) to perform a task via an object hierarchy that provides a representation of graphical user interface elements of the computer program(s) being controlled.

BACKGROUND

[0003] A user can control an application program by interacting with the application program via its graphical user interface (GUI). An application program may provide for the recording of a macroinstruction (sometimes termed a "macro"), which is a recording of the steps taken by the user in controlling the application through its GUI. The macro may be replayed at a later time to control the application program in the same way as the user had done at the time the recording was made. When an application program provides an application programming interface (API), the application program may be also controlled by another computer program via the API.

SUMMARY

[0004] Some embodiments provide for a system for remotely controlling multiple application programs executing on multiple respective virtual machines. The system comprises: a computing device, comprising a first processor, and configured to execute: a first virtual machine configured to execute a first application program; and a second virtual machine configured to execute a second application program; and a controller

communicatively coupled to the computing device and comprising a second processor, the controller configured to perform: generating, based on first information obtained from the first virtual machine and second information obtained from the second virtual machine device, a global object hierarchy comprising a plurality of objects

corresponding to active graphical user interface (GUI) elements of the first application program and the second application program; controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy; and controlling the second application program to perform the second sub-task at least in part by using the global object hierarchy.

[0005] Some embodiments provide for a system for remotely controlling one or more application programs executing on a respective one or more virtual machines. The system comprises a controller, comprising a processor, configured to perform: accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first virtual machine; identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; and providing an indication to the computing device to control the first application program to at least partially perform the first action; and a computing device, communicatively coupled to the controller, comprising a processor configured to execute the first virtual machine and to perform: generating a first object hierarchy comprising a first plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and in response to receiving the indication from the controller device to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action.

[0006] Some embodiments provide for a system for remotely controlling multiple application programs executing on multiple respective physical computing devices. The system comprises: a first computing device, comprising a first processor, and configured to execute a first application program; a second computing device, comprising a second processor, and configured to execute a second application program; and a controller communicatively coupled to the first and second computing devices and comprising a third processor, the controller configured to perform: generating, based on first information obtained from the first computing device and second information obtained

from the second computing device, a global object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program and the second application program; controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy; and controlling the second application program to perform the second sub-task at least in part by using the global object hierarchy.

[0007] Some embodiments provide for a system for remotely controlling one or more application programs executing on a respective one or more physical computing devices. The system comprises: a controller comprising a processor and configured to perform: accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first computing device external to the controller; identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; and providing an indication to the first computing device to control the first application program to at least partially perform the first action; and the first computing device, communicatively coupled to the controller and comprising a processor, configured to execute the first application program and to perform: generating a first object hierarchy comprising a first plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and in response to receiving the indication from the controller to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action.

[0008] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs; controlling the first

application program, via the software robot computer program, to perform the first sub-task at least in part by: automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by: automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising refreshing the object hierarchy; and automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

[0009] Some embodiments provide for a system comprising: at least one hardware processor; and at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and controlling the first application program to perform the first sub-task at least in part by: identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

[0010] Some embodiments provide for a method comprising using at least one computer hardware processor to perform: accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and controlling the first application program to perform the first sub-task at least in part by: identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

[0011] Some embodiments provide for at least one non-transitory computer-readable storage medium storing a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and controlling the first application program to perform the first sub-task at least in part by: identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

[0012] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs; controlling the first application program, via the software robot computer program, to perform the first sub- task at least in part by: automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by: automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar; and

automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

[0013] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the at least one application program to be controlled by the software robot computer program; controlling the first application program to perform the first sub-task at least in part by: identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

[0014] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage

medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program;

controlling the first application program to perform a first sub-task of a task at least in part by: automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

[0015] Some embodiments provide for at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs; controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by: automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by: automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar; and

automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

[0016] Some embodiments provide for a method, comprising: using at least one computer hardware processor to perform: accessing a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs; and controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by: automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by: automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar; and

automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

[0017] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one computer-readable storage medium storing: a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: accessing the software robot; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs; controlling the first application program to perform the first sub-task at least in part by using the software robot computer program and the object hierarchy; and controlling the second application program to perform the second sub-task at least in part by using software robot computer program and the object hierarchy, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

[0018] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program; controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

[0019] Some embodiments provide for at least one computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program; controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

[0020] Some embodiments provide for a method comprising: using at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program; controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

[0021] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; and during performance of the sequence of actions, generating a visual record of how a GUI of the at least one application program would appear on a display during the performance of the sequence of actions and a log of already-performed actions in the sequence of actions, the log containing at least some information not in the visual record and the visual record containing at least some information not in the log; storing the visual record and the log; determining than an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual record and the log.

[0022] Some embodiments provide for a system, comprising: at least one computer hardware processor; and at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; during performance of the sequence of actions, generating a log of already-performed actions in the sequence of actions; generating contextual information associated with the already performed actions, the contextual information containing at least some information not in the log and the log containing at least some information not in the contextual information; and providing to a user the log and the contextual information.

[0023] Some embodiments provide for a method, comprising: using at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; and during performance of the sequence of actions, generating a visual record of how a GUI of the at least one application program would appear on a display during the performance of the sequence of actions and a log of already-performed actions in the sequence of actions, the log containing at least some information not in the visual record and the visual record containing at least some information not in the log; storing the visual record and the log on the at least one storage device; determining than an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual record and the log.

[0024] Some embodiments provide for at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; and during performance of the sequence of actions, generating a visual record of how a GUI of the at least one application program would appear on a display during the performance of the sequence of actions and a log of

already-performed actions in the sequence of actions, the log containing at least some information not in the visual record and the visual record containing at least some information not in the log; storing the visual record and the log on the at least one storage device; determining than an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual record and the log.

[0025] Some embodiments provide for a method, comprising: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; during performance of the sequence of actions, generating a log of already-performed actions in the sequence of actions; generating contextual information associated with the already performed actions, the contextual information containing at least some information not in the log and the log containing at least some information not in the contextual

information; and providing to a user the log and the contextual information.

[0026] Some embodiments provide for at least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program; controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; during performance of the sequence of actions, generating a log of already-performed actions in the sequence of actions; generating contextual information associated with the already performed actions, the contextual information containing at least some information not in the log and the log containing at least some information not in the contextual information; and providing to a user the log and the contextual information.

[0027] Some embodiments provide for a system, comprising: at least one hardware processor; at least one non-transitory computer-readable storage medium storing: a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: accessing the software robot computer program; generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs, the generating comprising: generating, based on a first hierarchical representation of active GUI elements of the first application, a first portion of the object hierarchy corresponding to active GUI elements of the first application program;

generating, based on a second hierarchical representation of active GUI elements of the second application, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, and combining the first portion and the second portion; and controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task and the second application program to perform the second sub-task.

[0028] Some embodiments provide for a system, comprising: at least one hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising:

obtaining a hierarchical representation of the active GUI elements of the application program; generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and controlling, using software robot computer program and the generated object hierarchy, the first application to perform the first sub-task.

[0029] Some embodiments provide for a system for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the system comprising: at least one hardware processor; at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program and a second application program, the generating comprising: generating, based on a first hierarchical representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program; generating, based on a second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, and combining the first portion and the second portion; and controlling, using the generated object hierarchy, the first and second application program, respectively, to perform the first sub-task and the second sub-task.

[0030] Some embodiments provide for a method for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the method comprising: using at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program and a second application program, the generating comprising: generating, based on a first hierarchical

representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program; generating, based on a second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, combining the first portion and the second portion; and controlling, using the generated object hierarchy, the first and second application programs, respectively, to perform the first and second sub-tasks.

[0031] Some embodiments provide for a method comprising using at least one hardware processor to perform: generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising: obtaining a hierarchical representation of the active GUI elements of the application program; generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and controlling, using the generated object hierarchy, the first application program to perform the first sub-task.

[0032] Some embodiments provide for at least one non-transitory computer-readable storage medium storing processor-executable instructions to perform any one of the foregoing methods.

[0033] The foregoing is a non-limiting summary of the invention, which is defined by the attached claims.

BRIEF DESCRIPTION OF DRAWINGS

[0034] Various aspects and embodiments of the application will be described with reference to the following figures. Items appearing in multiple figures are indicated by the same or a similar reference number in all the figures in which they appear.

[0035] Fig. 1A is a diagram of an illustrative object hierarchy including objects corresponding to GUI elements of the calculator application program shown in Fig. IB, in accordance with some embodiments of the technology described herein.

[0036] Fig. IB is a diagram of an illustrative user interface showing GUI elements of one or more computer programs, including the GUI elements of the calculator program, corresponding to objects in the object hierarchy of Fig. 1 A, in accordance with some embodiments of the technology described herein.

[0037] Fig. 2A is a diagram of another illustrative object hierarchy including objects corresponding to active GUI elements of the calculator program shown in Fig. 2B, in accordance with some embodiments of the technology described herein.

[0038] Fig. 2B is a diagram of a GUI of the calculator program that includes GUI elements corresponding to objects in the object hierarchy of Fig. 2A, in accordance with some embodiments of the technology described herein.

[0039] Fig. 3A is a diagram of an illustrative object hierarchy including objects corresponding to active GUI elements of the calculator program shown in Fig. 3B, in accordance with some embodiments of the technology described herein.

[0040] Fig. 3B is a diagram of a GUI of the calculator program that includes GUI elements corresponding to objects in the object hierarchy of Fig. 3A, in accordance with some embodiments of the technology described herein.

[0041] Fig. 4A is a diagram of an illustrative object hierarchy including objects corresponding to GUI elements of the calculator program shown in Fig. 4B, in accordance with some embodiments of the technology described herein.

[0042] Fig. 4B is a diagram of a GUI of the calculator program that includes GUI elements corresponding to objects in the object hierarchy of Fig. 4A, in accordance with some embodiments of the technology described herein.

[0043] Fig. 5A is a diagram of an illustrative object hierarchy comprising objects corresponding to active GUI elements of multiple application programs, in accordance with some embodiments of the technology described herein.

[0044] Fig. 5B is a diagram of an illustrative example of the object hierarchy of

Fig. 5A, in accordance with some embodiments of the technology described herein.

[0045] Fig. 5C is a diagram of an illustrative example of the object hierarchy of

Fig. 5B, in accordance with some embodiments of the technology described herein.

[0046] Fig. 5D is a diagram of another illustrative example of the object hierarchy of Fig. 5A that includes objects corresponding to active GUI elements of multiple application programs that are configured to execute at least in part by using different GUI application libraries, in accordance with some embodiments of the technology described herein.

[0047] Fig. 6 is a diagram of an illustrative object hierarchy comprising objects corresponding to active GUI elements implemented using different GUI technologies, in accordance with some embodiments of the technology described herein.

[0048] Fig. 7 is a diagram of another illustrative object hierarchy comprising objects corresponding to active GUI elements implemented different GUI application libraries, in accordance with some embodiments of the technology described herein.

[0049] Fig. 8 is a diagram illustrating a unified "Button" object, in accordance with some embodiments of the technology described herein.

[0050] Fig. 9 is a diagram illustrating aspects of a unified object in an object hierarchy, in accordance with some embodiments of the technology described herein. [0051] Fig. 10A is a flowchart of an illustrative process for using an object hierarchy to control one or multiple computer programs to perform a task, in accordance with some embodiments of the technology described herein.

[0052] Fig. 10B is a flowchart of an illustrative process for generating an object hierarchy, in accordance with some embodiments of the technology described herein.

[0053] Fig. IOC is a flowchart of an illustrative process for using an object hierarchy to control a computer program to perform one or more sub-tasks of a task, in accordance with some embodiments of the technology described herein.

[0054] Fig. 11 illustrates component libraries, in accordance with some embodiments of the technology described herein.

[0055] Fig. 12A is a diagram of an illustrative software robot for controlling application programs implemented using different GUI application libraries to perform a task, in accordance with some embodiments of the technology described herein.

CLAIMS

What is claimed is:

1. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing:

a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and

processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

accessing the software robot computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs;

controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by:

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and

automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by:

automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising refreshing the object hierarchy; and

automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

2. The system of claim 1, wherein refreshing the object hierarchy comprises:

refreshing at least one object in the object hierarchy, adding at least one object to the object hierarchy, and/or removing at least one objects from the object hierarchy.

3. The system of claim 1 or any other preceding claim, wherein refreshing the object hierarchy comprises:

refreshing only a subset of objects in the object hierarchy.

4. The system of claim 1 or any other preceding claim, wherein refreshing the object hierarchy comprises:

refreshing only one or more parent objects of the first object in the object hierarchy, one or more descendant objects of the first object in the object hierarchy, and/or the first object.

5. The system of claim 1 or any other preceding claim, wherein refreshing the object hierarchy comprises:

adding a new object to the object hierarchy, wherein the new object is associated with an active GUI element of the first application program.

6. The system of claim 1 or any other preceding claim, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

7. The system of claim 1 or any other preceding claim, wherein the first active GUI element of the first application program is a hidden GUI element of the first application program.

8. A system, comprising:

at least one hardware processor; and

at least one non-transitory computer-readable storage medium storing:

a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and

processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:

accessing the software robot computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and

controlling the first application program to perform the first sub- task at least in part by:

identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and

automatically using the first object to cause the first application program to at least partially perform the first action.

9. The system of claim 8 or any other preceding claim, wherein the task further comprises a second sub-task to be performed by a second application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the second application program to perform the second sub-task at least in part by:

identifying, using the software robot computer program, a second action to perform in furtherance of the second sub-task;

automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising refreshing the object hierarchy; and

automatically using the second object to cause the second application program to at least partially perform the second action.

10. The system of claim 9 or any other preceding claim, wherein the task further comprises a third sub-task to be performed by a third application program, and wherein the processor-executable instructions further cause the at least one hardware processor to control the third application program to perform the third sub-task at least in part by: identifying, using the software robot computer program, a third action to perform in furtherance of the third sub-task;

automatically accessing, in the object hierarchy, a third object corresponding to a third active GUI element of the third application program, the accessing comprising refreshing the object hierarchy; and

automatically using the third object to cause the third application program to at least partially perform the second action.

11. The system of claim 8 or any other preceding claim, wherein refreshing the object hierarchy comprises refreshing only a subset of objects in the object hierarchy.

12. The system of claim 8 or any other preceding claim, wherein refreshing the object hierarchy comprises refreshing only the first object and/or one or more descendant objects of the first object in the object hierarchy.

13. The system of claim 8 or any other preceding claim, wherein refreshing the object hierarchy comprises adding to the object hierarchy a new object associated with an active GUI element of the first application program.

14. The system of claim 8 or any other preceding claim, wherein refreshing the object hierarchy comprises removing a particular object from the object hierarchy, wherein the particular object is associated with an inactive GUI element of the first application program.

15. The system of claim 8 or any other preceding claim, wherein refreshing the object hierarchy comprises updating at least one property of at least one object in the object hierarchy.

16. The system of claim 8 or any other preceding claim, wherein refreshing the object hierarchy comprises:

determining that a particular object in the object hierarchy is invalid; and refreshing the particular object using a reverse tree traversal technique.

17. The system of claim 8 or any other preceding claim, wherein the accessing further comprises searching for the first object in the object hierarchy using a search string constructed in accordance with a search grammar, the search grammar comprising at least one token operator, at least one action operator, and at least one relationship operator.

18. The system of claim 8, wherein refreshing the object hierarchy comprises:

refreshing a plurality of objects in the object hierarchy, wherein each of the plurality of objects is a descendant of the first object, and wherein refreshing the plurality of objects is performed using a plurality of threads.

19. The system of claim 18 or any other preceding claim, wherein the plurality of objects includes a first child object of the first object and a second child object of the first object, and wherein refreshing the plurality of objects comprises:

refreshing objects in the object hierarchy that are descendants of the first child object using a first thread in the plurality of threads; and

refreshing objects in the object hierarchy that are descendants of the second child object using a second thread in the plurality of threads.

20. The system of claim 19 or any other preceding claim, wherein the object hierarchy includes a first set of objects associated with the first application program and a second set of objects associated with a second application program, wherein the first set of objects includes more objects than that of the second set of objects, and wherein refreshing objects in the object hierarchy using the plurality of threads comprises using more threads in the plurality of threads to refresh objects in the first set of objects than that to refresh objects in the second set of objects.

21. A method, comprising:

using at least one computer hardware processor to perform:

accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and

controlling the first application program to perform the first sub-task at least in part by:

identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

22. At least one non-transitory computer-readable storage medium storing:

a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and

processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:

accessing the software robot computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and

controlling the first application program to perform the first sub-task at least in part by:

identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising refreshing the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform the first action.

A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing:

a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and

processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

accessing the software robot computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs;

controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by:

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application

program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar; and

automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by:

automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar; and

automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

24. The system of claim 23 or any other preceding claim, wherein the search grammar comprises a token operator, an action operator, and a relationship operator.

25. The system of claim 24 or any other preceding claim, wherein the first search string includes the action operator.

26. The system of claim 25 or any other preceding claim, wherein the action operator indicates how the object hierarchy is to be modified during performance of the search.

27. The system of claim 23 or any other preceding claim, wherein searching for the first object in the object hierarchy comprises modifying the object hierarchy.

28. The system of claim 23 or any other preceding claim, wherein searching for the first object further comprises:

searching for a third object in the object hierarchy;

modifying the object hierarchy by adding one or more new objects to the object hierarchy as a descendants of the third object to obtain a modified object hierarchy; and searching for the first object in the modified object hierarchy.

29. The system of claim 28 or any other preceding claim, wherein modifying the object hierarchy comprises:

using the third object to cause the first application program to perform an action causing one or more new GUI elements of the first application program to become active; and

modifying the object hierarchy by adding one or more new objects corresponding to the one or more new GUI elements as descendants of the third object in the object hierarchy.

30. The system of claim 28 or any other preceding claim, wherein:

searching for the third object is performed using a search string formatted in accordance with the search grammar, the search string including an action operator.

31. The system of claim 23 or any other preceding claim, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

32. The system of claim 23 or any other preceding claim, wherein the first active GUI element of the first application program is a hidden GUI element of the first application program.

33. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing:

a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program; and

processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

accessing the software robot computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the at least one application program to be controlled by the software robot computer program;

controlling the first application program to perform the first sub- task at least in part by:

identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task;

automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and

automatically using the first object to cause the first application program to at least partially perform the first action.

34. The system of claim 33 or any other preceding claim, wherein searching for the first object is performed using a first search string formatted in accordance with a search grammar.

35. The system of claim 34 or any other preceding claim, wherein the search grammar comprises a token operator, an action operator, and a relationship operator.

36. The system of claim 35 or any other preceding claim, wherein the search string includes the action operator.

37. The system of claim 36 or any other preceding claim, wherein the action operator indicates how the object hierarchy is to be modified during performance of the search.

38. The system of claim 33 or any other preceding claim, wherein searching for the first object further comprises:

searching for a second object in the object hierarchy;

modifying the object hierarchy by adding one or more new objects to the object hierarchy as descendants of the second object to obtain a modified object hierarchy; and searching for the first object in the modified object hierarchy.

39. The system of claim 38 or any other preceding claim, wherein modifying the object hierarchy comprises:

using the second object to cause the first application program to perform an action causing one or more new GUI elements of the first application program to become active; and

modifying the object hierarchy by adding one or more new objects corresponding to the one or more new GUI elements as descendants of the second object in the object hierarchy.

40. The system of claim 38 or any other preceding claim, wherein searching for the second object is performed using a search string formatted in accordance with a search grammar, the search string including an action operator.

41. The system of claim 33 or any other preceding claim, wherein the first active GUI element of the first application program is a visible GUI element of the first application program.

42. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects = corresponding to active graphical user interface (GUI) elements of a first application program;

controlling the first application program to perform a first sub-task of a least in part by:

automatically accessing, in the object hierarchy, a first object corresponding to an active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy, the searching comprising modifying the object hierarchy; and automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

43. At least one non-transitory computer-readable storage medium storing:

a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:

accessing the software robot computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs;

controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by:

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar; and

automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by:

automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar; and

automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

A method, comprising:

using at least one computer hardware processor to perform:

accessing a software robot computer program for controlling at least one application program to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs; and

controlling the first application program, via the software robot computer program, to perform the first sub-task at least in part by:

automatically accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program, the accessing comprising searching for the first object in the object hierarchy using a first search string formatted in accordance with a search grammar; and

automatically using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task; and

controlling the second application program, via the software robot computer program, to perform the second sub-task at least in part by:

automatically accessing, in the object hierarchy, a second object corresponding to a second active GUI element of the second application

program, the accessing comprising searching for the second object in the object hierarchy using a second search string formatted in accordance with the search grammar; and

automatically using the second object to cause the second application program to at least partially perform a second action in furtherance of the second sub-task.

A system, comprising:

at least one computer hardware processor; and

at least one computer-readable storage medium storing:

a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and

processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

accessing the software robot;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of the multiple application programs;

controlling the first application program to perform the first sub- task at least in part by using the software robot computer program and the object hierarchy; and

controlling the second application program to perform the second sub-task at least in part by using software robot computer program and the object hierarchy,

wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

46. The system of claim 45 or any other preceding claim, wherein, when the controlling of the second application is performed, the object hierarchy comprises plurality of objects corresponding to a respective plurality of active GUI elements of the first application program and a second plurality of objects corresponding to a respective plurality of active GUI elements of the second application program.

47. The system of claim 45 or any other preceding claim, wherein the object hierarchy comprises a first plurality of objects corresponding to a respective plurality of active GUI elements of the first application program and a second plurality of objects corresponding to a respective plurality of active GUI elements of the second application program.

48. The system of claim 45 or any other preceding claim, wherein controlling the first application program to perform the first sub-task comprises:

automatically accessing a first object in the object hierarchy corresponding to a first active GUI element of the first application program; and

automatically using the first object to cause the first application to perform an action in furtherance of the first sub-task.

49. The system of claim 48 or any other preceding claim, wherein controlling the second application program to perform the second sub-task comprises:

automatically accessing a second object in the object hierarchy corresponding to a second active GUI element of the second application program; and

automatically using the second object to cause the second application to perform an action in furtherance of the second sub-task.

50. The system of claim 48 or any other preceding claim, wherein:

automatically accessing the first object comprises searching for the first object in the object hierarchy using a search grammar; and

automatically accessing the second object comprises searching for the second object in the object hierarchy using the search grammar.

51. The system of claim 45 or any other preceding claim, wherein the first and second application programs are configured to execute using different operating systems.

52. The system of claim 45 or any other preceding claim, wherein the first application program is a WINDOWS application program configured to execute on a WINDOWS operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

53. The system of claim 45 or any other preceding claim, wherein the first application program is a WINDOWS application program and the second application program is a JAVA application program.

54. The system of claim 45 or any other preceding claim, wherein the first application program is an application program selected from the group consisting of an application program configured to execute on a mobile device, an application program configured to execute on a server, a WINDOWS application program, a LINUX GTK application program, an application program having component object model (COM) access, a JAVA application program, a FLASH application program, a SILVERLIGHT application program, a SAP application program, an Android application program, an ORACLE application program, a web-based application program, an HTML application program, a Javascript application program, an AJAX application program, a WINDOWS MOBILE application program, an IOS application program, a SOLARIS application program, an IBM AIX application program, and a proxy application program that may be used as a proxy for controlling another application program.

55. The system of claim 45 or any other preceding claim, wherein the first and second application programs are configured to execute at least in part by using different software platforms.

56. The system of claim 55 or any other preceding claim, wherein the different software platforms include a first software platform selected from the group consisting of a WINDOWS operating system, a LINUX operating system, a JAVA virtual machine, a

web-browser, an operating system configured to execute on a mobile device, a SAP platform, an ORACLE platform.

57. A system, comprising:

at least one computer hardware processor; and

at least one computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program;

controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and

controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy,

wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

58. The system of claim 57 or any other preceding claim, wherein the first and second application programs are configured to execute using different operating systems.

59. The system of claim 57 or any other preceding claim, wherein the first application program is a WINDOWS application program configured to execute on a WINDOWS operating system and the second application program is a web-based application program configured to execute at least in part by using a web-browser application program.

60. The system of claim 57 or any other preceding claim, wherein the first application program is a WINDOWS application program and the second application program is a JAVA application program.

61. The system of claim 57 or any other preceding claim, wherein the first application program is an application program selected from the group consisting of an application program configured to execute on a mobile device, an application program configured to execute on a server, a WINDOWS application program, a LINUX GTK application program, an application program having component object model (COM) access, a JAVA application program, a FLASH application program, a SILVERLIGHT application program, a SAP application program, an Android application program, an ORACLE application program, a web-based application program, an HTML application program, a Javascript application program, an AJAX application program, a WINDOWS MOBILE application program, an IOS application program, a SOLARIS application program, an IBM AIX application program, and a proxy application program that may be used as a proxy for controlling another application program.

62. The system of claim 57 or any other preceding claim, wherein the first and second application programs are configured to execute at least in part by using different software platforms.

63. The system of claim 62 or any other preceding claim, wherein the different software platforms include a first software platform selected from the group consisting of a WINDOWS operating system, a LINUX operating system, a JAVA virtual machine, a web-browser, an operating system configured to execute on a mobile device, a SAP platform, an ORACLE platform.

64. At least one computer-readable storage medium storing processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program;

controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and

controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy,

wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

65. A method, comprising:

using at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one of a first application program and a second application program;

controlling the first application program to perform a first sub-task of a task at least in part by using the object hierarchy; and

controlling the second application program to perform a second sub-task of the task at least in part by using the object hierarchy,

wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

66. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program;

controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; and

during performance of the sequence of actions,

generating a visual record of how a GUI of the at least one application program would appear on a display during the performance of the sequence of actions and a log of already-performed actions in the

sequence of actions, the log containing at least some information not in the visual record and the visual record containing at least some information not in the log;

storing the visual record and the log;

determining than an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual record and the log.

67. The system of claim 66 or any other preceding claim, further comprising the display, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

displaying the GUI of the at least one application program on the display during the performance of the sequence of actions.

68. The system of claim 66 or any other preceding claim, wherein the GUI of the at least one application program is not displayed on the display during performance of the sequence of actions.

69. The system of claim 66 or any other preceding claim, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

following the user taking corrective action responsive to the error, resuming automated performance of the sequence of actions.

70. The system of claim 66 or any other preceding claim, wherein the visual record comprises a video, and wherein providing the visual record to the user comprises providing the user with a graphical user interface through which the user ability to control playback of the video.

71. The system of claim 66 or any other preceding claim, wherein controlling the at least one application program to perform the sequence of actions comprises:

accessing a first object in the object hierarchy corresponding to a first active GUI element of the at least one application program; and

using the first object to cause the at least one application program to perform a first action in the sequence of actions.

72. A system, comprising:

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program;

controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions;

during performance of the sequence of actions,

generating a log of already-performed actions in the sequence of actions;

generating contextual information associated with the already performed actions, the contextual information containing at least some information not in the log and the log containing at least some information not in the contextual information; and

providing to a user the log and the contextual information.

73. The system of claim 72 or any other preceding claim, wherein generating the contextual information comprises:

generating a visual record of how a GUI of the at least one application program would appear on a display during performance of the already-performed actions.

74. The system of claim 73 or any other preceding claim, wherein the visual record comprises a video, and wherein providing the contextual information to the user comprises providing the user with a graphical user interface through which the user can control playback of the video.

75. The system of claim 72 or any other preceding claim, wherein generating the contextual information comprises:

generating a textual record comprising a natural language description of the already-performed actions in the sequence of actions.

76. The system of claim 72 or any other preceding claim, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

determining whether an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and

providing the contextual and the log to the user, when it is determined that the error in performance of the sequence of actions has occurred.

77. The system of claim 76 or any other preceding claim, wherein the processor executable instructions further cause the at least one computer hardware processor to perform:

following the user taking corrective action responsive to the error, resuming automated performance of the sequence of actions.

78. The system of claim 72 or any other preceding claim, further comprising the display, wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

displaying a GUI of the at least one application program on the display during the performance of the sequence of actions.

79. The system of claim 72 or any other preceding claim, wherein a GUI of the at least one application program are not displayed on the display during performance of the sequence of actions.

80. The system of claim 72 or any other preceding claim, wherein automatically controlling the at least one application program to perform the sequence of actions comprises:

accessing a first object in the object hierarchy corresponding to a first active GUI element of the at least one application program; and

using the first object to cause the at least one application program to perform a first action in the sequence of actions.

81. The system of claim 72 or any other preceding claim, further comprising at least one storage device, and wherein the processor-executable instructions further cause the at least one computer hardware processor to store the contextual information and the log on the at least one storage device.

82. A method, comprising:

using at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program;

controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; and

during performance of the sequence of actions,

generating a visual record of how a GUI of the at least one application program would appear on a display during the performance of the sequence of actions and a log of already-performed actions in the sequence of actions, the log containing at least some information not in the visual record and the visual record containing at least some

information not in the log;

storing the visual record and the log on the at least one storage device;

determining than an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual record and the log.

83. At least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program;

controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions; and

during performance of the sequence of actions,

generating a visual record of how a GUI of the at least one application program would appear on a display during the performance of the sequence of actions and a log of already-performed actions in the sequence of actions, the log containing at least some information not in the visual record and the visual record containing at least some information not in the log;

storing the visual record and the log on the at least one storage device; determining than an error in performance of the sequence of actions has occurred, the error preventing completing automated performance of the sequence of actions without human intervention; and

when it is determined that the error in performance of the sequence of actions has occurred, providing to a user the visual record and the log.

A method, comprising:

using at least one hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program;

controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions;

during performance of the sequence of actions,

generating a log of already-performed actions in the sequence of actions;

generating contextual information associated with the already performed actions, the contextual information containing at least some information not in the log and the log containing at least some information not in the contextual information; and

providing to a user the log and the contextual information.

85. At least one non-transitory computer-readable storage medium storing processor executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of at least one application program;

controlling the at least one application program to perform a task comprising a sequence of actions at least in part by using the object hierarchy to invoke actions in the sequence of actions;

during performance of the sequence of actions,

generating a log of already-performed actions in the sequence of actions; generating contextual information associated with the already performed actions, the contextual information containing at least some information not in the log and the log containing at least some information not in the contextual information; and

providing to a user the log and the contextual information.

86. A system for remotely controlling multiple application programs executing on multiple respective virtual machines, the system comprising:

a computing device, comprising a first processor, and configured to execute: a first virtual machine configured to execute a first application program; and

a second virtual machine configured to execute a second application program; and

a controller communicatively coupled to the computing device and comprising a second processor, the controller configured to perform:

generating, based on first information obtained from the first virtual machine and second information obtained from the second virtual machine device, a global object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program and the second application program;

controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy; and

controlling the second application program to perform the second sub- task at least in part by using the global object hierarchy.

87. The system of claim 86 or any other preceding claim, wherein the first information includes information generated on the first computing device about the active GUI elements of the first application program; wherein the second information includes information generated on the second computing device about the active GUI elements of the second application program; and wherein generating the global object hierarchy comprises combining the first local object hierarchy and the second local object hierarchy to obtain the global object hierarchy.

88. The system of claim 86 or any other preceding claim, wherein controlling the first application program comprises:

accessing, in the global object hierarchy, a first object corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to at least partially perform the first action.

89. The system of claim 88 or any other preceding claim, wherein the first virtual machine maintains a local object hierarchy including a first local object corresponding to the first object in the global hierarchy, and wherein using the first object comprises providing an indication to the first computing device to use the first local object to cause the first application program to at least partially perform the first action.

90. The system of claim 86, wherein the computing device is configured to execute a hypervisor to manage execution of the first virtual machine and the second virtual machine.

91. A system for remotely controlling one or more application programs executing on a respective one or more virtual machines, the system comprising:

a controller, comprising a processor, configured to perform:

accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first virtual machine;

identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; and

providing an indication to the computing device to control the first application program to at least partially perform the first action; and

a computing device, communicatively coupled to the controller, comprising a processor configured to execute the first virtual machine and to perform:

generating a first object hierarchy comprising a first plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and

in response to receiving the indication from the controller device to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action.

92. The system of claim 91 or any other preceding claim,

wherein the task further includes a second sub-task to be performed by a second application program executing on a second virtual machine executing on the computing device; and

wherein the controller is further configured to provide an indication to the computing device to control the second application program to at least partially perform a second action in furtherance of the second sub-task.

93. The system of claim 92 or any other preceding claim, wherein the computing device is configured to execute the second virtual machine and to perform:

generating a second object hierarchy comprising a second plurality of objects corresponding to active graphical user interface (GUI) elements of the second application program; and

in response to receiving the indication from the controller to at least partially perform the second action, using the second object hierarchy to cause the second application program to at least partially perform the second action.

94. The system of claim 91 or any other preceding claim, wherein using the first object hierarchy to cause the first application program to at least partially perform the first action comprises:

accessing in the first object hierarchy, a first object corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to at least partially perform the first action.

95. The system of claim 94 or any other preceding claim, wherein the computing device is configured to execute a hypervisor to manage execution of the first virtual machine and the second virtual machine.

96. A system for remotely controlling multiple application programs executing on multiple respective physical computing devices, the system comprising:

a first computing device, comprising a first processor, and configured to execute a first application program;

a second computing device, comprising a second processor, and configured to execute a second application program; and

a controller communicatively coupled to the first and second computing devices and comprising a third processor, the controller configured to perform:

generating, based on first information obtained from the first computing device and second information obtained from the second computing device, a global object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program and the second application program;

controlling the first application program to perform the first sub-task at least in part by using the global object hierarchy; and

controlling the second application program to perform the second sub- task at least in part by using the global object hierarchy.

97. The system of claim 96 or any other preceding claim, wherein the first information includes information generated on the first computing device about the active GUI elements of the first application program; wherein the second information includes information generated on the second computing device about the active GUI elements of the second application program; and wherein generating the global object hierarchy comprises combining the first local object hierarchy and the local second object hierarchy to obtain the global object hierarchy.

98. The system of claim 96 or any other preceding claim, wherein controlling the first application program comprises:

accessing, in the global object hierarchy, a first object corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to at least partially perform the first action.

99. The system of claim 98 or any other preceding claim, wherein the first computing device maintains a local object hierarchy including a first local object corresponding to the first object in the global hierarchy, and wherein using the first object comprises providing an indication to the first computing device to use the first local object to cause the first application program to at least partially perform the first action.

100. The system of claim 96 or any other preceding claim, wherein accessing the first object comprises refreshing the global object hierarchy.

101. A system for remotely controlling one or more application programs executing on a respective one or more physical computing devices, the system comprising:

a controller comprising a processor and configured to perform:

accessing a software robot computer program for controlling at least one application program to perform a task including a first sub-task to be performed by a first application program executing on a first computing device external to the controller;

identifying, using the software robot computer program, a first action to perform in furtherance of the first sub-task; and

providing an indication to the first computing device to control the first application program to at least partially perform the first action; and

the first computing device, communicatively coupled to the controller and comprising a processor, configured to execute the first application program and to perform:

generating a first object hierarchy comprising a first plurality of objects corresponding to active graphical user interface (GUI) elements of the first application program; and

in response to receiving the indication from the controller to at least partially perform the first action, using the first object hierarchy to cause the first application program to at least partially perform the first action.

102. The system of claim 101 or any other preceding claim,

wherein the task further includes a second sub-task to be performed by a second application program executing on a second computing device different from the first computing device; and

wherein the controller is further configured to provide an indication to the second computing device to control the second application program to at least partially perform a second action in furtherance of the second sub-task.

103. The system of claim 102 or any other preceding claim, further comprising:

a second computing device, communicatively coupled to the controller, comprising a second processor configured to execute the second application program and to perform:

generating a second object hierarchy comprising a second plurality of objects corresponding to active graphical user interface (GUI) elements of the second application program; and

in response to receiving the indication from the controller to at least partially perform the second action, using the second object hierarchy to cause the second application program to at least partially perform the second action.

104. The system of claim 101 or any other preceding claim, wherein using the first object hierarchy to cause the first application program to at least partially perform the first action comprises:

accessing in the first object hierarchy, a first object corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to at least partially perform the first action.

105. The system of claim 104 or any other preceding claim, wherein accessing the first object comprises refreshing the first object hierarchy.

106. A system, comprising:

at least one hardware processor;

at least one non-transitory computer-readable storage medium storing:

a software robot computer program for controlling multiple application programs to perform a task comprising a first sub-task to be performed by a first application program and a second sub-task to be performed by a second application program; and

processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:

accessing the software robot computer program;

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of the first and second application programs, the generating comprising:

generating, based on a first hierarchical representation of active GUI elements of the first application, a first portion of the object hierarchy corresponding to active GUI elements of the first application program;

generating, based on a second hierarchical representation of active GUI elements of the second application, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, and

combining the first portion and the second portion; and controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub- task and the second application program to perform the second sub-task.

107. The system of claim 106 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation of the active GUI elements of the first application program from an operating system.

108. The system of claim 107 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the second hierarchical representation from the second application program.

109. The system of claim 106 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation by using the WINDOWS Automation API.

110. The system of claim 106 or any other preceding claim, wherein the first application program is an application program implemented using a WINDOWS GUI application library and the second application program is an Internet browser program, and wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

obtaining the first hierarchical representation via the WINDOWS Automation API; and

obtaining the second hierarchical representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.

111. The system of claim 106 or any other preceding claim, wherein the first and second application programs are configured to execute at least in part by using different GUI application libraries.

112. The system of claim 106 or any other preceding claim, wherein controlling the first application program to perform the first sub-task comprises:

accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

113. The system of claim 112 or any other preceding claim, wherein accessing the first object comprises refreshing the object hierarchy.

114. A system, comprising:

at least one hardware processor;

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising:

obtaining a hierarchical representation of the active GUI elements of the application program;

generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and

controlling, using software robot computer program and the generated object hierarchy, the first application program to perform the first sub-task.

115. The system of claim 114 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the hierarchical representation of the active GUI elements of the first application program from an operating system.

116. The system of claim 114 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the hierarchical representation of the active GUI elements of the first application program via the WINDOWS Automation API.

117. The system of claim 114 or any other preceding claim, wherein controlling the first application program to perform the first sub-task comprises:

accessing, in the object hierarchy, a first object corresponding to a first active GUI element of the first application program; and

using the first object to cause the first application program to at least partially perform a first action in furtherance of the first sub-task.

118. The system of claim 114 or any other preceding claim, wherein accessing the first object comprises refreshing the object hierarchy.

119. A system for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the system comprising:

at least one hardware processor;

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program and a second application program, the generating

comprising:

generating, based on a first hierarchical representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program;

generating, based on a second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program, and

combining the first portion and the second portion; and

controlling, using the generated object hierarchy, the first and second application program, respectively, to perform the first sub-task and the second sub-task.

120. The system of claim 119 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation of the active GUI elements of the first application program from an operating system.

121. The system of claim 120 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the second hierarchical representation from the second application program.

122. The system of claim 119 or any other preceding claim, wherein the processor-executable instructions further cause the at least one hardware processor to obtain the first hierarchical representation via the WINDOWS Automation API.

123. The system of claim 119 or any other preceding claim, wherein the first application program is an application program implemented using a WINDOWS GUI application library and the second application program is an Internet browser program, and wherein the processor-executable instructions further cause the at least one computer hardware processor to perform:

obtaining the first hierarchical representation via the WINDOWS Automation API; and

obtaining the second hierarchical representation via a Document Object Model (DOM) representation of a webpage displayed by the Internet browser program.

124. A method for controlling multiple application programs to perform a task comprising a first sub-task and a second sub-task, the method comprising:

using at least one hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program and a second application program, the generating

comprising:

generating, based on a first hierarchical representation of active GUI elements of the first application program, a first portion of the object hierarchy corresponding to active GUI elements of the first application program;

generating, based on a second hierarchical representation of active GUI elements of the second application program, a second portion of the object hierarchy corresponding to the active GUI elements of the second application program,

combining the first portion and the second portion; and controlling, using the generated object hierarchy, the first and second application programs, respectively, to perform the first and second sub-tasks.

A method, comprising:

using at least one hardware processor to perform:

generating an object hierarchy comprising a plurality of objects corresponding to active graphical user interface (GUI) elements of a first application program, the generating comprising:

obtaining a hierarchical representation of the active GUI elements of the application program;

generating an object for each of at least some of the active GUI elements represented in the hierarchical representation; and

organizing the generated objects into the object hierarchy based on relationships among active GUI elements represented in the hierarchical representation; and

controlling, using the generated object hierarchy, the first application program to perform the first sub-task.