Sign In to Follow Application
View All Documents & Correspondence

"An Accessibility System For Providing User Interface Information To A Client"

Abstract: An accessibility system for providing user interface information to a client, the accessibility system comprising, a client which requests user interface information; a server which provides user interface information to the client, characterized by a client side interface comprising a logical tree for revealing user interface information that is interesting to the client and for hiding user interface information that is not interesting to the client, an accessibility system core comprising user interface automation services for filtering information based on whether the user interface information is interesting to the client and a server side interface for facilitating information transfer from a server side application regardless of the user interface engine used to build that application.

Get Free WhatsApp Updates!
Notices, Deadlines & Correspondence

Patent Information

Application #
Filing Date
17 February 2005
Publication Number
11/2009
Publication Type
INA
Invention Field
PHYSICS
Status
Email
Parent Application
Patent Number
Legal Status
Grant Date
2010-07-24
Renewal Date

Applicants

MICROSOFT CORPORATION
ONE MICROSOFT WAY,REDMONG,WASHINGTON 98052,U.S.A

Inventors

1. ROBERT SINCLAIR
24136 NE 6th PLACE,SAMMAMISH,WA 98074,U.S.A
2. PATRICIA M. WAGONER
23723 NE 61st STREET,REDMOND,WA 98053,U.S.A
3. BRENDAN MCKEON
1705 SUMMIT AVENUE,#307,SEATTLE,WA 98122,U.S.A

Specification

The present invention relates to an accessibility system for providing user interface information to a client. The invention relates to the field of assistive technology and automated testing products and the interaction of these products with user interface information. BACKGROUND OF THE INVENTION Assistive technology (AT) products exist to help computer users who have a need for assistance in areas of learning, communication and access to information contained in and presented by computer software. These products have a need for information relevant to the computer interface. Similarly, existing automated testing products and user interface commanding utilities also have a need for information about the user interface. Currently, these products have no sufficient source of user interface (Ul) information. These three types of products (clients) are required to have necessary support from elsewhere to enable them to: (1) gather information about an application's user interface; (2) progranunatically discover and interrogate UI elements regardless of the technology used to build the UI; (3) generate keyboard and pointer input; and (4) understand what type of behavior or functionality is currendy available. No single technology is available currently that gives an AT product all of these capabilities. Furthermore, current AT products are not always compatible with all graphical operating system (OS) technologies and lack the ability to filter and coordinate redundant or misleading notifications in a centralized way. An additional disadvantage is that current automation and accessibility infrastructures are not extensible and therefore require OS level changes to add new functionality. Furthermore, currently to gather information about an application's user interface, the AT product must write application-specific code to obtain information for the user. The process of writing this apphcation-specific code is time consuming and requires continuous maintenance. Current automation infrastracture also lacks the abUity to filter and coordinate redundant or misleading event notifications in a consistent manner. Thus, event consumers are required to independently filter information.Current systems allow AT products to request event notifications in three levels of granularity: (1) everything on a desktop; (2) in a particular process (such as opening of a word processor); or (3) in a thread in the particular process (multiple objects doing work in the process). Currently, when the client receives an event, it receives a window handle for a specific window in which the event occurred and other bits of information to indicate where the event occurred. A client can make a cross process call to retrieve the UI object that is related to the event. With this object, the client can make additional cross process calls to ask for information about that object. If the client needs five pieces of information, then the client must make five cross process caUs. Cross process calls are exceedingly slow, so the performance cost of collecting UI information using current accessibility infrastructure is high. This type of known scenario is shown in FIG. 8. A server application 12 fires an event 6. A kernel 14 determines which clients must be notified and sends an event notification 18 to an interested client 10. The client 10 makes a request 16 from the server application 12 across a process boundary 2 for the object related to the event notification 18. The server application 12 returns the object 20 and then the client 10 can begin sending requests 16 for information about the UI control which fired the event. The server application 12 returns the requested information 20 across the process boundary 2 to the client 10. Another current option allows client code to be loaded as a dynamic link library (.DLX.) within a process. This option has several drawbacks. First, it requires cooperation firom the system to load client code into a process. Second, it gives rise to security issues because once in the client code is loaded into an appUcation's process, it is difficult to restrict the information it gathers. Third, to be an effective technique for the client, it must be loaded into every process on the system. Optimally, only trusted clients should be loaded into another application's process. Furthermore, a system is needed that gives the client the ability to specify what event notifications it wants to receive. In known systems, a client may need to make a number of cross process calls and then analyze the information to determine if it is interested in the event. A mechanism is needed that can perform this event filtering in a mote performant maimer and that can be easily updated to support new system or application events. Furthermore, a system is needed that uses only trusted components in order to alleviate security concerns. Currently, when seeking information about a user interface, the AT product is required to access trees that are native to a particular UI framework. Accordingly, multiple trees are required to convey user interface information for multiple UI frameworks. These differing trees may contain information which is not of interest or is not visible to the user, such as hidden container objects which manage the visible UI controls manipulated by the end user. Therefore, a need exists for a single unified tree having on! y those nodes that are of interest to the user. A solution is needed tliat addresses the needs of AT products, automated testing tools, and commanding utHities. The solution should be usable by all graphical OS technologies and should allow all forms of UI and UI components to become accessible. BRIEF SUMMARY OF THE INVENTION The present invention is directed to a method and computer application for providing a chent with user interface information. In one aspect of the invention, an accessibility system for providing user interface information to a client is provided. The accessibility system includes an accessibility system core including user interface automation services for filtering information based on whether the user interface information is interesting to the client. The accessibihty system additionally includes a client side interface including a logical tree for revealing user interface information that is interesting to the client and for hiding user interface information that is not interesting to the client. The accessibility system also includes a server side interface for facilitating information transfer from a server side application regardless of the user interface engine used to build that application. In yet another aspect of the invention, a computer-implemented method for providing user interface information to a client is provided. The method includes monitoring user interface information with accessibility system automation services and transferring user interface information over a server side interface regardless of server side technology. The method further includes determining specific user interface information that is interesting to the client using a logical element tree that forms a part of the client-side interface. Additional advantages and novel features of the invention will be set forth in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following, or may be learned from practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS The present invention is described in detail below with reference to the attached drawing figures, wherein: FIG. 1 is a block diagram of a computing system environment suitable for use in implementing the present invention; FIG. 2 is a block diagram of interaction between an accessibility system, a client environment, and a server environment; FIG. 3 is a block diagram illustrating components of the accessibility system core; FIGs. 4(A)-4(D) illustrate the creation of a logical tree from native elements; FIG. 5 is a flowchart showing a sequence of procedures for building a logical tree; FIG. 6 shows a dialog box and its components forming logical elements; FIG. 7 is a flowchart illustrating procedures involved in activating an event mechanism of the invention; and FIG. 8 shows a known system for event notification. DETAILED DESCRIPTION OF THE INVENTION Exemplary Operating Environment FIG. 1 illustrates an example of a suitable computing system environment 100 on which the invention may be implemented. The computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or fuctionality of the invention. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 100. The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data stractures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or progranimable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices. With reference to HG. 1, an exemplary system 100 for implementing the invention includes a general purpose computing device in the form of a computer 110 including a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. Computer 110 typically includes a variety of computer readable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as reaid only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modtiles that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation, HG. 1 illustrates operating system 134, application programs 135, other program modules 136, and program data 137. The computer 110 may also include other removable/nonremovable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1 illustrates a hard disk drive 141 that reads from or writes to nomemovable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152, and an optical disk drive 155 that reads from or writes to a removable, nonvolatile optical disk 156 such as a CD ROM or other optical media. Other removable/nonremovable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 141 is typically cormected to the system bus 121 through an nonremovable memory interface such as interface 140, and magnetic disk drive 151 and optical disk drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150. The drives and their associated computer storage media discussed above and illustrated in FIG. 1, provide storage of computer readable instructions, data structures, program modules and other data for the computer 110. In FIG. 1, for example, hard disk drive 141 is illustrated as storing operating system 144, application programs 145, other program modules 146, and program data 147. Note that these components can either be the same as or different from operating system 134, application programs 135, other program modules 136, and program data 137. Operating system 144, application programs 145, other program modules 146, and program data 147 are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer 110 through input devices such as a keyboard 162 and pointing device 161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 195. The computer 110 in the present invention may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 171 and a wide area network (WAN) 173, but may also include other networks. When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking enviromnent, the computer 110 typically includes a modem 172 or other means for establishing commimications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user-input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1 illustrates remote application programs 185 as residing on memory device 181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications Hnk between the computers may be used. Although many other internal components of the computer 110 are not shown, those of ordinary skill in the art will appreciate that such components and the interconnection are well known. Accordingly, additional details conceming the internal construction of the computer 110 need not be disclosed in connection with the present invention. Accessibility System Structure As shown in FIG. 2, an accessibility system 200 interacts with a client environment 300 and a server environment 400. The accessibility system 200 may be implemented in the computer environment 100 described above with respect to FIG. 1. The accessibility system 200 includes a chent side accessibility interface 220 for facilitating interaction with the client 300, a server side accessibility interface 230 for facilitating interaction with the server side 400, and an accessibility system core 201. The accessibihty system 200 of the invention provides new application program interfaces (APIs), interfaces, and metaphors for programmatically accessing a user interface (UI). The accessibility system 200 allows applications to make themselves and any components they use accessible. The client environment 300 preferably includes an assistive technology (AT) product or automated UI testing tool. The server side 400 may implement a variety of different technologies as shown in FIG. 2. A server system 410 includes an adapter 412 and a core 414, which may be found in a first type of UI. A server system 420 includes a proxies component 422 and controls 424 as may be found in a second type of UI, such as a Win32 UI available in Microsoft Operating System products, from the Microsoft Corporation of Redmond, Washington. The server system 430 includes an adapter 432 and internal OM 434, which may be found in an alternative third type of UI. As shown in FIG. 3, an event mechanism 210, which is included in the accessibility system 200, relies on a UI automation client 202 and a UI automation server 204 for facilitating interaction with the client environment 300 and the server environment 400. The UI automation client 202 and the UI automation server 204 are described in greater detail below with reference to the events mechanism 210 of the invention. The accessibility system 200 of the invention gives the client (AT Product) 300 the capability to: (1) gather information about an application's user interface; (2) progranunaticaUy discover and interrogate UI elements regardless of the technology used to build the UI.; (3) generate keyboard and pointer input; and (4) understand what type of behavior or functionality is currently available. The accessibility system 200 allows appUcations to make themselves and their components accessible. The stracture shown in FIGs. 2 and 3 enables five major aspects of the accessibility system 200 including: (1) logical UI tree; (2) Control Patterns; (3) Event Mechanism; (4) Input API (not covered in this document); and (5) properties. UI access logical tree 222 An integral component of the accessibility system 200 is the logical tree 222, an example of which is shown in FIG. 4(D). The tree 222 is included in the chent side accessibility interface 220. The logical tree 222 is a iiltered view of the underlying structural hierarchy of UT elements, not a separate tree that must be implemented by the control or application developer. Instead, it keys off a few well-defined properties, interesting and uninteresting, which indicate whether a structural element should be exposed in the logical tree 222. The accessibility system core 201 consumes this information to produce the filtered UI logical tree 222 that is, in turn, presented to the AT products or test script. The logical tree 222 is a tree of elements, each of which represents a control, an item in a control, or a grouping structure, which may be a dialog, pane, or frame. The structure of the logical tree 222 should represent the UI of the application as perceived by the user (even if the controls are actually implemented using a different underlying structure). The tree should be stable over time. As long as an application looks the same to a user, the logical tree 222 representing that application should remain the same, even if implementation details of the application behind the scenes have changed. Native elements that exist for structural and implementation reasons, such as a shell's "ShDocView" window in the Microsoft OS products should not appear in this tree, since the user does not perceive them. The logical tree 222 is a single tree built from a plurality of fragments that is capable of unifying a plurality of different processes so that they are the same to the client. The logical tree 222 enables bulk retrieval and is capable of obtaining a value for a list of properties. By the time a user normally would have invoked a cross process call to ask for values, the accessibility system 200 will have fetched them through the use of the logical tree 222. Instead of being constructed in one step as in the known systems, the logical tree 222 is constructed from fragments that are used to build a raw tree. As shown in FIG. 5, three main procedures build the logical tree 222. In procedure 72, the accessibility system 200 locates native elements of underlying technologies and airives at the native trees shown in FIG. 4(A). In procedure 74, the accessibility system 200 combines native elements to form the raw tree 20 as shown in FIG. 4(B). Finally, in procedure 76, the logical tree 222 is obtained by hiding uninteresting components in the raw tree 20 as shown in FIG. 4(D). FIG. 4(A) illustrates two native trees 10 and 14, which are constructed from native elements of underlying technologies such as the Win32 UI, or any other available UI. The native tree 10 includes a parent node 11 and a plurality of descendants 12 having various relationships with one another. Similarly, the native tree 14 includes a parent node 15 having a plurality of child nodes 16. The child nodes 16 may be described as siblings of one another. As shown in FIG. 4(B), the native trees 10 and 14 may be combined to form a raw tree 20. The raw tree 20 includes a parent node 21, having two child nodes 22 and 30. The child node 22 has descendants 23- 29 and the child node 30 has descendants 31-33. This raw tree 20 is a combination of tiie native trees 10 and 14, with the nodes of the native tree 10 forming nodes 22-29 and the nodes of the native tree 14 forming nodes 30-33. Through a method broadly shown in HGs. 4(C) and 4(D), the raw tree 20 is converted into the logical tree 222. To move from the raw tree 20 to the logical tree 222, a developer can insert hints in the raw tree. The developer can marie nodes within the raw tree 20 as "hide self' or "hide self and children" or as "hide children of node", etc. The developer can also move nodes sideways or place nodes before children. These "hints" and modifications in the raw tree 20 are used to form the logical tree 222. For example, in FIG. 4(C), a developer has marked nodes 24-26 and 33 of the raw tree 20 as uninteresting as indicated by blocks 40 and 41. Typically, nodes that contain elements that will not be visible to the user are marked as uninteresting. Nodes related to the visible UI are typically considered to be interesting and will be included in the logical tree 222 for use by the AT client 300. As shown in FIG. 4(D), the nodes marked as uninteresting are not included in the logical tree 222. The accessibility system 200 uses the logical tree 222 to find information about events, the state of the system, the location of objects and information about controls. Known systems have not had the capability to scope within their trees. The logical tree 222 can be navigated based on the preferences of the client 300 and is capable of providing information regardless of the server side application in use. In operation, if a client 300 needs to obtain information for a user about an application, the client 300 looks for a button to press and observes text on the button. The client 300 would call an API "find an element". The API will return a value that is referenced to a position in the logical tree 222 of the client side interface 220. Through the logical tree 222, the accessibility system 200 gives the client 300 an abstract view of the UI regardless of the application in use. The abstract model includes structures, properties, events, and functionality that a list box, button or other UI component can expect to have in common with one another. The logical tree 222 is a single unifying tree that is a logical representation of the UI and is formed into a shape that includes only elements of interest to clients 300. Accordingly, instead of forcing AT products to filter the structural hierarchy of UI elements and guess at the model being presented to the end user, the logical tree 222 presents a hierarchy that closely maps to the structure being presented to the end user. This greatly simplifies the AT product's task of describing the UI to the user and helps the user interact with the application. Because this logical UI tree 222 is a fundamental part of the accessibiUty system 200, all of the other components of the accessibility system 200 are designed to work firom the logical tree 222. For example, HG. 6 shows a simple dialog box 60 that appears to have a very simple structure. However, when viewed through the currently available accessibility technology, the structure of this dialog box 60 is surprisingly complex. It contains 264 objects that an AT product must filter through to discover those that are meaningful to the end user. With the accessibility system 200 and its support for the logical UI Tree 222, the developer who owns this dialog box 60 can set a few properties to present the following structure shown in FIG. 6 to the AT products 300. As shown in FIG. 6, for a "Run" dialog, the developer can indicate as interesting, the flying window graphic 62 and 'Type the name of program, folder, document, or internet resource and windows will open it for you" at 63. The developer can also indicate as interesting the combo box 64 including notepad, word, calculator, etc., and the OK 65, cancel 66 and browse 67 buttons. This offere developers a low-cost mechanism to tag their element hierarchy and thereby produce a logical representation of their application's UI through the UI accessibility system 200. Each of the features shown may be represented by a node that ""has a specified relationship to each other node in the logical tree 222. This logical representation offers an immediate benefit to a testing team and to AT products or cUents 300. A set of APIs allows the client 300 to get to the tree. Functionality includes: (1) logical element from point to point; (2) logical element from event; and (3) currently focused logical element. As set forth above, a logical element represents a UI component, possibly a control, a part of a control, or a container or logical grouping (i.e. dialog, pane, or frame). Controls can vary greatly in terms of their functionality. Accordingly, different interfaces are used to represent functionaHty associated with particular types of controls. These control-specific interfaces derive from a common base interface that represents functionality conmion to aU controls. The common base interface contains: (1) methods for navigating the logical tree 222; (2) a general method for getting property values; and (3) methods for accessing supported control-specific interfaces. In navigating the logical tree 222, each underlying application UI technology will provide its own technique for navigation. Although the logical tree 222 is of ultimate interest to the user, the raw element tree 20 also serves some important fiinctions. While the logical tree 222 contains only elements that the user can relate to, the raw element tree 20 contains nodes, such as 22, that represent the implementation structure of the underlying framework. For a Win32 UI fragment, for example, this tree would contain nodes that represent HWNDs. In some respects, the raw element tree 20 is a 'half-way house' between the logical element tree 222 and the underlying frameworks' own native element trees. The raw element tree 20 is used as a basis for building the logical element tree 222, and it is where native elements first plug into the system. The raw element tree 20 can also be used for debugging and testing. It is useful for pinpointing or describing where a particular problematic node is. Functionality on a base raw element node includes: methods for navigating the raw element tree; a method for jumping to a corresponding logical element (if one exists); property containing 'debug string' for this element - e.g. "HWND 0x483FE" for HWND nodes; and other 'behind the scenes infrastmcture' methods. These other methods enable hit testing and location; events; and exposing properties that frameworks can easily provide (e.g. focused, enabled). The raw element tree 20 contains nodes 22-33 that represent elements from various rendering engines. The raw element tree is used as a starting point for rendering engines to plug themselves into the accessibility system 200 and is built from lightweight adapter objects which adapt native elements, such as HWNDs from Win32, into a unified logical tree 222. It is additionally used for handling hosting transitions, where one technology hosts another. Since the raw element tree 20 is the base on which the logical tree 222 is built, it can be used to check that the logical tree 222 is complete and connected and can be used to check for unaccounted-for elements. The raw element tree 20 may further be used for other infrastmcture-like tasks: such as providing some basic element ID and providing some basic framework-provided element properties, such as focused, enabled, and location. The raw element tree 20 is not the primary source of information for AT products or clients 300, is not used for logical navigation and is not exposed to end-users. The raw element tree 20 also caimot be used to capture an element's position in tree so that it can be returned to at some future point in time. The logical element tree 222 performs all these functions. The raw element tree 20 can typically be built mechanically from the raw elements of the underlying rendering technology (HWNDs, Elements) without knowledge of the logical elements represented. It can therefore be used to look for raw elements, which have not been accounted for in the logical tree 222. The raw element tree 20 is a useful debugging and diagnostic tool, as it allows a 'stack dump'-Hke description of a node's location to be captured. Furthermore, known systems base their trees on code specific criteria and are difficult to implement with diverse technologies. The present approach uses a general abstract 'raw element' type, which can be implemented by or on behalf of any underlying rendering technology. In order to obtain the raw element tree, calling a raw element root will get a desktop element, verified by making sure that its parent is NULL and all other nodes have it as their ultimate ancestor. To obtain other elements, calling a method to obtain a raw element from a specified point will return the element using valid screen coordinates. After obtaining the raw element tree, it can be checked and verified by checking the elements (parents, siblings and children). In operation, the client 300 can navigate the raw element tree 20 using relationships such as: parent; next sibling, previous sibling, first child, last child, etc. The client 300 can then jump from the raw element to the corresponding logical element in the logical tree 222. Events Mechanism When a client 300 wants to keep informed of events, the client 300 is able to register through the UI automation client 202 as shown in HG. 3 to obtain the information. The client 300 specifies object information it wants to receive, where it wants the information to go, and Ihe list of properties it wants to get back. The client request goes to UI automation client 202. UI automation client 202 can monitor any process on the desktop. The UI automation server 204 keeps track of which clients 300 are Hstening and knows how to get back to UI automation client 202. The UI automation client 202 advises the UI engine 206 of client interest, so the UI engine 206 knows when to tell the UI automation server 204 of the event. The UI engine does not have to utilize the client advice but may choose instead to always notify the UI automation server 204 of events or notify the UI automation server only if a client is listening for any events. The advice is useful if the UI engine wants to turn on UI automation server notification only if there is a client listening for events. The UI engine would do this to avoid possible degradation of speed of the UI and to avoid loading code modules it doesn't otherwise need. The UI Engine 206 then informs the UI automation server 204 of a UI event. UI automation server 204 returns the requested logical element to the client 300 and sends information to the client 300 including properties of the event that the client 300 requested. The UI automation server 204 decides what information is within the scope of client request and only forms a logical element if the information is of interest. Forming a logical element includes pre-fetching, on the UI automation server side, the set of properties that the client has indicated it will use when handling the event. For example, the UI automation server 204 can discover a logical element for a combo box. The scope would be the combo box or its children. The client 300 can request children/parent/dependents to define scope during the registration phase. After the UI automation server 204 determines whether information is within the requested scope, it builds a logical element. The UI automation client 202 serves the client 300 by talking to target apphcations receiving the requested information from the UI automation server 204 and routing objects to a proper space on the client 300. The UI automation server 204 is created when the client 300 registers to receive event notification. As an example, a UI engine 206 is running the Microsoft Word word processing application. The client 300 has registered for name property change. The client's registration causes the UI automation server 204 to be created. The client's registration also advises the UI engine 206 to start notifying the UI automation server 204 for the name property. The UI engine 206 doesn't get scope information. The UI engine 206 calls one of the APIs for the server side. The UI engine 206 specifies (1) what property changed; (2) the new value of the property; and (3) perhaps the old value. The UI automation server 204 is created based on events of interest to the client 300 and therefore knows events, properties, clients, and scope of interest so it knows if any client 300 is interested in the created logical element If more than one client 300 has registered for events with a particular UI automation server 204 and the clients 300 have registered for the same event and have asked for properties to be bulk fetched with the returned logical element, when the UI automation server 204 sends an event back to the clients 300, each client 300 will get the union of the requested bulk fetch properties returned with the logical element. For each client 300 listening, the UI automation server 204 notifies the client 300 passing the chent the logical element associated with the event. The UI automation server 204 creates only one logical element. This is an improvement over the current technology in which each client 300 would be required to ask for their own copy of the object that is the source of the event. If the UT engine 206 does not utilize the UI automation client's advice when clients register for events, the UI engine 206 can ask the UI automation server 204 if there are any accessibility clients 300 listening and if no one is hstening, then can avoid the work of creating the information and sending it to the UI automation server 206. For example, a screen reader is die client 300 and specifies where it wants information to go, the focus change object to receive events, and the specific list of properties of interest. The UI engine 206 is advised and knows it should send events to the UI automation server 204. Upon detecting focus changes, the UI engine 206 notifies the UI automation server 204. The UI automation server 204 converts to a well-known interface and sends the event and object to the UI automation client 202. The UI automation client 202 routes the object to an appropriate space on the chent 300. The above-described components improve upon the known systems by eliminating the central repository in the kernel for events. Instead, the UI automation server 204 knows all clients 300 interested in getting information about the context in which it is mnning. The elimination of the kernel repository also creates a more peer-to-peer interaction, since the UI automation server 204 fidfills the function previously performed in the kernel. The accessibility system 200 of the invention gives client 300 the abihty to specify what it wants to see such that filtering is accomplished on the server side using the UI automation server 204. FIG. 7 is a flow chart showing the procedures involved in the event registration and notification method. In step 80, the client 300 requests event notification. In step 82, the UI automation chent 202 communicates the request to the UI automation server 204. In step 84, the UI automation client advises the UI engine 206 that it wants notification. In step 86, the UI automation server 204 receives notification from the UI engine 206. In step 88, the UI automation server 204 filters the received information. If the received information is found uninteresting to the user in step 90, the UI automation server 204 discards the information and continues to wait for notification in step 92. Alternatively, if the information is found to be interesting in step 90, the UI automation server 204 creates a logical element and sends it to the UI automation client 202 in step 94. In step 96, the UI automation client 202 puts the received information in its proper place on the client 300. The event mechanism 210 of the accessibility system 200 allows the client 300 to register to receive event notifications for property changes in the UI, tree changes in a control's structure, multimedia events, and related information. Without these capabilities, clients 300 have to continually poU aU the UI elements in the system to see if any information, structure, or state has changed. The accessibility system 200 events mechanism 210 also allows clients 300 to receive events out-of-process, request a collection of properties to be returned with the event notification, and to register for events on multiple elements. The event mechanism 210 exposes: the interfaces the AT product or test application uses to register for events; interfaces the AT product implements on objects used to receive event notifications; and the interfaces control implementers use to notify the event engine of UI events. The event mechanism 210 is used to allow AT products and test applications to receive events independently of the UI engines used to render UI and allows AT products and test applications to track top-level application windows and focus without regard to the underlying technology. FIG. 2 shows how clients 300 interact with the accessibility system 200. There is no global management of events across processes. Instead, each accessibility system client 300 or server 400 has its own context. Client applications will only receive events for applications that support the accessibility system 200. To start using accessibility system events in an application, the client 300 can do one of four things: (1) Use the "AddTopLevelWindowListener" API to discover new UI appearing on the desktop and in the handler for that event, register for other events and by this means receive events from any process; (2) Use one of the Find APIs to locate interesting UI and target a specific piece of UI; (3) Use some other means to discover interesting UI such as finding a window handle or a point on the screen and, using that handle or point, acquire a logical element to use as a reference for listBning to events; or (4) Use the "AddFocusChangedListener" API to track the input focus and register for events on the UI that currently has focus. A top-level window is a window whose parent is the desktop. The use of the term "opened" indicates that a new top-level window has appeared to the user. The use of the term "closed" indicates that a top-level window has been destroyed. The accessibility system server side interface 230 includes two APIs for notifying the accessibility system 200 of events being added and removed. The UI automation client calls these APIs when clients register and unregister for events. This allows the UI engine 206 to get information about what accessibility events are being requested in its context. Where applicable, events are associated with logical elements from the application's logical element tree 222. Where logical elements are not applicable, events are associated with a human readable string or other well-known object that represents the source of an event. The event mechanism 210 provides filtering based on user supplied preferences during event registration. By using the client's filtering preferences at the server, before creating the event-related data and sending it cross process to the client, the event mechanism 210 inherently improves out-of-process performance by reducing the number of cross process calls. The event mechanism 210 provides a way to specify the properties of a logical element that are interesting for the event during event registration. This further reduces the number of cross-process caUs. The event mechanism 210 is extensible without requiring major operating system (OS) changes. Although the event mechanism 210 may be implemented using managed cQ

Documents

Application Documents

# Name Date
1 636-DELNP-2005-GPA-(31-05-2010).pdf 2010-05-31
1 636-DELNP-2005-RELEVANT DOCUMENTS [27-03-2020(online)].pdf 2020-03-27
2 636-DELNP-2005-Correspondence-Others-(31-05-2010).pdf 2010-05-31
2 636-DELNP-2005-RELEVANT DOCUMENTS [29-05-2019(online)].pdf 2019-05-29
3 636-DELNP-2005-RELEVANT DOCUMENTS [27-03-2019(online)].pdf 2019-03-27
3 636-delnp-2005-petition-137.pdf 2011-08-21
4 636-DELNP-2005-RELEVANT DOCUMENTS [28-03-2018(online)].pdf 2018-03-28
4 636-delnp-2005-pct-416.pdf 2011-08-21
5 636-DELNP-2005-RELEVANT DOCUMENTS [19-03-2018(online)].pdf 2018-03-19
5 636-delnp-2005-pct-409.pdf 2011-08-21
6 Patent No. 241772-Notification of alteration ur 94(2),94(3) and registration us 69-(11-12-2017).pdf 2017-12-11
6 636-delnp-2005-pct-408.pdf 2011-08-21
7 Form 27 [25-03-2017(online)].pdf 2017-03-25
7 636-delnp-2005-pct-401.pdf 2011-08-21
8 Form 27 [21-03-2017(online)].pdf 2017-03-21
8 636-delnp-2005-pct-308.pdf 2011-08-21
9 636-delnp-2005-pct-306.pdf 2011-08-21
9 ENTRY_OF_ADDITIONAL_ADDRESS.pdf_1.pdf 2016-08-16
10 636-delnp-2005-pct-304.pdf 2011-08-21
10 ENTRY_OF_ADDITIONAL_ADDRESS.pdf 2015-12-08
11 636-DELNP-2005-94-(1)-PROCESSED-CASE.pdf 2015-08-12
11 636-delnp-2005-pct-301.pdf 2011-08-21
12 241772.pdf 2015-04-13
12 636-delnp-2005-pct-210.pdf 2011-08-21
13 241772_AoS.pdf 2015-03-13
13 636-DELNP-2005-PCT-105.pdf 2011-08-21
14 241772_F-16.pdf 2015-03-13
14 636-delnp-2005-pct-101.pdf 2011-08-21
15 636-delnp-2005-gpa.pdf 2011-08-21
15 DOA_Attested.pdf 2015-03-13
16 241772-Assignment-(10-03-2015).pdf 2015-03-10
16 636-delnp-2005-form-5.pdf 2011-08-21
17 636-delnp-2005-form-3.pdf 2011-08-21
17 241772-Correspondence Others-(10-03-2015).pdf 2015-03-10
18 241772-Form-16-(10-03-2015).pdf 2015-03-10
18 636-DELNP-2005-Form-2.pdf 2011-08-21
19 241772-GPA-(10-03-2015).pdf 2015-03-10
19 636-delnp-2005-form-18.pdf 2011-08-21
20 241772-Others-(10-03-2015).pdf 2015-03-10
20 636-delnp-2005-form-1.pdf 2011-08-21
21 241772_AoS.pdf ONLINE 2015-02-18
21 636-delnp-2005-drawings.pdf 2011-08-21
22 241772_F-16.pdf ONLINE 2015-02-18
22 636-DELNP-2005-Description (Complete).pdf 2011-08-21
23 636-delnp-2005-correspondence-po.pdf 2011-08-21
23 DOA_Attested.pdf ONLINE 2015-02-18
24 636-delnp-2005-correspondence-others.pdf 2011-08-21
24 241772_FORM 27.pdf 2014-04-28
25 636-delnp-2005-complete specification (granted).pdf 2011-08-21
25 636-delnp-2005-GPA-(28-05-2013).pdf 2013-05-28
26 636-delnp-2005-complete specification (as files).pdf 2011-08-21
26 636-delnp-2005-Petition Others-(28-05-2013).pdf 2013-05-28
27 636-DELNP-2005-Claims.pdf 2011-08-21
27 636-delnp-2005-Correspondence Others-(28-03-2013).pdf 2013-03-28
28 636-delnp-2005-abstract.pdf 2011-08-21
28 636-delnp-2005-Form-27-(28-03-2013).pdf 2013-03-28
29 636-delnp-2005-abstract.pdf 2011-08-21
29 636-delnp-2005-Form-27-(28-03-2013).pdf 2013-03-28
30 636-DELNP-2005-Claims.pdf 2011-08-21
30 636-delnp-2005-Correspondence Others-(28-03-2013).pdf 2013-03-28
31 636-delnp-2005-complete specification (as files).pdf 2011-08-21
31 636-delnp-2005-Petition Others-(28-05-2013).pdf 2013-05-28
32 636-delnp-2005-complete specification (granted).pdf 2011-08-21
32 636-delnp-2005-GPA-(28-05-2013).pdf 2013-05-28
33 241772_FORM 27.pdf 2014-04-28
33 636-delnp-2005-correspondence-others.pdf 2011-08-21
34 636-delnp-2005-correspondence-po.pdf 2011-08-21
34 DOA_Attested.pdf ONLINE 2015-02-18
35 241772_F-16.pdf ONLINE 2015-02-18
35 636-DELNP-2005-Description (Complete).pdf 2011-08-21
36 636-delnp-2005-drawings.pdf 2011-08-21
36 241772_AoS.pdf ONLINE 2015-02-18
37 241772-Others-(10-03-2015).pdf 2015-03-10
37 636-delnp-2005-form-1.pdf 2011-08-21
38 241772-GPA-(10-03-2015).pdf 2015-03-10
38 636-delnp-2005-form-18.pdf 2011-08-21
39 241772-Form-16-(10-03-2015).pdf 2015-03-10
39 636-DELNP-2005-Form-2.pdf 2011-08-21
40 241772-Correspondence Others-(10-03-2015).pdf 2015-03-10
40 636-delnp-2005-form-3.pdf 2011-08-21
41 241772-Assignment-(10-03-2015).pdf 2015-03-10
41 636-delnp-2005-form-5.pdf 2011-08-21
42 636-delnp-2005-gpa.pdf 2011-08-21
42 DOA_Attested.pdf 2015-03-13
43 241772_F-16.pdf 2015-03-13
43 636-delnp-2005-pct-101.pdf 2011-08-21
44 241772_AoS.pdf 2015-03-13
44 636-DELNP-2005-PCT-105.pdf 2011-08-21
45 241772.pdf 2015-04-13
45 636-delnp-2005-pct-210.pdf 2011-08-21
46 636-delnp-2005-pct-301.pdf 2011-08-21
46 636-DELNP-2005-94-(1)-PROCESSED-CASE.pdf 2015-08-12
47 636-delnp-2005-pct-304.pdf 2011-08-21
47 ENTRY_OF_ADDITIONAL_ADDRESS.pdf 2015-12-08
48 636-delnp-2005-pct-306.pdf 2011-08-21
48 ENTRY_OF_ADDITIONAL_ADDRESS.pdf_1.pdf 2016-08-16
49 636-delnp-2005-pct-308.pdf 2011-08-21
49 Form 27 [21-03-2017(online)].pdf 2017-03-21
50 636-delnp-2005-pct-401.pdf 2011-08-21
50 Form 27 [25-03-2017(online)].pdf 2017-03-25
51 Patent No. 241772-Notification of alteration ur 94(2),94(3) and registration us 69-(11-12-2017).pdf 2017-12-11
51 636-delnp-2005-pct-408.pdf 2011-08-21
52 636-DELNP-2005-RELEVANT DOCUMENTS [19-03-2018(online)].pdf 2018-03-19
52 636-delnp-2005-pct-409.pdf 2011-08-21
53 636-DELNP-2005-RELEVANT DOCUMENTS [28-03-2018(online)].pdf 2018-03-28
53 636-delnp-2005-pct-416.pdf 2011-08-21
54 636-DELNP-2005-RELEVANT DOCUMENTS [27-03-2019(online)].pdf 2019-03-27
54 636-delnp-2005-petition-137.pdf 2011-08-21
55 636-DELNP-2005-Correspondence-Others-(31-05-2010).pdf 2010-05-31
55 636-DELNP-2005-RELEVANT DOCUMENTS [29-05-2019(online)].pdf 2019-05-29
56 636-DELNP-2005-GPA-(31-05-2010).pdf 2010-05-31
56 636-DELNP-2005-RELEVANT DOCUMENTS [27-03-2020(online)].pdf 2020-03-27

ERegister / Renewals

3rd: 05 Aug 2010

From 16/05/2005 - To 16/05/2006

4th: 05 Aug 2010

From 16/05/2006 - To 16/05/2007

5th: 05 Aug 2010

From 16/05/2007 - To 16/05/2008

6th: 05 Aug 2010

From 16/05/2008 - To 16/05/2009

7th: 05 Aug 2010

From 16/05/2009 - To 16/05/2010

8th: 05 Aug 2010

From 16/05/2010 - To 16/05/2011

9th: 25 Apr 2011

From 16/05/2011 - To 16/05/2012

10th: 19 Apr 2012

From 16/05/2012 - To 16/05/2013

11th: 07 May 2013

From 16/05/2013 - To 16/05/2014

12th: 14 May 2014

From 16/05/2014 - To 16/05/2015

13th: 20 Apr 2015

From 16/05/2015 - To 16/05/2016

14th: 06 Apr 2016

From 16/05/2016 - To 16/05/2017

15th: 05 Apr 2017

From 16/05/2017 - To 16/05/2018

16th: 05 Apr 2018

From 16/05/2018 - To 16/05/2019