CLIAMS:What is claimed is:
1. A method for checking compliance of a website with at least one of web standards and target display environments, the method comprising of
splitting a job received from a user interface into a plurality of tasks by a controller, wherein the job is entered by an end user using the user interface;
updating the plurality of tasks in a database by the controller;
obtaining rendered HyperText Markup Language (HTML) source of links present in a task by an agent module using a Document Object Model (DOM) component, on the agent module finding at least one available task in the database;
crawling the HTML source to find a list of Uniform Resource Locators (URLs) till a level by the agent module, wherein the level is as specified by the end user in the job;
filtering the list of URLs by the agent module according to the job;
obtaining rendered HTML source for the list of URLs using DOM component by the agent module;
verifying the rendered HTML source for rules as specified in the job by the agent module; and
marking the task as completed by the agent module.
2. The method, as claimed in claim 1, wherein the job comprises of a list of URLs; authentication related details; and rules.
3. The method, as claimed in claim 1, wherein the method further comprises of
prompting the end user to enter the base URL for which singed-in crawling and verification is required;
prompting the end user to enter required authentication details;
encrypting the required authentication details;
decrypting the required authentication details;
logging-in to the base URL using the required authentication details; and
logging out of the base URL after collecting the rendered HTML code.
4. The method, as claimed in claim 1, wherein the agent module marks the task as scheduled, on the agent module finding a task.
5. The method, as claimed in claim 1, wherein the method further comprises of the agent module validating the URLs.
6. The method, as claimed in claim 1, wherein the agent module checks if the task is marked for signed-in crawling and verification.
7. The method, as claimed in claim 1, wherein filtering the list of URLs by the agent module on basis of base URLs, any host URLs; or selected URL types.
8. The method, as claimed in claim 1, wherein rules comprises of web page verification rules; and web accessibility rules.
9. The method, as claimed in claim 8, wherein the web page verification rules comprises of Extensible HyperText Markup Language (XHTML) standards; Cascading Style Sheets (CSS) standards, must have and not have content search with regular expression and text; bad links search inside the webpage and rules related to plugins.
10. The method, as claimed in claim 8, wherein the web accessibility rules are based on Web Content Accessibility Guidelines (WCAG) guidelines.
11. The method, as claimed in claim 1, wherein the method further comprises of storing results of the verification in at least one of a database; and an unstructured data storage.
12. The method, as claimed in claim 11, wherein the method further comprises of making the results available to the end user for viewing.
13. The method, as claimed in claim 11, wherein the method further comprises of making the results available to the end user for downloading.
14. The method, as claimed in claim 1, wherein the method further comprises of scaling the agent module.
15. A system for checking compliance of a website with at least one of web standards and target display environments, the system comprising of
a controller configured for
splitting a job received from a user interface into a plurality of tasks, wherein the job is entered by an end user using the user interface;
updating the plurality of tasks in a database;
an agent module configured for
obtaining rendered HyperText Markup Language (HTML) source of links present in a task using a Document Object Model (DOM) component, on the agent module finding at least one available task in the database;
crawling the HTML source to find a list of Uniform Resource Locators (URLs) till a level, wherein the level is as specified by the end user in the job;
filtering the list of URLs according to the job;
obtaining rendered HTML source for the list of URLs using DOM component;
verifying the rendered HTML source for rules as specified in the job; and
marking the task as completed.
16. The system, as claimed in claim 15, wherein the system is further configured for
prompting the end user to enter the base URL for which singed-in crawling and verification is required;
prompting the end user to enter required authentication details;
encrypting the required authentication details;
decrypting the required authentication details;
logging-in to the base URL using the required authentication details; and
logging out of the base URL after collecting the rendered HTML code.
17. The system, as claimed in claim 15, wherein the agent module is further configured for marking the task as scheduled, on the agent module finding a task.
18. The system, as claimed in claim 15, wherein the agent module is further configured for validating the URLs.
19. The system, as claimed in claim 15, wherein the agent module is further configured for checking if the task is marked for signed-in crawling and verification.
20. The system, as claimed in claim 15, wherein the agent module is further configured for filtering the list of URLs by the agent module on basis of base URLs, any host URLs; or selected URL types.
21. The system, as claimed in claim 15, wherein the agent module is further configured for storing results of the verification in at least one of a database; and an unstructured data storage.
22. The system, as claimed in claim 15, wherein the system is further configured for making the results available to the end user for viewing.
23. The system, as claimed in claim 15, wherein the system is further configured for making the results available to the end user for downloading.
24. The system, as claimed in claim 15, wherein the system is further configured for enabling the agent module to be scaled.

25. A computer program product for checking compliance of a website with at least one of web standards and target display environments, the product comprising:
an integrated circuit comprising at least one processor;
at least one memory having a computer program code within said circuit, wherein said at least one memory and said computer program code with said at least one processor cause said product to:
split a job received from a user interface into a plurality of tasks, wherein the job is entered by an end user using the user interface; and
update the plurality of tasks in a database.
26. The computer program product, as claimed in claim 25, wherein said at least one processor further causes said product to monitor status of the plurality of tasks in the database.
27. The computer program product, as claimed in claim 25, wherein said at least one processor further causes said product to manage the plurality of tasks, wherein the plurality of tasks are executed by an agent module.
28. A computer program product for checking compliance of a website with at least one of web standards and target display environments, the product comprising:
an integrated circuit comprising at least one processor;
at least one memory having a computer program code within said circuit, wherein said at least one memory and said computer program code with said at least one processor cause said product to:
obtain rendered HyperText Markup Language (HTML) source of links present in a task using a Document Object Model (DOM) component, on the agent module finding at least one available task in a database;
crawl the HTML source to find a list of Uniform Resource Locators (URLs) till a level, wherein the level is as specified by the end user in the job;
filter the list of URLs according to the job;
obtain rendered HTML source for the list of URLs using DOM component;
verify the rendered HTML source for rules as specified in the job; and
mark the task as completed.
29. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to mark the task as scheduled, on the agent module finding a task.
30. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to validate the URLs.
31. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to check if the task is marked for signed-in crawling and verification.
32. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to filter the list of URLs by the agent module on basis of base URLs, any host URLs; or selected URL types.
33. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to store results of the verification in at least one of a database; and an unstructured data storage.

Dated: 6th May, 2013 Signature
Vikram Pratap Singh Thakur
Patent Agent ,TagSPECI:FORM 2
The Patent Act 1970
(39 of 1970)
&
The Patent Rules, 2005

COMPLETE SPECIFICATION
(SEE MEMBER 10 AND RULE 13)

TITLE OF THE INVENTION

System and method for crawling, verification and traceability of websites
APPLICANTS:

Name : HCL Technologies Limited

Nationality : Indian

Address : HCL Technologies Ltd., 50-53 Greams Road, Chennai – 600006, Tamil Nadu, India

The following specification particularly describes and ascertains the nature of this invention and the manner in which it is to be performed:-

TECHNICAL FIELD
[001] The embodiments herein relate to crawling, verification and traceability of websites and, more particularly, to ensuring compliance of websites with web standards and target display environments.

BACKGROUND
[002] Currently, users may use a wide range of devices to view a website. The devices may vary from phones, tablets, laptops, desktop computers, televisions and so on. The display present on such devices may range from as small as 2 inches (as present on mobile phones) to 42 inches (as present on televisions capable of accessing the internet). The browser being used on these devices may also vary. The user should be able to have a uniform and standard user experience when viewing the content on the websites, irrespective of the device being used, with the website being compatible with all the devices, irrespective of the nature of the device or the browser being used to access the website.
[003] Standards have been specified for website, wherein the website has to comply with standards such as World Wide Web Consortium (W3C) standard. Typically, compatibility with the standards will ensure that the website is able to be viewed by devices.
[004] To ensure that websites are compatible, several approaches may be adopted such as checking for website incompatibility issue as a part of the internal development and test process, outsourcing the checking to Information Technology Enabled Service (ITES) providers, doing a one-shot big bang compatibility check exercise and so on.
[005] But the above approaches are tedious, costly, time taking and one time exercises will not work. Moreover generating new test suites for escalating and changing compatibility requirements is an added overhead in the website development cycle, which can affect timelines, and with the fast changing web standards there is no predictability in how end users will adopt new technologies.

BRIEF DESCRIPTION OF THE FIGURES
[006] The embodiments herein will be better understood from the following detailed description with reference to the drawings, in which:
[007] FIG. 1 depicts components for performing crawling, verification and traceability of websites , according to embodiments as disclosed herein;
[008] FIG. 2 depicts a controller, according to embodiments as disclosed herein;
[009] FIG. 3 depicts an agent module, according to embodiments as disclosed herein;
[0010] FIGs. 4a, 4b, 4c and 4d are flowcharts depicting the process of online bulk crawling, verification and traceability of anonymous and authenticated websites for continually ensuring the websites are compatible with critical web standards and target display environments, according to embodiments as disclosed herein;
[0011] FIG. 5 depicts the process of an authorized person adding/modifying rules, according to embodiments as disclosed herein;
[0012] FIG. 6 is a screenshot illustrating a user interface for crawler in which option for selecting and entering single or list of URLs, according to embodiments as disclosed herein;
[0013] FIG. 7 is a screenshot illustrating a user interface for verifier in which option for selecting and entering single or list of URLs, according to embodiments as disclosed herein;
[0014] FIG. 8 shows the user interface for search where a person can search, compare and export search results, according to embodiments as disclosed herein; and
[0015] FIG. 9 shows user interface for requesting details for single sign on, according to embodiments as disclosed herein.

DETAILED DESCRIPTION OF EMBODIMENTS
[0016] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well-known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.
[0017] The embodiments herein relate to online bulk crawling, verification and traceability of anonymous and authenticated websites for continually ensuring the websites are compatible with critical web standards and target display environments. Referring now to the drawings, and more particularly to FIGS. 1 through 9, where similar reference characters denote corresponding features consistently throughout the figures, there are shown embodiments.
[0018] FIG. 1 depicts components for performing crawling, verification and traceability of websites, according to embodiments as disclosed herein. The system, as depicted, comprises of a user interface 101, a controller 102, an agent module 103, a database 104 and an unstructured data storage 105.
[0019] The user interface 101 receives a list of websites Uniform Resource Locators (URLs), crawling levels, filters for crawling, rules set for website verification and single sign-on related details (if authentication is required). The user interface 101 may receive these details from an end user. The end user may access the user interface 101 using a web based application, a computer application and so on. The end user may be a user authorized to check a website, a user registered with the website using a suitable means, a lay person who may be checking the website and so on. The user interface 101 filters the list of URLs on the basis of their host URL and creates a list of base URLs. The user interface 101 then prompts the end user to select the Base URL for which signed-in crawling and verification is required. The user interface 101 also prompts the end user to enter the authentication related details. The user interface 101 then creates a job execution request and sends the job execution request to the controller 102, wherein the job execution request comprises of a job id, rules related to job management, rules on handling result related query operations and so on.
[0020] The controller 102 breaks the job into a plurality of tasks and updates the tasks and other details in the database 104. The controller 102 may communicate with the database 104 via a data layer. The database 104 may be co-located with the controller 102. The database 104 may be present remotely from the controller 102 and connected to the controller 102 using a suitable means.
[0021] The agent module 103 is a multi-threaded module which is responsible for crawling and verification of list of URLs. The agent module 103 periodically checks for available tasks in the database 104. On finding an available task, the agent module 103 marks the task as scheduled. On initiating a task, the agent module 103 validates whether the URLs in the task are not broken or bad links. On validating the URLs, the agent module 103 checks whether the task is marked for signed-in crawling and verification. If the tasks are marked, then the agent module 103 selects a base URL and renders the selected base URL using a Document Object Model (DOM) component. The agent module 103 decrypts the authentication related details and logs in using the provided details. The agent module 103 selects all valid links having host URL same as the selected base URL and obtains the rendered HTML source of these links using DOM component in signed-in mode. Then, the agent module 103 crawls URLs in the HyperText Markup Language (HTML) source of webpage till the specified level (wherein the maximum and minimum levels may be specified by an authorized person using a suitable means) and also filters the results as per the options set by an end user. The agent module 103 may filter the crawled results on base URL, any host URL and for any other selected URL types. The agent module 103 saves the result into a database and crawls related log files in the unstructured data storage 105 against the task. The agent module 103 obtains the rendered HTML source for all crawled URLs in signed-in mode using DOM component. Then, the agent module 103 verifies the HTML source for the rules, as selected by the end user. The verification rules may comprise of Extensible HyperText Markup Language (XHTML) standards, Cascading Style Sheets (CSS) standards, must have and not have content search with regular expression and text, bad links search inside the webpage and rule set containing rules related to plugins, web accessibility (based on Web Content Accessibility Guidelines (WCAG) guidelines) and other custom rules. The agent module 103 may use an external service for performing XHTML and CSS verification. The agent module 103 updates the verifier related log files in the unstructured data storage 105 and the results in the database 104. The agent module 103 may store the intermediate data in the database 104. The agent module 103 may also store the results for CSS and XHTML verification with final result summary. The agent module 103 further marks the task as completed.
[0022] The controller 102 periodically checks in the database 104 whether all the tasks of the job are completed or not. On all the tasks being marked as completed, the controller 102 aggregates and optimizes the results, wherein the results comprises of execution log files and excel result files. The controller 102 provides results to the user interface 101. The user interface 101 makes the results available to end users in the form of a URL. The results may be available for download in a suitable format. The user interface 101 also enables log files for the executed job to be downloaded using the URL.
[0023] The controller 102 enables an authorized person to add or modify rules using a suitable means, such as an Application Programming Interface (API). To add a new rule, the authorized person defines structure using the appropriate rule grammar, such as a XSD file that contains definition of rules. The controller 102 parses the rule grammar to generate rule XML and message XML, and then the controller 102 updates the XMLs into the database 104. The controller 102 further generates a token and sends the token along with job execution request. The controller 102 while breaking a job into tasks, also updates the rule token in the database 104. The agent module 103 before starting a new task execution, matches its current rule token with the available token. If the tokens are different, then the agent module 103 updates the rule XML and message XML at its end, before starting the task execution.
[0024] The controller 102 along with the user interface 101 enables an authorized person to view a list of recently aborted, failed and current running jobs and also list of available agents and these jobs can be aborted or rescheduled and agent can be scaled up or scaled down. The controller 102 along with the user interface 101 also enables websites traceability.
[0025] FIG. 2 depicts a controller, according to embodiments as disclosed herein. The controller 102, as depicted comprises of a task manager 201, a data aggregator module 202, a data indexing module 203, a query handling module 204 and a rule engine 205.
[0026] The end user enters a list of websites URLs, crawling levels, filters for crawling, rules set for website verification and single sign-on related details (if authentication is required) via the user interface 101.
[0027] On receiving the job execution request from the user interface 101, the task manager 201 breaks the job into a plurality of tasks and updates the tasks and other details in the database 104, via the data layer.
[0028] The task manager 201 periodically checks in the database 104 whether all the tasks of the job are completed or not. On all the tasks being marked as completed, the data aggregator module 202 aggregates saved data of agents for each job. The data indexing module 203 processes the data to index and stores the results in the database 104, via the data layer.
[0029] The query handling module 204 processes and responds with data result against any query request comes from an end user.
[0030] The task manager 201 makes the results available to end users in the form of a URL, wherein the results may be viewed via the user interface. The task manager 201 may make the results available for download in a suitable format. The task manager 201 also enables log files for the executed job to be downloaded using the URL.
[0031] The rules engine 205 enables addition or modification of custom rules. The rules engine 205 parses the rule grammar to generate rule XML and message XML, and then the rules engine 205 updates the XMLs into the database 104. The rules engine 205 further generates a token and sends the token along with job execution request. The task manager 201 while breaking a job into tasks, also updates the rule token in the database 104.
[0032] FIG. 3 depicts an agent module, according to embodiments as disclosed herein. The agent module 103 comprises of an agent manager 301, a crawling engine 302 and a verification engine 303.
[0033] The agent manager 301 periodically checks for available tasks in the database 104. On finding an available task, the agent manager 301 marks the task as scheduled. The agent manager 301 before initiating a new task execution, matches its current rule token with the available token. If the tokens are different, then agent manager 301 updates the rule XML and message XML at its end, before starting the task execution. On initiating the task, the agent manager 301 validates whether the URLs in the task are not broken or bad links. On validating the URLs, the agent manager 301 checks whether the task is marked for signed-in crawling and verification. If the tasks are marked, then the agent manager 301 selects a base URL and renders the selected base URL using a Document Object Model (DOM) component. The agent manager 301 decrypts the authentication related details and logs in using the provided details. The agent manager 301 selects all valid links having host URL same as the selected base URL and obtains the rendered HTML source of these links using DOM component in signed-in mode. Then, the crawling engine 302 crawls URLs in the HyperText Markup Language (HTML) source of webpage till the specified level (wherein the maximum and minimum levels may be specified by an authorized person using a suitable means). The crawling engine 302 further filters the results as per the options set by an end user. The crawling engine 302 may perform the crawling based on a selected scrap level. The crawling engine 302 may filter the crawled results on base URL, any host URL and for any other selected URL types. The agent manager 301 saves the result into a database and crawls related log files in the unstructured data storage 105 against the task. The agent manager 301 obtains the rendered HTML source for all crawled URLs in signed-in mode using DOM component. Then, the verification module 303 verifies the HTML source for the rules, as selected by the end user using the user interface 101. The verification module 303 may use an external service for performing XHTML and CSS verification. The agent manager 301 updates the verifier related log files in the unstructured data storage 105 and the results in the database 104. The agent manager 301 may store the intermediate data in the database 104. The agent manager 301 further marks the task as completed.
[0034] FIGs. 4a, 4b, 4c and 4d are flowcharts depicting the process of online bulk crawling, verification and traceability of anonymous and authenticated websites for continually ensuring the websites are compatible with critical web standards and target display environments, according to embodiments as disclosed herein. The user interface 101 receives (401) a list of websites Uniform Resource Locators (URLs), crawling levels, filters for crawling, rules set for website verification and single sign-on related details (if authentication is required) from an end user. The user interface 101 filters (402) the list of URLs on the basis of their host URL and creates (403) a list of base URLs. The user interface 101 creates (404) a job execution request, wherein the job execution request comprises of a job id, rules related to job management, rules on handling result related query operations and so on. The controller 102 breaks (405) the job into a plurality of tasks and updates (406) the tasks and other details in the database 104. The agent module 103 periodically checks (407) for available tasks in the database 104. On finding an available task, the agent module 103 marks (408) the task as scheduled. On initiating a task, the agent module 103 validates (409) whether the URLs in the task are not broken or bad links. On validating the URLs, the agent module 103 checks (410) whether the task is marked for signed-in crawling and verification. If the tasks are marked, then the agent module 103 selects (411) the base URL and renders (412) the selected base URL using the DOM component. The agent module 103 decrypts (413) the authentication related details and logs (414) in using the provided details. The agent module 103 selects (415) all valid links having host URL same as the selected base URL and obtains (416) the rendered HTML source of these links using the DOM component in signed-in mode. Then, the agent module 103 crawls (417) URLs in the HTML source of webpage till the specified level and filters (418) the results as per the options set by an end user. The agent module 103 may perform the crawling based on a selected scrap level. The agent module 103 may filter the crawled results on base URL, any host URL and for any other selected URL types. The agent module 103 saves (419) the result into the database 104 and crawls (420) related log files in the unstructured data storage 105 against the task. Then, the agent module 103 verifies (421) the HTML source for the rules, as selected by the end user. The agent module 103 updates (422) the verifier related log files in the unstructured data storage 105 and the results in the database 104. The agent module 103 may store the intermediate data in the database 104. The agent module 103 further marks (423) the task as completed. The controller 102 periodically checks (424) in the database 104 whether all the tasks of the job are completed or not. On all the tasks being marked as completed the controller 102 aggregates (425) and optimizes (426) the results. The user interface 101 makes (427) the results available to end users in a suitable format. The various actions in method 400 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in FIGs. 4a, 4b, 4c and 4d may be omitted.
[0035] FIG. 5 depicts the process of adding/modifying rules by authorized person, according to embodiments as disclosed herein. To add a new rule, the authorized person defines (501) the structure using the appropriate rule grammar, such as a XSD file that contains definition of rules. The controller 102 parses (502) the rule grammar to generate rule XML and message XML, and then the controller 102 updates (503) the XMLs into the database 104. The controller 102 further generates (504) a token and sends (505) the token along with job execution request to the database. The controller 102 while breaking a job into tasks, also updates (506) the rule token in the database 104. The various actions in method 500 may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some actions listed in FIG. 5 may be omitted.
[0036] FIG. 6 is a screenshot illustrating a user interface for crawler in which option for selecting and entering single or list of URLs, according to embodiments as disclosed herein. Using the crawler tab, an end user can select crawling level, enter Base or Host URL filter and select URL type filter option.
[0037] FIG. 7 is a screenshot illustrating a user interface for verifier in which option for selecting and entering single or list of URLs, according to embodiments as disclosed herein. The user interface in verifier tab user can select and enter text or regular expression validation details for must have or have not verification in webpage content, can also select other validation options for webpage content verification like XHTML, CSS and Bad links. Then for other rules, the person can further select rules from the list of available rules which are related to Plugin, Accessibility and other Custom Rules.
[0038] FIG. 8 shows the user interface for search where a person can search, compare and export search results, according to embodiments as disclosed herein.
[0039] FIG. 9 shows a user interface for requesting details for single sign on, according to embodiments as disclosed herein.
[0040] Embodiments disclosed herein may be implemented in a cloud. Embodiments herein disclose a combination of Website Compatibility Engine and Expert Services, hereby providing an end-to-end service, thus maximizing impact and reducing overall cost of compatibility testing. Embodiments as disclosed herein are multi-threaded and high performance engine ensures speed of analysis, thus minimizing impact of additional test cycles on release dates. Embodiments herein also support anonymous and authenticated analysis. Embodiments herein obtain the rendered web page content for the detection of plugins and other verification. Embodiments herein disclose a generic rule engine ensures acceleration for most common compatibility requirements and flexibility to use the engine for other upcoming or custom compatibility requirements. Embodiments disclosed herein are optimized for running longer jobs & heavy workloads through user interface and backend decoupling. Embodiments herein support real-time reporting of current state of the system and are compatible to multiple standards, devices and browsers.
[0041] The embodiments disclosed herein can be implemented through at least one software program running on at least one hardware device and performing network management functions to control the network elements. The network elements shown in Figs. 1, 2 and 3 include blocks which can be at least one of a hardware device, or a combination of hardware device and software module.
[0042] The embodiments herein relate to online bulk crawling, verification and traceability of anonymous and authenticated websites for continually ensuring the websites are compatible with critical web standards and target display environments. Therefore, it is understood that the scope of the protection is extended to such a program and in addition to a computer readable means having a message therein, such computer readable storage means contain program code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The method is implemented in a preferred embodiment through or together with a software program written in e.g. Very high speed integrated circuit Hardware Description Language (VHDL) another programming language, or implemented by one or more VHDL or several software modules being executed on at least one hardware device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof, e.g. one processor and two FPGAs. The device may also include means which could be e.g. hardware means like e.g. an ASIC, or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means are at least one hardware means and/or at least one software means. The method embodiments described herein could be implemented in pure hardware or partly in hardware and partly in software. The device may also include only software means. Alternatively, the invention may be implemented on different hardware devices, e.g. using a plurality of CPUs.
[0043] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the claims as described herein.

CLAIMS
What is claimed is:
1. A method for checking compliance of a website with at least one of web standards and target display environments, the method comprising of
splitting a job received from a user interface into a plurality of tasks by a controller, wherein the job is entered by an end user using the user interface;
updating the plurality of tasks in a database by the controller;
obtaining rendered HyperText Markup Language (HTML) source of links present in a task by an agent module using a Document Object Model (DOM) component, on the agent module finding at least one available task in the database;
crawling the HTML source to find a list of Uniform Resource Locators (URLs) till a level by the agent module, wherein the level is as specified by the end user in the job;
filtering the list of URLs by the agent module according to the job;
obtaining rendered HTML source for the list of URLs using DOM component by the agent module;
verifying the rendered HTML source for rules as specified in the job by the agent module; and
marking the task as completed by the agent module.
2. The method, as claimed in claim 1, wherein the job comprises of a list of URLs; authentication related details; and rules.
3. The method, as claimed in claim 1, wherein the method further comprises of
prompting the end user to enter the base URL for which singed-in crawling and verification is required;
prompting the end user to enter required authentication details;
encrypting the required authentication details;
decrypting the required authentication details;
logging-in to the base URL using the required authentication details; and
logging out of the base URL after collecting the rendered HTML code.
4. The method, as claimed in claim 1, wherein the agent module marks the task as scheduled, on the agent module finding a task.
5. The method, as claimed in claim 1, wherein the method further comprises of the agent module validating the URLs.
6. The method, as claimed in claim 1, wherein the agent module checks if the task is marked for signed-in crawling and verification.
7. The method, as claimed in claim 1, wherein filtering the list of URLs by the agent module on basis of base URLs, any host URLs; or selected URL types.
8. The method, as claimed in claim 1, wherein rules comprises of web page verification rules; and web accessibility rules.
9. The method, as claimed in claim 8, wherein the web page verification rules comprises of Extensible HyperText Markup Language (XHTML) standards; Cascading Style Sheets (CSS) standards, must have and not have content search with regular expression and text; bad links search inside the webpage and rules related to plugins.
10. The method, as claimed in claim 8, wherein the web accessibility rules are based on Web Content Accessibility Guidelines (WCAG) guidelines.
11. The method, as claimed in claim 1, wherein the method further comprises of storing results of the verification in at least one of a database; and an unstructured data storage.
12. The method, as claimed in claim 11, wherein the method further comprises of making the results available to the end user for viewing.
13. The method, as claimed in claim 11, wherein the method further comprises of making the results available to the end user for downloading.
14. The method, as claimed in claim 1, wherein the method further comprises of scaling the agent module.
15. A system for checking compliance of a website with at least one of web standards and target display environments, the system comprising of
a controller configured for
splitting a job received from a user interface into a plurality of tasks, wherein the job is entered by an end user using the user interface;
updating the plurality of tasks in a database;
an agent module configured for
obtaining rendered HyperText Markup Language (HTML) source of links present in a task using a Document Object Model (DOM) component, on the agent module finding at least one available task in the database;
crawling the HTML source to find a list of Uniform Resource Locators (URLs) till a level, wherein the level is as specified by the end user in the job;
filtering the list of URLs according to the job;
obtaining rendered HTML source for the list of URLs using DOM component;
verifying the rendered HTML source for rules as specified in the job; and
marking the task as completed.
16. The system, as claimed in claim 15, wherein the system is further configured for
prompting the end user to enter the base URL for which singed-in crawling and verification is required;
prompting the end user to enter required authentication details;
encrypting the required authentication details;
decrypting the required authentication details;
logging-in to the base URL using the required authentication details; and
logging out of the base URL after collecting the rendered HTML code.
17. The system, as claimed in claim 15, wherein the agent module is further configured for marking the task as scheduled, on the agent module finding a task.
18. The system, as claimed in claim 15, wherein the agent module is further configured for validating the URLs.
19. The system, as claimed in claim 15, wherein the agent module is further configured for checking if the task is marked for signed-in crawling and verification.
20. The system, as claimed in claim 15, wherein the agent module is further configured for filtering the list of URLs by the agent module on basis of base URLs, any host URLs; or selected URL types.
21. The system, as claimed in claim 15, wherein the agent module is further configured for storing results of the verification in at least one of a database; and an unstructured data storage.
22. The system, as claimed in claim 15, wherein the system is further configured for making the results available to the end user for viewing.
23. The system, as claimed in claim 15, wherein the system is further configured for making the results available to the end user for downloading.
24. The system, as claimed in claim 15, wherein the system is further configured for enabling the agent module to be scaled.

25. A computer program product for checking compliance of a website with at least one of web standards and target display environments, the product comprising:
an integrated circuit comprising at least one processor;
at least one memory having a computer program code within said circuit, wherein said at least one memory and said computer program code with said at least one processor cause said product to:
split a job received from a user interface into a plurality of tasks, wherein the job is entered by an end user using the user interface; and
update the plurality of tasks in a database.
26. The computer program product, as claimed in claim 25, wherein said at least one processor further causes said product to monitor status of the plurality of tasks in the database.
27. The computer program product, as claimed in claim 25, wherein said at least one processor further causes said product to manage the plurality of tasks, wherein the plurality of tasks are executed by an agent module.
28. A computer program product for checking compliance of a website with at least one of web standards and target display environments, the product comprising:
an integrated circuit comprising at least one processor;
at least one memory having a computer program code within said circuit, wherein said at least one memory and said computer program code with said at least one processor cause said product to:
obtain rendered HyperText Markup Language (HTML) source of links present in a task using a Document Object Model (DOM) component, on the agent module finding at least one available task in a database;
crawl the HTML source to find a list of Uniform Resource Locators (URLs) till a level, wherein the level is as specified by the end user in the job;
filter the list of URLs according to the job;
obtain rendered HTML source for the list of URLs using DOM component;
verify the rendered HTML source for rules as specified in the job; and
mark the task as completed.
29. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to mark the task as scheduled, on the agent module finding a task.
30. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to validate the URLs.
31. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to check if the task is marked for signed-in crawling and verification.
32. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to filter the list of URLs by the agent module on basis of base URLs, any host URLs; or selected URL types.
33. The computer program product, as claimed in claim 28, wherein said at least one processor further causes said product to store results of the verification in at least one of a database; and an unstructured data storage.

Dated: 6th May, 2013 Signature
Vikram Pratap Singh Thakur
Patent Agent

ABSTRACT
The embodiments herein relate to online bulk crawling, verification and traceability of anonymous and authenticated websites for continually ensuring the websites are compatible with critical web standards and target display environments.

FIG. 1