RoboGen is a visual editor for Robot Exclusion Files. It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public. RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable. RoboGen maintains a database of over 180 search engine user-agents, which are selectable from a drop down menu.
robotstxt analytics tool 1.0 This robots.txt analysis tool can easily verify the content of "robots.txt" of an arbitrary web server (file that specifies behavior on the server for the robot of the search...
Do-download.com do not supply any crack, patches,torrent, password, serial numbers, registration codes, key generators, cd key, hacks or keygen for the software,and please consult directly with program authors (Rietta Solutions) if you have any problem with the software.