Home » Forum
  • If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.


No announcement yet.

V5 development progress - Limit pages per start point

  • Filter
  • Time
  • Show
Clear All
new posts

  • V5 development progress - Limit pages per start point

    It is already possible to limit the number of pages indexed per web site (start point), but in V4 of the software the limit had to be the same for all sites.

    In V5 you can specify a different number of files to index for each start point. When indexing multiple web sites this allows some sites to be indexed deeply and other to have just a few pages indexed. If you are running a search engine portal type site, there is also the possibility you can sell web site owners the service of deep indexing.

    This option is set in the "Edit start point" window, by clicking on "More" in the Spider tab, and selecting the URL with "Edit". Note that you will also be able to import this option in as part of an imported text file containing your start points. They will have to be in the format of:

    http://www.mysite.com/, INDEX_AND_FOLLOW, http://www.mysite.com/, USELIMIT, 15
    http://www.wrensoft.com/, INDEX_AND_FOLLOW, http://www.wrensoft.com/, NOLIMIT, 25

    The parameters are optional and can be omitted, in which case it will default to no limit.