Zoom Search Engine FAQ - Indexing message boards, forums, etc.
Q. How should I index my site if it features a message board, forum, or calendar and other similarly complex scripts?
If your site features a message board, discussion forum (eg. phpBB, UltimateBB, vBulletin, etc.), a dynamically generated calendar, or other similarly complex scripts, please read the following.
Due to the nature of these web scripts, it is important to configure Zoom carefully if you wish to index them as part of your search engine. While spider mode indexing allows you to index these features, careful configuration is required to avoid indexing too much irrelevant information. When you allow the spider to index the large number of content-irrelevant pages created by these scripts, you are reducing the effectiveness of your search results (by returning too many pages that a user would not find useful), as well as significantly extending the time required to index your site, and wasting resources in terms of bandwidth and disk space.
The reason that this is required, is because spider mode is designed to follow every legitimately different link on a web page. But in the case of these scripts, there can often be many useless pages which are simply user options (eg. login procedures, sorting options, various display modes of the same page) and in some cases, they can even create an infinite number of pages - for example: a calendar script which shows the days of the month, and allows you to browse "next" and "previous" months indefinitely. The spider could potentially follow these links and index every day of the month that the web script allows (most do not have a limit and you'll end up indexing all the months from the year 0 up to the year 99999 and onwards).
In most cases, it would make sense to avoid indexing these sections of the site. Most forums feature their own search facility so you can usually exclude the entire forum from the main site's search engine. You can do this easily by using the "Skip pages and folder list" in the Configuration window, under the "Skip Options" tab. For example, if your forum is hosted at "http://www.mysite.com/forum/", you can skip the entire forums with a skip page entry of "/forum/". Similarly, you can skip calendars and alternate views by specifying the folder, filename, or any recognizable parameters in the URL (eg. "&month=", etc.). See the Users Guide for more information on the skipping pages.
If you actually want to index your forum/script as part of your site's search engine, then you will need to carefully consider which links should be omitted and which needs to be indexed.
Below are some example skip lists for some known forum packages. Note that the same concepts apply to other scripts such as VBulletin, Invision, Ikonboard, Gallery, Calendar, etc. If you wish to include them as part of your site's search engine, take the time to run some tests and determine all the different pages that you should skip so that you don't run the risk of over-indexing your website.
As an example, the phpBB V2 script contains many pages which you may want to exclude, such as the members list, profile pages, private message pages, login pages, etc. The following is a small example skip list of pages and parameters that you would most likely want to skip when indexing a phpBB website:
This should allow Zoom to index all the topic pages (as opposed to individual posts) and assumes you have the forums in a sub-directory named "forum". Note that this is not necessarily a comprehensive list, and you may wish to add/remove pages to suit your site. Run some test indexing sessions and analyse the log window to determine if there are other pages indexed which you do not need.
Here is a example skip page list for indexing UBB sites. Please see the above explanation for more information on why this is necessary or what this achieves.
Note that this is not necessarily a comprehensive list, and you may wish to add/remove pages to suit your site. Run some test indexing sessions and analyse the log window to determine if there are other pages indexed which you do not need.
Here is a example skip page list for spidering vBulletin forums. Please see the above explanation for more information on why this is necessary or what this achieves. This assumes your forum is installed in a directory called /forum/
Note that this is not necessarily a comprehensive list, and you may wish to add/remove pages to suit your site and your version of vB. Run some test indexing sessions and analyse the log window to determine if there are other pages indexed which you do not need.
This section has been moved to the CMS page. Please go here.
Return to the Zoom Search Engine Support page