V5 development progress - Google Site Map Generator

[Ray]

This new option in the Zoom Search Engine allows you to generate a plain text sitemap (compatible with Yahoo Sitemaps) and a XML sitemap (compatible with Google Sitemaps) for your website at the same time as having Zoom index your website.

This means in a single 'spidering' of your web site you can generate a site map, a search engine index plus check for broken links.

Sitemaps can be useful for submitting your websites to Internet-wide search engines such as Google and Yahoo, helping their spiders index your website more quickly and increasing your web presence. You may also find it useful for maintenance or other development purposes. The sitemap includes information such as the last modified date and the relative priority of the page within the website. Note that you should only create sitemaps for individual websites (and not sitemaps that span multiple websites).

The XML sitemap can also be automatically uploaded at the end of indexing, along with your search files.

Unlike the other rather lame site map generators on the market our solution should

• Scale to truly huge sites of up to a million pages per site map.
• Generate multiple 'small' site map files (50,000 URLs in each) and then generate a master index of all the site maps. As required by Google site maps for large sites.
• Support incremental additions to the map without needing to re-spider the entire site.
• Support the last update date and priority fields automatically based on data from the Zoom Search index files.

We think this Google Site Map generator will be one of the killer features in V5 of Zoom. Please try it out and let us know what you think.

[pjtester]

If I were to index several sites would v5 generate a site map file for each site or attempt to build one file for all sites that I have scanned?

Thanks

[Ray]

It will only attempt to build one sitemap (for either the TXT or XML/Google option selected) for all sites you have scanned.

As you may know, Google Sitemaps require a sitemap file to only contain URLs under the base URL where the sitemap file itself is located. This means that it is not possible to have a sitemap file containing pages from different sites (presumably Google will just ignore the external URLs in the sitemap).

We currently ask the user to be aware of this and only generate sitemaps for a configuration where they are indexing a single site. You should create a different ZCFG config file per site, and build each one separately. This is really the most straight forward way to do it, considering that with each site, you will need to upload them to different FTP servers as well.

We may consider changing this in the future so that it would automatically create multiple sitemaps when more than one site is indexed. However, it would still be up to the user to upload the right sitemap to the right server, and this could be a fairly confusing process for the user to manage. Nonetheless, it would be something we could look into based on user feedback.

[pjtester]

Originally Posted by Ray

We may consider changing this in the future so that it would automatically create multiple sitemaps when more than one site is indexed. However, it would still be up to the user to upload the right sitemap to the right server, and this could be a fairly confusing process for the user to manage. Nonetheless, it would be something we could look into based on user feedback.

Thanks Ray, if Zoom Search was being used to provide a service which spidered several sites then each site's webmaster might appreciate access to a site map for their site. A sweetner in exchange for agreeing to their site being spidered perhaps.

If sitemaps could be written to a specific file location each webmaster could come and get their sitemap file themselves. Maybe one for v6 or v5.1.