Internal Server Error indexing CFM pages

[zwink]

I have the professional version of Zoom v6, and am trying to use the "Spider Mode" to crawl our internal Cold Fusion site.

We are able to use offline mode without any problems, and Spider Mode works fine with HTML files, but on any CFM file, we get the "Internal Server Error" which in the IIS logs is error code 500 0 0. It will not crawl beyond 1 CFM file.

My first thought was that some script in the file was causing this. I took the code from an indexable (working) HTML file, pasted it into a CFM file, and got the same error.

We are running IIS6 with Cold Fusion 8. I've tried both Javascript and CGI (CGI is what we use for offline mode, and it works well... for our knowledge base).

I've read all the topics on this subject and none of the other solutions appeared to help.

Thanks

[zwink]

Additional Comment

One more thing... all of the pages we are trying to index are viewable in IE6, IE7, and Firefox 3. Below is the actual IIS log error code:

2009-03-03 17:39:15 W3SVC890044797 10.82.64.89 GET /intranet/index.cfm - 80 - 10.82.64.89 ZoomSpider+-+wrensoft.com+[ZSEBOT] 500 0 0

Thanks

[David]

It is hard to comment on why your index.cfm file might be crashing without having access to the code within this file. Plus we have never written any CF code, so even if you did post it, we probably couldn't quickly debug your code for you.

It might be that your script is doing browser sniffing. That is to say the behaviour of the script will vary depending on the browser or client hitting the page.

I would try downloading the page using the public domain wget tool. They try forcing different user agent strings in wget to see how your script behaves. (Or you just look at your code until you find and fix the bug)

[zwink]

Not the code

Well its not the code, thats the problem. Lets say I have a file called "index.htm", and inside that file I place several hyperlinks. That file is properly crawled as 1 page and then stops, because the links are fake.

So I take the HTML code out of that page, paste it into a CFM file, and then the file is NOT crawled due to the Internal Server Error. The problem appears more related to premissions or IIS than anything script wise.

As an aside, this thing did work once about a month ago and properly crawled some 56 CFM pages. I was going to update the CGI information, and started getting these errors. It won't crawl ANY cfm pages.

I hate 500 errors because they are so generic. I can crawl through CFM files in offline mode fine, its the online mode that causes the issues.

[David]

The 500 error is from your server, not Zoom. So this is why offline mode works, there is no server invovled in offline mode.

Yes, server side 500 errors are very generic, there can be dozens of causes.

Again, I would suggest trying wget, then try looking for more detailed errors in the server log (I am not sure what is available from IIS).

[zwink]

I think it may be permissions related. What system account does Zoom use when trying to crawl a website, or does it use the IUSR (internet user) account?

[zwink]

Problem Resolved

You all may want to make a note of this for Cold Fusion users:

In the application.cfc or applicationSettings.cfm file, you must have the following line present:

<cfset this.sessionManagement = true>

Otherwise, Cold Fusion administrator will throw an error:

"coldfusion.runtime.NoOperScope$ScopeDisabledExcep tion : The requested scope session has not been enabled."

This error will return to IIS6/7 or Apache as error 500, which is useless, and will thus throw a 500 error in Zoom Search Engine as well (on CFM files only).

I was able to track this error down using the Cold Fusion server monitor, pulling up the buried log file, and searching the Ben Forta forums on what the heck that error even meant

Cheers

[David]

Thanks for posting the solution