Is it possible (I don't think so without customizing the code) to open only certain file types in a new window?
I want .doc, .pdf, .xls, etc files to open in a new window, but .asp, .htm, .html, etc files to open in the same window.
I'm using the ASP version.
As you expected, no, it's not possible without customizing the search script code. You can only specify if all search results should be opened in a new window, frame, or same window.
However, you should note that if you wish to open the other file types in a new window of their native application (eg. PDF opens in Acrobat Reader, rather than an IE window with the Acrobat plugin), then you should refer to my answer in this previous thread here:
Wrensoft Web Software
Zoom Search Engine