Home » Forum
  • If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.


No announcement yet.

Known issue with server load mechanisms and "core engine is not responding..."

  • Filter
  • Time
  • Show
Clear All
new posts

  • Known issue with server load mechanisms and "core engine is not responding..."

    We've come across a situation where a web server may be configured to avoid a DoS (Denial of Service) or similar issues with being overloaded by too many requests from a single client, and this is being triggered when using Zoom with all 10 threads to spider the web site.

    If setup properly, this mechanism should simply stop responding gracefully and Zoom will start reporting that the server is not responding.

    In some cases, the server has a mechanism that is overly drastic and starts terminating the communication midway. This could lead to Zoom not responding as it in turn is waiting for its own HTTP component (curl) that never comes back.

    In either of these cases, you can avoid this problem by configuring Zoom to use less threads (e.g. 1 or 2) or to insert a delay between requests. You can find these settings under "Configure"->"Spider options"

    It may also help to talk to your web host or admin, to determine what policy is in place and what number of connections within x number of seconds may provoke such behaviour so you can optimize your crawling configuration.
    Wrensoft Web Software
    Sydney, Australia
    Zoom Search Engine