Home » Forum
  • If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Announcement

Collapse
No announcement yet.

Connection timed out

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Connection timed out

    I have notice that we are getting a few Connection timed out errors in log like the one below. Should we change how soon the spider starts on the next thread? Will that help reduce the amount of timeouts the indexer is receiving maybe?

    11|01/05/12 12:15:47|Queued URL: http://www.ncsl.org/default.aspx?tabid=22336
    14|01/05/12 13:34:23|DL Thread #1, got URL (http://www.ncsl.org/default.aspx?tabid=22336) off queue
    04|01/05/12 13:34:23|Downloading file http://www.ncsl.org/default.aspx?tabid=22336
    09|01/05/12 13:35:23|Could not download file:
    http://www.ncsl.org/default.aspx?tabid=22336 (Connection timed out)

  • #2
    Hard to know the solution without knowing the cause.

    Possible causes are,
    - A unreliable network connection. Wireless is notorious for this.
    - A server that is overloaded some of the time.
    - A bug in the ASPX code that generates the page in question on your site.
    - Overactive security software (e.g. some type of firewall that starts blocking after a certain number of requests per second).
    - Throttling on the server itself. e.g. maybe the site only serves 1 page per sec per IP address as a way to stopping to much load on the server.
    - A bug in the web server software (as apposed to the web page). For example we have seen IIS leak resources and then crash from time to time.

    Depending on the cause reducing the threads and increasing spider throttling in Zoom might help.

    Deeper investigation is really required. E.g. looking at error logs on the server, taking TCP/IP traces, looking at Zoom logs to see if just a specific subset of pages are effected or if it is random or seeing if the problem occurs periodically, e.g. each 500 pages indexed.

    Comment


    • #3
      I changed it to 1 sec delay and it did help but still had a few timeouts. Going to look at make a little longer to see if that will help out. Thanks for all the ideas to look at. Chris

      Comment


      • #4
        A modern server should be able to serve up more than 10 pages per second. So if there are any failures at 1 page / sec I think there is something else going wrong.

        Comment


        • #5
          It has been better at 8 threads and 1/2 sec delay. I am going to push that to a full 1 sec to see if I can caputre the last few documents. The Previous settings I had it at 10 threads with no delay.

          Comment


          • #6
            Finally figured out the problem is with the session login options. I would setup IE with a search login and it seems to be what created the problem with timeouts.

            Comment

            Working...
            X