PassMark Logo
Home » Forum

Announcement

Collapse
No announcement yet.

Updating index files fails

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Updating index files fails

    I've got the indexer set up to run directly on the web server itself and to deposit its index files right in the root directory of the site. It ran perfectly well in testing this way, but after a move to production, the indexing process (running nightly) fails with these kinds of errors at the end of the log:

    08|11/01/10 01:52:31|Failed to delete existing file E:\Inetpub\[..]\zoom_dictionary.zdat (File may be held open by search engine - attempt 3 of 3)
    08|11/01/10 01:52:32|Failed to delete existing file E:\Inetpub\[..]\zoom_dictionary.zdat (File is held open by another application)

    It says this for four of the .zdat files. The site is running on IIS 6.0 in its own application pool. In order to release the hold on the .zdat files that is causing the problem, I have to stop the website AND recycle the application pool. After that, I can delete the files and replace them with new ones.

    I want to automate this process without having to shut down the site and recycle the app pool on a daily basis. Is there something else that can be done to release the hold on these files?

    chinkle

  • #2
    Are you using Classic ASP, ASP.NET or the CGI/Win32 version on your IIS server?
    --Ray
    Wrensoft Web Software
    Sydney, Australia
    Zoom Search Engine

    Comment


    • #3
      Originally posted by Ray View Post
      Are you using Classic ASP, ASP.NET or the CGI/Win32 version on your IIS server?
      I'm using ASP.NET.

      Comment


      • #4
        Can you confirm that you have the latest ASP.NET Server Control installed? This means you have uninstalled the previous control on your web server, and downloaded and installed the latest from here, onto your web server:
        http://www.wrensoft.com/zoom/aspdotnet.html

        I can't reproduce this behaviour here, and I'm using the latest version of the server control. Do you know if there's any caching settings in place which might be changing behaviour? What version of Windows are you using?
        --Ray
        Wrensoft Web Software
        Sydney, Australia
        Zoom Search Engine

        Comment


        • #5
          I had actually corresponded with you by email on 25 OCT 2010 about another issue wherein you advised me to update my server control. I did so, and it resolved the issue I had... so I'm sure that my software is up-to-date.

          I don't believe any caching settings are in place on the web server. We're using IIS 6.0 on Windows 2003 Server (fully patched).

          Comment


          • #6
            Is there another ASP.NET application running in the same application pool? It's hard to imagine how it might affect it, but I'm just looking at anything related. If so, try creating a separate application pool for Zoom.

            I'm guessing you're saying that right after you restart the application pool (and make a request), immediately after that, the files are locked, and you cannot rename or delete any these ZDAT files from the folder (until you stop the pool). Or is it only after a number of searches have been made? Or is it intermittent and sometimes the files are locked and sometimes they're not?

            Just so you know, we still can't reproduce this problem, and we just tried it on IIS 6.0 on Windows 2003 Server.
            --Ray
            Wrensoft Web Software
            Sydney, Australia
            Zoom Search Engine

            Comment


            • #7
              The application is the only one in its pool. I haven't explicitly tried testing the locked-ness (as opposed to Loch Ness!) of the files because this is a production site. It didn't seem to me that the files were locking immediately because this wasn't a problem during testing or, indeed, in the initial few days the site was online.

              I wonder if the problem may be related to high volume... since clearly the test site isn't going to get the same traffic as the production site, and we launched on a Friday, so the first couple days were the weekend (our traffic is higher during the week than on the weekends).

              Comment


              • #8
                Can you turn on search word logging in the Zoom configuration. At least then you can see how many searches are being done on the site and the duration of the search (which will more of less correspond to the duration that the files are being locked).

                If you have near continuous overlapping searches being done, then this might be linked to the problem. It also might highlight other issues, like searches taking too long to perform on your server.

                Comment


                • #9
                  Apologies for the very late reply. My boss turned my attention elsewhere, and I'm not getting email when this thread updates.

                  I actually did have the search logging turned on. We don't get that many searches, especially not at 1:00am, which is when the scheduled task runs. We get a few searches every 5 or 10 minutes. Most people search on a couple things, so there are clusters of 3 or 4 searches at a time by the same user, but the clusters are separated by about 5 to 10 minutes.

                  Comment


                  • #10
                    Doesn't sound like this is the cause of the locking then.
                    How long does each search take?

                    Does the user who is running the scheduled job have permission to delete files from this folder. Might be just permissions related.

                    Comment


                    • #11
                      I've never seen a query take longer than 0.7s. Most queries run faster than 0.1s, with the majority taking around 0.06s. Also, I don't think it's a permissions problem because the process was running fine for a while, and then it intermittently ran into problems. However, when it does run into problems, the files become corrupted (because only a couple of the files get copied), and the search mechanism fails until I can regenerate the files manually and copy them over.

                      Something else I'm seeing... occasionally the indexing will halt right in the middle of what it's doing. I have the indexer outputting a log file, and successful runs have log files of about 1,418KB. Occasionally (the 16th, 19th, and 20th of this month) the log files were 9 to 13KB with them spontaneously ending in the middle of a word, e.g.:
                      14|11/16/10 01:50:06|Index Thread got ready buffer for http://my.domain.c

                      01|11/19/10 01:50:06|Skipping http://my.domain.com/images/News/select_tab.gif (Blocked by extensi

                      14|11/20/10 01:50:06|Index Thread got ready buffer for h
                      Since I've been outputting the index files to a different directory and manually copying them over, it's not the same problem I had before.
                      Last edited by chinkle; Nov-22-2010, 07:14 PM.

                      Comment


                      • #12
                        If it is a scheduled task, make sure you do not have any setting enabled which may terminate the scheduled task if it does not complete within a certain amount of time (or other conditions such as if the computer ceases to be idle, etc.). Check this via Windows Control Panel -> Administrative Tools -> Task Scheduler.

                        Otherwise we have not had any other reports for freeze/crashes with the current build. Make sure you are using the latest build available (V6.0 build 1023 as of time of post) at this page:
                        http://www.wrensoft.com/zoom/whatsnew.html

                        With the updating index files problem -- just reviewing your original post, it has to be a difference in configuration between your development server and your production server. Since you said it never exhibited this problem until you moved to your production server. So if you can find out what is different (especially in terms of policies applied, user accounts used, etc.) then it will help.
                        --Ray
                        Wrensoft Web Software
                        Sydney, Australia
                        Zoom Search Engine

                        Comment


                        • #13
                          Originally posted by Ray View Post
                          If it is a scheduled task, make sure you do not have any setting enabled which may terminate the scheduled task if it does not complete within a certain amount of time (or other conditions such as if the computer ceases to be idle, etc.). Check this via Windows Control Panel -> Administrative Tools -> Task Scheduler.

                          Otherwise we have not had any other reports for freeze/crashes with the current build. Make sure you are using the latest build available (V6.0 build 1023 as of time of post) at this page:
                          http://www.wrensoft.com/zoom/whatsnew.html
                          The scheduled tasks aren't being terminated... that's not the issue. ZoomIndexer.exe is crashing. I just found some entries in the Application event log:

                          Event Type: Information
                          Event Source: Application Error
                          Event Category: (100)
                          Event ID: 1004
                          Date: 11/24/2010
                          Time: 1:52:45 AM
                          User: N/A
                          Computer: SERVER
                          Description:
                          Reporting queued error: faulting application ZoomIndexer.exe, version 6.0.0.0, faulting module ntdll.dll, version 5.2.3790.4455, fault address 0x0004cde2.

                          For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
                          Data:
                          0000: 41 70 70 6c 69 63 61 74 Applicat
                          0008: 69 6f 6e 20 46 61 69 6c ion Fail
                          0010: 75 72 65 20 20 5a 6f 6f ure Zoo
                          0018: 6d 49 6e 64 65 78 65 72 mIndexer
                          0020: 2e 65 78 65 20 36 2e 30 .exe 6.0
                          0028: 2e 30 2e 30 20 69 6e 20 .0.0 in
                          0030: 6e 74 64 6c 6c 2e 64 6c ntdll.dl
                          0038: 6c 20 35 2e 32 2e 33 37 l 5.2.37
                          0040: 39 30 2e 34 34 35 35 20 90.4455
                          0048: 61 74 20 6f 66 66 73 65 at offse
                          0050: 74 20 30 30 30 34 63 64 t 0004cd
                          0058: 65 32 e2
                          and
                          Event Type: Information
                          Event Source: Application Error
                          Event Category: (100)
                          Event ID: 1004
                          Date: 11/24/2010
                          Time: 1:52:52 AM
                          User: N/A
                          Computer: SERVER
                          Description:
                          Reporting queued error: faulting application ZoomIndexer.exe, version 6.0.0.0, faulting module rpcrt4.dll, version 5.2.3790.4759, fault address 0x000326da.

                          For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
                          Data:
                          0000: 41 70 70 6c 69 63 61 74 Applicat
                          0008: 69 6f 6e 20 46 61 69 6c ion Fail
                          0010: 75 72 65 20 20 5a 6f 6f ure Zoo
                          0018: 6d 49 6e 64 65 78 65 72 mIndexer
                          0020: 2e 65 78 65 20 36 2e 30 .exe 6.0
                          0028: 2e 30 2e 30 20 69 6e 20 .0.0 in
                          0030: 72 70 63 72 74 34 2e 64 rpcrt4.d
                          0038: 6c 6c 20 35 2e 32 2e 33 ll 5.2.3
                          0040: 37 39 30 2e 34 37 35 39 790.4759
                          0048: 20 61 74 20 6f 66 66 73 at offs
                          0050: 65 74 20 30 30 30 33 32 et 00032
                          0058: 36 64 61 6da
                          Let me know if we should discuss this further by email -- perhaps the forum isn't the place for this type of support?

                          Comment


                          • #14
                            Yes, please e-mail us with more information including your ZCFG configuration file.

                            Try to create a debug log file to see if it crashes on the same file each time ("Configure"->"Index log" and enable "Save index log to file" and "Debug mode"). Make sure to save your configuration for this to apply to any scheduled task (and "append date and time" if you want multiple files to be created instead of overwriting). Include this log file with your e-mail.

                            I presume you have also made sure you are using the latest build as mentioned before.
                            --Ray
                            Wrensoft Web Software
                            Sydney, Australia
                            Zoom Search Engine

                            Comment


                            • #15
                              Will do... thanks.

                              Originally posted by Ray View Post
                              I presume you have also made sure you are using the latest build as mentioned before.
                              Yes.

                              Comment

                              Working...
                              X