We are using WMware 4.1

When I run that command is shows:
[boitho@searchdaimon ~]$ df -h /boithoData
Filesystem         Size Used Avail Use% Mounted on
/dev/sdb           79G 5.2G   73G   7% /boithoData
So it does show plenty of space and seems to be seeing the additional space I added in VMware. but doesn’t seem to be using it. Now this would be on Hard Drive Two. Could it be related to space needed on hard drive one?
I just used a program called Gparted that allowed me manage the three partitions on Disk One. I increased their size since changing the size in VMware left that additional space as unallocated. So I redistributed some additional space to the main partition, the TMP and Swap to see if that should help. I just did this so time will tell. The Swap was only 2GB and the temp was less than 1GB as I recall. Could increasing those partitions on main drive help?

I just received the Insufficient Space message again after it crawled about 100 documents. The message was there for a few minutes and then it started trying to crawl again. This is the what usually happens it will take quite a while and crawl a few files and then give the error and then start again. This also seems to make the CPU usage begin to rise over time.

Also when looking in the crawling log I see these types of events:
Jul 30 15:15:59 searchdaimon crawlManager[10609]: crawled url: “file://10.11.20.7/research/HOME/#External_Research_By_Company/Skanska/2011/NC-SC-VA/Reed/NC/NC_Transportation pcn.xlsx”, size: 17396 b, ACL: allow “S-1-5-18,S-1-5-21-268664334-1329476581-367356602-1307,S-1-5-21-268664334-1329476581-367356602-1314,S-1-5-21-268664334-1329476581-367356602-512,S-1-5-21-268664334-1329476581-367356602-513”, denied “”
And
Jul 30 15:15:49 searchdaimon crawlManager[10609]: ERROR: can’t sent to bbdn! Tryed to send doc “file://10.11.20.7/research/HOME/#External_Research_By_Company/Skanska/2011/NC-SC-VA/Reed/NC/NC_Lodging pcn.xlsx” Will sleep and then reconect. Wont send same doc again.
Not sure if those mean anything.

I haven’t come across the out of space issue with my setup, but I am seeing it seem to stop crawling for quite some time then start again. The first message you list above is it performing normal indexing. The second appears to be an issue, and I’m seeing the same one - along with another:
Jul 30 16:46:18 searchdaimon crawlManager[29376]: crawled url: “file://192.168.238.249/Projects/SUPPORT/drv/Archive/CheckList_drv2.doc”, size: 40960 b, ACL: allow “S-1-5-11,S-1-5-32-544,S-1-5-21-3581580000-1769619486-3757737635-1233,S-1-5-21-3581580000-1769619486-3757737635-1303,S-1-5-21-3581580000-1769619486-3757737635-1382,S-1-5-21-3581580000-1769619486-3757737635-1384”, denied “”
Jul 30 16:46:18 searchdaimon crawlManager[29376]: ERROR: can’t sent to bbdn! Tryed to send doc “file://192.168.238.249/Projects/SUPPORT/drv/Archive/ItemsToCheckOn_drv.doc” Will sleep and then reconect. Wont send same doc again.
Jul 30 16:46:57 searchdaimon crawlManager[29376]: ERROR: documentError: level: 1
Jul 30 16:46:57 searchdaimon crawlManager[29376]: ERROR: crawlsmb.c: Error! Could not open directory //192.168.238.249/Projects/Template/SomeDirectory for dir “//192.168.238.249/Projects/Template/SomeDirectory” at src/crawlSMB/crawlsmb.c:243
Jul 30 16:50:00 searchdaimon crawlManager[4718]: ERROR: Starting crawl of collection “Projects” (id 197).
Jul 30 16:50:00 searchdaimon crawlManager[4718]: ERROR: Error: Can’t crawl collection “Projects”. We are already crawling it.
Jul 30 16:55:26 searchdaimon crawlManager[4899]: ERROR: Starting crawl of collection “Projects” (id 197).
Jul 30 16:55:26 searchdaimon crawlManager[4899]: ERROR: Error: Can’t crawl collection “Projects”. We are already crawling it.
Jul 30 17:00:55 searchdaimon crawlManager[5082]: ERROR: Starting crawl of collection “Projects” (id 197).
Jul 30 17:00:55 searchdaimon crawlManager[5082]: ERROR: Error: Can’t crawl collection “Projects”. We are already crawling it.
Jul 30 17:06:21 searchdaimon crawlManager[5271]: ERROR: Starting crawl of collection “Projects” (id 197).
Jul 30 17:06:21 searchdaimon crawlManager[5271]: ERROR: Error: Can’t crawl collection “Projects”. We are already crawling it.
Jul 30 17:10:11 searchdaimon crawlManager[29376]: ERROR: documentError: level: 1
Jul 30 17:10:11 searchdaimon crawlManager[29376]: ERROR: crawlsmb.c: Error! We don’t have access to //192.168.1.249/Projects/Documents/someexcelsheet.xls.
Jul 30 17:11:46 searchdaimon crawlManager[5454]: ERROR: Starting crawl of collection “Projects” (id 197).
Jul 30 17:11:46 searchdaimon crawlManager[5454]: ERROR: Error: Can’t crawl collection “Projects”. We are already crawling it.
Jul 30 17:17:14 searchdaimon crawlManager[5661]: ERROR: Starting crawl of collection “Projects” (id 197).
Jul 30 17:17:14 searchdaimon crawlManager[5661]: ERROR: Error: Can’t crawl collection “Projects”. We are already crawling it.

After a while, it resumes it’s normal crawling - but it seems to be doing the above more often than it does active crawling. And note - it’s quite possible I don’t’ have access to the documents it’s referring to - I’m not using an admin account for my SMB. I wouldn’t expect that to hold it up for 30 minutes though.

When I run this Command to resize filesystem dev/sbd
Resize2fs –p /dev/sbd it shows:
The filesystem is already 20971520 blocks long.  Nothing to do! Which is only 20GB or so, when I’ve given it 80GB.
Because it’s mounted it may not let me resize it. I’ve tried to umount it but it shows Device Busy. Are there services I can stop to allow me to umount the dev/sdb filesystem? Once it’s umounted maybe I can then resize to get the full 80GB is what I’m hoping. Or will that not work?

Unfortunately we have never seen this at any of our clients where we have access to their installation. We also have made some large test sets we uses to test this, but never seen the out of disk space error. When we can’t simulate it, it is hard to track it down, bu I see we will have to do a fresh effort on fixing this.

Since we don’t see it, and it doesn’t appear all the time, maybe it is related to how a specifically file is handled, that has an error we haven’t come across yet? The ES uses many 3-party connectors that may contain bugs. They run in separate process, but maybe some bug still manages to break the system…

To test this I have started to download this torrent: http://thepiratebay.sx/torrent/6353395/Geocities_-_The_PATCHED_Torrent . It contains the remains of Geocities, and should contain some of the worst set of html and other files seen to man. I will try to index it and see what happens.

ERROR: can’t send to bbdn! Tried to send doc “file://XXXXXX” Will sleep and then reconnect. Won’t send the same document again.

This error message (with some horrible typos that has been fixed in the next version) is normally harmless. It indicates that the document manager fail to handle the document. If your collection of data is big enough you will find that some document are corrupt or otherwise can’t be read. If this happens the converter possess restarts and then handle the next document. The crawler will then print this error message.

Because it’s mounted it may not let me resize it. I’ve tried to umount it but it shows Device Busy. Are there services I can stop to allow me to umount the dev/sdb filesystem? Once it’s umounted maybe I can then resize to get the full 80GB is what I’m hoping. Or will that not work?

Try to stop all the services listed under “Manage services” in the ES adm panel.

Any thoughts on what could cause the gap in progress between the last-good file (a 40k .doc) at 16:46:18 and the error messages where it’s not doing anything (up to 17:17:14 last time I looked)? Are there any other log locations or temp files or something I could look at that might give me a better idea of what’s going on during that time?

We’re currently only crawling SMB shares, and these are only internal documents on a US based (only) system, so I wouldn’t expect much in the way of Unicode characters. The occasional corrupt document is expected, but not much out of the ordinary in our datasets.

I’m afraid I just ran into the out of space error again myself (after greatly increasing the drive space a couple days ago). The system thinks I have plenty:
[boitho@searchdaimon ~]$ df -h
Filesystem         Size Used Avail Use% Mounted on
/dev/sdb1         9.5G 1.9G 7.1G 21% /
/dev/sdb2         996M   35M 910M   4% /tmp
tmpfs           4.0G   0 4.0G   0% /dev/shm
/dev/sda           79G 6.4G   72G   9% /boithoData

Since you’re not able to replicate this on your datasets (and have referenced OS-related issues in the past), I’m thinking perhaps the best approach would be to reinstall on a new linux OS rather than trying to use the provided VM’s? My system is still in the test phase at the moment, so if there’s any test I can run for you to help in troubleshooting I’d be happy to.

I’ve setup a new (parallel) VM on the same host, using the same memory and processor configuration. I expanded the second hard drive to 80GB, as I had done on the original. I used your kickstarter process on this one (as opposed to the original which was a VMWare vmdk). I pointed it to the same share giving me problems, and it’s currently indexing… I’ll let you know the results in a couple days.

The one main difference I see is that the sda and sdb drives are indeed swapped between the two installs. I don’t yet see any other difference.

The problem seems to be occurring once it reaches about 400,000 documents in a single share, which takes a good full day to get to, so I won’t know if this resolves the problem for a while.

I do see my original VM still indexing - it indexes for a bit, then reports it’s out of disk space and sits there for a while, then indexes a little more after a few minutes - and that process repeats over and over again.

Strangely enough on mine yesterday afternoon I was watching and all of a sudden started crawling at an increased rate and finished up the last 20,000 documents very quickly, in about 30 minutes. So I setup a new crawl of another space and it too crawled very quickly reaching 150,000 + documents very quickly. But then it slowed way down and the CPU’s all peaked at 100% and never let them go until I rebooted it this morning. It then shortly there after finished up the last 40,000 documents. But now when the recrawls are running of the previous Collections I’m getting out of Space messages again even though I’m showing plenty still. I start the Crawl again and this time it finishes without and error. Here is my configuration:
4 CPU’s
16GB of RAM
Disk one: 50GB and Disk 2 80GB. Both with less than 7% used.
All my crawls have been of MS Shares using the SMB connector.
I currently have three Collections, with 11.000, 272,000 and 85,000 files. I had a fourth one that I mentioned above but deleted it when I shouldn’t have, but it was 191,000 files (see separate post about that one). I don’t recall any issues until after the first two collections finished. When doing the last two I started getting out of space errors and CPU peaking.
After rebooting the CPU usage goes down again, but slowly begins rising until it peaks out and doesn’t go away without a reboot. This sounds like a memory leak in the application or maybe the SMB connector. Could this be part of the problem? I’m willing to provide any other information you need to help resolve the problem as this is still in the test phase as well.

Thank you for your help bout of you. Hopefully we can finally track this down. Sadly I don’t have a good explanation on why you are seeing these changes in speed, but a memory leak comes to mind.

When you crawl, can you run “top” from the command line and see if the VIRT or %MEM columns for crawlManager2 or boitho-bbdn increases.

It is interesting that bout of you are experiencing this problem with the smb crawler. I have gone back in out notes, and for what I can see all other complains has also been when people have been using smb. However it can just be that smb is where one has the most documents.

I have written a quick custom crawler that makes random html documents and add it to the index so I can stress test the crawling possess. It can add about 800 documents pr. second. Just added 1 million to see what would happen, and didn’t see any memory leaks. This may mean that it is related to a specific crawler.

Ok I’ve been running a crawl now for about 2 hours. I rebooted the Search server in VM first before starting and it has been running pretty consistent so far, which seems to be normal. It has crawled 25,000 plus documents so far, so it’s running at a good clip. Generally this will be the case and then several hours in the crawl slows down and the CPU usage climbs, and I start getting insufficient space messages. I have the Top running in the command line and I’m monitoring. So far CPU usage is running between 10-20%. This is what I’m also seeing the VM Console on the performance graph. I will keep an eye on this and post when I start to see the Virt or %Mem usage increasing.

My original VM did complete indexing overnight (finished with about 520,000 docs) despite the constant pauses and errors about being out of space. I ran a number of CPU, disk space, and memory commands last night but didn’t see anything out of the ordinary and no CPU spikes like itgroup is reporting. My 2nd version of the VM (installed using the kickstart process) is still indexing, and currently seeing long delays between some documents - but not yet reporting out of space.

OK at about 5 hours and 54656 file crawled I now have my first Insufficient Disk space message and the crawl has stopped. I’m for now not seeing any high utilization of CPU, Virt or Mem in Top on on the VM performance tab. I do that this message in the Crawling log.
Jul 31 14:05:47 searchdaimon crawlManager[2301]: Finished crawling of collection “Corporate” (id 185).
The crawl though isn’t done, as there are probably 270.000 total that needs to be crawled on this share.
The crawl then started again on it’s own about 10 minutes later with the Crawl numbers very slowly moving up again. In the crawling log I have today and in the past seen this a lot as well:
Jul 31 14:15:47 searchdaimon crawlManager[22840]: ERROR: Error: Can’t crawl collection “Corporate”. We are already crawling it.
Jul 31 14:15:47 searchdaimon crawlManager[22840]: ERROR: Starting crawl of collection “Corporate” (id 185).
Not sure why as I’m not manually starting the crawl its doing it on it own. It maybe nothing but I thought I mention it as I see this a lot.
CPU usage has at times hit over 50% but so far has always come down after a period of time. I did see this happen once where the process XSLTPROC was using 100% for about 5 minutes but then released it.
I continue to post updates when event happen.

Great news! I think I am able to reliably replicate this now:

It is not caused by a memory leak but an issue with the way processes communicates in Linux. To speed up the startup of some programs we uses a functions known as “SO_REUSEADDR”. It is generally considered safe, but as the manual sees “unexpected data can come in, it may confuse your server, but while this is possible, it is not likely”.  Likely or not, on rare occasions it does happen to us…

We are writing a fix now. It shouldn’t take more then 1-2 days before we have on you guys can test

That is great news. And I’ll be happy to test once you have a fix in place.

thanks…