Ok overnight the CPU maxed out again, twice. Here is what I saw:
There were 15 processes of the XSLTPROC running, all running for 45 minutes or more. Each one was using 373 Virt, 26% CPU and 2.2 MEM. The total processor usage was showing 96%. Nothing was crawling anymore and I had to stop the crawl and reboot the VM Host for the Search to get things running again.
Not sure if the XSLTPROC is related to the issue you found, but seems to be the cause of the CPU maxing out that I’m seeing.
Let me know if you need any other info.

Ok the CPU’s are peaking out again. And it’s the XSLTPROC again that appears to be the issue. Here is a screen capture of what I’m seeing. As you can see one has been running for 283 minutes:

login as: boitho
.(JavaScript must be enabled to view this email address)‘s password:
Last login: Wed Jul 31 22:49:17 2013 from dnw8-it2.fminet.com
[boitho@searchdaimon ~]$ su -
Password:
[root@searchdaimon ~]# top
top - 13:26:13 up 6:50,  2 users,  load average: 5.73, 5.22, 3.99
Tasks: 143 total,  7 running, 136 sleeping,  0 stopped,  0 zombie
Cpu(s): 99.6%us,  0.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi,  0.0%si,  0.0%
Mem:  16217388k total, 10729888k used,  5487500k free,  319536k buffers
Swap:  2337448k total,      0k used,  2337448k free,  8009076k cached

  PID USER     PR NI VIRT RES SHR S %CPU %MEM   TIME+  COMMAND
20086 boitho   20   0 168m 149m 1116 R   83 0.9   2:42.51 xsltproc
19618 boitho   20   0 168m 149m 1116 R   79 0.9   4:18.74 xsltproc
20558 boitho   20   0 168m 149m 1116 R   79 0.9   1:18.54 xsltproc
25598 boitho   20   0 535m 517m 1116 R   78 3.3 283:32.74 xsltproc
7880 boitho   20   0 531m 513m 1116 R   76 3.2 26:50.05 xsltproc
1896 apache   20   0 242m 5956 1576 S   0 0.0   0:01.31 httpd
  1 root     20   0 10308 692 584 S   0 0.0   0:00.46 init
  2 root     15 -5   0   0   0 S   0 0.0   0:00.00 kthreadd
  3 root     RT -5   0   0   0 S   0 0.0   0:00.60 migration/0
  4 root     15 -5   0   0   0 S   0 0.0   0:00.01 ksoftirqd/0
  5 root     RT -5   0   0   0 S   0 0.0   0:00.00 watchdog/0
  6 root     RT -5   0   0   0 S   0 0.0   0:00.66 migration/1
  7 root     15 -5   0   0   0 S   0 0.0   0:00.00 ksoftirqd/1
  8 root     RT -5   0   0   0 S   0 0.0   0:00.00 watchdog/1
  9 root     RT -5   0   0   0 S   0 0.0   0:00.41 migration/2
  10 root     15 -5   0   0   0 S   0 0.0   0:00.00 ksoftirqd/2
  11 root     RT -5   0   0   0 S   0 0.0   0:00.00 watchdog/2
  12 root     RT -5   0   0   0 S   0 0.0   0:00.27 migration/3
  13 root     15 -5   0   0   0 S   0 0.0   0:00.00 ksoftirqd/3
  14 root     RT -5   0   0   0 S   0 0.0   0:00.00 watchdog/3
  15 root     15 -5   0   0   0 S   0 0.0   0:00.01 events/0
  16 root     15 -5   0   0   0 S   0 0.0   0:00.00 events/1
  17 root     15 -5   0   0   0 S   0 0.0   0:00.00 events/2
  18 root     15 -5   0   0   0 S   0 0.0   0:00.00 events/3
  19 root     15 -5   0   0   0 S   0 0.0   0:00.00 khelper
  71 root     15 -5   0   0   0 S   0 0.0   0:00.05 kblockd/0
  72 root     15 -5   0   0   0 S   0 0.0   0:00.00 kblockd/1
  73 root     15 -5   0   0   0 S   0 0.0   0:00.00 kblockd/2
  74 root     15 -5   0   0   0 S   0 0.0   0:00.00 kblockd/3
  78 root     15 -5   0   0   0 S   0 0.0   0:00.00 cqueue/0
  79 root     15 -5   0   0   0 S   0 0.0   0:00.00 cqueue/1
  80 root     15 -5   0   0   0 S   0 0.0   0:00.00 cqueue/2
  81 root     15 -5   0   0   0 S   0 0.0   0:00.00 cqueue/3
  83 root     15 -5   0   0   0 S   0 0.0   0:00.00 ksuspend_usbd
  88 root     15 -5   0   0   0 S   0 0.0   0:00.00 khubd
  91 root     15 -5   0   0   0 S   0 0.0   0:00.00 kseriod
  167 root     20   0   0   0   0 S   0 0.0   0:00.59 pdflush
  168 root     20   0   0   0   0 S   0 0.0   0:00.80 pdflush
  169 root     15 -5   0   0   0 S   0 0.0   0:00.00 kswapd0
  219 root     15 -5   0   0   0 S   0 0.0   0:00.00 aio/0
  220 root     15 -5   0   0   0 S   0 0.0   0:00.00 aio/1

What I ended up doing is run Kill -9 and put in the PID number for the two processes that were chewing up all the resources. I then monitored and CPU usage went way down, but he crawling still continued. Hopefully that didn’t screw anything up but I’m pretty sure they weren’t doing anything anymore other than using up CPU, Virt and MEM cycles.

Thank you for your patience. I am very glad to get thus reports, they have been essential to tracking this down. Both this bugs are related to how documents are handled, and thus hard to prevent because we do not have every single type of buggy document to test with.

Out of disk space issue

We believe we have found the cause for this, and have produced a fix. If you guys would want to test it would be great.

* Download the file http://bbh-001.boitho.com/div/boitho-crawlManager-1.4.119-1.i386.rpm
* Then go to Software updates->Advanced in the admin panel and install the rpm
* Start the crawl again

The XSLTPROC issue

I have had a quick look at this and it is related to extraction of data from .xlsx files ( the newest format for Microsoft Excel ).

A will look into a more permanent solution, but a quick fix is to disable extraction of thus files. you can do so by editing the /home/boitho/boithoTools/fileFilter/ooxml/runinfo file. For example by using nano:

Find this text:

and add # befor it, so it reads:

Then go to the administrator interface and restart the “Document Management” services at the manage services page.

This should prevent the xlsx file filter from disrupting the crawling, but also means you will not be able to search inside .xlsx files.

Good timing on the out of space fix, as I just got it when I saw your email. I’ve just applied the fix and have restarted the crawl. I will continue to monitor and let you know if it returns.
Being able to search inside of Excel files is pretty important to us and we use Excel pretty extensively. I will at this time continue to let that function enabled. When it’s running for a long period of time like an hour or more, would you say it’s ok to Kill process as I did? Or will that prevent searches within Excel files as well?

It is safe to kill it when it has run for an extended period as you did.

File filters like this are supposed to have a monitor process that prevents it for running to long. I don’t yet know why the xlsx filter is able to avoid being handled by the monitor process if it runs forever, but I will find out

Thanks for the info. And I will add that we do have some very large Spreadsheets on our network, some as high at 1GB in size. probably most of which are XLSX though we have a lot of the older format XLS as well. Not sure if that is contributing to the issue, but I thought I’d mention it.

That is interesting; everything that is unnormally big has a way of triggering bugs in software.

How does a spreadsheet become 1 GB? Dos it has a lot of embedded content, or is it generated by some other software? Or did someone sit and type inn 1 GB of numbers in Excel?

Think I will make a couple really large one and test for this.

We have used Excel like a database, where to have large number of worksheets, with lots of formulas, embedded or link content, where you plug in data and then display results in a graphical format using various graphs, charts, pivot tables, and lots of macros. This is something that really should be done in Access, but our company has always done it in Excel for many of our projects because that is the application they know how to use. We are a Management Consulting company so we gather lots of data and crunch numbers to come to various conclusions to help our clients. Many times these large spreadsheets are buggy and don’t always work right. Good luck with your tests.

Well my early testing with the update on the Crawl I had running looked good. It rolled through the last 70,000 documents in about 3 hours with no Insufficient drive space message. The first 130,000 had run more than 24 hours. I plan on setting up another crawl for about 170,000 documents her in a few minutes and all report if I have any issues. But so far so good.

Ok I’ve done two other crawls now. One with 190,000 files took around 8 hours to complete. One with 139,000 documents just finished and took 5 hours. So this update seems to be working well for me.
So my question now is about the crawls done before that had problems with the insufficient disk messages, I worried how accurate the crawl was, and even on the re-crawls I would get this message. Should I just delete them and crawl then again? Or will the re-crawls that run now fix anything that might have been missed?

The new version is working very well for me - looks to have eliminated the delay problem I was seeing!

That is good the hear searcher It must have been related then.

I have traced the XSLTPROC issue down to a bug in the XSLT C library ( http://xmlsoft.org/XSLT/ ).

I have downloaded about 1 000 .xlsx files from the net and will use thus to investigate how often this happens. We have a timeout that shall prevent this type of issues with 3-party libraries, but the .xlsx file converter starts other processes also, that are not killed by the timeout.

If this is all about on single bug we may considering fixing the bug, but if there are more than one way this can happen we may also just make a timeout function that covert child processes also.

So my question now is about the crawls done before that had problems with the insufficient disk messages, I worried how accurate the crawl was, and even on the re-crawls I would get this message. Should I just delete them and crawl then again? Or will the re-crawls that run now fix anything that might have been missed?

The insufficient disk shouldn’t have effected witch document gets added. If some was left out they will be picked up by the next crawl if there is enough disk space. If you want to do a full recrawl anyway there is a function for that, called “Force full crawl now”. It does a delete, then a new crawl. So there is no need for deleting the collection, then adding it back again.

I am quite certain we have found what causes the XSLTPROC issue now. Unfortunately it is a little bit complex to fix, and involves sub-processes spanning new sub-processes. We will probably have to write a monitor system that traverse this tree of processes and kill any processes that don’t conform to the rules.

I will try to have a fix ready in the middle of next week.

Ok thanks for the update. It doesn’t seem to be an issue when re-crawling a Collection, only from doing the initial crawl, from what I’ve seen so far.

The reason for that is that this only happens to a fraction of the .xlsx documents. When you do the first time crawl you crawl more documents, so the chance of encountering a document that triggers this is higher.

I will keep you updated with any progress on a fix.