Crawls run out of space | Searchdaimon Forum

1 of 2

Crawls run out of space

spage@mspromotions.com

Posted: 13 October 2011 07:10 PM

[ Ignore ]

Newbie

Total Posts: 17

Joined 2011-10-13

I thought I’d start a new post, since this is a new, separate issue.

I have some a few large collections (>500 GB each). I’ve noticed that during the initial crawl, the process will stall out with an error saying that the crawl stopped because the disk is out of free space.

However when I run df -h from the command prompt, I see that there is no shortage of space. If I restart the crawl process it resumes, but the process stops again after a while with the same error.

Is it running out of swap space? Do I need to resize disks? If so, what is the best method to resolve this issue?

Thanks

spage@mspromotions.com

Posted: 13 October 2011 07:22 PM

[ Ignore ] [ # 1 ]

Newbie

Total Posts: 17

Joined 2011-10-13

Here’s the exact error: Insufficient disk space. Can’t crawl any more documents.

Image Attachments

Click thumbnail to see full-size image

Runar Buvik

Posted: 13 October 2011 07:45 PM

[ Ignore ] [ # 2 ]

Administrator

Total Posts: 253

Joined 2010-09-13

During crawling, and especially indexing the ES use temporary files that can grow up to 4gb in size. There is also some database files, that may behave quit unpredictably of they run out of space.

Therefor is it a minimum reserved space of 4gb on /boithoData. If you have less then 4 gb available in /boithoData you most increase it. Please see http://www.searchdaimon.com/documentation/C29 for how you can increase a virtual disk in VMware. If you are on some other platform then VMware it’s probably basically the same steps.

spage@mspromotions.com

Posted: 13 October 2011 07:56 PM

[ Ignore ] [ # 3 ]

Newbie

Total Posts: 17

Joined 2011-10-13

So is sda3 (/boithodata) is the most crucial partition in this case?

Runar Buvik

Posted: 13 October 2011 08:06 PM

[ Ignore ] [ # 4 ]

Administrator

Total Posts: 253

Joined 2010-09-13

Yes, the other partitions can grow to 99% without problems. The error message you posted is caused by /boithoData filling up.

What virtualization platform are running this on ?

spage@mspromotions.com

Posted: 13 October 2011 08:08 PM

[ Ignore ] [ # 5 ]

Newbie

Total Posts: 17

Joined 2011-10-13

I’m running this on a VMWare ESXi HA cluster. I actually just ended up resizing the vmdk, then booted up the appliance with a gparted gui boot ISO and resized /tmp and /boithodata.

So far so good. I’ll let you know how it goes.

spage@mspromotions.com

Posted: 13 October 2011 08:56 PM

[ Ignore ] [ # 6 ]

Newbie

Total Posts: 17

Joined 2011-10-13

I resized the partitions as follows:

Increased the vmdk to 40GB
Expanded sda1 (/) to 14GB
Expanded sda2 (/tmp) to 10GB
Expanded sda3 (swap) to 16GB (when booting up, the OS mounts this as /boithoData

This appears to function, however it seems like collection crawls run extremely slowly now.

Runar Buvik

Posted: 14 October 2011 11:38 AM

[ Ignore ] [ # 7 ]

Administrator

Total Posts: 253

Joined 2010-09-13

I am glad you did get it to work. But forfurther references, you shouldn’t have to manually re-partition the disk. Just increase the second (10GB) disk from the VMware client, then reboot the ES. The ES will automatically increase the /boithoData partition during boot.

When it comes to your crawling speed that should pick up after a while. For Windows file shares recrawling and crawling is basically the same. Bout is a scan of the file system looking for new or updated files. Is the crawl got stooped for some reason, the crawler will have to start the scan from the beginning. It will then take some time before it encounter new files.

spage@mspromotions.com

Posted: 17 October 2011 11:23 PM

[ Ignore ] [ # 8 ]

Newbie

Total Posts: 17

Joined 2011-10-13

For your reference, partitioning the disk my way was problematic. The VM never functioned properly after that.

I went ahead and torched that VM and rebuilt it. Per your instruction, I just enlarged the 10GB /boithoData vmdk (I chose 50 GB to be safe). I also granted an additional proc and 4GB of memory. This appears to be handling these very large SMB shares much more easily.

Thanks.

Generalservices

Posted: 16 March 2016 02:24 PM

[ Ignore ] [ # 9 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

Life following a Chapter 7 bankruptcy isn’t as daunting as numerous would perhaps you have think. Actually, most from the negative information available is spewed through the credit card issuers that miss out when you seek bankruptcy relief protection. www.skincarepearls.net

Honesty as well as integrity are crucial to the actual building associated with society. These types of cornerstones construct prosperity as well as peaceful interpersonal order. Dishonesty as well as greed tend to be destructive causes that rip society lower. www.fashionismydrug.org

Mongolia includes a significant confirmed area with regard to coal exploration sector. Within Mongolia the actual annual fossil fuel production is actually approximate regarding 5 zillion metric lots. Almost 85% of the production can be used for vapor and electrical power generation. www.healthbeautycare.net

During the last decade Taupo within New Zealand offers continued in order to attract brand new house contractors and show that not every small cities are decreasing. Read this short article to see three explanations why Taupo keeps growing both within population dimension and financially. www.mensmeds.net

There tend to be some easy steps you may follow to construct a prosperous lawn treatment business. Knowing these actions, it is essential to continuously remind yourself of these or you might find yourself declining and gradually lose clients. www.interior2han.net

Home wellness service these days offers a lot to patients who’re recuperating through surgery or even illness. These providers offered permit patients to recuperate comfortably within their own house, without worrying families that can’t provide a lot of those services. www.fatburningfurnacereport.org

In the following paragraphs I clarify research We conducted to be able to understand the reason why people believe healthy food choices is much more expensive compared to unhealthy meals. I examine the information I carried out and form a viewpoint in line with the results through my test. www.insurancehealthquote.org

Narrowly described, “moderate alcoholic beverages consumption” reduces health danger from coronary disease. But this raises danger for numerous cancers, hepatitis, brittle bones, immune reductions, accidents, as well as suicide. They do not tell a person that component. www.intuitionandhealing.org

The second 1 / 2 of getting ill is fussing using the health insurance coverage claims procedure. I possess some tips in order to pass-on from my very own health statements experiences that may help a person navigate this particular maze. www.dailyhealthtrends.net

Changes within the health treatment environment tend to be here. Because physician frontrunners, are you prepared to build powerful, flexible healthcare teams which will thrive? Three methods are presented in the following paragraphs that may position you as well as your team to achieve success. www.salesmanagementsystems.net

Many home owners are getting rid of high requirements grass using their yards and choosing a enjoyable garden rather. If you aren’t ready to stop on your own green carpeting, here tend to be some strategies for a a proper green, bud free yard. www.cheapclothesshoppingonline.net

Generalservices

Posted: 16 March 2016 02:24 PM

[ Ignore ] [ # 10 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

Education with regard to Sustainable Improvement (ESD) is really a rather brand new field associated with education. We are able to see it being an innovative type of future training for colleges linking the actual child’s development using the future problems of culture. www.ehealthplanners.net

There really are a small number of young Americans who’re already benefiting from education abroad. They obtain credits towards their degree at the cheaper cost, get to see the existence overseas as well as travel close to Asia. www.heavenhomes.org

Discover here the reason why large companies, small types and professionals each one is benefiting from cloud company computing for a number of reasons. The impair system is actually described as well as predictions are created for it’s future growth. www.gloucesterplumbing.net

Law isn’t the starting place, nor could it be the closing place with regards to ethics. Our ethical ideas frequently shape the smoothness of the laws. Criticism associated with laws as well as proposals with regard to change within our laws in many cases are based on the shared ethical ideal that isn’t being accomplished. www.weightlossshakeshq.org

This article is actually directed from men however ladies if you wish to comment after that feel free of charge! Understand exactly what it will be a guy! I’m not referring to what a person learned out of your dad or your pals, what I am talking about is the way you as a guy project you to ultimately others. www.prolawteam.net

This past year the UNITED KINGDOM winter caused a large freeze, it was unusual, as the majority of winters in the united kingdom are given by moderate westerly winds in the Atlantic sea, but the wintertime of 2009/2010 was to become much not the same as normal. www.clubsimenahotel.net

Fashion includes a rippling impact on people through different strolls of existence. It progressively gains recognition among various strata associated with society. To assist fashion get in touch with the customers, a many professionals are essential. www.certifiedhealthcoach.net

Shopping with regard to camping chairs might seem simple sufficient, but knowing the very best places to appear can help you find the main one you would like while helping you save both period and cash. Hint: don’t begin your search in the supermarket! www.marketyourpractice.net

Food shopping on the budget? Whenever you enter any kind of supermarket, you’ll fall target to advertising. Marketing is merely the nicely researched technology of why is people purchase. Don’t fall under the advertising trap. Here tend to be 14 wise tips that will help you save 10—15% of the money whenever food buying… www.beautyandstyling.net

The Pappa Rocking Seaside Chair is in which the beach fulfills the patio! So in case your buying beach seat that rubble likes the porch chair and it is light, transportable and retracts flat, then this particular beach chair may be the one for you personally! www.safeguardfinancialusa.com

Generalservices

Posted: 16 March 2016 02:24 PM

[ Ignore ] [ # 11 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

If you’re like me personally and adore designer clothing at low cost prices then you will discover this post useful. It is filled with tips on where you can shop in Nyc and ways to get Designer labeling at discount prices, in addition to on ways to get around the town. www.firstcallservices.org

Shopping with regard to gifts never been simpler. This is particularly true should you go online to look for your own gifts. Thousands and thousands of merchants can sell really stunning, quality presents at sensible prices and you will shop in the comfort of the living space and laptop computer! www.veghealth.net

Everyone is actually buzzing about being enviromentally friendly these days and also the home is among the best places to begin. Since all of us spend a lot time inside our homes, it just is sensible to begin being much more eco-friendly presently there. Here, we’ll take a look at five ways your family can start being enviromentally friendly with a good eco house. www.seniorsinsuranceadvice.net

So what is it prefer to own a part of a home inside a corner from the largest wine-producing area in Portugal? Imagine a number of your best wines as well as similar descriptions enter into play: background, depth, intricacy, uniqueness associated with character. How sweet it’s. www.shopfisher.net

If you had been to purchase a property about the beachfront, wouldn’t you select the location that combines an increasing role about the international picture with obtainable prices? Should you said indeed, consider a house or condominium in Cancun. www.lyfood.net

Since we’ve protected IT equipment, computer protection and catastrophe recovery (make sure you see components 1 & 2 of the series), let’s consider the vast variety of other equipment you’ll want to make points happen rapidly and efficiently in your house business. www.betaeducation.net

Luxury property in Victoria, B . C ., is truly a few of the very best on the planet. Macleans journal selected Victoria as the main city within Canada by which to relocate a household business. Conde Nast Tourist magazine voted Victoria the very best over-all town in Canada because of its environment as well as ambiance. www.childrensoralhealthohio.net

Dramatic changes took place within the housing market in Marbella. In truth over the entire Costa delete Sol. Property here happens to be in need by North Europeans since the climate is extremely warm as well as sunny the majority of the year and the price of living may be historically reduced. www.myattorneyatlaw.net

Buyers as well as sellers, please make sure to entrust this particular transaction to some full time property professional along with experience along with a track report. Someone who’s capable of give a person expert suggestions about homes within Summit NEW JERSEY or your own market region. www.buddha-yoga.net

During yesteryear week, the globe has witnessed a number of events which influenced the different sectors from the growing worldwide economy. Similarly, some occasions promised much more developments as well as expansion not just in the actual international trade but additionally in particular countries. www.ansaricapital.com