Hello.

New to SearchDaimon (but not to enterprise search - long time user of Copernic/Coveo ES).  Congratulations on a great product.

So I set things up, added some collections, did some test crawls & then full crawls. I’m seeing some results in the search, but not as much as I’d expect. (For example, when I look for data in files that should have been crawled, it’s not there for some, but is there for others).  It’s almost as if the indexing has stopped.  Can anyone explain the flow of how things work?  Does the indexing run in parallel to the crawl? Does it only begin on a whole collection after a successful crawl complete?  I’ve looked at the log in Statistics & Logs->log files->document indexing log, but can’t make much of it (there are no timestamps so I can’t tell if it’s doing anything, and the entries don’t describe the indexing and the target).

Is there a way to tell if the collection has been fully indexed, or if it’s in process, or if it failed or is stuck?  An operations dashboard with this information (similar to Coveo) would be a great addition.

Any thoughts?

Thanks.

To prevent fragmentation of the index (something similar to file system fragmentation) the ES first crawls the whole collection, then index it.

The collection is finished when it gets a green chin at the overview section like this: http://www.searchdaimon.com/documentation/overview/

You can then select manage for each collection to get more information. For example a raw dump of documents are available under the “Documents” tab.

To hunt down issues with documents not being correctly indexed I normal start by searching for the file name of the document, to see if it was indexed at all. If the document was not indexed you should try indexing just a small portion of documents. For example the folder the document persist in, to verify that the ES has access ( no robots.txt blocking on websites or lack of rights on a smb file share).

If you can find the document when searching for it, try to open the cached version from the search results to verify that the document was converted into text correctly.

Specific question on an indexing issue.  I see the file in question listed in the bucket as having been indexed. However, it’s not showing up in search (even by filename).  It’s a simple MS Word doc.

Any troubleshooting tips?

Thanks.

Insufficient rights are a frequent reason that files don’t show up in the search results, so that is the first I would check for.

What data source have you crawled?

Have you connected your ES to Microsoft Active Directory ?

Is so you should check that your user has sufficient rights to read the file. You can list witch groups a user is a members of from the “end user” page in the admin pannel. Then check that one of thus groups are listed in the “Acl allow” for this file.

I do not have it connected to AD.  I am using the public search and have the collection access level set to anonymous.

I can access the file from the client that I’m running the search from, and the file does show up on SD has having been indexed (in the bucket).

Any other suggestions? 

As a side note - when not using AD, I assumed permissions were irrelevant. ie: If a file is indexed, it’s going to show up in the public search results regardless of whether the requester has permission.  Only when they click the file will they know that they don’t have permission to that resource.  Correct?

Thanks for the help.

You are correct; when you haven’t connected the ES to an Active Directory access permissions are irrelevant. I only brought it up because access permissions are a very common source for missing files in the search results.

From your descriptions it isn’t obvious for me what may be going on her, but some steps you can consider is:

* You can try recrawling the collection and see if the error persist. If the file is available in the search result after the recrawl this may be caused by a bug in the ES software. There is a function called “force full recrawl” in the setting for each collection.

* Dos the file has any properties that sets it apart from the other files you can search? Is it unusually large? Does it have a special file name with letters like ~@/[] in it or Unicode characters like øæå in its file name? Is it unusually old? Etc. The ES should handle all of these cases, but you never know…

I tried the first step that you suggested - “force full crawl” on the collection.  The document now appears in search!  Is there anything by way of logs that I could provide to you to help track down the issue with this file?  It makes me wonder how many other documents are hidden from search results.  Would it be a good practice to do a “force full crawl” every so often?  Maybe this should be an option in settings to schedule.  However, I can see how this would be a big deal on a big collection ( could take days).

To answer your other question, it was a Word 2003 file from 2012, with nothing strange in the filename.

Thanks for the help and let me know what I can do to help identify the bug.

Unfortunately it sounds like you may have come across some kind of bug.

Thank you for offering to help us look into this. Unfortunately the only feasible way to debug this would be to give Searchdaimon access to the server so we could do a forensic investigation, but that it too late now because the index gets written over when one dos a “force full crawl”.

Generally this shouldn’t happened and you should not have to do a full recrawl regularly, so there is no option to schedule full recrawls.

So the mystery now continues…

As mentioned above, the file now shows in search.  Before I forced the full crawl, I wanted to cover your second point - so I copied the file to the same directory & renamed it “test file.doc”.  When I did the full crawl, both the original & the test file showed in the bucket and in the search results.

I came back here and reported success.

However - I then went back and deleted “test file.doc” from the directory, and selected “crawl now” from admin.  After the crawl, the file count did not go down by 1, and the test file still shows up in the search results and even the bucket - even though it is deleted from the target system.

I waited 24 hours for a “normal, scheduled” crawl to occur, which it did this morning, but the test file is still there.

Any thoughts?

Thanks.

That a file is available for some time after it has been deleted from the source system is perfectly normal. You have stumble across one of the side effects of garbage collection ( http://en.wikipedia.org/wiki/Garbage_collection ).

Most systems the ES index has no way of alerting us that a file is gone, instead it just disappears. We handle this by routinely go thru all files/webpages/objects and see if some is missing. If it is gone several days in a row, the file is then deleted permanently from the ES also.

We used to delete files at the first crawl previously, but we have found that that is unwise. Many source systems we index may suffer from partially temporary errors from time to time. (Php scripts fails, nfs links stalls, hard disks fails, web servers goes down etc.). If we then delete thus files that is affected we must then recrawl them when the system works again. That may then put to much load on the system when the ES and other systems try to catch up.

That makes sense.  Thanks for explaining.

Everyone likes to be essentially the most stunning legend. One in the efficient approaches is by simply catching way up with manner trends. There are several tips along with tricks that you can keep up because of the latest tendencies. fashoinstyle

Raising children while looking to make ends meet is something home-based organization moms can now accomplish with additional ease through the assistance of the Net. If you approach it, most moms currently have what can be done to run an enterprise. businessmom

Imagine planning a life with all the Law involving Attraction plus the Law involving Allowing as an alternative to taking precisely what comes on your path. What in case, by just changing how your experience, you could build a miracle as part of your finances, your own life, plus your career? lawcorner

As well as thought the amount your day-to-day or once a week shopping could seriously help improve your current Spanish words? There is merely one technique of learning Spanish that is certainly speaking your language. shoppingtalk

Finances remodeling can be key, but that is certainly doesn’t mean you will need to settle for the cheapest merchandise, materials along with labor. It only means smart diy. smarthomeimprovement

You in fact were given birth to naked, nevertheless minutes after a midwife wraps anyone in swaddling (precisely how trendy the outfit can be that! )#), and you could have been donning clothes since that time. While a number of females get mastered your art involving African fashion and search fabulous continuously, many still fight to figure all this out. fashion101

What include the actions to maintain a terrific health and also a perfect system? In this information we will certainly discuss the best way to follow a sound body practices leading a good living lifestyle, to keep or regain your well being and have a very perfect in shape body. 101health

This too includes threat management along with insurance leads also. One has to know some standard concepts with regards to small financing business management before commencing any company in order that the company could grow properly. smallfinance

With countless online corporations, it is just not even difficult to acquire what you desire within minutes and have the idea delivered right for your doorstep. Convenience can be what on-line buying is centered on. onlinebuy

Looking for a lot of easy interior diy ideas which could improve the style in your home without enormous expenses? We’ve got some economical ideas in your case. homeimprovementideas

Tourist books are helpful nonetheless they are a high priced choice along with moreover due to know no matter whether to have confidence in them as well as not. Regardless of whether you use a man guide, you can not reduce a true travel guidebook, mytravelguide

Nowadays adult men are be subject to even additional style critique than girls. If you happen to be stuck in a very fashion ditch, or are actually told one lots of times which you are required a guy style guidebook, read on for a lot of great options for fashion suggestions online. fashionadvice

Selecting the best health insurance agency depends on the age, wellbeing, medical requires, location along with finances. There isn’t a single provider which could lay assert to being the top for anyone. besthealth

Knowing regarding the law involving attraction and the intention of our emotions will be the key for you to understanding precisely why things are when they are. It is just a deep knowing in the nature on this universe that will serves you operational and just about any arena involving life. aboutlaw

As a great magic formula shopper is simple, but it doesn’t mean that it must be easy. The tips on this page will cause you to be a wonderful secret customer and help you to get more along with better magic formula shopper work opportunities. proshopper

There are generally literally a huge number of possible doggy house concepts waiting that you can make them your. Whether it’s actually a complex developer dog kennel, or a fairly easy traditional doggy house, the chances are countless. hosueideas

This is a directory of creative take a trip blogs i read along with follow. These are written by simply independent take a trip writers, the listing include people who I look at as heavy-weights throughout travel writing a blog. mrtravel

Cultural impact on are stronger if we consider your economic element of the manner industry. Business requires market where it might present fresh new ideas along with products divorce lawyers atlanta new morning. fashionhosue

Well, I’m here to see you that this secret to having healthy meals accessible anytime and whenever is arranging and having appropriate food choices staples. healthiness

There is something no one particular else is hinting about legislation of Destination and it’s about time you seen it. It can be what My spouse and i call your WOW component. It is with all the law involving attraction via inside out and about. insidelaw

Not merely does a reduced interest charge mean lower monthly bills but it will also save anyone substantial money in the life of your respective mortgage. However seeking the best rates might be difficult if you do not have a quality mortgage charge shopping suggestions. shoppingadvice

Everyday is often a day with the beach! ” because old expressing goes, even though you know we are unable to really spend daily at your beach. On the other hand, you could experience daily at your beach using beach property decor. housedecor

On account of it’s prosperous heritage, hot friendly men and women and unspoiled mother nature, Cambodia is often a fabulous desired destination for high-class tours. luxurytour

A new fashion symbol is a person whose outfits, hair, or all round appeal have been imitated by simply everyone and possesses survived ages after. fashionicon

That is certainly what counts essentially the most. Once anyone develop this specific healthy behavior and keep it going consistently, planning out and about your balanced day the night time before can become your ritual the ones pounds will certainly really start off falling off one’s body. healthyday

Why you may need professional legislations advice regarding accident injury is often a query of countless across the globe. This question is just not limited for you to any place or land as injuries occur everywhere and the wonderful would like to know about legal representatives. lawadvice

The emergence in the Internet as being a booming market outlines the importance for on-line shopping websites. All customers resort on the Internet when you should pick way up something brand-new. shoppingblog

When it relates to college existing, the non-profit student homes development option can often be overlooked. On the other hand, there are lots of benefits to planning this option, starting while using development’s core focus: your young higher education scholar. housedevelopment

There are many of stuff make the holiday just that will bit greater when choosing such a travel. Here are several of the rewards that My spouse and i find create a wonderful small class tour. grouptour

Personal Bags are actually around for a huge number of years. Bags stand for a mixture of technological along with social changes and they also express national advancements throughout fashionable terminology. fashionkeeper