not finding files via SMB

Posted: 04 December 2013 03:45 PM

[ Ignore ]

Newbie

Total Posts: 8

Joined 2013-11-25

The vmware search image has now index something like 6 millions files (yes it took a few days). the total number is around 8 million, but I guess the missing 2 million are not index-able.

Now if I search those files for the company name as my normal user, I get no results. If I use my domain login (the same as account used to search). I get 82 matches, but only 3 results “79 of the expected results were filtered out by the system. This can happen if the resource has been moved, deleted or the rights have been change after the most recent indexing.” which I don’t believe at all. I expect the majority of the word documents we have (196721 at last count) to contain our company name.

Current initial testing looks like it is not working, what could be wrong?

Runar Buvik

Posted: 05 December 2013 06:40 PM

[ Ignore ] [ # 1 ]

Administrator

Total Posts: 253

Joined 2010-09-13

It may be that you having come across an issue with the real time security access system.

For some crawlers, like smb, the ES will ask the source system for access rights for every file during search. This is done since the access rights of a file can have changes since the ES crawled it. If the end-user doesn’t have access anymore we don’t want to show him the file either.

While an important part of security and allows for removing results from the search system in real time, this has been known for casing some errors on rear occasions.

Can you please try the following 3 things:

* Have you supply the username used for crawling in the format “domain\user” ? For example we have a domain called sdtest.local here. My user is “runarb”. Some file servers accept only runarb as username, some needs sdtest\runarb. If you are using the “domain\user” format, please go to “Manage” on the collection in questioning and check that the “user prefix” is someting like “domain\”. Then do a new search and see what happens.

* Temporarily change the security model used for each collection to something other than system permissions. To do so, go to the “Manage” function for a collection, and open the “Access level“ tab. Then use something like “User” to allow access for all logged in users. Then do a new search and see what happens.

* There is also a way to see why documents are being filtered out. If you add &debug=showxml to the url in the address bar you will view the raw xml (if there is a “#” sign at the end, remove it first). Find the text between the “<searchnode>” and “</searchnode>” tags and post it her for interpretation.

Example:

Url: http://demo.searchdaimon.com/webclient2/indexanon.pl?query=test&lang=en&debug=showxml

Searchnode profile:

<searchnode>
<nodename>localhost</nodename>
<totaltime>0.059973</totaltime>
<hits>24</hits>
<filtered>0</filtered>
<shown>10</shown>
<time_profile>
<AuthorSearch>0.000000</AuthorSearch>
<UrlSearch>0.000000</UrlSearch>
<MainSearch>0.001999</MainSearch>
<MainAuthorMerge>0.000000</MainAuthorMerge>
<popRank>0.000000</popRank>
<responseShortening>0.000000</responseShortening>
<allrankCalc>0.000000</allrankCalc>
<indexSort>0.000000</indexSort>
<searchSimple>0.000000</searchSimple>
<popResult>0.042981</popResult>
<adultcalc>0.000000</adultcalc>
<filetypes>0.000000</filetypes>
<iintegerGetValueDate>0.000000</iintegerGetValueDate>
<dateview>0.000000</dateview>
<FilterCount>0.000000</FilterCount>
<pathaccess>0.000000</pathaccess>
<urlrewrite>0.095958</urlrewrite>
<getUserObjekt>0.000000</getUserObjekt>
<cmc_connect>0.003998</cmc_connect>
<html_parser_run>0.000000</html_parser_run>
<generate_snippet>0.000000</generate_snippet>
</time_profile>
<filters>
<filterAdultWeight_bool>0</filterAdultWeight_bool>
<filterAdultWeight_value>0</filterAdultWeight_value>
<filterSameCrc32_1>0</filterSameCrc32_1>
<filterSameUrl>0</filterSameUrl>
<filterNoUrl>0</filterNoUrl>
<find_domain_no_subname>0</find_domain_no_subname>
<filterSameDomain>0</filterSameDomain>
<filterTLDs>0</filterTLDs>
<filterResponse>0</filterResponse>
<cantpopResult>0</cantpopResult>
<cmc_pathaccess>0</cmc_pathaccess>
<filterSameCrc32_2>0</filterSameCrc32_2>
</filters>
</searchnode> 

jmcentee

Posted: 05 December 2013 08:02 PM

[ Ignore ] [ # 2 ]

Newbie

Total Posts: 8

Joined 2013-11-25

Seems, my initial post was a little wrong. If I search using my normal account I get the 82 hits, but can only see 3, 79 filtered out. If I search as my admin account I get 476 hits but 36 filtered out. If I change the security model I get 13,220 hits, this is a believable number but this should be no different to my admin account.

* Have you supply the username used for crawling in the format “domain\user” ? For example we have a domain called sdtest.local here. My user is “runarb”. Some file servers accept only runarb as username, some needs sdtest\runarb. If you are using the “domain\user” format, please go to “Manage” on the collection in questioning and check that the “user prefix” is someting like “domain\”. Then do a new search and see what happens.

It is all on one share. I have it as DOMAIN\ but when I delete it and click save an go back in it has reappeared.

* Temporarily change the security model used for each collection to something other than system permissions. To do so, go to the “Manage” function for a collection, and open the “Access level“ tab. Then use something like “User” to allow access for all logged in users. Then do a new search and see what happens.

I get 13,220 as stated above

* There is also a way to see why documents are being filtered out. If you add &debug=showxml to the url in the address bar you will view the raw xml (if there is a “#” sign at the end, remove it first). Find the text between the “<searchnode>” and “</searchnode>” tags and post it her for interpretation.

I now get 0 results and 305 filtered files with my admin account, but the tab says 476 still.

<searchnode>
<nodename>localhost</nodename>
<totaltime>12.855716</totaltime>
<hits>476</hits>
<filtered>305</filtered><shown>0</shown><time_profile><AuthorSearch>0.000000</AuthorSearch><UrlSearch>0.000000</UrlSearch><MainSearch>0.432956</MainSearch><MainAuthorMerge>0.000000</MainAuthorMerge><popRank>0.000000</popRank><responseShortening>0.000000</responseShortening><allrankCalc>0.000000</allrankCalc><indexSort>0.009000</indexSort><searchSimple>0.000000</searchSimple><popResult>12.380764</popResult><adultcalc>0.000000</adultcalc><filetypes>0.002000</filetypes><iintegerGetValueDate>0.001000</iintegerGetValueDate><dateview>0.000000</dateview><FilterCount>0.004000</FilterCount><pathaccess>0.000000</pathaccess><urlrewrite>0.000000</urlrewrite><getUserObjekt>0.000000</getUserObjekt><cmc_connect>0.000999</cmc_connect><html_parser_run>0.000000</html_parser_run><generate_snippet>0.000000</generate_snippet></time_profile><filters><filterAdultWeight_bool>0</filterAdultWeight_bool><filterAdultWeight_value>0</filterAdultWeight_value><filterSameCrc32_1>0</filterSameCrc32_1><filterSameUrl>0</filterSameUrl><filterNoUrl>0</filterNoUrl><find_domain_no_subname>0</find_domain_no_subname><filterSameDomain>0</filterSameDomain><filterTLDs>0</filterTLDs><filterResponse>0</filterResponse><cantpopResult>0</cantpopResult><cmc_pathaccess>305</cmc_pathaccess><filterSameCrc32_2>0</filterSameCrc32_2></filters></searchnode> 

-<searchnode>

<nodename>localhost</nodename>

<totaltime>7.993204</totaltime>

<hits>476</hits>

<filtered>305</filtered>

<shown>0</shown>

-<time_profile>

<AuthorSearch>0.000000</AuthorSearch>

<UrlSearch>0.000000</UrlSearch>

<MainSearch>0.396960</MainSearch>

<MainAuthorMerge>0.000000</MainAuthorMerge>

<popRank>0.000000</popRank>

<responseShortening>0.000000</responseShortening>

<allrankCalc>0.000000</allrankCalc>

<indexSort>0.007000</indexSort>

<searchSimple>0.000000</searchSimple>

<popResult>7.554248</popResult>

<adultcalc>0.000000</adultcalc>

<filetypes>0.003000</filetypes>

<iintegerGetValueDate>0.001000</iintegerGetValueDate>

<dateview>0.000000</dateview>

<FilterCount>0.003999</FilterCount>

<pathaccess>0.000000</pathaccess>

<urlrewrite>0.000000</urlrewrite>

<getUserObjekt>0.000000</getUserObjekt>

<cmc_connect>0.002000</cmc_connect>

<html_parser_run>0.000000</html_parser_run>

<generate_snippet>0.000000</generate_snippet>

</time_profile>

-<filters>

<filterAdultWeight_bool>0</filterAdultWeight_bool>

<filterAdultWeight_value>0</filterAdultWeight_value>

<filterSameCrc32_1>0</filterSameCrc32_1>

<filterSameUrl>0</filterSameUrl>

<filterNoUrl>0</filterNoUrl>

<find_domain_no_subname>0</find_domain_no_subname>

<filterSameDomain>0</filterSameDomain>

<filterTLDs>0</filterTLDs>

<filterResponse>0</filterResponse>

<cantpopResult>0</cantpopResult>

<cmc_pathaccess>305</cmc_pathaccess>

<filterSameCrc32_2>0</filterSameCrc32_2>

</filters>

</searchnode>

Runar Buvik

Posted: 05 December 2013 11:33 PM

[ Ignore ] [ # 3 ]

Administrator

Total Posts: 253

Joined 2010-09-13

From your data I see that the filtering is done in the “cmc_pathaccess” function. It is the function that looks up user’s access rights for every file during search, and is unfortunately relying on the user prefix file do be correct.

It is all on one share. I have it as DOMAIN\ but when I delete it and click save an go back in it has reappeared.

That is odd. We use a small JavaScript function to set user prefix from the crawler user, but you are supposed to be able to override it. What web browser are you using?

jmcentee

Posted: 06 December 2013 01:12 PM

[ Ignore ] [ # 4 ]

Newbie

Total Posts: 8

Joined 2013-11-25

I have used both IE and firefox with the same problem.

jmcentee

Posted: 18 December 2013 12:59 PM

[ Ignore ] [ # 5 ]

Newbie

Total Posts: 8

Joined 2013-11-25

I am getting closer. I guess the original crawl had errors and problems, but still manage to mark itself as complete. I have remove the share from the crawl and added each sub-directory instead. This now seems to work, and gives me more results on searches and not filtered results.

Jarle Hansen

Posted: 18 December 2013 04:05 PM

[ Ignore ] [ # 6 ]

Administrator

Total Posts: 47

Joined 2011-10-19

Ok, dividing the share in subdirectories shouldn’t not be necessary, but if that make it works for you, you can of coursers do so.

Do you experience some of these symptoms on some of these new collections? Dividing the shares like that may actually be a good way of tracking down what is wrong. If the symptoms don’t appear in all the collections, you can focus on what is different with the one that fails.

jmcentee

Posted: 15 January 2014 03:36 PM

[ Ignore ] [ # 7 ]

Newbie

Total Posts: 8

Joined 2013-11-25

I have been doing a bit of further testing. one of the collections stop working. I think the cause was out of disk space, but fixing that and a rescan did not fix the problem. I had to remove the collection and re-add it and no it is working.

Is it possible to get it to list the file i thinks the permission has been change on. We now have another collection which find 7 files but 1 is filtered out, but rescanning collection does not fix it either. and I going to have to remove and re-add the collections on a regular basis? Or will running out of disk space corrupt all the collections?

Runar Buvik

Posted: 20 January 2014 07:06 PM

[ Ignore ] [ # 8 ]

Administrator

Total Posts: 253

Joined 2010-09-13

You do not need to delete the collection and add it again to force a completely new crawl. Instead simply use the “Force full crawl now” button under the “Advanced management” tab for the collection to do a complete new crawl (first do a delete of the underlying data, then crawl it again as it was a new collection).

However having to force a full recrawl should not be necessary. The ES should detect changes and handle them automatically. Frankly I don’t know why you get thus filter messages, but I will talk to the other developers here and try to come up with a strategy to debug this.

Generalservices

Posted: 16 March 2016 12:54 PM

[ Ignore ] [ # 9 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

Life after having a Chapter 7 bankruptcy isn’t as daunting as numerous would perhaps you have think. Actually, most from the negative information available is spewed through the credit card providers that miss out when you seek bankruptcy relief protection. skincarepearls

Honesty as well as integrity are important to the actual building associated with society. These types of cornerstones construct prosperity as well as peaceful interpersonal order. Dishonesty as well as greed tend to be destructive causes that disparaging offer society lower. fashionismydrug

Mongolia includes a significant confirmed area with regard to coal exploration sector. Within Mongolia the actual annual fossil fuel production is actually approximate regarding 5 zillion metric lots. Almost 85% of the production can be used for water vapor and electrical power generation. healthbeautycare

Throughout the last decade Taupo within New Zealand offers continued in order to attract brand new house building contractors and show that not every small cities are suffering. Read this short article and discover three explanations why Taupo keeps growing both within population dimension and fiscally. mensmeds

There tend to be some stages you may follow to construct a prosperous lawn treatment business. Knowing these actions, it is essential to continuously remind yourself of these or you might find yourself screwing up and gradually lose clients. interior2han

Home wellness service these days offers a lot to patients who’re recuperating through surgery or even illness. These providers offered permit patients to recuperate comfortably within their own house, without troubled families that can’t provide a lot of those services. fatburningfurnacereport

In the following paragraphs I clarify research We conducted to be able to understand the reason why people believe appropriate food choices is now more expensive compared to unhealthy meals. I examine the information I carried out and form a feeling in line with the results through my test. insurancehealthquote

Narrowly described, “moderate alcoholic beverages consumption” decreases health danger from heart disease. But this raises danger for numerous cancers, hepatitis, weakening of bones, immune reductions, accidents, as well as suicide. They do not tell a person that component. intuitionandhealing

The second 1 / 2 of getting ill is fussing using the health insurance coverage claims procedure. I possess some tips in order to pass-on from my very own health statements experiences that may help a person navigate this particular maze. dailyhealthtrends

Changes within the health treatment environment tend to be here. Because physician frontrunners, are you prepared to build powerful, flexible healthcare teams which will thrive? Three methods are presented in the following paragraphs that may position you as well as your team achievement. salesmanagementsystems

Many home owners are getting rid of high requirements grass using their yards and choosing a enjoyable garden rather. If you just aren’t ready to stop on your own green carpeting, here tend to be some strategies for a a proper green, pot free yard. cheapclothesshoppingonline

Generalservices

Posted: 16 March 2016 12:54 PM

[ Ignore ] [ # 10 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

Education with regard to Sustainable Improvement (ESD) is really a rather brand new field associated with education. We are able to see it being an innovative type of future training for colleges linking the actual child’s development using the future problems of culture. ehealthplanners

There really are a small number of young Americans who’re already benefiting from education abroad. They obtain credits towards their degree at the cheaper cost, get to see the existence overseas as well as travel close to Asia. heavenhomes

Discover here the reason why large companies, small types and professionals each one is benefiting from cloud company computing for a number of reasons. The foriegn system is actually described as well as predictions are created for it’s future growth. gloucesterplumbing

Law isn’t the starting place, nor could it be the closing place with regards to ethics. Our meaningful ideas frequently shape the type of the laws. Criticism associated with laws as well as proposals with regard to change within our laws in many cases are based on the shared meaningful ideal which is not being accomplished. weightlossshakeshq

This article is actually directed from men however ladies if you wish to comment after that feel free of charge! Understand exactly what it will be a guy! I’m not referring to what a person learned out of your dad or your pals, what After all is the way you as a guy project you to ultimately others. prolawteam

This past year the UNITED KINGDOM winter created a large freeze, i thought this was unusual, as the majority of winters in the united kingdom are provided by moderate westerly winds in the Atlantic marine, but the cold winter months of 2009/2010 was to become much distinctive from normal. clubsimenahotel

Fashion includes a rippling relation to people through different strolls of existence. It progressively gains recognition among various strata associated with society. To assist fashion find the customers, a many professionals are expected. certifiedhealthcoach

Shopping with regard to camping chairs might seem simple sufficient, but knowing the very best places to appear can help you find the main one you would like while helping you save both period and cash. Hint: don’t begin your search in the supermarket! marketyourpractice

Food shopping on the budget? Whenever you enter any kind of supermarket, you’ll fall target to advertising. Marketing is merely the nicely researched technology of why is people purchase. Don’t fall under the advertising trap. Here tend to be 14 wise tips that will help you save 10 : 15% of the money whenever food buying… beautyandstyling

The Pappa Rocking Seaside Chair is in which the beach fulfills the outdoor patio! So in case your searching for a beach seat that rubble likes the porch chair and it is light, transportable and folds over flat, then this particular beach chair may be the one for you personally! safeguardfinancialusa

Generalservices

Posted: 16 March 2016 12:54 PM

[ Ignore ] [ # 11 ]

Sr. Member

Total Posts: 805

Joined 2016-03-16

If you’re like me personally and adore designer clothing at low cost prices then you will discover this post useful. It is filled with tips on where you can shop in Nyc and ways to get Designer labeling at discount prices, in addition to on ways to get around the town. firstcallservices

Shopping with regard to gifts has not been simpler. This is specially true should you go online to surf for your own gifts. Thousands and thousands of merchants can advertise really stunning, quality presents at sensible prices and you will shop in the comfort of the living space and laptop computer! veghealth

Everyone is actually buzzing about being enviromentally friendly these days and also the home is among the best places to begin. Since all of us spend a lot time inside our homes, it just is sensible to begin being much more eco-friendly presently there. Here, we’ll take a look at five ways your family can start being enviromentally friendly with a good eco house. seniorsinsuranceadvice

So what is it prefer to own a part of a home inside a corner from the largest wine-producing area in People from france? Imagine a number of your best wines as well as similar descriptions enter into play: background, depth, difficulty, uniqueness associated with character. How sweet it’s. shopfisher

If you are to purchase a property about the beachfront, wouldn’t you select the location that combines a developing role about the international picture with obtainable prices? Should you said indeed, consider a house or property in Cancun. lyfood

Since we’ve protected IT equipment, computer protection and catastrophe recovery (make sure you see components 1 & 2 of the series), let’s consider the vast choice of other equipment you’ll want to make points happen rapidly and efficiently in your house business. betaeducation

Luxury property in Victoria, B . C ., is truly a few of the very best on the planet. Macleans newspaper selected Victoria as the main city within Canada by which to relocate a household business. Conde Nast Criminal magazine voted Victoria the very best over-all town in Canada because of its environment as well as ambiance. childrensoralhealthohio

Dramatic changes took place within the market in Marbella. In truth over all the islands Costa delete Sol. Property here happens to be in need by N . Europeans since the climate is extremely warm as well as sunny the majority of the year and the price of living may be historically reduced. myattorneyatlaw

Buyers as well as sellers, please make sure to entrust this particular transaction to some full time property professional along with experience along with a track report. Someone who’s capable of give a person expert suggestions about homes within Summit NJ-NEW JERSEY or your own market region. buddha-yoga

During yesteryear week, the globe has witnessed a number of events which influenced the different sectors from the growing worldwide economy. Similarly, some occasions promised much more developments as well as expansion not just in the actual international trade but additionally in particular countries. ansaricapital