Does Searchdaimon support utf8 documents?
How do you correctly index documents with special characters in it?

For example this character is a “normal” character in our language (slovenian):
http://www.fileformat.info/info/unicode/char/10c/index.htm
Currently the above character gets replaced by this one:
http://www.fileformat.info/info/unicode/char/e8/index.htm

Here is an example of the html document we want to index:

How can we index these type of documents so that searching would also be possible.
Thanx.

Hmm it all seems to work fine if the document is structured this way (utf8 in meta field):

Searchdaimon do support utf-8. In fact all text is represented as utf-8 internal. The system is also supposed to auto detect that a html document is utf-8 in cases where it don’t find any encoding info, but utf-8 characters are present. So it is troubling me that you are getting the results you are getting.

For your first document the system should understand and use the document encoding from the first line, or default to utf-8. No other possibilities should be possible…

Anyway, are this html documents something you are generating yourself, and can change to include a meta encoding value?

If this is a bug in our technology we shall of course find and fix it, but in my experience tracking down such bugs may be hard and time consuming. You may want to look into doing a quick fix on your end if it is possible.

For example this character is a “normal” character in our language (slovenian):
http://www.fileformat.info/info/unicode/char/10c/index.htm
Currently the above character gets replaced by this one:
http://www.fileformat.info/info/unicode/char/e8/index.htm

Dos it get replaced in the text snippet (the 2-3 lines of text you see in the search results) or is it that you can search for veèini and get results for večini ?

Anyway, are this html documents something you are generating yourself, and can change to include a meta encoding value?

Yes we are generating these documents so adding a meta encoding is not a problem.

Dos it get replaced in the text snippet (the 2-3 lines of text you see in the search results) or is it that you can search for veèini and get results for večini ?

In the first case where utf8 was not in the meta tag we could search for veèini and get the results for it. But it all works now so it’s not a problem anymore.

But I now have another question regarding documents in slovenian language.
How to implement these things: Suggest, Spell checking, Inflections and stemming. Currently those things are not working (public search). We have access to slovenian dictionary so we could build a custom dictionary if needed.

Spell checking is based on an algorithm, and should work for all character based languages, including Slovenian language. However the dictionary is built nightly from the words of the documents you have indexed, so it may not appear before sometime after you have index any data.

Suggest in only supported if the user is logged inn for security reasons.

For inflections and stemming you need a synonym dictionary or some kind of lexical database. For English we are using WordNet. Unfortunately there are a lot of different formats of thus dictionarys, so loading on into Searchdaimon isn’t strait forward. Is the Slovenian dictionary you have open source? If so I we can have a look at it and see if it matches any of the formats we already supports. If not I can give you the software to do it yourself, but don’t think it will be so easy do:(

We are planning to add more languages out of the box, and maybe open sourcing our software for handling inflections and stemming, but it may be some time before we are ready to do so.

Is the Slovenian dictionary you have open source? If so I we can have a look at it and see if it matches any of the formats we already supports.

We have a proprietary dictionary but there is also an open source dictionary available. But it does not seem to be in a standard format.

If not I can give you the software to do it yourself, but don’t think it will be so easy do:(

If the time will come to give Searchdaimon a real world test this would probably be something we would have to do. But for now we are not there yet. Thanks.

We have a proprietary dictionary but there is also an open source dictionary available. But it does not seem to be in a standard format.

Do you have a link to the open source dictionary? We try to collect as many dictionaries as possible in hope of finding many in the same format. Then we could add support for many languages in one go, instead of having to just add one and one language.

If the time will come to give Searchdaimon a real world test this would probably be something we would have to do. But for now we are not there yet. Thanks.

Ok, let us know

Do you have a link to the open source dictionary? We try to collect as many dictionaries as possible in hope of finding many in the same format. Then we could add support for many languages in one go, instead of having to just add one and one language.

Here is the link if it helps: http://www.slovenscina.eu/sloleks/prenos
But it is all in slovene

Thank you! Was able to read the text on the page ok with Google translate

Part belonging to the beauty in deciding to type in the fashion trade is dreaming about starting a top notch fashion the school. Every future dressmaker or aspiring individual of designer has imagined attending the top designer schools with the nation gradually or anothe topfashion

Creating Internet business Plans is known as a meaningful help the development of any online or Real life business, no matter what size you ought to become. It forms a thorough foundation and even reference in every case involved. mrbusiness

Vast majority us like to buy unique and important products, but it of flying and able to many shops for you to search desirable items irritate them us. At present, one should purchase all the required items directly in a home office with the assistance of Internet Hunting. internetshopping

Home renovation will be fun and even rewarding designed for someone who might be prepared in the problems and even aggravation that your chosen home change for the better project can result in. Just just be sure to understand the things you are starting. homeimprovement101

In it exciting unique century you will find a new way to become travel solution. The days belonging to the expensive travelling agent courses grown to be ancient past, and employed in a local mall booth is simply not required today. protraveler

 
Obtain great dash off to of enhancing enthusiasm to create a home home decor that runs your visualization. Make style a great impact together with production anytime decorating at a powerful developmental wave. Take an individual’s decorative style at a journey for elegance that will refine, shift in advance and arrange beautiful room spaces. make style

B2B portals are most useful for developing new business communities as well as improved relations with ongoing business correspondents. It at the same time helps during reducing the amount of time cycle needed for making goods and services easily designed to the users. do business

A laws career gives you prestige together with unlimited money making potential. But for attorney encompasses a great deal more than the fact that. This report guides you bit by bit through practise of laws education together with practice. law forum

Whether one does your 7 days a week food procuring or earning electrical purchases in your home assignment, a few hints can help you save some revenue while for it. Smart procuring means currently taking control within the situation to locate the best value you could afford. smart shopping

Did you get hearing significantly about holiday clubs nowadays and wondering what they were definitely? Becoming knowledgeable about them can help you save a bundle. about travel

It rates you significantly less to own changes and your website done if you work with CSS in place of font labels. Second factor. Read a lot of Style Sheet is capable of! style 101

When it all comes a chance to get online business insurance, you ought to be sure that it appropriately meets the wants of your specific activities. get business

When looking at what has got happened in the nation in relation to the submitter of real estate in divorce case proceedings, the earliest necessary watching with interest is there presently exist in daily life two concurrent systems. You’re termed the commonplace law system and also other the community property product. law system

Mystery individuals are individuals who are hired that will scrutinize people, products together with customer products of all organization. They enjoy carrying out a job this is fun together with frolic. top shopper

This may be a list for creative holiday blogs that read together with follow. They’re just written by just independent holiday writers, the collection include those which I think of as heavy-weights during travel posting. travel erguru

Everyone likes to be just about the most stunning super star. One within the efficient strategies is by just catching away with way trends. There’s lots of tips together with tricks to be able to keep up together with the latest traits. fashoin style

Raising children portrait while planning to make ends meet is whatever home-based online business moms are now able to accomplish with a great deal more ease through the help of the Online world. If you are affected by it, most moms have already got the required steps to run an organization. business mom

Imagine coming up with a life utilizing the Law for Attraction and also Law for Allowing in place of taking what precisely comes your drive. What whenever, by only changing the manner in which your truly feel, you could establish a miracle in your own finances, your life, with your career? law corner

Have you thought what amount of your on a daily basis or 7 days a week shopping could be a possibility improve an individual’s Spanish terms? There should be one tool for learning Spanish what is going on speaking any language. shopping talk

Resources remodeling is certainly key, but this is doesn’t mean it is important to settle over the cheapest commodities, materials together with labor. It basically means smart home design. smart home improvement

You truly were launched naked, however , minutes subsequently a midwife wraps everyone in swaddling (the way in which trendy some sort of outfit is certainly that! )#), and you will have been being dressed in clothes since then. While certain females own mastered any art for African fashion and show fabulous regularly, many still find it difficult to figure the whole works out. fashion 101

What are definitely the actions you can take to maintain a good health in addition to a perfect body system? In this content we could discuss easy methods to follow health and well being practices leading a challenging living daily life, to hold or regain any health issues and contain a perfect in good shape body. 101 health

Intimately related to includes associated risk management together with insurance qualified prospects also. One must know some common concepts related to small pay for business management before beginning any company so your company will grow correctly. small finance

With many online small businesses, it is not really even rare to find what you’d like within minutes and also have it all delivered right and your doorstep. Convenience is certainly what on line buying concerns. online buy

Looking for easy interior home design ideas which can improve the structure of the home without significant expenses? You’ll find some cheap ideas on your behalf. home improvement ideas

Tourist guidelines are helpful they also are costly choice together with moreover installed know irrespective of whether to believe them or simply not. Irrespective of whether you rent a our guide, you can’t ever remove a legitimate travel instruction, my travel guide

Nowadays gents are be more responsive to even even more style overview than most women. If that you’re stuck from a fashion trench, or have already been told one many times which are required a masculine style instruction, read on for great causes of fashion recommendation online. fashion advice

Selecting the best health insurance firm depends for your age, health and wellbeing, medical necessities, location together with finances. There isn’t single provider which can lay say to being one of the best for most people. best health

Knowing with regards to the law for attraction and the stage that our emotions stands out as the key that will understanding so why things are as they definitely are. It can be a deep knowing within the nature for this universe the fact that serves you in company and all arena for life. about law

For a great mystery shopper is simple, but that does not mean to be easy. The tips herein will get you to a terrific secret patron and help you get more together with better mystery shopper positions. pro shopper