No I’ve got my search engine prototype completed i need to invest in some good hardware. Can anyone recommend any servers with high storage (above 600gb), but also have a powerful cpu? Or do you think i’m better of going for lots of smaller servers, like google?

I’ve currently got 200thousand files in my database (but it’s a file search engine so each file can be as big as 600mb, most are about 1-2mb), I hope to expand to about 10x this so i’m going to need huge storage.

If you continue to expand, you will have to use many small servers sooner or later. Using one big server will not scale economically. Because you don’t get a doubling in performance when you double the price.

The problem is that on order to use many computers in parallel, instead of one, you will have to create a system for them to work together.

For example to search, each server could hold a portion of all documents, and an index for them. You can then make a meta search engine that search all your servers, and then combine the results.

To create such a system takes time, sow many end up using one powerful server to get the first version up and running. Then changes it to use many small servers later.

At Boitho we uses Super Micro 5014C-MT 1u server to hold data and searching. See http://www.searchdaimon.com/forum/viewthread/10/ . The servers has Intel P4 or Intel dual cure possessors.

If you need a cheap server with a lot of possessor power, tak a look one the Dell PowerEdge SC1425. A setup with two Xeon processors is relatively cheap. Unfortunately it only has space for two harddrives.

Thanks for advice. I’m not sure which route to go down, whether I have a few individual servers with a crawler, indexer and query servers on each one and then one server to handle all outside requests that will query all the servers at once (meta search type thing). If i went with this option i would go for the PowerEdge SC1425.

Or i have one big storage server and a seperate crawler/indexer and two or more query servers.

If i go for the 2nd option i was looking at these: http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=5877262661&ssPageName=MERC_VI_RSCC_Pr4_PcY_BIN_Stores_IT for the query and indexers and maybe something like this: http://cgi.ebay.com/4-Terabyte-RAID-Storage-Server-64-Bit-3U-Rack_W0QQitemZ5878663070QQcategoryZ64072QQrdZ1QQcmdZViewItem  for the storage server.

What do you think?

Search speed can be a problem using one big server. If you for example use 5 small servers, you have 5 cpus working on the problem in parallel. Each working on 1/5 of the problem.

If you uses on big server you either have to use only one processor at a time. Even if you get the most expensive one, it won’t be as powerful as 5 cheap Intel p4’s working in parallel.

Or you can go for a server with more then one possessor, and use treads or fork to work in parallel, but then you have to first split up the problem, then merge the results. Getting that to work will probably be as hard as writing a system based on parallel servers.

An other problem is disk i/o. If you uses separate server, each only have to work on local harddrives. If you have one query server, and one storage servers, you hav to move a lot of data across the network. This can be slow, especially if you plan to use NFS.

Also when having separate server you can grow your search engine by adding new servers. And if on of the servers is down you service isent down, you can still return answers from the remaining servers.

The conclusion is that separate server is cheaper, more scaleable and more robust. But using on big server is essayer to develop for. At Boitho we first used one Dual Xeon server and SM 5014C-MT for storage, by NFS. Then we swapped to that the SM 5014C-MT did the searching.

I agree with you about it being more scallable to use several small server.

But say you are skimming the top 10 results from 3 servers then this is not going to to be the top 30 most relevant results. Say if one server has crawled part of the web which is most relevant to the search term then result 11 on that server could still be more relevant than the first result on another server, however it will not be returned. Do you have a way to get round this at boitho?

One solution could be if you wanted the top 30 results then you could query 3 server for the top 20 then combine the results and discard the last 30. But this way would not be very efficient and you could still miss more relevant results.

I am using Clucene for my database, so it cannot span more than one hard drive unless I use a raid array, but then I can’t use seperate servers. I might look into a distributed file system like lustre.

Merging the results to get the best, is as you have identified, on of the problem with a parallel architecture. At Boitho we now just get the same number of results from each node as we shall display. So to display 20 results, each node has to return 20 results.

This is flexible and easy. As long as at least on node is up we can serve some result.

so it cannot span more than one hard drive unless I use a raid array

One can use LVM (Logical Volume Manager) if you are on Linux to make on virtual disk of many physical disks.

I don’t think using distributed file system is specially smart. What happens if one harddisk crashes? Dos the entire filsystem then become corrupted? It also probably pretty slow.

Making a system with independent nodes, will be more flexible. The “share nothing architecture” where no node share anything with an other node, is the cheap architecture on the planet.

There’s a very interesting discussion on webmasterworld about the server hardware needed for high traffic sites - if it’s of any interest. Find it here: http://www.webmasterworld.com/forum23/4496.htm .

Simon

Therefore the following are the reasons I do think most people sign up for the bloggers’ wellbeing program: Nearly all bloggers are generally misinformed; Nearly all bloggers are generally lazy; Nearly all bloggers don’t have skills; Nearly all bloggers don’t have Commitment; Nearly all bloggers don’t have business state of mind. legendsoflaw

From your BlackBerry on the light bulb on the telephone, Canadian pioneer technology are an element of the normal life of the globe. But most finish up in US primarily based companies. We count number down the superior ten Canadian technological inventions that will stayed you will come to home. financeformanagers

Session of Ruth Porat, while Chief Fiscal Officer in Google, could possibly be symbolized just as one empowerment involving women throughout tech promoting jobs. Nevertheless, it’s sad until this glory can be dimmed down while using actual files showing your decline inside ratio involving tech work opportunities held by simply women. medicomtech

Taking your business known inside local area might be hard. Have you ever put basic principles in position? Follow these kind of five straightforward local SEARCH ENGINE OPTIMISATION principles to start with to have the exposure your small business needs. charterschoolpolicy

Have you ever faced the challenge of finding exactly the right doggy house? Although thousands of patterns flood the market industry now, choosing a dog house might be tough, especially for those who have a certain design planned or when you would like some actually unique doggy house. thebeautyresource

Hollywood in the North. There’s justified reason why Vancouver can be honoured with a real glamourous pseudonym. In fact, it has become a filmmaking centre for up to a one hundred year. savenow4college

Traveling in foreign countries is high priced. Some take a trip companies take your dollars and are not able to manage your current trip sufficiently. Some tricks for contemplating the easiest way a firm that cares to the traveler underneath their enjoy. bkkshopping

Faith, expect, and love include the components involving life. These about three alone get us from examine point in your lives, and supply us to be able to live your virtuous lifestyle. With these kind of three you can easliy move throughout alignment on the will involving God. top—designer-brands

Finding take a trip deals throughout Europe is often a general topic however, many very crystal clear paths to visit. Lose them and will also be lost into a large world involving not exciting opportunities. A huge number of offers are around, with quite tight price ranges one to each other. lifestyle-navi

Earlier in the 19th one hundred year in Luxembourg, people widely used horse utilized carriages pertaining to transportation. There ended up obviously zero wristwatches then. So metal carriage clocks were employed to monitor time. womenfashionblog

Internet online marketers discuss the system they uses of their SEO ways of log what are the traffic routes they stimulate when operating traffic to your new area. They in addition outline your default issues they like about the content that they can generate. 7jewelry

Determing the best person a great Infertility Advising with is usually a daunting activity. But when you have chosen the correct person, you may be amazed at what we can find out. There are generally many items to consider when deciding on a therapist. shop24hr

Mobile Value Hunt Advertising is often a new promoting concept. It makes it possible for advertisers to set their makes, ads along with free giveaways onto a universal prize road for portable users to recover via NAVIGATION. thepetcompanion

Today, the athletics industry is considered to be the 11th largest sector in U . s ., producing over $213 billion dollars annually. It is just a growing along with diverse discipline, including a number of segments similar to professional squads, private along with public companies, en-architects

There are generally many motor vehicle insurance possibilities online. Nevertheless before acquiring, try to watch out for various websites and have quotes via different insurance agencies for your automobile. Of course as being a consumer, you wish to have a good suggestion of the retail price ranges available. dressydresses

Growing number of people today depend upon services that offers hassle no cost recovery involving assets along with bring ways of long imminent cases in connection with locating investors, heirs for you to estates, people, due refunds, and many others. visonictech

As technological innovation becomes a greater part of our own children’s existence, pediatricians are generally seeing a lot more young people for similar stress incidents. How could parents aid the prevention of injury? cardsforbusiness

Factors that will influence your fee and the price tag on real house management. What on earth is the difference relating to the fee plus the cost? The price is what we sign inside real house management commitment whereas the charge is just how much you will need to pay all year round to obtain the service you may need. sdgl-edu

Traveling is surely an interesting interest. Travelers have the advantages of learning along with gathering know-how about different spots. Not only that they can see along with explore distinct places nonetheless they also discover those spots intentionally and also unintentionally. house-blueprints

Medical Accounts receivable technology features witnessed carried on expansion in the first 8 weeks of this summer, as verified by pr releases about a number of ninety solutions accumulated throughout BillingWiki. A train manager may fight to select the top product from a real large product or service variety. newharbortech

If you would like luxury real estate property for sale made, then it could be good determination to search for online. The net is stuffed with a great deal of information regarding real estate property for the unique area you wish to buy throughout. There are a great deal of real auctions online who could seriously help in your current hunt. returneeshome

Were you aware that there are generally over fityfive, 000 plumbers inside state involving California? In case you did certainly not, then anyone likely be aware that this job is speedily growing throughout popularity. A plumbing engineer can gain $16 for you to $35 and quite often more 1 hour using the field along with years involving training. simplythebusiness

You’ll be able to increase your sales of your respective company by utilizing manufacturer’s distributors or unbiased sales distributors. This web marketing strategy is wonderful for small companies when you only shell out your salesman based in performance. Allow me to share the principles. businesstrainingteam

Whenever we mention fashion, a brilliantly hot hunting lady commences reverberating in your mind. The grounds behind this can be fashion is often a replica involving beauty, facial foundation and fashion, that is the most suitable seen throughout women. Nevertheless it doesn’t imply male are generally behind the style scene. managementconsultingbusiness

Your straightforward sense pertaining to fashion genuinely isn’t asking that much would it be? Trying to take a look good often is not a real bad thing especially when many societies are created on 1st impressions. Whatever you do for the living as well as what higher education you attended, looking your very best self is critical. finance-perso

When it relates to creating a new happy household, most individuals recognize the benefit of actions, diet, along with health. But these critical factors are only perhaps the equation. Sometimes people feel that if everyone in a very home loves one other enough, delight will only happen. However, that’s not invariably true. acceptbusiness

An enterprise law contract is utilized between a pair of entities, normally both firms, to format the scope associated with an arrangement, agreement as well as deal relating to the two corporations. If you happen to be about to start a commitment agreement using another organization, here can be a few things you must look for. handybusiness

Understanding your acceptance process in a Jamaica real estate property sales contract is important to precisely what maintains involving signing along with taking possession in the property. If just about any party entered right contract, they must be informed about the mother nature and mode associated with an acceptance. registermylocalbusiness

For every single disease, we expect to identify a corresponding treat and cure. However, even with the luxury of advanced technological innovation, cure remains to be not intended for many ailments. The solutions for Along syndrome are an illustration of this that. social-finance

How anyone decorate your own home can be be extremely important for a emotional well-being. Your surroundings give you a daily depiction to who you happen to be and how where you will live. A well-planned indoor decor may help to increase your impression of well-being and also be extraordinary to guests. successeducationsystem

Probably a small server will do? Depending on the needs of your business. If your system would need more, then its okay to increase server capacity. I also thought of this same situation before, but with the advice from my website developer va , I kept it as simple as I can. Without any uneccessary expenses.