Hello,

I was just referred here from WebmasterWorld. I would love to be able to converse openly with other SE operators so I’m glad to be here…

I operate a string of SE’s, with my most popular site being SearchSight.com. My sites runs mostly on Windows-based machines, so I’m in an up-hill battle from the start.

I’ve been doing this for 6 years, and the SE business has been good to me. I have always considered this somewhat of a hobby, though I do have employees and due to my wife’s prompting, I try to run it as a real business.

I’m fascinated with data aggregation, and I hope to be able to contribute something to the Internet on a World-wide scale. I just haven’t been able to do it yet!

Which SE sites are represented here?

I run www.boitho.com, Fischerlaender runs www.neomo.de . Both is algorithmic, crawler based search engines based on own file structures (not a sql database), writen in Perl and C.

What languages are you developing in ?

( Please don’t judge me based on the search engine that you find at www.boitho.com , it is a 1.5 years old demo we made to show to potential investors how thumbnail pictures can be used in search engines. )

What languages are you developing in ?

Well, please don’t judge me based on the languages that we use!

Our crawler uses Visual Basic, mostly because of the thumbnail image retrieval, speed has not been something we’ve been concerned with. You just can’t get very fast if you want to download an entire web page instead of just the source code.

Our database runs on SQL Server…. again probably the worst choice for running a search engine. We are testing an index manager from surfinity that creates a much more efficient search on SQL Server and we hope to have Microsofts full-text search replaced with this product within the next few days.

Ease of development has been the primary concern up until this point. Since I am in charge of programming, marketing, and eveything in between I don’t have the time to spend working on developing the fastest system.

I do have some pretty good marketing systems in place right now and I’d be interested in working with someone to create a new search engine based on a new platform. Anyone interested?

Well, a belated reply to this post, but what the heck! :rolleyes:

Well, I’ve ended up writing the system for searchserf.market-uk.com.
I know… it’s on a subdomain. :rolleyes: I’ll get a domain name for it shortly…

See, I actually just got side-tracked while writing myself a little app for something else, and somehow ended up with this thing!

Anyhow, I’ve been finding it really fascinating, so I’ve stuck with it and am doing it as a hobby, with a view to perhaps creating a job for myself with it so I can quit chasing the end of the month and start chasing my dreams, man!!! 

I’m actually quite shocked I even got this far… having zero idea about search engines other than some basic SEO, etc.

So basically it’s 4 pieces of software I wrote with Delphi. One does the crawling, with seeds added from another piece of software that handles submissions (as well as sending out email for submission confirmation), another handles the indexing on keywords, and yet another that monitors that everything is up & running & email me reports daily.

It’s running on one of those ol’ wind-up computers, so it’s sure gonna be interesting to see it run on its own server one day! And, of course, a whole lot bigger connection would work wonders! But… one step at a time I suppose. I’m learning as I go, sort of really just dreaming up how it should work - it’s really such a trip doing it - I wish I had a job that was this much fun!!! :blink:

I have been using some Delphi to. The crawler for boitho.com is for example in Delphi.

How many pages have you crawled sow fare?

I have been using some Delphi to. The crawler for boitho.com is for example in Delphi.

How many pages have you crawled sow fare?

Hi Runarb,

I’ve had a couple of test runs… debugging can be a pain, as you know.

Considering I first just banged a bit of code together really, I’ve been learning more and more from the experiment as I go along, and I adapt the thing further as I go along.

The crawler does maybe up to 60000 or a bit more a day - not good at all, but it’s doing too much of the work, filtering unwanted sites, parsing entire pages & feeding itself URL’s from every page it downloads. Backend is MySQL, running on same pc as crawler. So not a fast setup by any standard.

But the results are encouraging enough for me to be planning a complete rewrite based upon the things I’ve learned so far from its behaviour.

For example, I’d like to perhaps multi-thread it so that it can be caching pages while waiting for pages to download. At the moment, it waits for the page to download, then caches it, then gets the next url to parse, waiting for MySQL to execute the query, etc etc.
Another thing I’d like to do is have it crawl only a limited length route from the initial seeds, eg max xyz links away from seed. The thing is I’m finding that the further away from a good seed you go, the more 404’s you find! I was actually quite shocked to see how many outdated links many people have on their sites!

Any suggestions?

Just having one thread that do the crawling in serial wont be optimal.

You should look into either using multi-thread, as you mention, or asynchronous io.

You can implement asynchronous io by having a large array of non blocking sockets, and sent request through each, one by one, but not wait for it to finish, just go to the next one. Then start at the first to check if all the data have come in. If it has, give it an other page to download.

Unfortunately this is more complicated, and may be tricky to get working.

Also se http://en.wikipedia.org/wiki/Asynchronous_I/O

Thanks runarb!

How many sockets in the array would you recommend, or should I just play around & monitor the difference in results.

Also, how would you recommend feeding & fniding good seeds? I am thinking of keeping a scoreboard of all links spawned from crawled pages & cutting entire chains of those that fall below a certain score (eg site “A” gives x links of which y spawn y2 good links and z that spawn z2 bad links (404’s etc) )

I’m definitely going to do a complete rewrite of the crawler so I’m really keen on hearing any ideas I can before I start. B)

Try with 500 for starts, and se how that performs. Then adjust to you have a good cpu utilization.

For seeds most search engines uses the Open Directory RDF Dump, from http://rdf.dmoz.org/ . That is a XML like file containing all the links in dmoz.

Therefore listed below are the reasons I believe most people make an application for the bloggers’ health program: The majority of bloggers tend to be misinformed; The majority of bloggers tend to be lazy; The majority of bloggers absence skills; The majority of bloggers absence Commitment; The majority of bloggers absence business way of thinking. legendsoflaw

From the actual BlackBerry towards the light bulb towards the telephone, Canadian creations are part of the normal life of the planet. But most finish up in US dependent companies. We depend down the very best ten Canadian technology inventions which stayed you’ll come to home. financeformanagers

Visit of Ruth Porat, because Chief Monetary Officer from Google, might be symbolized being an empowerment associated with women within tech advertising jobs. However, it’s sad this glory is actually dimmed down using the actual information showing the actual decline within the ratio associated with tech work held through women. medicomtech

Having your business known within the local area could be hard. Perhaps you have put the fundamentals in location? Follow these types of five easy local SEARCH ENGINE OPTIMIZATION principles to start to find the exposure your company needs. charterschoolpolicy

Perhaps you have faced the issue of finding the right canine house? Although thousands of styles flood the marketplace now, picking out a dog house could be tough, especially if you have a specific design in your mind or when you are interested in some genuinely unique canine house. thebeautyresource

Hollywood from the North. There’s valid reason why Vancouver is actually honoured with this type of glamourous pseudonym. In the end, it is a filmmaking centre for up to a hundred years. savenow4college

Traveling overseas is pricey. Some journey companies take your hard earned money and neglect to manage your own trip properly. Some strategies for contemplating choosing a organization that cares for that traveler below their view. bkkshopping

Faith, wish, and love would be the components associated with life. These 3 alone consider us from denote point within our lives, and provide us a chance to live the actual virtuous existence. With these types of three we could move within alignment towards the will associated with God. top—designer-brands

Finding journey deals within Europe is really a general topic however, many very obvious paths to follow along with. Lose them and will also be lost into a massive world associated with not fascinating opportunities. A large number of offers are available, with really tight costs one together. lifestyle-navi

Earlier throughout the 19th hundred years in Luxembourg, people took horse attracted carriages with regard to transportation. There had been obviously absolutely no wristwatches in those days. So metal carriage clocks were accustomed to keep an eye on time. womenfashionblog

Internet entrepreneurs discuss the device they uses within their SEO ways of log what kind of traffic stations they trigger when generating traffic to some new site. They additionally outline the actual default points they like about the content they generate. 7jewelry

Choosing the best person to have Infertility Guidance with could be a daunting job. But after you have chosen the best person, you’ll be amazed at that which you can discover. There tend to be many points to consider when selecting a therapist. shop24hr

Mobile Cherish Hunt Advertising is really a new advertising concept. It allows advertisers to place their manufacturers, ads as well as free special gifts onto a worldwide prize chart for cellular users to build up via GPS NAVIGATION. thepetcompanion

Today, the sports activities industry is considered to be the 11th largest business in Usa, producing a lot more than $213 billion dollars annually. It’s a growing as well as diverse area, including several segments such as professional groups, private as well as public businesses, en-architects

There tend to be many car insurance choices online. However before purchasing, try to consider various websites and obtain quotes through different insurance providers for your vehicle. Of course like a consumer, you need to have advisable of the cost ranges available on the market. dressydresses

Growing amount of people today depend on services which supply hassle free of charge recovery associated with assets as well as bring ways of long impending cases associated with locating investors, heirs in order to estates, traders, due refunds, and so on. visonictech

As technologies becomes a greater part individuals children’s life, pediatricians tend to be seeing increasingly more young sufferers for repeated stress accidental injuries. How may parents aid the prevention of injury? cardsforbusiness

Factors which influence the actual fee and the price of real house management. What’s the difference between your fee and also the cost? The charge is that which you sign within the real house management agreement whereas the price is the total amount you need to pay all year long to obtain service you’ll need. sdgl-edu

Traveling is definitely an interesting pastime. Travelers have the advantage of learning as well as gathering understanding of different locations. Not only they see as well as explore various places however they also find out about those locations intentionally in addition to unintentionally. house-blueprints

Medical Payments technology offers witnessed ongoing expansion throughout the first 8 weeks of come early july, as substaniated by pr releases about a few ninety items accumulated within BillingWiki. A exercise manager may fight to select the very best product from this type of large item variety. newharbortech

If you are searching for luxury property for purchase, then it might be good choice to search online. The web is full of lots of information regarding property for the precise area you need to buy within. There will also be lots of real auctions online who may help you in your own hunt. returneeshome

Are you aware that there tend to be over fityfive, 000 plumbers within the state associated with California? Should you did not really, then a person likely realize that this occupation is quickly growing within popularity. A local plumber can generate $16 in order to $35 and frequently more an hour or so in line with the field as well as years associated with training. simplythebusiness

You are able to increase the actual sales of the company by dealing with manufacturer’s representatives or impartial sales representatives. This web marketing strategy is fantastic for small companies since you only spend your salesman based upon performance. Listed here are the fundamentals. businesstrainingteam

Whenever we discuss fashion, a fantastically hot searching lady begins reverberating within our mind. The excuse for this is actually fashion is really a replica associated with beauty, make-up and design, that is better seen within women. However it doesn’t imply that male tend to be behind the style scene. managementconsultingbusiness

Your easy sense with regard to fashion truly isn’t asking everything much could it be? Trying to appear good regularly is not this type of bad thing specially when many societies are made on very first impressions. Regardless of what you do for any living or even what university you visited, looking your very best is essential. finance-perso

When it involves creating the happy house, most people recognize the benefits of conduct, diet, as well as health. But these critical factors are only area of the equation. Sometimes people believe that if everyone inside a home loves each other enough, joy will simply happen. Regrettably, that’s not at all times true. acceptbusiness

A company law contract can be used between 2 entities, generally both businesses, to describe the scope of the arrangement, agreement or even deal between your two companies. If you’re about to enter a agreement agreement along with another company, here really are a few things you need to look out for. handybusiness

Understanding the actual acceptance process inside a Jamaica property sales contract is important to exactly what maintains in between signing as well as taking possession from the property. If any kind of party entered right into a contract, they must know about the character and mode of the acceptance. registermylocalbusiness

For each disease, we expect to locate a corresponding remedy and remedy. However, even with the advantages of advanced technologies, cure continues to be not readily available for many illnesses. The remedies for Lower syndrome are a good example of that. social-finance

How a person decorate your house can be quite important for your emotional well-being. Your surroundings give a daily expression to who you’re and how where you will live. A well-planned inside decor may increase your feeling of well-being in addition to be amazing to site visitors. successeducationsystem