While there are different ways to organize web content, every crawling search
engine has the same basic parts:
• a crawler
• an index (or catalog)
• a search interface

Crawler (or Spider)

The crawler does just what its name implies. It scours the web following links,
updating pages, and adding new pages when it comes across them. Each search
engine has periods of deep crawling and periods of shallow crawling. There is also
a scheduler mechanism to prevent a spider from overloading servers and to tell the
spider what documents to crawl next and how frequently to crawl them.
Rapidly changing or highly important documents are more likely to get crawled
frequently. The frequency of crawl should typically have little effect on search
relevancy; it simply helps the search engines keep fresh content in their index. The
home page of CNN.com might get crawled once every ten minutes. A popular,
rapidly growing forum might get crawled a few dozen times each day. A static site
with little link popularity and rarely changing content might only get crawled once
or twice a month.
The best benefit of having a frequently crawled page is that you can get your new
sites, pages, or projects crawled quickly by linking to them from a powerful or
frequently changing page.

Search Engine Relevancy
Many people think search engines have a hidden agenda. This simply is not true.
The goal of the search engine is to provide high-quality content to people searching
the Internet.
Search engines with the broadest distribution network sell the most advertising
space. As I write this, Google is considered the search engine with the best
relevancy. Their technologies power the bulk of web searches.
The Problem Listing a New Site
The biggest problem new websites have is that search engines have no idea they
exist. Even when a search engine finds a new document, it has a hard time
determining its quality. Search engines rely on links to help determine the quality
of a document. Some engines, such as Google, also trust websites more as they
age.
The following bits may contain a few advanced search topics. It is fine if you do
not necessarily understand them right away; the average webmaster does not need
to know search technology in depth. Some might be interested in it, so I have
written a bit about it with those people in mind. (If you are new to the web and
uninterested in algorithms, you may want to skip past this to the search result
image on page 35.)
I will cover some of the parts of the search engine in the next few pages while
trying to keep it somewhat basic. It is not important that you fully understand all
of it (in fact, I think it is better for most webmasters if they do not worry about
things like Inverse Document Frequency, as I ranked well for competitive SEOrelated
terms without knowing anything about the technical bits of search);
however, I would not feel right leaving the information out.

Some people ask me to do ten hours worth of competitive analysis for free. Generally, I cannot do hours of consulting for free, but if you have a quick question or do not understand something, please make sure you e-mail me so I can try my best to help you.
You bought this book hoping to get useful and easy-to-understand information, and my goal is to give that to you. I have probably read millions of forum posts and web pages, so many of the random thoughts scattered throughout this e-book might not make a bunch of sense to someone who has not done the same.of the ideas came from errors I made on my sites, and some of them came from errors friends made.
Most sites do not end up getting banned or penalized from search engines without reason, although on rare occasion it does happen. Sometimes engines are quick to
respond, and sometimes they do not care much. As you learn more about why they do certain things and how the business aspects play into the algorithms, it becomes easier to evaluate where they might go and how they might try to go there.
SEO is both reactive and proactive.
If a section of my e-book is not clear to you or does not make sense, then that is my fault and I should explain it to you. Feel free to send me an e-mail if you have ANY questions.
This book is primarily my voice and how I understand the web. To help give you a broader perspective, I also interviewed many search, marketing, and web experts. I recommend you also read the bonus interviews at http://www.seobook.com/seointerviews.pdf.
Best of luck with your sites, -Aaron seobook@gmail.com

What if you are starting with nothing? Can you still compete? Of course you can. At the end of 2002, I got kicked out of the military for using drugs. At that point, I was experiencing a number of things:
 Suicidal depression  Financial bankruptcy, living on credit cards  Social isolation  Ignorance to the web, SEO, and marketing (slightly less serious, I know)
Within 4 years, I had pulled myself out of this emotional and psychological slump, and had achieved success:
 I was fairly knowledgeable about the web, SEO, and marketing.  I had made lots of friends.  I was getting mentioned in the Wall Street Journal (and many other newspapers).  I was speaking at colleges about SEO (one college even wanted to hire me to become a professor).  I had venture capitalists offering to invest in this site.  I had a mainstream publisher offer to publish this book.  I got married to the most wonderful woman in the world
What did I have that allowed me to do well? I had a passion for learning. That passion helped me attract great friends who took me under their wing and helped me far more than I could have ever expected. It takes time to do well, but if you keep pushing, keep learning, keep sharing, and are trying to help others, eventually you will do well on the web.
Many true web authorities started out as topical hubs. People who had no intent of creating a business would just freely talk about a subject they loved, and linked out to related websites they found useful. Every web marketer should read this post:

http://chartreuse.wordpress.com/2006/09/18/why-paris-hilton-is-famous-or-understanding-valuein-a-post-madonna-world/

You become a platform worthy of attention by talking about others who are worthy of attention. Getting people to pay attention is a real cost. You have to get people to pay attention before you can extract value from your work.
To most people, the single most relevant and important thing in the world is themselves.
Here is a quote from Radiohead’s Meeting People is Easy: If you have been rejected many times in your life, then one more rejection isn’t going to make much difference. If you’re rejected, don’t automatically assume it’s your fault. The other person may have several reasons for not doing what you are asking her to do: none of it may have anything to do with you. Perhaps the person is busy or not feeling well or genuinely not interested in spending time with you. Rejections are part of everyday life. Don’t let them bother you. Keep reaching out to others. When you begin to receive positive responses then you are on the right track. It’s all a matter of numbers. Count the positive responses and forget about the rejections.
You are not always going to be able to predict what will work and what doesn’t, but the more you keep learning and the more things you try the better the odds are that something will stick. Internet marketing is just like offline marketing, but cheaper, faster, and more scalable.
Social scientists have studied why things become popular, and many things are popular only because they are already popular. In Is Justin Timberlake a Product of Cumulative Advantage Duncan J. Watts wrote about how groups tended to like the same things, but random different things in each group. Even if success is random and unpredictable there is a self reinforcing effect to marketing. http://www.nytimes.com/2007/04/15/magazine/15wwlnidealab.t.html
If you keep reaching out to people you will be successful. It might take 3 months. It might take 5 years. But eventually it will happen.