Site-maps, robots, cloaking, spiders...


education directory
Ed the Webmaster

 Webmasters' Education:
Start here - Why me?
What you have to know
2 keywords = 1000 phrases
Bad titles, low traffic
Free meta tags
Speaking in tongues
Slow pages equal more traffic
Free and Easy
Yahoo for free
Master of your own domain?
Spiders cannot fill in forms
Good and bad doorways
Order the engines around
Do you use frames?
Are you a magnet?
Visitors by default
Lasso your lost visitors
$100 = 10,000 visitors
Others work - you earn
Ads - Don't click here?
Invite the competitors over
E-mails and warnings
Infectious or immunized?
All the best submission tools
100s of search engines
The "must have" web tools
Free for all sites
Pressing the editor's button
Legal or e-legal?
Where do I start?
A to Z index

 Finance Partners:
Cover Base Life Insurance
Finding Life Insurance Cover
Annuity Comparisons
Aviva Annuities
Buy To Let Mortgages
Commercial Insurance
Inheritance Tax
Pension Misselling
Annuities Comparison
Enhanced Annuities
Pension Annuity Answers
Annuity Office
Annuity Plan
Annuities Plan
Annuities Central
Open Market Option
Capped Drawdown
Mortgage Comparisons
Hidden Mortgages
Barclaycard PPI Claims
PPI Claims
PPI Complaints
PPI Refunds
Barclays PPI Claim
Lloyds PPI Claim
RBS PPI Claims
Wrongly Sold PPI
PPI Justice

  Web Guest Writers:
Web kiosks for the poor
Online Price Mistakes
Thesaurus Ranking
HTML or text e-mails?
ALTered Images
Renaming your Images
Free tools to create your site
Invasion of the Amazons
Advertising v. PR
Will content be profitable?
Responding to spam
What is spam?
Pay for position
Web host tips
Webmaster beauty
Bad web design: Animation
CGI: What the heck is that?
Long live banners
Great press releases
E-mail Sigs - 7 Key Items
Ad tips for affiliate marketing
I want to keep my frames
Creating Doorway Pages
Measuring Link Popularity
Domains and hyphens
Why have a domain name

 General Education:
Homepage | Web Search
Education Resources
Active Online Pupil Forums
Art | Film | Dance | Bands
Countries of the World
Digs UK | US | Canada
Finance | Economics/Biz
Homework Helpers | Exams
Kiddies Korner | Tots | Pets
Jobs UK/EU | US | Canada
Bartending Work
Medicine and Health
Museums and Galleries
Mystery Shopping
PE | Sports | Sporting Events
Power & Politics | Civil Rights
Print Media, TV and Radio
Problems and Advice
Shops | Fashion | Books
Subjects | Religion | Language
Technology Education & ICT
Teens | Just for Fun
Schools UK | US | Canada
Unis/Colls UK | US | Canada

 Special Features:
Essays - Full Writing Course
What is Bullying?
Stress in Teaching
Online Education

 Guest Contributors:
The best Dad?
You're an Idiot!
Slave Caster of Freedom
Out of the mouths of babes
The Right to Life?
The Nostradamus Hoaxes
Explaining terrorism to a child
Internet 2 a scam?
Break a Rule, Bad Girl
Britannica near extinction?
The 1st time I really lied
Nigerian Scam Letters
Singular turns plural
English Writing
In debt?
The Secret Shopper
Too busy at work?
Bullying... Our Stories
Start to live your dreams
Recognize your potential
Stop worrying, please!
Public speaking
Elegant resumes
In praise of black sheep
Ritalin - Straight-jacketing?'s full list of pages

Increase Traffic to your Web-site - Traffic Tips and Secrets...

IMPORTANT INFORMATION ppi Have you taken out a credit card, mortgage, secured loan, unsecured loan or hire purchase agreement in the last ten years? If you have (or have had) a mortgage, loan or credit card with providers such as Abbey, Barclaycard, MBNA, Halifax, HSBC, HBOS, Lloyds TSB, Natwest, RBS or in fact any other credit provider, you may be able to reclaim up to 15,000 if you were sold PPI insurance - in most cases even if you've lost the paperwork. Learn more about PPI Claims now!

BECOME A MYSTERY SHOPPER Are you a student over 18? Part-time teacher? Or maybe a parent or just someone that needs some extra income? Some free food and drinks perhaps? Would you like to pick your own hours? Casual work is available now.

The Mystery Shopping Club provides you with an EXCEPTIONAL collation of intelligence that is crucial for anyone with an interest in Mystery Shopping. Become a Mystery Shopper now!

EXCEPTIONAL EDUCATION ADVERTISING OPPORTUNITY It is simple to get your site listed on! You simply pay a small one-time-only administration charge for a PERMANENT lifetime advert! Learn more about advertising on now!

Kate the Webmistress

Importance of site maps

A site-map can be a search engine crawler friendly page.

Create a HTML page and fill it with all the links on your site (see's site map for an example).

Linking to a site map page from your homepage helps the search engine spiders find and index all the pages of your site easier. It has the added benefit of getting a lost visitor back on track.

If you have hundreds of pages, you could split up the site map pages to include only 50 URLs (Uniform Resource Locators / web addresses) each and make sure all these pages are linked to each other. See also how have utilised crawl pages at - click on any one of their links to understand the structure.


  • If your links are embedded in javascript, search engines have problems travelling through them.

  • Avoid using image maps, most search engines cannot crawl them.

    The Robots.txt File

    The robots.txt file is a file that can tell search engine robots which areas of your site not to index.

    It is particularly useful if you want to keep some areas of your web-site away from public view, such as password protected pages or "office use only" pages.

    A robot is a small software program that visits sites and automatically requests documents from them for indexing. They use the hypertext links found within documents to travel to their next destination. A search engine is a database of HTML pages gathered by a robot. Robots are also known as webbots, bots, web crawlers or spiders.

    More on the robots.txt file:

  • Robots Exclusion
  • Database of Robots
  • Robots
  • Internet Tips Robots
  • Robots Check
  • The Web Robots Database

    Come into my parlour said the spider...

    Cloaking and Spiders
    Below are some web-sites that contain information about the risky subject of page cloaking and hiding your HTML from the search engines (not for the novice).

  • Spider Food
  • Spiderhunter
  • Search Engine Base
  • Search Engine Forums
  • Fantomaster Cloaking FAQ

    Beware, even though you may think you are cloaking ethically, you might make a mistake and be flagged as a spammer. The need to keep up to date with changing spider IP addresses is also a problem.

    MSN say that they will ban sites that use cloaking technology and will share information about companies they've caught with Inktomi (the search provider for many major portals) and with other search engines. calls cloaking "egregious" and says "it goes against our ideal of us seeing the same thing a search engine sees".

    Do you use frames on your site? Click here to find out why frames are maybe not such a great idea...

    Save this site

    Please don't forget to add to your list of favorites/bookmarks. Internet Explorer users please click here and others, right click here - - - - Also, you can learn how to make any page your start page by clicking here.

    Share this site

    Click here to tell your friends, family or colleagues about


  • education directory

    An Ed-U-Kate production. This page was produced 28 January 2001 and last edited 9th June 2015.
    This site is for information only; responsibility for its use rests with user., its characters, names & related indicia ©
    Add URL | Link to us | Terms of Use | Privacy Policy | Contact Information | ArtFair | Home