What are “Dynamic Websites”?

Dynamic websites are websites whose pages are generated on the fly. Unlike static pages (primarily .htm/.html pages), dynamic pages are generated when an user triggers an action through that particular page.

Here is a sample dynamic URL-

http://www.bbc.co.uk/cgi-bin/search/results.pl?q=%22dynamic+websites%22&tab=news&go=homepage

As per the above example of http://www.bbc.co.uk, the dynamic part (i.e. the part) of the URL which changes as per surfer request is the part after the question mark (?)

What are the problems that search engines face in indexing Dynamic URLs?

1. Search engines often consider a dynamic URL as an infinite set of links.

2. Since dynamic URLs find maximum application in online shopping carts, there is a possibility of incorporating a session id to a particular page. As session ids of that particular page change, the search engine spider needs to index an infinite number of copies of the same page, which is a Herculean task for them.

3. Proceeding with the same logic presented in point # 2, indexing the same dynamic page might overload the servers of the search engines and therefore prevent the search engines to present with the most relevant information in the fastest possible time.

Here is what Google says about indexing of dynamic websites –

Reasons your site may not be included: Your pages are dynamically generated. We are able to index dynamically generated pages. However, because our web crawler can easily overwhelm and crash sites serving dynamic content, we limit the amount of dynamic pages we index. (Source – http://www.google.com/webmasters/)

What are the options that you have in order to make a search engine spider index your Dynamic URLs?

1. Use of softwares – Exception Digital Enterprise Solutions (http://www.xde.net) offers a software which can change the dynamic URLs to static ones. Named XQASP, it will remove the “?” in the Query String and replace it with “/”, thereby allowing the search engine spiders to index the dynamic content.

Example –
http://www.my-online-store.com/books.asp?id=1190 will change to
http://www.my-online-store.com/books/1190.

The latter being a static URL, it can easily be indexed by the search engine spiders.

2. Use of CGI/Perl scripts – One of the easiest ways to get your dynamic sites indexed by search engines is using CGI/Perl scripts. Path_Info or Script_Name is a variable in a dynamic application that contains the complete URL address (including the query string information). In order to fix this problem, you’ll need to write a script that will pull all the information before the query string and set the rest of the information equal to a variable. You can then use this variable in your URL address.

Example – http://www.my-online-store.com/books.asp?id=1190

When you are using CGI/Perl scripts, the query part of the dynamic URL is assigned a variable.
So, in the above example “?id=1190” is assigned a variable, say “A”. The dynamuc URL http://www.my-online-store.com/coolpage.asp?id=1190
will change to http://www.my-online-store.com/books/A through CGI/Perl scripts which can easily be indexed by the search engines.

3. Re-configuring your web servers –

(i) Apache Server – Apache has a rewrite module (mod_rewrite) that enables you to turn URLs containing query strings into URLs that search engines can index. This module however, isn’t installed with Apache software by default, so you need to check with your web hosting company for installation.

(ii) ColdFusion – You’ll need to reconfigure ColdFusion on your server so that the “?” in a query string is replaced with a ‘/’ and pass the value to the URL.

4. Creation of a Static Page linked to an array of dynamic Pages –

This approach is very effective, especially if you are the owner of a small online store selling a few products online. Just create a static page linking to all your dynamic pages. Optimize this static page for search engine rankings. Include a link title for all the product categories, place appropriate “alt” tag for the product images along with product description containing highly popular keywords relevant to your business (You can conduct keyword research for your site through http://www.wordtracker.com). Submit this static page along with all the dynamic pages in various search engines, conforming to the search engine submission guidelines.

How Amazon.com, Earth’s Biggest Bookstore, coped with the issue of indexing of dynamic URLs?

A search in Google for internet marketing books, yielded a result that takes you directly to the appropriate dynamic page at Amazon: http://www.amazon.com/exec/obidos/ISBN%3D0395683297/103-0475212-8205437.

Since the above URL does not contain any query strings, all search engines can index Amazon.com’s products. Amazon.com uses this method to get its product selections indexed by search engines. This is very important for Amazon, because being an online bookstore, it is very natural for them to adopt dynamic URLs yet it was equally important for them to make their dynamic URLs search engine index friendly.

Conclusion

Even a few years back, most of the major search engines did not index dynamic URLs, thereby often preventing top search engine rankings for the online stores. With Google starting to index dynamic URLs a few months ago, the picture is going to change in the coming days. This is more so because Google’s numero uno position is currently being threatened by Microsoft’s MSN (developing its own search engine) and Yahoo! who recently acquired Overture, the biggest player in the PPC Search Engine industry.

########################