| Analyzing Search Engines Essay, Research Paper
1. Formulate five criteria for the evaluation of search enginesTo effectively evaluate three different search engines from the perspective of an advanced web user, the following criteria were established:
1. Relevance and accuracy of search results
2. Search speed
3. Advanced search options
4. Other services
5. Site design and layout
More information about these criteria and how they were used is available below.
2. Test three search engines against your criteria
For this evaluation, the following three search engines were tested:
1. AltaVista – http://www.altavista.com/
2. Google – http://www.google.com/
3. DirectHit – http://www.directhit.com/
3. Explain your methodologyThe criteria outlined in Part 1 were reached by considering the things that would be important to an advanced user. Of course, factors like Relevance and Speed are critical to such a user (particularly in a work environment). The search options criteria is particularly important to advanced users who are after a specific page and do not want to waste time ‘wading’ through results from a ’standard’ search. The other criteria are important to most users and mean that the search engines are more usable and useful.
The search results from the three test search engines (Part 2) were evaluate according the criteria, explained it more detail below.
Criteria 1 – Relevance and Accuracy of ResultsThis criterion was to evaluate the usefulness of a particular search engine according to the accuracy of its results. I defined the relevance, according to personal standards, after browsing the pages in the results, for three separate search terms. The Relevance Percentage is calculated by counting the number of relevant pages that appear in the first 20 search ‘hits’ and then multiplying by 20 (for a percentage).
Criteria 2 – Search SpeedThis criterion was to evaluate which of the test search engines returns results quicker.
However, it is important to note that the speed test is both a test of the Internet connection to the remote machine, as well as the time taken by the search engine to perform the search. Furthermore, tests of Internet speeds are very variable based on time of day, other internet traffic etc and are therefore of only limited accuracy. For this reason, no ‘hard numbers’ were calculated (ie timed searches). Instead, I rated each search engine based on 5 searches per engine at the same time, same day. The possible Speed Ratings are as follows: Excellent, Good, Poor
Criteria 3 – Search OptionsThis criteria was to evaluate the search engines advanced search options. This is important for advanced web users because it allows them to track down specific pages, and weed out irrelevant pages from their searches, meaning they find the most relevant pages quickly.
In evaluating this criteria, the three test search engines were tested to see if they had the following search options:
Variable number of results per page
Search for “phrases”
Boolean Searches (AND, OR, NOT)
Wildcard (*) searches
Search for text in only title or body
Search for pages in different languages
Search for pages located in specific countries
Search for pages under a certain domain
Search for pages containing links to a URL (eg link:http://www.yahoo.com/)
Search for pages LIKE a certain page (eg like:http://www.yahoo.com/)
The number of search options each search engine had available was calculated and then recorded out of 10.
Criteria 4 – Additional ServicesThis criteria was to evaluate the other services offered by the search engine to make web searching and browsing easier for advanced users. These features make the search engines more useful and more of a ‘one-stop destination’ for finding information, including news, sport results, TV guides, weather reports and other information in a ’start page’ design to allow the user to easily access the information he needs quickly. These services are particularly relevant to the advanced and regular web users because they use the web frequently, making such a page very convenient.
In evaluating this criteria, the three test search engines were tested to see if they offered the following additional services/features:
Online shopping guide
Suggested searches. Searches based on what you have searched for
Other searches – whitepages, yellowpages, maps/directions etc
Hand-picked categories or channels for web browsing
Personalisation of search results, layout, information etc
The number of additional services offered by each of the test search engines was calculated and then recorded out of a score of 5.
Criteria 5 – Page Design and LayoutThis criteria was to evaluate the usefulness and usability of the page design and layouts of the search engines. This criteria is important for all users, especially advanced users because they visit search engines regularly and therefore want a page that looks nice, is easy to get around, and loads quickly, even on low-end machines. Because although advanced web users would probably be able to navigate around and use a poorly designed site, a well designed site will mean reduced time using the site and quicker access to the information they’re after.
Note: Page time is a factor of consideration for this criteria. This is referring specifically to the time the HTML page loads (based on page elements such as tables (and nested tables), images, CSS, JavaScripts etc
In evaluating this criteria the three test search engines were tested against the following points:
Easy navigation
Good Appearance/Aesthetics
Quick Page load time
Each of these points were graded out of 3 points with 1 for poor, 2 for average, 3 for excellent.
Search Engine SelectionThe three test search engines that the previous criteria were tested against were selected as being the three search engines most likely to be used by an advanced web user.
AltaVista was selected because of its powerful search syntax options (eg link:, url: etc) and the fact that it claims to have the largest page index of all search engines. It also provides other features and services useful to a corporate web user and therefore is likely to be one of the first choices for such a user.
Google was selected because its ‘no-noise’ approach. This ideology would be appreciated by many advanced users who don’t need fancy graphical interfaces, or comprehensive search help. The first page is simply a search screen, with other features (directory etc) tucked away, where users can access it if they need it; however the main emphasis of Google is on searching. Google’s simple search results screen also ensures that the results are returned quickly and its search options (with the possibility of saving preferences to a cookie) means that advanced users can easily have Google configured to their specific needs. Google is very much an advanced user’s search engine and it is for this reason it was included for evaluation.
DirectHit was selected because of its new approach to ranking search results. Most search engines base the results on keywords, keyword location, keyword density etc, as does DirectHit, but this search engine also takes notice of the links that users click when they perform a search, and how much time they then spend on the sites. These results are then applied to future searches, effectively making real people the judge of the search order (http://www.directhit.com/help/score.html). This different approach is one that is likely to interest advanced web users who are often seeking information quickly and easily.
4. Report on resultsThe following table represents the results of the three test search engines (Part 2) against the evaluation criteria (Part 1 and 3).
Accuracy + RelevanceSpeedSearch OptionsAdditional ServicesLayout + DesignAltaVista60%Excellent9/105/56/10Google55%Excellent8/102/56/10
DirectHit85%Good5/105/57/10Specific Details of the search engine scores for the Search Options, Additional Services and Layout and Design criteria are available in Appendix 1.
5. Evaluate your criteria and MethodologyAfter evaluating three search engines according to the criteria established in Part 1, the usefulness of some of the criteria was questionable.
The Accuracy and Relevance score was useful, and seemed to accurately represent how accurate I felt the search engine was. However, in evaluating the accuracy and relevance of pages, no ’scientific’ method could be employed – rather, it was a purely personal decision, of whether I thought the page was relevant to someone using the search term used. For a more in-depth analysis of search engines, I would have to suggest a more tangible method of evaluating this criteria. However, I can’t see relevance of search results being measured according to ‘rules’; I think personal opinion is an important part of determining relevancy (what may be relevant to one person may be irrelevant to another). Therefore, I don’t think a definitive statement of a search engine’s accuracy and relevance for all people is possible.
Due to the short time-span for this assignment and a lack of inside knowledge of the operation of search engines, the Speed criteria could be measured only by my own guess. This meant that no specific numbers about search speed could be returned and the results I came to are very much debateable. In retrospect, I think the Speed criteria should have been evaluated according to the following points:
CPU time taken to perform search (based on 100 searches). Of course, finding these details would probably require access to the search engine server.
Average ping time (ping search server from about 10 worldwide locations). Again, this point would have been very difficult to do, especially in the limited timeframe.
I think a measurement of speed according to the above points would be more accurate and useful, and would provide more specific results.
I think the Search Options and Additional Services Criteria were very useful and establishing specific points that the search engines could be easily evaluated against (either true or false) made establishing a score for each search engine easier. The only flaw with the criteria as set out in Part 2 was probably a lack of points, and the scope of these points may have been too narrow. For a more in-depth study, I think more criteria points would have to be established.
I think the Layout and Design Criteria were too broad. I think for a more in-depth study, more (and more specific) criteria would have to be established. I would also suggest a true/false style points system (rather than three points per option system) for fairness and less personal opinion. Furthermore, I think the criteria addressed layout and design too generally, and not specifically for the advanced web user.
In retrospect, I stand by my selection of search engines. I think they represented the search engines used by advanced web users, although I think a more in-depth study would have to evaluate more search engines (I think Excite and Infoseek should also have been evaluated) because many advanced web users would also use the mainstream search engines such as Yahoo, Excite, AltaVista, Infoseek, Metacrawler etc.
Overall, I think the evaluation performed was adequate as a more simplistic evaluation. While some problems with the criteria became apparent after use, I think the criteria were generally fair and useful for an advanced web surfer.
6. Useful URLS dealing with search engine evaluationC|Net: Search Engine Shoot-Out – http://coverage.cnet.com/Content/Reviews/Compare/Search2/How to Search the web: A Guide to Search Tools – http://daphne.palomar.edu/TGSEARCH/Evaluating Internet Search Engines – http://www.udmercy.edu/htmls/Academics/library/searchSearch Engine Statistics – http://searchengineshowdown.com/stats/Yahoo: Comparing Search Engines – http://dir.yahoo.com/Computers_and_Internet/Internet/World_Wide_Web/Searching_the_Web/Search_Engines/Comparing_Search_Engines/SearchIQ – http://www.zdnet.com/searchiq/directory/general.htmlAPPENDIX 1Specific Results for Search Options:AltaVista – 9/10:Variable number of results per page
Search for “phrases”
Boolean Searches (AND, OR, NOT)
Wildcard (*) searches
Search for pages in different languages
Search for pages located in specific countries
Search for pages under a certain domain
Search for pages containing links to a URL (eg link:http://www.yahoo.com/)
Search for pages LIKE a certain page (eg like:http://www.yahoo.com/)
Google – 8/10:Variable number of results per page
Search for “phrases”
Boolean Searches (AND, OR, NOT)
Wildcard (*) searches
Search for pages in different languages
Search for pages located in specific countries
Search for pages under a certain domain
Search for pages LIKE a certain page (eg like:http://www.yahoo.com/)
DirectHit – 5/10:Search for “phrases”
Boolean Searches (AND, OR, NOT)
Search for text in only title or body
Search for pages in different languages
Search for pages located in specific countries
Specific Results for Additional Services:AltaVista:Online shopping guide
Suggested searches. Searches based on what you have searched for
Other searches – whitepages, yellowpages, maps/directions etc
Hand-picked categories or channels for web browsing
Personalisation of search results, layout, information etc
Google:Hand-picked categories or channels for web browsing
Personalisation of search results, layout, information etc
DirectHit:Online shopping guide
Suggested searches. Searches based on what you have searched for
Other searches – whitepages, yellowpages, maps/directions etc
Hand-picked categories or channels for web browsing
Personalisation of search results, layout, information etc
Specific Results for Layout and Design:AltaVista:Easy navigation – 3
Good Aesthetics – 2
Quick PAGE load time – 1
Google:Easy navigation – 2
Good Aesthetics – 1
Quick PAGE load time – 3
Directhit:Easy navigation – 3
Good Aesthetics – 3
Quick PAGE load time – 1
|