One thing that's great about Google is that its paid services are constantly updated with new features, but users don't have to pay more. For example, when it was launched in February 2007, Google Apps Premier Edition didn't include Google Sites, a presentation app, message recovery, email migration and Gmail accounts had 10 GB of storage. For the same 50ドル/user/year, Google Apps Premier Edition offers a lot more features, so the incremental cost of significantly upgrading a product is very low.

Google Site Search, a service previously known as Custom Search Business Edition, falls in the same category. Google offers a hosted alternative for small businesses that want a powerful site search, but don't need all the features offered by Google's expensive search appliances, which are mostly useful for intranet search. Google Site Search is based on the Custom Search platform, but is fully customizable through XML feeds, includes support and doesn't have ads (they're actually optional).


One of the major problems with Google Custom Search was that it used the same index as Google's web search engine, so the site search was just a restriction of Google's web search results. Google realized that people were unhappy with these limitations and decided to create a separate Custom Search index. "We're now maintaining a CSE-specific index in addition to the Google.com index for enhancing the performance of search on your site. If you submit a Sitemap, it's likely that we will crawl those pages and include them in the additional index we build."

The most important new feature offered by Google Site Search is an enhanced indexing, but Google is quick to mention that this doesn't affect Google's main index. "Any additional pages in the Google Site Search index aren't included in Google's index and won't impact a site's behavior in the Google search results."

The other 3 additions to Google Site Search offer simple ways to influence search results. You can upload a dictionary of synonymous to improve Google's query expansion, it's also possible to give more weight to recent web pages or to pages from a specific section.

Unlike many of its competitors, Google Site Search still can't index password-protected pages and it doesn't let you decide how often to index your site's pages, but the price is hard to beat: starting from 100ドル/year for 5,000 documents.

If you don't need support or advanced customization, Google Custom Search is still free. There's also a competition for winning a subscription to Site Search and Google T-Shirts: the only condition is to submit the address of a site that doesn't have search features (quickly remove the search box!).

Google has been experimenting with letting users reorder and remove search results. This may seem strange if you think that Google cares a lot about algorithmically ranking search results. The new options allow you to promote some of the results at the top of the page, hide the results you think are not relevant and add new web pages that are missing from Google's results. These changes are saved in your Google account and only influence your results. For now, the users who see the experimental feature are randomly selected.


These new personalization options can be partly recreated using a custom search engine. You can build a custom search engine for the entire web, that should include all the sites from Google's index. Every time you find web pages or web sites that are not very useful, but have good rankings, you can edit the search engine and them to the exclusion list.


To promote search results, check this option in the custom search engine's settings: "Add my Subscribed Link to this Custom Search Engine". Then create subscribed links for some of the terms you search often. Alternatively, you could use Google Spreadsheets to define a list of subscribed links. The subscribed links are also included if you use Google's standard search engine, but they're displayed after the third search result.


The custom search engine lacks the user interface options from Google's experimental feature, but it's a pretty powerful way to customize your search experience.

Useful links:
Manage your custom search engines
Manage your subscribed links

{ The first screenshot is from Nimish. }

So you've found a great web page that lists a lot of interesting links about a certain topic and now you're ready to explore them, but you don't have enough time to read everything. A good option would be to create a custom search engine, but it's too time-consuming. A faster way is to create a custom search engine on-the-fly from all the links included in the page you've found. That means you'll have a search engine restricted to all the domains, subdomains and web pages linked from your page.

For example, del.icio.us search results pages are a good place to obtain a list of authoritative sites on a topic like AJAX. Once you have the del.icio.us URL, paste it at Google's on-the-fly search engine factory and enter your query.


Other good starting points could be: Wikipedia articles, other wikis, directories, scholar papers, pages with many references etc. Maybe Google should create a separate search engine for pages that have a lot of external links related to a topic (like these pages).

I reported in October that Google added Subscribed Links to the preferences page and renamed them as search add-ons. Google removed that feature a couple of days after I wrote that post, but now it's back for good, at least in the US English version of Google.

You'll notice at the bottom of the preferences page a list of subscriptions and a link to a directory of subscriptions. Each Subscribed Link matches some of your queries (for example, queries about weather, nutrition, traffic) and shows information extracted from a database. It's like a Google OneBox created by third-parties, but you can decide if you want to see it and there's always an option to unsubscribe.


According to the FAQ, "using Subscribed Links, you can add information created by providers you trust to your Google search results pages. Whenever you search on Google in an area of their expertise, you'll see a custom result from those providers in your search results. (...) Your Subscribed Links will appear in the fourth search result position", even if they aren't counted as a real search result.

Here's what happens if I subscribe to the Nutrition Facts knowledge source and search for [calories in cream of celery] or even [cream of celery]:


Even if other results offer similar information, I get the benefit to find what I want without having to click on a search result. In some cases, it's not necessary to enter the context of my search because my subscriptions already provide that context ([cream of celery] shows information about calories because I'm interested in nutrition).

The Subscribed Links are some special search results with custom snippets and a limited range of queries that are displayed based on your preferences. They allow you to perform specialized searches on a general-purpose search site.

And if you're not satisfied with the limited number of Subscribed Links that are available, you can create one and promote it on your site.

Google uses many signals to rank search results and, in some cases, it filters some of them based on your location, device or preferences. Here are some ways to disable these filters or to create custom filters.

1. Google automatically expands some queries and also includes results that contain different morphological forms or related words. If you search for [av], Google will highlight antivirus and anti-virus. To prevent this, add + in front of the word: [+av]. This is also useful if a certain keyword is very important and must be included in each search result.

2. Google filters duplicate search results, so if you want to find a better estimation for the number of search results add &filter=0 at the end of Google's URL. This parameter disables the following two filters:

* Duplicate Content — If multiple documents contain the same information, then only the most relevant document of that set is included in your search results.

* Host Crowding — If there are many search results from the same site, Google may not show all the results from that site or may show the results lower in the ranking than they otherwise would have been.

3. Here are two useful URLs for Google's homepage:

* http://www.google.com/webhp - if Google redirects you to the mobile version, but you want the standard homepage, this URL bypasses the detection

* http://www.google.com/ncr - if Google redirects you to a localized version, based on your IP address or your browser's settings, you can go to the global version using this URL (or by clicking on "Google.com in English").

4. To introduce a local bias, add the gl parameter to Google's search results URL. For example, http://www.google.com/search?q=roma&gl=it shows the results for [Roma], but gives better rankings to pages related to Italy.

5. If you don't like Google's personalized results you can log out from your Google account, disable the Web History service or turn off the personalization bias by adding &pws=0 to Google's URL. Note that the parameter is not persistent and it only works for the current search.

6. To restrict your search to the high-quality (?) web sites included in the Open Directory Project, you can append &cat=gwd/Top to Google's URL or perform your search at Google Directory.

7. Find the right custom search engine for your query and enter your query there. A good custom search engine restricts the search space to a number of authoritative sites from a domain.

8. Use the site: operator to restrict your search to a TLD (like .edu), domain, subdomain or even a pattern. For example, to restrict your search to YouTube pages that contain videos, try: [site:youtube.com/watch/v?=].

9. Search only the web pages and sites linked from a page using Google Co-op's on-the-fly feature. Use the "try it out" option.

10. A version of Google which identifies and prioritizes "search results that are more easily usable by blind and visually impaired users" is Accessible Web Search.

Other filters are available in the advanced search page: restrict the search results to pages written in a language, from a certain country, to non-adult sites, or to pages licensed using a flavor of Creative Commons.

Google Subscribed Links, launched as part of Google Co-op, is a service that allows you to extend Google's search capabilities with data from specialized sites. A Google patent explains the idea behind this service:

When users search for websites and other information on the Internet, they may not always obtain the desired results. In many cases, users must carefully formulate their queries in order to obtain the information they are seeking. (...) Website authors, on the other hand, often have such expertise and are able to formulate queries that will provide information likely to be of use to visitors of those websites. Specialized search functionality often appears on websites, allowing visitors to those websites to see search results tailored to the particular content they are likely to be interested in, based on the fact that they are searching from that website. For example, a website devoted to traffic information can interpret a query such as "Interstate 280" differently than a general-purpose search site would provide. The fact that a user is visiting the traffic information website means that he or she is interested in traffic information.

However, it is not always convenient for users to visit a particular website in order to perform such a specialized search. Users may wish to perform all (or most) of their searches on a general-purpose search site, such as www.google.com, without having to visit different websites to perform different searches. Accordingly, it would be useful if third-party content providers could enable specialized searches on general-purpose search sites.

Subscribed links are basically custom OneBoxes you choose to have on your search results pages. They're triggered when your query matches a pattern, for example a distance measuring subscribed link could be triggered when you search for "distance from London to Moscow".

This service is now ready for prime time, has been rebranded as Google Search Add-Ons and can be found at the bottom of Google's preferences page.


Google recommends you a list of popular search add-ons, but you can add others from a directory or develop your own. For example, I was recommended a Unix manpages add-on that shows-up if you type "man" followed by a Unix command. If I search for [man man], Google's results aren't extremely useful, but the new box that shows at the top matches my query with a pattern and displays a short description of the man command, followed by links to the entire entry.


Other things you can add: information about weather, ISBN look-ups, facts from Wikipedia or automatic translations. Here's the weather add-on, that shows more information than Google's OneBox, but there's no way to deactivate Google's weather OneBox so they both show up (you'll notice it's actually a Google Gadget):


The directory doesn't include too many add-ons (around 50), but the public launch of the service will increase its exposure and more websites will build their own add-ons. After all, if they convince you to subscribe to their service, they get a placement at the top of Google's search results for any query that matches their patterns. If you don't like the results provided by an add-on or if it shows-up too often, you can always remove it.


It's a great way to expose specialized content to those who are interested, a brilliant way of combining bookmarks with search results. Once again, Google shows it wants to become the central point for all your search-related activities and this service allows it to outsource specialized results to third-party developers.

Update: The new section from Google's preferences page has vanished, so maybe the feature wasn't ready for prime time.

I told you last week that Google is experimenting with a new feature that lets you place custom web pages at the top of search results for some queries. Even if this feature may not be released to everyone, there's a way to define custom results using Google Co-op's subscribed links.

For example, when I search for "onebox", I may want to read this page from my blog. The colored box from the screenshot below includes a user-defined web page.


You can define a list of web pages by creating a new Google Spreadsheet using this format:

* the first row contains: # subscribed links TSV file
* the second row contains: author
* all the other rows should contain four columns: in the first column type item, the second column contains your query, the third column includes the web address and in the last column you could write a small description of the web page.


Then you should publish the spreadsheet, enable "automatically re-publish when changes are made", click on "more publishing options", select TXT from the drop-down and copy the URL. To add this file to Google Co-op, enter the URL here and choose a name for your subscribed links. Google will visit the file in less than a minute and you could test if everything went OK by entering one of your queries at Google.com.

The nice thing is that you can edit the spreadsheet and Google will refresh the list of links in a couple of minutes. You can only define a custom link for a query and the web page doesn't replace the top search result: it's displayed in a special OneBox at the top of the page only for you and only when you're logged in.

Google launched a business edition of the Custom Search program that will allow small businesses to add powerful search features for their sites. Google offers the same benefits as in the consumer edition, but also technical support, customization of search results using an XML API and no ads. The service is not very cheap: 100ドル per year for websites of up to 5,000 web pages, and 500ドル per year for websites of up to 50,000 pages, but it's much cheaper than other similar services or Google Search Appliance, a hardware solution that creates a local index and lets you search the intranet as well. The most affordable search appliance from Google costs 1,995ドル for 50,000 web pages. Here are the prices for other hosted site search solutions (* means that the product lets you control the crawling frequency, ** means the search tool also indexes password-protected pages):


Because Google uses its index to deliver results for the custom search engine, it can't guarantee it will index a web site or a certain page. Most other products try to completely index the site and let you control how often they re-index the pages.

Google realized that the ad-supported model, that works very well for consumers, isn't an option for businesses, that want technical support, custom-tailored solutions and are willing to pay for this.

If you liked the way Google implemented search in most of their official blogs, now you can get something similar without writing complicated code. Blogger blogs that migrated to layouts can include a search widget that shows results from the blog, its blogroll, the pages linked from the blog and the web. The feature uses the AJAX Search API and Google Custom Search, so the search results are loaded at the top of the current post, without reloading the page.

The widget is available at Blogger in Draft, the place where Blogger showcases experimental features.


The widget may show sponsored links, but you can't earn money from them. In fact, Google doesn't earn money from the ads either and includes them for free. "Ads in the AJAX Search API are basically an experiment still. We are trying to figure out if ads are useful in this model, and how do our customers want to use the ads and benefit from the ads? We don't know the answers yet. Yeah, Google is not benefiting from these ads, because advertisers are not being charged for them. The only guys that actually benefit, to be honest with you here, are the advertisers. They are basically getting free impressions," explained in an interview Mark Lucovsky.

Blogger also added a poll widget and enclosures for podcasts and videocasts. "Enclosure links let you turn your blog into a podcast. If you've uploaded your audio or video to the web, you can link to it as an enclosure so that your readers can subscribe and download it using iTunes or another podcatcher."

Google launched an interesting way to search for custom search engines created using Google Co-op: a Google Base structured search that only includes the most popular search engines. The cool thing is that you can restrict the results based on the search engine's name, language, its description, the most popular queries or the number of sites that are included. "If you're interested in finding a search engine to contribute to, search specifically for search engines that allow volunteers. For instance, if you're most interested in non-profit organizations, search only for non-profit search engines," suggests the CSE Blog.


Google Base has a very powerful search interface and it would be nice to see it in other products like Blog search, Gmail or Google Calendar. When you perform a search or select an option, Google Base will only show the options that are available in the current state and that's useful to dynamically build a complex query.

Google's custom search engines have received a huge upgrade. Now when you add a site to your search engine, you can also include all the pages linked from that site. So even if you use the custom search engine only for a single site, you can transform it into a whole ecosystem of pages relevant to your site.

All you have to do is choose the option to "dynamically extract links from this page and add them to my search engine" for your site. You have three options: include only the individual pages linked from your site, include the subdomains or the entire sites, even if you only link to a page.


If you don't have this feature you can test it at this page or in this blog's search box.



A search for "Safari" returns the homepage of Apple's browser and a site that offers a way to preview a site in Safari before my post about Safari and that's a fair enough. That post linked to both sites that are ranked higher.


The feature has already been available in Lijit, a site that let you "easily create your own search engine, which searches your blog, blogroll, bookmarks, photos, and more", but now it's a simple option in Google Co-op. Here are the results for "Safari" in a custom search engine built using Lijit.

You can use this for feeds, directories or for sites that collect a lot of interesting links. Bloggers could dig deeper in their archives and rediscover useful links, while readers have more comprehensive search results.

An important drawback is that if you enter the homepage of a site, Google will retrieve the links only from that page, even if it will continue to monitor it for new links. Also the search seems to be somewhat slower and the results may appear diluted.

But, as the Custom Search blog says, lazy people can rejoice. "If you have a blog or a directory-like site and don't feel like listing all of the URLs you want to search across, you can leave the work to us. With this new feature we'll automatically generate and update your CSE for you."

Last month, Google started to test a new way to present search results that groups the pages by category and displays the top two results from each category. For example, if you search for a movie, you could see reviews, news, blogs, encyclopedia pages, stores. Each category can be expanded.

Google already shows refinements for a small list of topics like: health, computer games, travel, cars and photo equipments. "Topics are specific search areas that Google is developing with the help of expert contributors. Contributors to topics annotate websites that they think are especially useful, relevant, or authoritative to a topic with pre-defined category labels." But not only experts can contribute. You can select the topics that interest you and use Google Marker to annotate sites as you browse the web.

Google also has a list of general labels and those labels were used to categorize search results in the experimental view from last month. To annotate pages with the general labels, you can create a custom search engine and add labels from the list. I made a custom search engine to test the quality of the labels and I noticed that in many cases the labels didn't describe well the content of the page.

Many people enter very broad queries (like "Vista") and search engines try to guide them to the right results using refinements. One way to refine a query is by adding another word that eliminates the ambiguity (for example, "Vista reviews"), but this is not always the best idea because someone could review Windows Vista without including the word "review(s)" in the page. Having a list of pages that offer reviews could save the day.

Hakia, a semantic search engine that will officially launch this year, uses categories for general queries, like iPod or Italy, but they appear to be dynamically generated and look more like an encyclopedia entry.

It seems that Google Co-op's secret mission was to rebuild at a larger scale Google Directory (ODP), which also organized the web by topic into categories. One thing is for sure: this year, Google's interface will undergo radical transformations.

Related:
New ways to visualize search results
SearchMash

Google Subscribed Links are a way to build your own OneBox results, by defining direct answers for a list of queries. For example, you could check the status of a flight or find a book if you know the ISBN by entering a simple query.


Now the subscribed links can have a richer interface by including Google gadgets. You can't add inline gadgets or built-in gadgets like the preview of Gmail's inbox, but the good news is that your gadget can access the query and display something relevant.

A good example of gadget that's now included in a subscribed link is Google Translate. After subscribing to the gadget, you can translate some text directly from Google's search box. A query like [translate "I love you" to German] will trigger a direct answer.


This could be used to show search results from other search engines, the latest posts from a blog, solve complicated equations, show relevant photos from your Flickr albums and more.

Google lets you create custom search engines, by defining a list of sites and web pages that are experts on a topic. For example, you love poetry and you know a great deal of sites that aggregate poetry. Compile a list of those sites (you can also use URL patterns to better describe the pages) and create a poetry search engine.

For the moment, Google doesn't aggregate all the custom search engines (I made a simple tool for that), but here's an idea that could improve search results. Google likes to use OneBox results to glue information from different sources. A OneBox for custom searches could match your query with the most popular custom search engines based on their description, their most frequent queries and their list of sites. Then it could show the top results from these search engines for that query.


This concept of custom search engines might replace web directories, that can't scale because of the exponentially growth of the web pages and the limited number of editors. With custom search engines, anyone could become an editor for his/her topics of interest.

Here's an interesting description of Google Directory, a Google product that searches the sites listed by Open Directory Project, which is now almost obsolete:

"Google's directory engine also lets you search within a category once you've decided on the specific subsection of the web that interests you. In this way, you'll get only responses that fall within that category. For example, you may want to search for teams named Cougars within the college basketball section of the directory only, instead of across the entire web."

Instead of creating hierarchical categories, Google could use the labels you can attach to each site.

Google doesn't provide a way to find custom search engines built by other users, but you can restrict a normal Google search to the homepages of those search engines. You just have to add [site:google.com inurl:coop/cse] to your query. For example, you can use [site:google.com inurl:coop/cse css] to find custom search engines for CSS.

This might be useful if you have to do some research on a narrow topic and the regular Google search doesn't return good results. By restricting your search to a list of sites about CSS or web programming, you'll definitely find great results even for ambiguous queries like "position" or "display".

To make things even easier, I created a custom search engine that searches through more than 100,000 custom search engines.

If you have a custom search engine at Google Co-op and you're not the only one who uses it, you might be interested to know how popular it is. Google offers a report that shows the number of queries for the last day, week or month and a list of popular queries (in case there are queries that occurred sufficiently often).

There's also a list of the most used custom search engines.


{ Via Radioactive Yak. }

If you use del.icio.us to save your bookmarks, you might want to search inside those pages, not just their titles, tags and descriptions. While you could import them into Google Bookmarks, there's an even more interesting solution: create a custom search engine from your bookmarks.

Vik Singh, who helped Google build the Co-op platform, developed a powerful tool that does more than only importing your bookmarks:

* it generates refinements from the most important tags using a machine learning algorithm

* you can expand links that point to a site, so you can search the entire site

* you can restrict the list of bookmarks to a single tag

* boost the ranking of a page depending on the number of bookmarks.

Related:
Google Custom Search Engines
Add search to Google Reader

Search is one of the most requested features for Google Reader. Until this feature is implemented, there's a simple workaround: you can use Google Co-op to create a search engine restricted only to the sites you're subscribed to. Here's how to do this:

1. Go to this page that shows your list of subscriptions (in OPML format) and save it to your desktop.

2. Create a search engine at Google Co-op. Enter a name, a description, a bogus keyword (like "blogs"), a bogus site in the list of sites to search (like "www.example.com"). Click "Next" and then "Finish".

3. Now click on "control panel" next to the search engine you've just created. Select "Advanced" and upload the file downloaded from Google Reader in the annotations section.

4. (Optional) To access your search engine, you have more options:

* Integrate it into Google Reader using a Greasemonkey script

* Go to "Code" section and copy some code you can add to your site or add a gadget to Google Personalized Homepage.

* Go to "Preview" section and bookmark the homepage of your search engine.

* You can also go to the homepage of your custom search engine, right-click on the search box and add the search engine to your browser (for example: if you have Google Toolbar, right-click and select "Generate custom search"; if you use Opera, select "Create search").

You can use this to search the sites from any OPML file (most feed readers have an option to export the subscribed feeds to OPML). Of course, if you add or remove feeds, you need to reupload the OPML to Google Co-op to keep the search engine in sync.

Update (Sept. 2007): Google Reader added search.

Google knows that people are selfish and act in their own interests. Google also knows that good ideas don't have to be killed if they're not successful, they just need a new twist.

Google Co-op, a place "where users can contribute their knowledge and expertise to improve Google search for everyone", didn't have too much success. Google wanted to collect some meta information about web pages, so it would be able to refine results for general queries. People would find good sites and add one or more labels from a predefined set. The task is cumbersome and the motivation almost non-existent.

If people can create their own search engine, the motivation increases:
* they actually create something tangible
* the search engine might actually become successful
* people can work together for a smaller common cause with someone they trust
* there's a monetary incentive

While "search engine creators" improve their sites and add more labels and sites, Google collects this information, combines it, and uses it to label the entire web. If you think about it, the process has a lot in common with Google Image Labeler (the ESP game).

"Google is launching Customized Search Engine, a program where individuals' and organizations' can tailor the search results on their personal Web sites to better accommodate audience requests. The Custom Search Engines also enable users to monetize their search results using Google's AdSense – a program where Web site publishers can make money when visitors click on the Google ads displayed on their site. Google officials say the Custom Search Engine also allows its users to modify their search results so they are branded to look like their own."

Google Custom Search Engine, available at www.google.com/coop/cse lets you actually create a search engine. You give it a name, a description, some keywords, a list of sites where to search, and you may allow other people to improve the list. You can restrict the search to the list of the sites you specified, or just emphasize them in the search results. The style of the search result pages can be customized and you can add a logo, like in AdSense for Search. The search results can be hosted on a Google page or can be displayed in a page from your site. You can even use Google Ajax Search instead of the standard formats. The ads displayed in the SERPs can be connected to you if you have an AdSense account.

The search engine can be improved using annotations (a set of labels that describe a page), that work like in Google Co-op, and can be added using Google Marker.

Here are some custom search engines from: Macworld, Intuit and a small search engine about Google created by me.

The new product is a combination of AdSense for Search and Google Co-op that makes it easy to create a search engine focused on specific topics.