Finding things on the web with BOSS

Finding things on the
web with BOSS

Christian Heilmann | http://wait-till-i.com | http://scriptingenabled.org

Open Hack Day 2009 - Bangalore, India

Innovation is improving the
current state of affairs.

In the best of all cases this
means that it gets easier for
the person using a product
to reach a goal.

This can be achieved by
connecting several different
products and turning them
into one. (pssst... Mashup).

I’ve seen this followed
cleverly on several levels
here in India.

Finding things on the web with BOSS

BOSS is your chance to make
the web an easier to navigate
space.

You can help us turn a search
engine into a find engine.

Web search is broken!

http://luckyrobot.com/2009/02/06/search-is-broken-%E2%80%93-really-broken/

http://www.borthwick.com/weblog/2009/02/05/creative-destruction-google-slayed-by-the-notificator/

Back in the
days the
web was
small and
largely static
documents.

Nowadays it
is huge and
its content
and the
publication
speed
increased a
lot.

This is one challenge of
search engines these days.

The others are old, but also
increasing

Now, say you have
the most awesome
idea of a search
engine that works
around these
issues.

Where you will get stuck is
the overhead of indexing,
storing and filtering the data
of the web.

And this is where BOSS
comes in.

It is an API interface to our
data stores for search.

BOSS is Build Your Own
Search Service:
http://developer.yahoo.com/search/boss/

To use it, you need a
Application ID:
https://developer.yahoo.com/wsregapp/

And there is full
documentation available:
http://developer.yahoo.com/search/boss/boss_guide/

You can get the code
examples I will show here:
http://isithackday.com/hacks/
bangalore/bosscode.zip

Say you want to search the
web for donkeys.

Because
Donkeys alright then ...
... oh
Rock!

Using BOSS you can do this
with a REST API and display
the results any way you want!

The REST API:
boss.yahooapis.com/ysearch/{type}/v1/
{search}

The REST API:
{search}

type is what you want to
search:
web: the interwebs
news: new stuff
images: pictures

The REST API:
{search}

search is the term to look for
(url-encoded)
Put “” around terms to ensure the right order, f.e. “donkey
fur” (you don’t want to see cats, do you?)
Filter with a -, f.e. donkey -shrek
Restrict to a site using site:, f.e. donkey site:flickr.com

The REST API:
{search}

Other parameters:
appid: your app ID (needed)
count: amount of results
start: where to start the counting
region / lang: country and language
format: xml or json
sites: restrict to certain sites (comma separated)

Web search REST API:
boss.yahooapis.com/ysearch/web/v1/{search}

Extra parameters:
filter: To filter out nasties, use filter=-porn-hate
type: to search different types. You can use html, text, pdf,
xl, msword, ppt or groups like msoffice and nonhtml. You
can also do a type=msoffice,-xl

Image search REST API:
boss.yahooapis.com/ysearch/images/v1/
{search}

Extra parameters:
filter: no nudies
dimensions: all, small, medium, large, wallpaper,
widewallpaper
refererurl: all images in that url
url: image at that url

News search REST API:
boss.yahooapis.com/ysearch/news/v1/{search}

Extra parameters:
age: how old the news are in days. Last five days would be
“5d”

There are restrictions how to
display results and
information as to what data
comes back.

For this, read the guide!
http://developer.yahoo.com/search/boss/boss_guide/

The easiest way to use BOSS
is using JavaScript.

http://boss.yahooapis.com/
ysearch/web/v1/donkeys?
format=json&appid={id}

{quot;ysearchresponsequot;:
{quot;responsecodequot;:quot;200quot;,quot;nextpagequot;:quot;/ysearch/web/
v1/donkeys?
format=json&appid=[...]&start=10quot;,quot;totalhitsquot;:quot;492
215quot;,quot;deephitsquot;:quot;15700000quot;,quot;countquot;:quot;10quot;,quot;startquot;:quot;0
quot;,quot;resultset_webquot;:[{quot;abstractquot;:quot;Hyperlinked
description of the domesticated mammal discussing
its appearance, relationship to horses, economic
<b>...</b> horses and <b>donkeys</b> were
brought back <b>...</b>quot;,quot;clickurlquot;:quot;http://
lrd.yahooapis.com/
_ylc=X3oDMTU4b2NoaDR2BF9TAzIwMjMxNTI3MDIEYXBwaWQDV
Fg2YjRYSFYzNEVuUFhXMHNZRXI1MWhQMXBuNU84S0FHcy5MUVN
YZXIxWjdSbW1WclpvdXo1U3Z5WGtXc1ZrLQRwb3MDMARzZXJ2a
WNlA1lTZWFyY2hXZWIEc2xrA3RpdGxlBHNyY3B2aWQDR3lDaEg
wU081cTlmSktUNG1ndTVUUUJNdlNjaS4wa1ZUVndBQVF5Sw--
/SIG=11820sato/**http%3A//en.wikipedia.org

To use this across domains,
simply define a callback
parameter:

ysearch/web/v1/donkeys?
format=json&callback=foundd
onkeys&appid={id}

founddonkeys({quot;ysearchresponsequot;:
{quot;responsecodequot;:quot;200quot;,quot;nextpagequot;:quot;/ysearch/web
/v1/donkeys?
format=json&callback=founddonkeys&appid=TX6b4XHV3
4EnPXW0sYEr51hP1pn5O8KAGs.LQSXer1Z7RmmVrZouz5SvyX
kWsVk-
&start=10quot;,quot;totalhitsquot;:quot;492215quot;,quot;deephitsquot;:quot;15700
000quot;,quot;countquot;:quot;10quot;,quot;startquot;:quot;0quot;,quot;resultset_webquot;:
[{quot;abstractquot;:quot;Hyperlinked description of the
domesticated mammal discussing its appearance,
relationship to horses, economic <b>...</b>
horses and <b>donkeys</b> were brought back
<b>...</b>quot;,quot;clickurlquot;:quot;http://
lrd.yahooapis.com/
_ylc=X3oDMTU4cG05cXJwBF9TAzIwMjMxNTI3MDIEYXBwaWQD
VFg2YjRYSFYzNEVuUFhXMHNZRXI1MWhQMXBuNU84S0FHcy5MU
VNYZXIxWjdSbW1WclpvdXo1U3Z5WGtXc1ZrLQRwb3MDMARzZX

All you then need to do is put
this url in a script node and
write the founddonkeys
function:

<div id=quot;searchresultsquot;></div>
<script type=quot;text/javascriptquot;>
function founddonkeys(o){
var donkeys = o.ysearchresponse.resultset_web;
var results = document.createElement('ul');
for(var i=0,j=donkeys.length;i<j;i++){
var item = document.createElement('li');
var link = document.createElement('a');
var abstract = document.createElement('p');
link.setAttribute('href',donkeys[i].clickurl);
link.innerHTML = donkeys[i].title;
item.appendChild(link);
abstract.innerHTML = donkeys[i].abstract;
item.appendChild(abstract);
results.appendChild(item);
}

document.getElementById('searchresults').appendChild(r
esults);
}
</script>
<script type=quot;text/javascriptquot; charset=quot;utf-8quot;
src=quot;http://boss.yahooapis.com/ysearch/web/v1/donkeys?
format=json&callback=founddonkeys&appid=xxxquot;></script>

First of all - without
JavaScript there are no
donkeys!

Secondly - you can only find
donkeys!

The solution: Event Handling
and dynamic script
generation.

<p>Warning: this is terrible code, USE A LIBRARY INSTEAD!</p>
<ul id=quot;searchesquot;>
<li><a href=quot;http://search.yahoo.com/search?va=donkeysquot;>
Search for Donkeys
</a>
</li>
<li><a href=quot;http://search.yahoo.com/search?va=kittensquot;>
Search for kittens
</a>
</li>
</ul>
<div id=quot;searchresultsquot;></div>

var APIkey = 'TX6b4XHV34EnPXW0sYEr51hP1pn5O8KAGs'+
'.LQSXer1Z7RmmVrZouz5SvyXkWsVk-';
var searchlinks = document.getElementById('searches').
getElementsByTagName('a');
for(var i=0;searchlinks[i];i++){
searchlinks[i].onclick = function(){
var searchterm = this.href.split('va=')[1];
var url = 'http://boss.yahooapis.com/ysearch/web/v1/' +
searchterm + '?format=json&' +
'callback=founddonkeys' + '&appid=' + APIkey
var s = document.createElement('script');
s.setAttribute('type','text/javascript');
s.setAttribute('src',url);
document.getElementsByTagName('head')[0].appendChild(s);
return false;
}
}
</script>

Using the YUI library (YUI3
JavaScript and CSS grids) you
can easily make this much
cooler:

To make using BOSS with
JavaScript easier, I’ve written
a BOSS wrapper called
YBOSS:
http://icant.co.uk/sandbox/yboss/

<div id=quot;resultsquot;></div>
<script type=quot;text/javascriptquot; src=quot;yboss-lib.jsquot;>
</script>
<script type=quot;text/javascriptquot;>
YBOSS.get({
searches:'search,news',
query:'obama',
count:10,
callback:seedpics
});
function seedpics(o){
var all = '<h4>Web Sites</h4>' + o.webHTML +
'<h4>News</h4>' + o.newsHTML;
var out = document.getElementById('results');
out.innerHTML = all;
}
</script>

For server side code there are
code examples available at:
http://www.saurabhsahni.com/boss-
examples.zip

There’s also the Python
mashup framework that
allows for SQL for remixing
arbitrary XML/JSON sources:
http://developer.yahoo.com/search/boss/mashup.html

And there’s an easy way to
deploy BOSS apps at Google
App Engine:
http://zooie.wordpress.com/2008/08/04/yahoo-boss-
google-app-engine-integrated/

All this
has been
around for
a while.

Here are some new things
added lately:

People are trying to make the
web a less messier place by
adding semantic data to
HTML documents.

Using
SearchMonkey
technology
BOSS now
lists this
information in
the results.
http://www.flickr.com/photos/glenscott/3273401181/

view=searchmonkey_feed

view=searchmonkey_rdf

http://developer.yahoo.com/search/boss/
structureddata.html

Using the view=keyterms
parameter you can get
keywords associated with
each result.

In order to get longer
descriptions of each result
you can now use the
abstract=long parameter to
get up to 300 characters
instead of 130.

Another thing we’ve done is
using the Yahoo Site Explorer
and bundle it with BOSS.

This way you can now get
page information and page
inlink information with two
new BOSS services.

ysearch/se_inlink/v1/{URL}?
appid={APPID}
Gets you a list of pages that
linked to the URL provided.

ysearch/se_pagedata/v1/
{URL}?appid={APPID}
Gets you a list of all pages
belonging to a domain in the
Yahoo! index.

So what has been done using
BOSS?

The client side is where
strange things happen.

http://isithackday.com/hacks/web-the-adventure/

The motherlode of BOSS
implementations:
http://mashable.com/boss/
http://delicious.com/tag/bossmashup

Add yours by tagging it with
“bossmashup” on Del.icio.us!

THANKS!
Keep in touch:
Christian Heilmann
http://wait-till-i.com
http://scriptingenabled.org
http://twitter.com/codepo8

Finding things on the web with BOSS

More Related Content

Finding things on the web with BOSS