[philiptellis] /bb|[^b]{2}/
Never stop Grokking


Showing posts with label twitter. Show all posts
Showing posts with label twitter. Show all posts

Monday, May 02, 2011

Twitter, SSL and #poopin

Watching the twitter feed about #jsconf shows a lot of people tweeting about #poopin. Turns out that someone's been stealing twitter cookies using a firesheep like tool and tweeting on their behalf. The tweets aren't malicious in nature, and are geared more at educating the user about the need to use SSL or some kind of encrypted tunnel when tweeting over untrusted wireless connections.

Here's the problem. Even people who do know the risks, and take the trouble to use twitter over SSL will get caught because of certain bugs with twitter's handling of their SSL pages.

If you visit https://mobile.twitter.com/, this is what you'd get: (shown in a browser so I could hilight the URL bar)

Click Sign in and this is what you get:

Sign in, and this is what you get:

Notice that the post sign-in URL is no longer https, but is now http.

At various points of time, trying this through my mobile phone, I get redirected from an https site to an http site when I do some of the following:
  • Replying to a tweet
  • Replying to a direct message (seems to be fixed)
  • Retweeting (with/without? JavaScript)

If you search through the page source on https://mobile.twitter.com/ for the string "http://", you'll find a few instances in comments, but then these interesting ones in a JSON object:
"twitterApiBase":"http://api.twitter.com"
"apiBase":"http://api.local.twitter.com:9000"
"twitterBase":"http://twitter.com"
"mobileBase":"http://mobile.twitter.com"
I haven't examined the code in detail to see how these are used, but it seems to suggest that at least some calls are going out over http, and since they're all on the twitter.com domain, your twitter cookies get sent along.

Now this is only the twitter mobile website. Mobile clients could be another matter, and the desktop site could also have problems. I haven't tested. Personally, I try to either use a VPN, or only tweet using SMS, but I have been caught by something like this before (at FOSS.IN/2010) which is when I started to study the problem.

Also, it doesn't matter if you've configured twitter to always use HTTPS. It still has this problem.

Monday, November 22, 2010

Stream of Collaboration and the Unified InBox

Back in 2003, I'd published a report on the state of computer mediated collaboration at the time. The report did not contain any original research, but was a list of references to other research on the topic. I was also working on ayttm and doing some work on automated project management at the time, which led to my talks on Fallback Messaging and Project Management with Bugzilla, CVS and mailing lists.

The state of technology has changed a lot over the years, and we're getting closer to the Unified Message InBox. The idea has been mulling in my mind for a while, and I've prototyped various implementations as a subset of what I call a stream of collaboration.

Communication

As technologically aware humans, we communicate in a variety of ways. Face-to-face, through the grapevine, handwritten letters and post-it notes, instant messaging, SMS, telephone calls, email, discussion boards, twitter, blogs, smoke signals, morse code using signalling lights, semaphore flags and more. Some of us prefer one form over another, and some of us will completely boycott a particular form of communication purely on principle, some of us won't use a form of communication because simpler methods exist and some of us will use a particular form of communication purely because it isn't the simplest one available. That's what makes each of us uniquely human. It also pushes us into groups and cliques that may or may not intersect. Who among you has a separate group of friends on twitter and facebook even though the two media are not vastly different? Do you also belong to a HAM radio club and a book reading group?

Now some forms of communication are well suited to automated archiving while others might end up being a game of Chinese whispers. Each form of communication also results in a different signal to noise ratio. Depending on the context of the conversation, accuracy and efficiency may or may not be a concern.

Degrees of Communication

Collaboration and Cooperation
Collaboration between a group of people involves communication, often requires archiving of all communications, and a high signal to noise ratio. Discussion lists, IRC logs, wikis, whiteboards, video/audio conferences, project trackers, bug trackers, source control commits, and sometimes email all provide good archiving capabilities. With proper time-stamping of each interaction, the archiving service can interleave events showing the exact chronological order of decisions being made. A Bayesian Filter, similar to the ones used for classifying spam can be used on a per topic basis to hide (but not remove) off-topic sections in an attempt to increase the signal to noise ratio. Once archived, even synchronous communication turns asynchronous [1]. Readers can later comment on sections of a communications log.

While some of the tools mentioned above are aimed at technical collaboration, many of them may also be used effectively for collaboration and support in a non-technical context [2,3] where immediate dissemination of information that can be archived and later used for reference is important.
Social Interaction
A form of communication that is more tolerant of a low signal to noise ratio, and in many cases does not require archival is social interaction. For example, at a party to watch a ball game, conversation may range from the actual events of the game to something completely irrelevant, and they're all socially acceptable in that context. Similarly, the corridor conversations at a conference may only be partially relevant to the subject matter of the conference. How does this translate to a scenario where people are geographically separated and need to communicate electronically rather than face to face?

Social television experiences [4], Co-browsing, LAN parties and digital backchannels [5] are examples where individuals communicate online while simultaneously engaging in a common task at geographically disparate locations. Computer mediated collaboration allows these individuals the ability to come close to the full party experience.
Casual Conversations
With more people moving their lives online [6,10], we're at a point where the volume of casual electronic conversations far exceeds that of technical collaboration. Fallback messaging is a good starting point to tie different forms of communication into a single thread, but it ignores current reality.

For starters, the fallback messaging idea assumes that users would use the same interface to communicate synchronously and asynchronously. In reality people use different methods for each. Asynchronous communication offers the ability, and often creates the necessity of longer messages and a larger amount of time devoted to communicating the message [7]. Should I say "Dear ..." or do I lead in with "Hi"? Should my background be pink or yellow? Do I want hearts, unicorns, birthday cakes or plain white in the background? Do I sign off with "Sincerely", "Regards", "Cheers" or "ttyl"? A different interface for each mode of communication makes it possible to customise the interface for the most common uses of that mode.

Fallback messaging was initially only applied to conversation between two persons, however a lot of casual communication happens between groups with the conversation sometimes forking into side-band conversations between a few (possibly two) members of the group and then merging back in to the parent conversation [7]. Some times the child conversation is made public to the group and some times it isn't. A messaging system must take this into consideration.

Enabling the Stream of Collaboration

Borrowed Ideas
The main idea behind fallback messaging was that service providers, communication protocols and user accounts were an implementation detail and the user should not be concerned with them. Users should only need to think of who they're communicating with, identifying them in any way they find comfortable (for example, using their names, nicknames, an avatar or favourite colour). The service needs to be able to determine based on context and availability, which messaging protocol to use.

Another idea that comes out of the original SMTP specification, is the now obsolete SOML [8] command. The SOML (Send Or MaiL) command would check to see if the recipient of the message was online when the message was sent, and if so, would echo the message to the user's terminal. If the user wasn't online at the time, it would append the message to their mailbox instead. Services like Yahoo! Messenger offer offline messaging capabilities that will hold a message on the server until the user comes online at which point it is delivered to them.

The problem with both these approaches is that they expect the user to create a message appropriate for the underlying service rather than the other way around. An entire email message in the case of SOML or a short text message in the case of Yahoo! Messenger. What we really need is a service that can decide based on what the user does, what kind of message needs to be sent, and how that message should be presented.

Facebook, GMail and Yahoo! Mail all offer a service where instant messaging and mail style messages can be sent from the same page, but with a different interface for each. Additionally, GMail provides the ability to see archived email messages and chat messages in the same context.
Proposed Interface
A messaging system is most useful to the user if it acts as a single point for them to reference and act on all past conversations, and provides an easy gateway to initiating new ones. The read interface must list all past conversations chronologically, with the ability to filter them based on topic and other participants in the conversation. It should be able to show where a conversation forked into separate threads, some of which may have been private, but all of which involved the user, and where these threads merged back into the primary stream.

The interface should include all kinds of communication including email, instant messages, co-browsing sessions, whiteboards, IRC, and anything else. Integrating with a service such as Google Voice, Skype or other VoIP solutions also allows it to tie in telephone conversations. Twitter and facebook notifications would tie in to this timeline as well.

The system should not rely only on message headers and meta-information, but also on message content to determine the topic and participants in a conversation [9]. Some participants, for example, may be involved indirectly, but not be privy to the details of the conversation, however it is useful to the reader to be able to filter messages with these details. Content analysis can also be used to identify messages as common Internet memes and tag them accordingly, possibly providing external links to more detailed information on the topic. Lastly, as has been proposed with fallback messaging, the system needs to aggregate all accounts of a contact across services into a single user defined identifier. It should still be possible for the user to identify which service is being used, but this should be done using colours, icons or similar markers.

Where are we today?

All major electronic communication services already provide some level of API acess to their messaging systems [11,12,13,14,15,16,17]. Services like Tweetdeck provide a single interface to multiple short message services (like twitter, identi.ca and the facebook wall), and Threadsy is supposed to unify your online social experience. Facebook seeks to unify email, IM, texting and Facebook messages through their own messaging service [18] which, like fallback messaging, is supposed to abstract out user accounts so that all you see is your contacts. I haven't seen the new messaging service yet, so I don't know if it also integrates with things like GMail, twitter, Yahoo! and other services that compete with Facebook [19]. If it does, that would be pretty cool. If it doesn't, there's an opportunity to build it. There are still other services that need to be tied in to enable full collaboration, but it doesn't seem too far away.

References

  1. Terry Jones. 2010. Dancing out of time: Thoughts on asynchronous communication. In O'Reilly Radar, October 26 2010. http://radar.oreilly.com/2010/10/dancing-out-of-time-thoughts-o.html
  2. Leysia Palen and Sarah Vieweg. 2008. The emergence of online widescale interaction in unexpected events: assistance, alliance & retreat. In Proceedings of the 2008 ACM conference on Computer supported cooperative work (CSCW '08). ACM, New York, NY, USA, 117-126.
  3. Sutton, J., Palen, L., & Shklovski, I. 2008. Back-Channels on the Front Lines: Emerging Use of Social Media in the 2007 Southern California Wildfires. In Proceedings of the Conference on Information Systems for Crisis Response and Management (ISCRAM).
  4. Crysta Metcalf, Gunnar Harboe, Joe Tullio, Noel Massey, Guy Romano, Elaine M. Huang, and Frank Bentley. 2008. Examining presence and lightweight messaging in a social television experience. ACM Trans. Multimedia Comput. Commun. Appl. 4, 4, Article 27 (November 2008), 16 pages.
  5. Joseph F. McCarthy, danah boyd, Elizabeth F. Churchill, William G. Griswold, Elizabeth Lawley, and Melora Zaner. 2004. Digital backchannels in shared physical spaces: attention, intention and contention. In Proceedings of the 2004 ACM conference on Computer supported cooperative work (CSCW '04). ACM, New York, NY, USA, 550-553.
  6. Donna L. Hoffman, Thomas P. Novak, and Alladi Venkatesh. 2004. Has the Internet become indispensable?. Commun. ACM 47, 7 (July 2004), 37-42. http://cacm.acm.org/magazines/2004/7/6471-has-the-internet-become-indispensable
  7. Rebecca E. Grinter and Leysia Palen. 2002. Instant messaging in teen life. In Proceedings of the 2002 ACM conference on Computer supported cooperative work (CSCW '02). ACM, New York, NY, USA, 21-30.
  8. Jonathan Postel. 1982. RFC 821: Simple Mail Transfer Protocol. IETF Network Working Group RFC Draft. http://www.ietf.org/rfc/rfc0821.txt, (August 1982).
  9. Dong Zhang; Gatica-Perez, D.; Roy, D.; Bengio, S.; "Modeling Interactions from Email Communication," Multimedia and Expo, 2006 IEEE International Conference on , vol., no., pp.2037-2040, 9-12 July 2006
  10. Rana Tassabehji and Maria Vakola. 2005. Business email: the killer impact. Commun. ACM 48, 11 (November 2005), 64-70. http://cacm.acm.org/magazines/2005/11/6081-business-email
  11. Yahoo! Mail API. http://developer.yahoo.com/mail/
  12. Yahoo! Messenger SDK. http://developer.yahoo.com/messenger/
  13. Twitter API. http://dev.twitter.com/doc
  14. Facebook Developer API. http://developers.facebook.com/docs/
  15. GMail API: http://code.google.com/apis/gmail/
  16. Google Voice APIs: http://thatsmith.com/2009/03/google-voice-add-on-for-firefox, http://code.google.com/p/pygooglevoice/
  17. Jabber protocol (Google Chat & Facebook Chat): http://xmpp.org/xmpp-protocols/
  18. MG Siegler. 2010. Facebook's Modern Messaging System: Seamless, History and a Social Inbox. Techcrunch, Nov 15 2010. http://techcrunch.com/2010/11/15/facebook-messaging/
  19. Alexia Tsotsis. 2010. Between Gmail, Twitter and now Facebook There is no Universal Inbox Yet. Techcrunch, Nov 22 2010. http://techcrunch.com/2010/11/21/facebook-messages-is-people/

Thursday, January 07, 2010

Handling document.write in dynamic script nodes

As a performance junkie, I generally want my pages to be as fast as I can make them. When I control the entire page, that typically means going all out. As a web user, I get annoyed when the little spinner on my browser keeps spinning even though I know that all the essential content on my page has loaded. When I redesigned my website late last year, I decided to address this issue.

Now if you take a look at my homepage, you'll notice a bunch of external resources:
  1. My Yahoo! Avatar
  2. My twitter feed
  3. My delicious bookmarks
  4. My upcoming events
  5. My dopplr badge
  6. My flickr photos
That's resources from six services whose performance I cannot control, nor rely upon. They're also six components on my page that aren't critical to the content of my page, and if one of them were unavailable for some amount of time, that wouldn't really hurt the purpose of the page.

Now I've been working with dynamic script nodes for a very long time to do background loading of scripts, but in all those cases, those scripts played nicely with other things on the page, and had JSONP responses. Not all the resources that I use now have this behaviour, so I had to come up with something else. Let's go through them one at a time.

To start with, I just included the javascript that all these services told me to include. Since the Avatar is just an image, I just used an img tag and left it at that. I've also never seen any performance issues with my Y! Avatar. The other services, however, all went down at some point or the other, and all had to be included as javascript.

Twitter

I started with the twitter widgets page. I copied the code, and pasted it where I wanted the twitter widget to show up. It's a lot of code, but that was okay to start with:
<script src="http://widgets.twimg.com/j/2/widget.js"></script>
<script>
new TWTR.Widget({
  version: 2,
  type: 'profile',
  rpp: 4,
  interval: 6000,
  width: 250,
  height: 300,
  theme: {
    shell: {
      background: '#333333',
      color: '#ffffff'
    },
    tweets: {
      background: '#000000',
      color: '#ffffff',
      links: '#4aed05'
    }
  },
  features: {
    scrollbar: false,
    loop: false,
    live: false,
    hashtags: true,
    timestamp: true,
    avatars: false,
    behavior: 'all'
  }
}).render().setUser('bluesmoon').start();
</script>
I then had to figure out if I could easily move the code to the bottom of my document so that it didn't block the rest of my page's load. twitter tends to go down more often than any of the other services.

I read through the source code for widget.js and found out that it creates a DIV into which it writes itself, however, you can create the DIV yourself, and pass its id to the widget constructor. The new code becomes:
<script src="http://widgets.twimg.com/j/2/widget.js"></script>
<script>
new TWTR.Widget({
  version: 2,
  type: 'profile',
  rpp: 4,
  id: 'twitter_widget',
  interval: 6000,
  width: 250,
  height: 300,
  theme: {
    shell: {
      background: '#333333',
      color: '#ffffff'
    },
    tweets: {
      background: '#000000',
      color: '#ffffff',
      links: '#4aed05'
    }
  },
  features: {
    scrollbar: false,
    loop: false,
    live: false,
    hashtags: true,
    timestamp: true,
    avatars: false,
    behavior: 'all'
  }
}).render().setUser('bluesmoon').start();
</script>
I could then create a DIV with an id of twitter_widget where I wanted the widget to go, and push the twitter code to the bottom of my page. This worked well. Kudos to Dustin Diaz for building a flexible widget, but really, you need to make those API docs available somewhere in that widget.

Anyway, we'll get back to twitter later, let's move on.

delicious

Finding the delicious badge was the toughest part. It's hidden on the help page under tools. Anyway, if you don't want to search for it, this is the link for delicious linkrolls.

After configuring the widget, I ended up with this javascript:
(I've split it onto multiple lines for readability)
<script type="text/javascript"
    src="http://feeds.delicious.com/v2/js/bluesmoon?
         title=My%20Delicious%20Bookmarks
         &icon=s
         &count=5
         &bullet=%E2%80%A2
         &sort=date
         &name
         &showadd"></script>
This code is not very nice, because I can't just move it elsewhere in the document. Looking at the page source, it seems that it initialises a Delicious namespace, and then loads two other javascript files. The first handles rendering of the linkroll, and the second is a JSONP feed of my links. Unfortunately, the first script chooses to write to the document using the document.write() javascript function.

Note that this function is the primary reason for javascript blocking page rendering -- it can modify the structure of the page as it loads. I decided to tackle this later.

Upcoming

Upcoming's badges are linked to from the page footer, so it was easy to start with this. I got the code, but decided to style it myself. The code points to the following javascript file (again wrapped for readability):
http://badge.upcoming.yahoo.com/v1/?
    badge_type=user
    &badge_size=sm
    &badge_layout=v
    &badge_styling=2
    &badge_no_background=
    &badge_venue=1
    &date_format_type=us_med
    &id=54783
The source of this badge shows that it also uses document.write() to write itself into the document. Solving this problem would tackle upcoming and delicious as well.

Dopplr

Dopplr was next, and was by far the easiest to work with. The account section points to a Blog badge which gives you a bunch of javascript to include onto your page among other things. The javascript link for my account is:
http://www.dopplr.com/blogbadge/script/6d1f4effa8fc5ac6db60160860ece8be?
    div-id=dopplr-blog-badge-for-bluesmoon
And the source code of that script had a lot of comments saying exactly what you can do with it. Brilliant. I just created a DIV with an id of my choice, and pushed this script to the bottom of the page.

Flickr

After the Avatar, delicious and upcoming, Flickr was the fourth Yahoo! property on my page. The previous two had already proved bad players, so my hopes weren't too high. Still, flickr has been good in the past, so I looked into it. The flickr badge page has a wizard to create the badge for you. This got me the following javascript:
http://www.flickr.com/badge_code_v2.gne?
    count=10
    &display=random
    &size=s
    &layout=x
    &source=user
    &user=57155801%40N00
Looking at the source of that showed the same old problem. document.write()

It was time to tackle this beast.

document.write()

To handle document.write(), I had to redefine what it did so that it would work asynchronously even after the entire page had loaded. I came up with this javascript:
document.writeln = document.write = function(s) {
 var id='';
 if(s.match(/\bupcoming_badge/)) {
  id='upb_events';
 }
 else if(s.match(/\bflickr_badge_image\b/)) {
  id='flickr_badge_wrapper';
 }
 else if(s.match(/\bdelicious\b/)) {
  id='delicious_widget';
 }
 else {
  id='overflow_div';
 }

 document.getElementById(id).innerHTML = s;
 return true;
};
It checks the content of the HTML to be written, and if it matches one of the three badges that I expect, I write it into the innerHTML of a DIV that I've already created for that badge. If it's something I don't recognise, then I assume that it doesn't matter where on the page it shows up, so just write it to an off-screen DIV.

I also had to make sure that there was no javascript being written out -- which is what the delicious badge was doing. In that case, I included the resulting javascript instead of including the javascript that the badge gave me.

Asynchronous loading

This worked, except that badges were still loaded in the order that I included the javascript, and if one blocked, all the others would wait, so I needed to make things asynchronous. This was easily accomplished using dynamic script nodes and handling the onreadystatechange event.

With this solved, I decided to parallelise downloads by moving all the code back into the head of the document. That way the script nodes would download in parallel with the document. Unfortunately, that also meant that some scripts might load up before the DIVs they need are available. Only dopplr handled this case well. For all the others, I had to handle it.

I ended up changing the write function above to defer until the required DIV was available.

Rather than include the entire Javascript here, I'll point you to the source file that you can see for yourself. It's not commented, but it's fairly small, so if you already know javascript, it should be easy to read.

With that, I had five badges on my page, all loaded asynchronously, and in parallel with each other and the page itself. My entire page is now reduced to four synchronous components: The HTML, CSS, Javascript and Avatar image (and possibly the favicon). Everything else is loaded asynchronously in the background and affects neither the firing of my onload handler, nor the browser's spinner.

You can see it in action on bluesmoon.info. Go ahead and view source.

Short URL: http://tr.im/docwritedynscr

Tuesday, October 27, 2009

Getting my twitter updates on this blog

I wanted my twitter updates to show up on the sidebar of my blog. At first I found that blogger already had a gadget for that, so I just included it. Unfortunately, this gadget loaded my timeline in an iframe. The iframe pointed to a script on someone else's domain, and every now and then that domain was unresponsive resulting in the rest of my page not loading.

It also did not work on Opera.

I decided to jump into twitter's API and figure out if I could get this to work on my own. I didn't bother with making it customisable, but if you want to reuse it, you'll need to change my username to your own. Here's what I did.

First, create a div that will hold my timeline:
<div id="twitter">
   <ul class="twitter-timeline">
   </ul>
   <div class="follow-me"><a href="http://twitter.com/bluesmoon">Follow me on twitter</a></div>
</div>

Put this wherever you want your timeline to go.

Next, write the Javascript that will draw the timeline. This is fairly simple:
function show_twitter(o) {
   var div = document.getElementById("twitter");
   var ul = div.getElementsByTagName("ul")[0];
   ul.innerHTML = "";
   for(var i=0; i<o.length; i++) {
       var li = document.createElement("li");
       li.innerHTML = o[i].text.replace(/@(\w+)/, "<a href='http://twitter.com/$1'>@$1</a>");
       ul.appendChild(li);
   }
}
I put this at the start of the document, but you can put it anywhere before you make a call to the twitter API, which is the next step:
<script src='http://twitter.com/statuses/user_timeline.json?id=bluesmoon&count=5&callback=show_twitter'
        type='text/javascript'></script>

A few things to note about this call:
  1. It's got my userid in it. It only works with userids that have public timelines
  2. count is limited to 5 items
  3. the callback parameter's value is the name of the function we defined in the previous step.
That's it. Put all this together and you have your latest 5 tweets on your blog. If your blogging software requires XML valid templates (like this blogger thing), then you'll need to either put your javascript inside a CDATA section or escape all quotes, &, < and >

Short URL: http://tr.im/bloggertwitter

...===...