On Web Applications, Web Architecture And Resource Identifiers

Table of Contents

• 1 On Web Applications, Web Architecture And Resource Identifiers
  • 1.1 Background
  • 1.2 Case Study: BBCiPlayer And BBC Backstage
  • 1.3 BBC IPlayer
  • 1.4 BBC Backstage
  • 1.5 How It Works At Present
  • 1.6 Observations
• 2 Conclusion

1 On Web Applications, Web Architecture And Resource Identifiers

1.1 Background

As we evolve from a Web of documents (Web 1.0) to a Web of applications (Web 2.0) and eventually Toward 2^W --- Beyond Web 2.0, key underpinnings of Web Architecture such as resource identifiers require careful re-examination. As a member of the W3C's Technical Architecture Group, I have been trying to define Web Architecture in the context of Web applications; a necessary first step toward that goal is to analyze how complex Web applications are implemented on the Web of today.

This article will carefully avoid abstract issues such as Resource vs Representation, URIs vs URLs, etc. - and instead focus on more practical considerations such as:

1. What is a URI and what can the user expect to do with it?
2. When dereferencing a URI, what pieces of software does one need to have to retrieve a useful representation of that resource?
3. Here, useful is defined from the perspective of the end-user. Thus, given a URI to a piece of media on the Web, relevant metadata is necessary but not sufficient to be useful - the user needs to be able to retrieve and play the media stream as well.

1.2 Case Study: BBCiPlayer And BBC Backstage

The British Broadcasting Corporation (BBC) provides streaming access to a large amount of radio and television content via a Web application called BBC iPlayer. In addition, BBC Backstage provides a rich data-oriented API to the underlying dataset in the form of linked data. Additionally, program schedules can be downloaded in a number of presentation independent formats such as XML, JSON and YAML. The remaining sections in this article detail what can (and cannot be done) with the information that is readily available from BBCiPlayer and BBC Backstage. In the process, we observe some design patterns (and anti-patterns) found on today's Web, and their efect on building richer Web applications from Web parts.

1.3 BBC IPlayer

Using the BBC iPlayer Web application requires:

1. A modern script-enabled browser such as Chrome, Firefox, Safari, or IE.
2. Browser plugins for media playback, such as Realplayer or Windows Media.
3. The Adobe Flash plugin for translating playback links on the BBC iPlayer page to their corresponding Realplayer or Windows Media resources.
4. Appropriate media player plugins based on the user's platform, e.g., Realplayer or Windows Media.

The Web application as implemented provides a rich, interactive visual interface that is sub-optimal for use from other programs.

1.4 BBC Backstage

Given the triple (radio-station, outlet, date) e.g.:

 (radio4, fm, 2010/04/14)

one can retrieve an XML representation of the program schedule using the URL:

 http://www.bbc.co.uk/radio4/programmes/schedules/fm/2010/04/14.xml

as documented on the BBC Backstage site. Alternative serializations such as JSON or YAML can be retrieved by appropriately replacing the .xml extension.

This retrieved schedule contains detailed metadata for each program that is broadcast, including a programme id pid that is used throughout the data store.

The BBCBackstage API assigns a persistent URI to each program of the form:

http://www.bbc.co.uk/iplayer/episode/<pid>

When retrieved, this persistent URI redirects appropriately to the BBC iPlayer page for that program. Note that the media streams for most programs are only available for a week.

As an example of the above, you can retrieve Midnight News from BBC Radio4 for April 14, 2010 by doing:

• Retrieve the program schedule via BBC Radio 4 FM Schedule
• Obtain the programme id pid for Midnight News b00rw6hf
• The BBC iPlayer URL for this episode is http://www.bbc.co.uk/iplayer/episode/b00rw6hf.

On the surface, this URL appears to satisfy many of the expectations that users might have:

1. Plays the relevant media when handed to a Web browser.
2. Can be bookmarked for later use (modulo the 1 week limit on archived media).
3. Can be passed around via email?

The final bullet above exposes some of the problems with the current implementation. Note the set of pre-requisites for the BBC iPlayer Web application enumerated earlier; all of these apply to the URI generated above.

1.5 How It Works At Present

It is instructive to turn on HTTP Request/Response tracking in the browser when opening URL http://www.bbc.co.uk/iplayer/episode/b00rw6hf. Here is a brief summary of some of the steps that the browser performs:

1. Receives an HTTP Response with content-type text/html.
2. The body of this response is an HTML document that in turn loads a number of JS libraries.
3. An embed tag in the retrieved HTML page invokes the Flash (shockwave) plugin.
4. The embedded shockwave player receives several mostly undocumented parameters that pass in details of the enclosing environment.
5. Once these steps have completed, the browser is automatically redirected to http://www.bbc.co.uk/iplayer/console/b00rw6hf, i.e., the earlier URI is transformed by replacing episode with console.
6. The HTTP conversation continues, and the browser is eventually sent to http://www.bbc.co.uk/mediaselector/4/mtis/stream/b00rw6g2 which resolves to the realplayer .ram file: http://www.bbc.co.uk//iplayer/aod/playlists/2g/6w/r0/0b/RadioBridge_intl_2300_bbc_radio_fourfm.ram.

Thus, the recipiant of the Midnight News URL would need to implement all of the above transforms (or have access to software that does those computations) in order to effectively consume the media stream that was addressed by the URL.

1.6 Observations

1. Web applications have gotten more complicated than they need to be: notice the multiple redundant layers between Flash, JS, HTML, and the complex interplay that results during the HTTP conversation between client and server.
2. Such complex interplay within multiple layers makes RESTful APIs difficult to achieve.
3. It is possible that the underlying media stream URLs are being intentionally obfuscated. It's hard to imagine anyone wanting to voluntarily inflict the pain inherent in steps 1..6 without a valid reason.
4. The obfuscation scheme makes it effectively impossible (on the surface) for interfaces other than the BBC iPlayer Web application to play the media.
5. Note on the surface in the above. As a testament to the robustness of the architecture of the Web, steps 1..6 can be hidden in a computational blackbox that surfaces a reliable URI that can be email.
6. As an implementation of the above, see this IPlayer Convertor found on the Web.
7. In addition to providing a simple HTML form that takes a pid and performs the trnaslation that happens during the client/server HTTP conversation, that site offers a persistent URL given a pid.
8. What's more, the persistent URL offered up by this convertor is guessable given the pid - this in its turn then becomes a RESTful API for accessing BBC media streams given a pid.
9. Thus, for the BBC Midnight News episode in question, the iPlayer convertor above serves up http://www.iplayerconverter.co.uk//pid/b00rw6hf/r/stream.aspx.
10. Notice that replacing %s in

    http://www.iplayerconverter.co.uk/pid/%s/r/stream.aspx
    

    in the above with a pid yeilds a persistent URL that can be handed off directly to a media player, where:
    1. The media player supports the codec in use.
    2. The media player supports the underlying streaming protocol, rtsp in this case.

2 Conclusion

So to conclude, let's ask the original question:

1. Given a URL, what can a user expect to be able to do with it, after having dereferenced the URI?
2. How does the user discover what software bits he needs in order to consume the received HTTP Response?
3. In Web 1.0 (Web of documents) the answer was simple --- HTTP Response header Content-Type specified the media type, which in turn specified what the recipiant needed to understand.
4. A recipiant who only understands mime-type text/html in this example is likely to flee screaming in terror if he makes the mistake of doing Show Source.
5. We all acknowledge that Show Source helped Web 1.0 succeed.
6. Q: What is the equivalent of Show Source that will help us collectively take the Web to the next level?

Author: T.V Raman <raman@google.com>

Date: 2010-04-14 Wed

HTML generated by org-mode 6.08c in emacs 23

Toward 2^W --- Beyond Web-2.0: The Rest Of The Story

A little more than a year ago, I blogged here about my thoughts around the second coming of the Web. I've now fleshed out some of those ideas in a full-length Communications Of The ACM article. You can find the rest of the story in terms of the thought process that led to the article on my personal Web site. Additionally, this blog post might also serve as a useful means to collect comments from across the Web, since the ACM Web site itself provides no such functionality. Note also that in general, users might find my personal copy of the article that I've placed on my Web site more accessible than navigating the Communications Of The ACM Web site.

Leveraging 2^W For Accessibility

A few weeks ago, I blogged about 2^W: The Second Coming Of The Web. That article talked about how once Web content was URL-addressable, it immediately became mash-up ready thereby opening up the way to creating integrated views of that content. Earlier this year, I blogged on Google's Official Blog about how Web-2.0 mashups presented an as yet unleveraged opportunity for providing accessibility across the Web by integrating custom views to present content in a form that was optimized to the end-user's needs. To reiterate --- this is in fact what Universal Accessibility was always supposed to mean --- rather than the somewhat short-sighted view that has become prevalent --- If it works with a screenreader it's accessible.

More recently, my Google officemate Charles L Chen and I have been working on a JavaScript based Web-2.0 framework called AxsJAX for injecting accessibility into Web applications. As an illustration of the ideas underlying 2^W coming to life in running code enabled via AxsJAX, we recently AxsJAX-ed the XKCD comic strip. Here, we bring together the XKCD sketches with the associated transcript to create a mashed-up view where the user gets to listen to the transcript while at the XKCD site. You can most easily experience this for yourself by installing a self-voicing plugin for the Firefox browser.

Note however that there is nothing Fire Vox specific about this XKCD mashup or any of the other AxsJAX extensions; all it needs is a relatively modern Web browser like Firefox that implements W3C ARIA and adaptive technology that has been updated to work with the event notifications raised by conformant AJAX applications.

Toward URL Equality For Web Clients And Web Servers

In response to my previous post on the second coming of the Web, Mark Birbeck commented on the power of the Web Command line and went on to right an extremely insightful article on Passing command line arguments via XPointer. His article provides a concrete example of how his Web application container (Sidewinder) leverages the XPointer syntax to pass parameters to the Web application that is being invoked. But his article also succinctly points out what has been a glaring inequality between Web servers and Web clients when it comes to URLs. The Web server and Web client started off as equals at the inception of the Web in this regard ? indicated server-side URL-params, while # stood for client-side URL-params. The the last 10 years have seen the Web successfully use URL parameters on the server-side to achieve the RESTful Web in all its glory. In stark contrast, URL intelligence on the client-side has been stagnant at where it was in 1992 in that the only client-side parameter that can be passed via a URL is the id of the element to which the client should jump upon opening the document.

To achieve equality among Web servers and Web clients in this regard, it is perhaps time that the Web started innovating with respect to the use of client-side URL parameters. Here is a wish-list of things that one might wish to achieve using such client-side URL params --- from here on, let's once again think ? for the server, # for the client.

Specify initial location

Already achieved via <URL>#id

Set style

Open a URL with a user-specified style that is chosen among the several alternatives offered by a site, e.g.: <URL>#style(high-contrast.css).

Filter a document

Filter a document by a given XPath expression: <url>#xpath-filter(expression).

And a lot more than can fit in this margin...

Note that all of the XPointer syntax sketched out above should be taken with a large grain of salt --- the goal here is to speculate, not specify;-).

2^W: The Second Coming Of The Web

Audio

Slides (PDF)

Slides (HTML)

I recently gave a talk entitled What Comes After Web 2.0? at the W3C Technical Plenary in Boston. The humor in the talk was well appreciated; I'm posting this entry to ensure that the technical meat underlying the talk does not get overlooked.

The Importance Of Being URLAddressable

The World Wide Web was built on the following architectural pieces:

HTTP

A simple client-server protocol

HTML

A simple markup language for authoring hypertext

URLs

A universal means of addressing Web content

Where Web 1.0 was about bringing useful content to the Web, Web 2.0 is about building Web artifacts out of Web parts. URLs play a central role in enabling such re-use --- notice that a necessary and sufficient condition for something to exist on the Web is that it be addressable via a URL. A key consequence of this design is that Web artifacts when deployed on the Web themselves become an integral part of the Web and are ready to be re-used in building higher-level Web components. Here are a few illustrative examples:

Google Search

Clearly, Google WebSearch would not exist without the Web. But notice that every Google Search in its turn has a URL; this makes it possible for hypertext documents across the Web to embed links to specific searches. Thus, not only is Google Search built on the Web; it itself becomes an integral part of the Web.

Auctions

Items available on auction sites such as eBay are URL addressable. This again makes these an integral partof the Web.

Online Shopping

When items in an online catalog have URLs, each item immediately becomes part of the Web.

The overall impact of the above design is profound; by ensuring that everything that exists on the Web has a unique URL, we ensure that it becomes possible to construct higher-level Web artifacts out of existing Web parts. This is what has led to the success of mashups on the Web; notice that the typical Web mashup accesses a multiplicity of data sources via the relevant URLs to deliver an integrated view to the user.

As we look forward to the second coming of the Web where mashups are not limited to pairwise combinations of Web resources, but instead allow general composition of arbitrary combinations of Web resources, it's important to stress the following:

• Separating data, presentation and interaction into their relevant layers (HTML, CSS, JavaScript) is crucial for content re-use on the Web.
• Data access is critical for mashups that bring together multiple data sources into an integrated view. However, mashups can do much more; for example, the same mashup technologies can be leveraged to integrate multiple views of the same data to produce custom views. A good example of this is seen in the context of XForms, where the model-view separation permits the binding of multiple synchronized views to a given data-set.
• Finally, retaining the separation between data, presentation and interaction in the process of creating mashups, and ensuring that the resulting Web artifact is itself URL addressable leads to the definition of easy to use REST APIs; this is crucial for the evolution of the Web Command Line.

XForms And AJAX Applications In Perspective

The following article by Kurt Cagle makes some excellent points that are worth summarizing here --- I highly recommend his article cited above, and his subsequent follow-ups that his article promises. His detailed write-up helps put Web developments of the last 6 years in perspective --- I'll summarize the salient bits below:

XForms Processing Model

When we set out to define the next generation of Web forms technology in early 2000, we had some very lofty goals which I believe we achieved very welll -- modulo the fact that our enthusiastic optimism at the time made us believe that we'd be done in 2 years. (Fact Of Life --- it's still happening). The primary underpinning of XForms is not the beloved XML angle-brackets, it's the processing model that does for structured XML what spread-sheets achieved for tabular data two decades ago.

XForms Submission

Everyone talks of XForms UI controls all the time, and depending on their perspective either praises it --- or as happens far too often --- mocks it for its abstract UI controls. Sometimes people even make it beyond this and get to talking of the XForms Model, and then fall into the tarpit of arguing over declarative vs imperative programming. The upshot of course is that they never get around to talking about the XForms Submission mechanism.

The XForms Submission mechanism was designed in early 2001 with the insight that the HTML model of replacing the entire page each time one did a client/server round-trip was suboptimal with respect to good user experience on the client. This is why XForms Submit allows you to say
<submission replace="instance">...</submission>
which means

Treat what you get in the server response as fresh form data.

Notice the close relationship to XML-HTTP --- this is not an accident. XForms Submission was invented right around the time XML-HTTP began showing up on the radar as a means to reduce angle-bracket client/server chatter that plagued older HTML Forms.

But getting XML Data from the server is only one half of the Web application story --- to be useful, the retrieved data needs to be made tangible to the user. In conjunction with the afore-mentioned XForms Processing Model, the above is sufficient to account for the several pages of application-specific JavaScript that each AJAX application requires for moving the data from the XML response into the UI tree and back.

Model Constraints

Kurt gives an excellent overview of how the XForms Model and the encapsulated constraints help the developer progressively evolve the model. He also points out how this is typically more difficult to do when the entire application is implemented as a program. I'd like to add the following in the hopes of finally laying the declarative vs imperative debate to rest --- at least in the context of XForms.

In fact, all declarative vs imperative debates are at best pointless religious debates, in practice, one finds that combining the two approaches as appropriate often leads to the best implementations. In this context, XForms declarative constraints reflect a refactoring of those parts of a Web application that need to change often from those aspects of Web interaction that follow a common design pattern. Thus, once the model constraints and the inter-dependencies among data items in the model have been refactored, what emerges is the XForms Processing Model --- which by the way is implemented in imperative languages ranging from C/C++ to JavaScript.

The net effect is to move the coding of the complex bits into the browser e.g., the Firefox XForms plugin or the various IE extensions, and free the Web developer to focus on key aspects of the application at hand. Now, it's probably understandable that browser vendors cried foul in the early days of XForms --- after all, now the browser had to do more; however, if you look back at the history of computing, compiler writers always cry foul when you raise the level of abstraction present in your language. The encouraging thing --- both from the history of computing, as well as from the last two years of XForms adoption is that over time the interests of the application developer moves the compiler (in this case browser) implementor to build support for the higher-level abstraction --- the former negative of I need to do more from the browser turns into a positive of the form Choose my browser platform because I do more for you the developer!.

Consistent Eventing

Web applications on the client come to life thanks to client-side eventing. Developing AJAX applications for legacy browsers is time-consuming because of inconsistent eventing --- though some of this pain is assuaged by using appropriate cross-browser JavaScript libraries whose implementors have bravely shouldered that pain --- similar to the compiler/browser developer taking the hit of implementing support for higher-level abstractions. When we developed XForms 1.0, we made a very concious decision that:

• We would require full DOM2 Events support --- including event bubbling and capture.
• To keep the language extensible, we used XML Events to author event binding.

Together, the above leads to a robust interaction model with the necessary flexibility of allowing application developers the needed extension points to hook custom behavior. It does suffer from the fact that browsers need to implement a consistent eventing model; once again, notice that this apparent negative is saying nothing more than

Over time, browsers will absorb the functionality present in cross-browser eventing libraries, thereby obviating the need for an application to download a library that provides this abstraction layer.

XForms Via AJAX

The primary attraction for using AJAX is that it is easy to deploy in an environment where legacy browsers co-exist with next-generation Web technologies. It is crucial here to realize that AJAX is an excellent deployment technology --- even though it may not be the best choice when it comes to authoring an application. In completing this circle, it is encouraging to see many XForms implementations beginning to process XForms markup on the server to deliver an AJAX deployment of the application to legacy browsers --- thereby reflecting yet another design point of XForms 1.0 --- namely --- the XForms Processing Model must be implementable in any of the following forms:

• Entirely on the client
• Entirely on the server
• Partially split between client and server

And still be able to deliver a consistent user experience. Here, consistency of user experience is intended to be more than screen-deep; consistency means that the user gets the same answer independent of the runtime deployment environment that is used to access the application.

Introduction

XML provides a simple, extensible means for serializing data, think S-Expressions. This Blog focuses on building component-based applications using XML technologies.