Web design at paulcarvill.com, the home of Paul Carvill on the web

Hi, I'm Paul Carvill, I'm a web developer. I'm currently working as Technical Lead at LBi, Europe's largest digital agency.

I also like walking, cooking, Bollywood and rock 'n' roll.

Archive for the ‘Web design’ Category

Internet time

Tuesday, June 8th, 2010

Me and @agentdeal have both recently been noticing an increased interest in analogue clock-based time interfaces on the web. Pleasingly, the practice has been implemented across web development disciplines, proving that it’s not just loopy Flash developers who spend their time making utterly pointless widget-y gizmos — even Steve Jobs and his panting army of fanboys can have a go!

The trend seems to have started with bbc.co.uk’s lovingly crafted piece of old test card nostalgia, made using Canvas:

It continues in spectacular fashion with the new Times paywall site thetimes.co.uk, which has a heartbreakingly pathetic laurel-wreathed carriage clock, rendered in Flash, updating in real time, presumably sitting atop a digital mantelpiece as a retirement gift to the online news industry:

The newest kid on the block, and definitely the showiest, is the timezone-straddling multi-clock Flash display of Globe Trotter, purveyor of hand made luxury luggage:

What fascinates me about all this is that surely every single example here and elsewhere is completely redundant. Is there a device anywhere that doesn’t display the time and/or date somewhere in the interface? I’m pretty sure that anything running a web browser also has a clock or calendar in view a mere eyeball movement away. But that is what makes these internet clocks so great — each is a conceit made of whimsy and I love them.

Tags: canvas, clocks, flash, time
Posted in Design, Development, Web, Web design | 1 Comment »

Increasing memory limit for PHP in Rackspace Cloud Sites

Sunday, February 28th, 2010

If you install more than a few modules in your Drupal implementation the chances are that it’ll run out of memory and you’ll start seeing blank pages and odd behaviour. The fix for this is to increase the amount of memory allocated to PHP, which you can usually do in your php.ini file. But if your hosting is Rackspace Cloud Sites you don’t have access to the php.ini file. You must instead put your PHP settings in a .htaccess file in the root directory of your hosting space.

Here’s some example settings:

php_value memory_limit 96M
php_value upload_max_filesize 50M
php_value post_max_size 50M

Now, here’s the important bit: once you’ve made put those lines in your .htaccess file and FTP’d it to your webspace, the changes might not appear to have taken effect. I had to delete the original .htaccess file and upload a fresh one for my changes to be picked up. Hopefully this might help somebody else in the same situation.

N.B. Rackspace’s support had next to no idea of what I was talking about. Their Livechat support service was as useful as telephoning someone in a library and waiting while they went off to find the information. They obviously don;t support individual applications hosted by them, but I’d have expected a little more help.

Tags: drupal memory rackspace
Posted in Computing, Development, Web design, Work | Comments Off

In bed with CSSEmbed – using data URIs and mhtml to reduce HTTP requests

Friday, February 12th, 2010

My team at LBi have added a pimped version of CSSEmbed into the continuous integration process of the project we’re working on. CSSEmbed is a tool by Yahoo’s Nicholas Zakas which takes CSS files as input, searches them for usages of image URLs, locates each image file being used via its URL or filepath, converts it into base64-encoded data and embeds that data back into the stylesheet in the form of a data URI, replacing the original URL. This has the effect of reducing the number of HTTP requests your browser has to make to load all of the images a page requires — everything in the stylesheet is downloaded in a single request. Because the number of concurrent HTTP requests your browser can make is limited, using CSSEmbed increases the browser’s chances of downloading all assets as quickly as possible.

Image by jek in the box

Performance

In practice getting the best performance out of cssembed is something of a balancing act, and takes some trial and error to get right. The size of the CSS file bloats pretty quickly once you start embedding even relatively small images in it. My first attempt increased one CSS file from around 80KB to almost 900KB! Now, while this file only takes one HTTP request to fetch, it also blocks downloads in that request channel for the entirety of the time it takes to come down the wire. After some experimentation we found the sweet spot was to embed images that were less than 1k in size, as the increased CSS file size was in a much better ratio to the number of HTTP requests saved.

We’ve tested this using PNG, GIF and JPEG in Firefox, Safari, Chrome and IE versions 6, 7 and 8.

Comparison

Here’s a comparison of a page I picked at random from our current project, showing data for using data URIs, both with and without Gzipping of the transfers:

	Page size (KB)	Number of HTTP requests	Avg cold cache download time (ms)	Avg warm cache download time (ms)
Using data URIs	540	45	792.30	726.60
Using data URIs, gzipped	235	45	802.00	748.30
Using original css/images, gzipped	194	56	882.60	771.00
Using original css/images	483	56	849.00	773.60

That’s a 19% fewer HTTP requests. Pretty impressive!

Pimping CSSEmbed

With some great help from my fellow LBi-ers Andrew and Raul, we amended CSSEmbed to take more arguments than the standard version, including a maximum image size to convert to data, a .txt file listing the CSS files to convert, a file extension to append to the outputted files and a directory to output to.

Continuous integration

Here’s a quick list of the steps we took to automate our CSSEmbed process:

1. Create a manifest of the stylesheets you want to embed image data into
2. Add an ant target that executes the cssembed jar twice, once to create files containing datauris (for modern browsers) and then again to create mhtml files (for older version of IE). Supply argument lines as necessary. Our ant targets do the following:
a. Supply root URLs to be prepended to relative URLs in the stylesheet, so the cssembed jar can locate the image files
b. Specify output directories: /datauri and /mhtml, respectively
c. Specify a file extension for the mhtml files: mht
d. Specify the location of the manifest .txt file
3. Amend your page’s head to serve the appropriate stylesheets to each browser

Here’s one of our ant targets:

<java jar="tools/cssembed-0.3.5.jar" fork="true">
     <arg line="--root '${deploy.home}/${project}-build/'" />
     <arg line="-o '${deploy.home}/${project}-build/assets/css/mhtml'" />
     <arg line="--maxsize 1"/>
     <arg line="--mhtml --mhtmlroot http://${project}.build/assets/css/mhtml/" />
     <arg line="--outextension mht" />
     <arg line="'tools/css-manifest.txt'"/>
</java>

A brief word on mhtml

The reason we output the mhtml files with a different file extension and put them in their own directory is to work around some presentation bugs in IE7, specifically IE7 on Windows Vista. IE7 on Vista doesn’t seem to like parsing PNG files as mhtml data unless the data is hosted in a separate file and served up as Content-Type: text/plain. Putting the mhtml files in a seperate directory with the ir own file extension means our client can use Apache directives either to serve either files with that extension as plain/text, or to serve any files from a particular directory as plain/text. It gives them the option and adds no performance overhead at all.

Serving it all up

Finally, with some clever use of conditional comments, we can produce valid markup that will let each browser download just those files it needs to. Here’s how to do it:

* Stylesheets for all browsers go here
<link rel="stylesheet" href="/path/to/styles.css" />

<!--[if gt IE 7]>-->
 * The datauri stylesheets go here.
 * Modern browsers and IE > 7 will see them.
 *
 * Note that the opening tag of the
 * conditional comment above is fully
 * enclosed in an HTML comment.
 *
 * All IE browsers read the conditional
 * comment and act on it, so only IE 8
 * sees these links

<link rel="stylesheet" href="/datauri/styles.css />

 * Note again that the closing tag of the
 * conditional comment below is fully
 * enclosed in an HTML comment

<!--<![endif]-->

<!--[if lt IE 8]>
 * The mhtml stylesheets go here.
 *
 * Only IE < 8 will see them.
 *
 * This whole section is inside an HTML
 * comment, so only IE browsers read the
 * conditional comment and act on it.
 *
 * Only IE < 8 sees these links

<link rel="stylesheet" href="/mhtml/styles.mht" />

 * Note the closing HTML comment tag below.

<![endif]-->

<!--[if lt IE 7]>
 * Here's an example of another conditional
 * comment, for supplying IE 6 stylesheets
 *
 * This is nothing to do with cssembed, it's
 * just here for illustration

<link rel="stylesheet" href="/ie6styles.css" />

<![endif]-->

Now whenever our build script runs, our CSS files are checked for image URLs and new CSS files are created with image data embedded. At runtime each browser downloads the appropriate stylesheets using almost 20% fewer requests than before. This means the page is ready faster than it was before we used data URIs, and we've freed up bandwidth for other activity to take place.

Posted in Internet, Performance, Web design, Work | Comments Off

Why front-end developers are so important to the future of businesses on the web

Thursday, September 24th, 2009

or How traditional businesses who have moved to the web regularly undervalue their front-end web developers, and are worse off because of that

Distinction
The roles of web developers and web designers have been around for over 15 years now, and the role of a client-side or front-end web developer started to mature into a distinct entity around 10 years ago, as the content-presentation-behaviour layer paradigm became embedded in people’s working methodologies (and, with the introduction of Google’s then-new search algorithm, when the need for cleanly structured, easily indexable pages became, for businesses, not just an aspiration but a necessity). Unfortunately the perception of the front-end developer’s role remains somewhat coloured by an early association in observers’ minds with the other, loosely related role of the web designer. The role of web designer is an extremely important and valid one, but it is very different to that of the web developer, and the lack of a clear distinction between the two, in some people’s perception, is unhelpful and does both roles an injustice.

Skill set
The web developer (sometimes also called a client-side developer, front-end developer, web architect or front-end engineer) has a huge skill set and a job description to match. They are often expected and required to excel in many disciplines, and have good working knowledge of many others. They exist at the point where art, design, interaction, programming and behavioural and performance analysis intersect. Given the time, support and ambition of a good business, being a web developer can be an extremely fulfilling job. However, the role of a web developer is often misunderstood within even the most progressive and well-meaning of businesses.

Perception
The danger can be that front-end developers, working in a user-focused area, are seen as performing a superficial function — applying a polish to the heavy lifting done by another developer, say, or that dread comment, “making things look nice”. Let’s be clear, making things look nice is the sole responsibility of the designer. When front-end developers spend much of their time deploying underlying data received from a backend database into their views, or pages, they might mistakenly be thought of as merely translators or interpreters, transferring a graphical image — the Photoshop-ed design — into markup and style rules, purveyors of what is sometimes almost mockingly referred to as a ‘black art’ of making pixels lay out correctly onscreen. While this perception is perhaps unfortunate, it is understandable. It is a particular problem where a development workflow is — some might say artificially — segregated into database infrastructure/domain modeling/server side workflows/front-end workflows. In smaller organisations a front-end developer has the opportunity, if she wishes, to input into any of these areas. In larger organisations, the increased granularity of functional areas means those opportunities are greatly reduced, and as you can see from the segregation model above, the front-end development work comes at the end of a long chain of events and decisions which essentially shape and restrict the front-end developer’s choices.

Frustration
In such cases the development workflow is one-way, negates the developer’s architectural, organisational and behavioural skills and occurs late in the development process. This chronology minimizes the opportunity for the front-end developer to have effective input into, and feedback from, the interaction design they are now expected to code. This is a sad state of affairs and undoubtedly leads to frustration, feelings of being undervalued or ignored, and an extreme cases disenfranchisement and resignation, either in the figurative or practical sense. A good business will understand how highly-nuanced user behaviour is, and value skilled interpretation and shaping of that behaviour in the interests of improving their digital offering.

Value
The modern web developer has huge amounts of value to offer a business. Indeed the type of professional you often find in this role encapsulates the very best the web has to offer:

up-to-date knowledge of available and emerging technologies
extensive experience of implementing de facto web standards and programming patterns
database configuration and data manipulation
implementation across multiple platforms and legacy software applications
provisioning for mobile devices
data aggregation
graphics sourcing and creation
search engine optimisation (SEO)
a thorough understanding of the aesthetics and parameters of designing for the web

Further, the best web developers have wealth of knowledge and understanding around interaction design, user needs, hierarchies of data, navigation systems, user journeys, wireframing, design brief interpretation, focus group and usability testing and the art of a finely polished product. Largely gone are the days of HTML-monkeys, spending days on end converting Photoshop comps to pixel-perfect layouts. A web developer’s role is broad: from developing in what Yahoo!’s Nate Koechley calls ‘the world’s most hostile development environment’ — the browser — and ensuring cross-platform and cross-browser consistency, to working with art directors and designers and remaining true to their vision, to considerations and implementations of accessibility, usability and the overall user experience. A web developer is responsible for everything that sits on the client side of the web stack — the content, presentation and behaviour layers. Few other roles touch so many other key aspects of a business as does a web developer’s. Good businesses realise what an asset they have in their front-end web development team, and welcome their input into the product development process. Even better businesses have a User Experience team which encapsulates all those values, skills and judgements necessary to make great websites. Members of those teams are part of a feedback loop that results in great products, not just acceptable implementations of the first good idea that came up.

Slow
Large businesses and organisations move slowly. They may find it hard to understand they have developers whose skills and interests cross the boundaries their job descriptions impose on them. In addition, large businesses like to modularise their development teams into clearly segmented areas for planning and accountability. End-to-end developers don’t really fit this business model. Web development, certainly rapid prototyping at least, is moving away from monolithic relational database installs and towards schema-free, fluid data repositories like CouchDB and MongoDB. Many other layers of the application stack are now capable of being managed by a web developer. Most developers of this ilk, who are able to own the whole process from from domain and model definition, through to server infrastructure and on to a useful and appealing user experience, are running their own consultancies or are employed by the more enlightened web properties. Some examples of this type of person are Jeff Croft, Dan Rubin, John Resig, Jeffrey Zeldman, among many others. Functionality, data storage and interaction are increasingly moving to the client side (HTML5, Gears, RIAs, iPhone and Android web apps). The web stack sits on top of any technology, making the web developer one of the most versatile members of any business, let alone the technology department.

Undervalued
Unfortunately my experience has been that most large businesses massively undervalue their web developers, employing them in narrowly defined roles as the guys who make the site look nice, or fix the Javascript bugs that make the page break in IE. Larger business and the public sector have made moves towards working seriously on accessibility and usability, but the thinking behind such strategies remains superficial, those conceptual areas misunderstood. Too many people think of them in terms of awards and rating levels, not in ongoing process of improvement. Most of this work is also likely to be outsourced to a specialist third-party.

Career progression
One further, worrying, complication is the lack of clear career progression for front-end developers. Once you’ve spent a year or ten working in every nook and cranny of every browser out there until you can code progressively enhanced web pages blindfolded, what next? Economics and management structures mean there’s only so many architect or senior engineer roles to go round. The other option is to specialize in something and move forward from there. Current trends would seem to be leaning towards a big need for performance specialists in the next 5 years, as the client side moves ever further towards accommodating distributed, complex web applications. Page views will also continue their inexorable rise, placing stress and demand on infrastructure, databases and hardware, and thus even greater stress on a fast, responsive user experience.

Know your value
What does this mean if you’re a front-end or client-side web developer? Know your value. What are your skills? Are you a developer, an engineer, a User Experience architect or an Interaction Designer? Advertise your value. Shout about it. Don’t be knowingly undermined or ignored. Create a User Experience team within your business. If they’ve already got one, join it! Your job is incredibly important, and your present employer needs to realise that, as they stand to benefit.

Updated, 29th September 2009: just fixed a couple of typos

Posted in Browsers, Design, Development, Internet, Usability, User Experience, Web design, Work | 55 Comments »

Wallpaper* Open House map is a prime example of awful Flash interface design

Saturday, September 19th, 2009

Further to my previous post regarding the Adobe-Omniture deal, and how it might improve the consistently awful Flash user experience people regularly have to deal with, this morning I found a very good example of bad Flash interface design to illustrate my point with.

The weekend of 19th and 20th September is London Open House weekend, when the public gets wide and varied access to the architectural pleasures of the City. Unfortunately, while Open House do publish a printed programme, the only way to discover locations of Open House participants online is through a fairly complex and ugly search form at http://www.londonopenhouse.org/public/london/find/. This method is not an intuitive way to find Open House locations, and does not encourage discovery or serendipity.

However, there is a map at Wallpaper magazine’s website. While this could have been a useful alternative to the official listings, it is instead problematic and discouraging. The Wallpaper site features a Google map, overlaid with a Flash map. There are several problems:

The Wallpaper Flash map features only a selection of the full listings. They call this the Wallpaper Edit. For the full listing you must still consult the printed programme or use the official search form.
The Wallpaper Flash map is extremely simplified, and lacks context, scale and travel information, as well as several other navigation aids.
The Wallpaper Flash map uses an entirely new and different navigation method. There is no drag control, and no zoom. Instead, the user needs to click on buttons near the top of the map which signify arbitrary areas of London. There is no option to show partial areas or the whole map at once.
The Wallpaper Flash map uses an entirely different scale to the accompanying standard image-based Google map. Changes in the orientation of one map are not reflected in the other, making switching between the two extremely disorienting.
The Wallpaper Flash map uses bespoke markers and a bespoke popup system, which opens a popup at the top left of the map, completely out of context of the click event. The Google map operates in an expected way, much as other web standards-based online maps do, by opening the popup at the location of the click event, where the user is focused.

The frustrating thing about Wallpaper using a custom Flash interface is that the Google map view already provides all the functionality the user needs, in an established, expected way, and in many ways in offers a superior user experience. Control of the Google map is more granular and direct, and the user benefits from the extra context and transport directions functionality. In addition, in case of the popup, the Google map provides more information, in the form of opening times. Even simple things, like the detail text being copyable, make the Google map a far more suitable implementation.

The Wallpaper Open House Flash map interface is a perfect example of the continued redundancy of Flash interface design, it’s ability to encourage design abuse, the lack of understanding of user behavioural patterns and the skewed importance of brand over usability.

Posted in Development, Internet, Usability, User Experience, Web design, flash | Comments Off

The Flash user experience is regularly awful

Friday, September 18th, 2009

If the Adobe-Omniture deal means anything it’s that with feedback from behaviour tracking Flash developers might finally get around to addressing their applications’ regularly awful user experience.

Designers of Flash apps and widgets seem to go back to the drawing board every single time they make something. They start from scratch each time, creating new and confusing user interfaces to reflect whatever the fashionable paradigm of the time is — whizzy, futuristic dashboard? No problem! Over-elaborate transitions? Easy-peasy! Flash intro on your splash page? No sooner said than done (but is it still 1999?).

The problem with this approach is that with each new app and its innovative bespoke interface, users are regularly forced to learn a new method of navigation. The web has well-established, native patterns of interaction that do everything you need them to. Discoverability, when implemented correctly, is a finely nuanced mechanism for delivering complex systems to the user. But Flash apps are too often clunky, unnavigable, hastily put together bits of software designed to capitalise on trend and satisfy business interests keen on some thing ‘jazzy’ or ‘funky’. Way back in 2000 Jakob Nielsen said that Flash was 99% bad, Flash encouraged design abuse, and made bad design more likely, and not much has changed since he made those statements.

Flash applications avoid many established design patterns, are often explicitly designed to be impermanent, and regularly lack fallback content in the instance of the correct plugin not being installed, but these issues are rarely considered by a business pushing for a snazzy new feature they can parade before their board.

Hopefully the Adobe-Omniture deal will set off a new era of closely-monitored stats which might prove useful in finding where Flash applications and interfaces could be improved, and where using web standards to build a long-term solution might prove a more fruitful endeavour. Of course, Flash analytics can currently be recorded by other analytics tools such as Google Analytics, but Omniture is a far bigger hitter on large scale websites, and where influential Flash design is needed the most in order for good practice to trickle down into common usage.

Tags: flash, webstandards
Posted in Design, Internet, Web design | Comments Off

A collection of HTML5 links, documents and discussions

Tuesday, September 15th, 2009

The movements around discussing, designing and implementing HTML5 are gaining real momentum now, and it was definitely one of the big buzzwords at the summer web conferences this year. I’ve started implementing one or two small pieces of the spec on production sites, and am looking forward to using some of the long-awaited functionality like the form-validation work that’s in there, as well as the offline storage that’s already been relatively widely deployed (well, by Google, at least: Cache pattern for offline HTML5 web applications). So I’ve been monitoring the specification documents a little more closely than usual, as one of the important things about working in some of the more developmental areas is knowing that the spec is likely to change.

As part of that, I’ve rounded up here some of the recent linkage around the HTML5 spec:

Zeldman
Zeldman, publisher of A List Apart, founder of Happy Cog Studios and “King of Web Standards” (according to Business Week), approves of the direction HTML5 is taking, although he does have a few issues with the spec. But, he says, “…many things we had previously considered serious problems were fixable issues related to language.”

“Half of standards making is minutia; the other half is politics.”

http://www.zeldman.com/2009/08/31/loving-html5/

HTML5 Super Friends
Zeldman’s friends like the direction of HTML5 too, and are happy to publicly say so, as part of the HTML5 Super Friends group:
http://www.zeldman.com/superfriends/

Although some people have griped that it would have been more conformant to use the proper channels to air some of these views, I agree with the group that the more public support and high-profile discussion we can get around the spec the better — it’ll eventually all filter down to the WHATWG mailing list anyway.

“We, the undersigned, wish to declare our support for the direction in which the HTML5 specification is heading. Its introduction of a limited set of additional semantic elements, its instructions on how to handle failure, and its integration of application development tools hold the promise of richer and more consistent user experiences, faster prototyping, and increased human and machine semantics.”

The HTML5 Super Friends have a list of technical hiccups and problems in the spec. Positively, they have also proposed solutions:
http://www.zeldman.com/superfriends/guide/

The Spec
Just the markup parts of the spec, omiting much of the browser implementation stuff
http://dev.w3.org/html5/markup/
and a one-page version
http://dev.w3.org/html5/markup/spec.html

Nicole Sullivan/Yahoo!
Nicole Sullivan of Yahoo!, and also a Super Friend, gives her thoughts on the HTML5 spec, with particular focus on bringing Standardistas and JavaScripters together

“It should be possible to do more with less javascript. I’d like to see browser support for common aspects of web pages as well as web applications. Practically every site in the known universe has toggle blocks, tabs, carousels, or accordion menus. I’d like to seen native browser support and CSS styling, so that these element incur no particular performance cost.”

http://www.stubbornella.org/content/2009/08/31/html5-who-is-bad-enough-to-take-on-canvas/

Tags: html w3c webstandards
Posted in Internet, Web design, html5 | Comments Off

IE6 state of play linkage

Friday, August 14th, 2009

A few links to recent IE6 coverage.

Digg the Blog: Much Ado About IE6
“This goes directly to why most folks use IE6: they don’t have a choice. Three out of four IE6 users on Digg said they can’t upgrade due to some technical or workplace reason…Giving them a message saying, ‘Hey! Upgrade!’ in this case is not only pointless; it’s sadistic.”

BBC News | Technology: Microsoft backs long life for IE6 [support will continue until 2014, four years beyond their original deadline]
“‘Friends do not let friends use IE6,’ said Amy Barzdukas, Microsoft’s general manager for Internet Explorer.
‘If you are in my social set and I have been to your house for dinner, you are not using IE6,’ she said. ‘But it is much more complicated when you move into a business setting.’”

blogs.msdn.com | IEBlog: The Engineering Point Of View
“The engineering point of view on IE6 starts as an operating systems supplier. Dropping support for IE6 is not an option because we committed to supporting the IE included with Windows for the lifespan of the product. We keep our commitments.”

MoD to stick with IE6 despite security concerns
“According to parliamentary written answers received by Labour MP Tom Watson, the majority of [government] departments still require staff to use IE6. Most have plans to upgrade to the more secure IE7, and some to IE8, but the Ministry of Defence (MoD) has no plans to change.”

Conclusion: IE6, at least in corporate environments, isn’t going away for a few years yet.

Tags: ie6, microsoft
Posted in Browsers, Development, Internet, Uncategorized, Web design, Work | Comments Off

Using Greasemonkey to Pimp the Telegraph’s MPs’ Expenses Data

Monday, July 13th, 2009

You may have noticed that the Telegraph have some useful MPs’ expenses data on their site. To go with it they’ve also got some nice graphs, which show an MP’s historical expense claim data. The trend of those claims is, as you might expect, mostly upwards. No surprises so far. But, until recently, inflation has always been a positive number, so you would be probably be right to expect expense claims to rise in tandem with the prices of everyday goods and services.

Above: the unpimped image of Michael Ancram’s expenses data chart

But I don’t find the graphs useful enough. I can easily see the trend of the claims just by looking at the accompanying data table. What I really want to know is how closely the rises in MPs’ expenses matched the standard inflation rate. My hunch was that the rise in value of the MPs’ claims would be ahead of the inflation rate, so how best to check this? And how to check it for a wide number of claimants? In addition, I wanted to visualise a breakdown of the claim types – for travel, stationary, 2nd home etc.

Above: an image of Michael Ancram’s expenses data chart following the addition of my Greasemonkey script.

I was interested enough to ask the Telegraph, via Twitter, if they had any plans to add some economic context to the data. They actually responded to let me know that the data was available in a spreadsheet, so why not go ahead and mash it up? Great! But on checking the spreadsheet I found that the data only went back to 2004, while the data on the Telegraph’s website went back to 2001. I’m aware that expenses data going back further are available from the Houses of Parliament website, but I didn’t really have to time to go searching for them, converting the format to what I needed etc. So I decided to write a quick Greasemonkey script to operate on the data already published by the Telegraph, on their website, which happened to be good enough markup to extract data from and use.

Greasemonkey is a Firefox plugin which enables you to write JavaScript scripts which can operate on the contents of a web page solely within your browser. Anything you do to the data does not alter the original in any way, and so it’s a great way of changing the way websites work, customising them to look how you like and generally mucking about with them.

Here’s some technical info on what I did:

First I checked the Office of National Statistics for the annual inflation data from the Retail Price Index, the broadest measure of inflation, which includes mortgage interest payments. I grabbed data for the years covered by the Telegraph data — 2001/02 to 2007/08 and added it to my script. Then I loaded in a couple of extra scripts which help to grab the data from the page, draw a new graph add it to the page in place of the original.

I extract the data from the MP’s expenses table, which is actually the same way the Telegraph are drawing the graph in the first place. The difference is that I extract each row of data rather than just the totals. I draw all the data to the graph, then add an extra line which extrapolates the MP’s first year total claim across the whole term of the graph, based on the inflation data for the subsequent years. Finally, I added a series of checkboxes to isolate different data sets within the graph, so you can compare, say, just an MP’s claims for food against her claims for stationery.

The script was fairly quickly put together, so if you’re handy with a bit of JavaScript you might be able to think of something I haven’t. For example I’d like to see a comparison of MPs’ claims against nursing or teaching salaries. Even if you just want to hack around you can download it and amend it as much as you like without any danger. But here’s the really great thing — I’ve actually maintained most of the structure of the Telegraph’s original graph code, so if they want to take my script back and use it on their page they’re free to do so, and it should work perfectly!

Was this hack useful? Yes. For example, it allowed me to see that Ian Austin, Labour MP for Dudley North, made below-inflation claims rises for 2006/07 and 2007/08. And that Labour MP for Leeds North East Fabian Hamilton’s 2007/08 total claim (£170,794.00) was almost double what it would have been if his first recorded claim (£75,437.00) from 2001/02 had risen in line with inflation.

You can download my Greasemonkey script here. If you don’t want to do that (or your IT department won’t let you) you can see from the images above how the script operates on the original website.

Some notes about using this script in Greasemonkey:

I wrote this script using Greasemonkey 0.8.2. As of version 0.8 you can use the @require metadata key to load JavaScript libraries into your user-script. Greasemonkey will download those scripts to your local directory, meaning your browser isn’t increasing network traffic every time you run the script. If for any reason you need to use an earlier version of the plugin you’ll find some useful information here on loading in scripts without the @require key.

We don’t want the script to execute except when we’re looking at a Telegraph MPs’ expenses page, so for best performance you should set this script to run on the following URLs:

http://parliament.telegraph.co.uk/mpsexpenses/expenses/*
http://parliament.telegraph.co.uk/mpsexpenses/expense-microsite/expenses/*

Some good links to try with the script installed:

Tags: mpsexpenses telegraph javascript greasemonkey
Posted in Internet, Web design | Comments Off

HTML5 HTML text semantics granularity

Thursday, June 18th, 2009

Wow, ~~HTML5~~ HTML semantic text description options are so granular I had to spend several minutes pondering whether my previous code snippet about relaxing Apache permissions warranted <code>, <kbd>, or <samp> elements, or a combination of all three.

In the end I settled on a <kbd> element for the bit I want you to type in (opening a file in vi from the command line), as that’s the bit you’re going to type. For the contents of the file I chose a <samp> element, as the text shown in the file is a sample of the output of my file, rather than a chunk of code you need to enter.

The difference between <code> and <samp> is very small, but it’s great that we actually ~~now~~ have this level of specificity, which should help ensure that ~~HTML5~~ HTML is robust enough to last well into the future.

Tags: html5 kbd pre code samp
Posted in Code snippets, Computing, Development, Web design, html5 | 2 Comments »