Internet at paulcarvill.com, the home of Paul Carvill on the web

link: paulcarvill on twitter

link: paulcarvill at flickr

paulcarvill.com

Hi, I'm Paul Carvill, I'm a web developer. I'm currently working as Technical Lead at LBi, Europe's largest digital agency.

I also like walking, cooking, Bollywood and rock 'n' roll.

Archive for the ‘Internet’ Category

Who should have a key to restart the internet?

Thursday, July 29th, 2010

Right, I’ve nommed my seven recipients for each of the 7 Keys To Restart The Internet In The Event Of A Catastrophic Failure. They are:

1. Richard Desmond. Because with his recent purchase of Five he’s really shown that he understands the high quality of service and content that a truly global medium demands. Also, I’m not sure there’s really enough boobs on the internet yet.

2. Ling of Ling’s Cars. Because she already has most of the Internet right there on her page.

3. Double Rainbow guy. Because the Internet, like a double rainbow right the way across the sky, is a beautiful thing.

4. Rupert Murdoch. Because he’ll do the internet properly this time. Delivered to your door daily by ‘internet-boys’ on bikes.

5. A randomly selected Mayor or badge-holder. Because that’s the responsibility that comes with being a pillar of the community or badge hobbyist, right? And by community I mean cliquey braggadocio service and self-selecting marketing database. But remember — no deputy mayors.

6. A HOT SPY. Because the key holders need to convene in a secret location, and you can’t have a secret location without spy, so you might as well get a HOT one.

7. Whoever has the highest Hot Or Not score. Because someone HOT should be able to restart the Internet, and Vint Cerf is NOT HOT.

I shall be sending these names to ICANN forthwith, and hopefully they will soon be installed either as replacements for the already-nominated 7 or at the very least as substitutes for them in the event of international travel problems following a large scale war or terrorist attack.

Have you nommed your 7 Recipients Of The Keys yet?

#whoShouldHaveAKeyToRestartTheInternet

Mobile network operators are just big dumb pipes in the sky

Monday, February 15th, 2010

I think the idea of 24 of the world’s largest mobile phone operators coming together to ‘make it easier to sell and distribute mobile phone apps’ is laughable. This is what’s being announced today at a huge mobile trade show in Barcelona.

Apparently the mobile phone networks fear that at the moment they are in danger of becoming little more than “dumb pipes in the air”, with all the revenues created by applications going to software developers and the companies that operate the stores that supply them.

Well here’s news, mobile network operators — I would like NOTHING MORE than for you to be dumb pipes. Which shouldn’t be a problem because THAT’S WHAT YOU ARE! I want you to act in the same way I want all my other utilities — water, sewage, gas — to act. Like great, big literal dumb pipes. Subservient conduits. Enslaved vessels. I don’t ask British Gas to get together with all the other gas companies and brainstorm new hot dinner ideas I might want to try out. You know, using their GAS and all? No, I’ll rely on trained chefs, food writers and my vast collection of suspiciously cleanly-paged cookbooks for that. Specialists, in other words. Just you keep the gas coming, guys. In fact I’m not even sure British Gas are the people who supply my gas — THAT’S how little I care about my gas utility supplier.

Your problem, mobile network operators, is you think you’re better than that. You want us to love you. You want us to wonder what we’d do without you. But I mean, to continue the utilities analogy, it’s not like you even manufacture the porcelain toilet, or the brushed steel taps, that connect to the dumb pipes. Nope, you just take care of all that stuff sloshing about inside them, the stuff everyone takes for granted. I really don’t want Thames Water forming coalitions with other water companies to provide me with new and improved cocktail ideas, that I could make using, you know, the water they send to my house. No thanks, fellas, just keep the wet stuff coming, though, you’re doing a great job.

But the mobile operators don’t really care about their customers. They’re just worried about their profits, and how they’ll steadily lose them to every innovative company that encroaches into their tradtional space. Customers have literally zero loyalty to any of their carefully nurtured and obviously humanised brands. People are interested in technology and functionality, which are provided by the handset manufacturers and the software developers respectively. Those two groups are both also driven by profit, but, importantly, their mission is to deliver better technology and functionality than their rivals. Mobile operators, on the other hand, offer a binary proposition — you can either telephone, SMS or get on the web, or you can’t. Not a hard job, youd have thought.

Worse still, the mobile operators got themselves into this mess in the first place by offering their customers closed systems and restricted access; they attempted to own content and access to content through walled-garden web access. Their very offering relies on restriction and artificial handicapping, as they attempt to categorise and upsell customers into various arbitrary packages, the difference in whose actual cost is negligible. Mobile phone operators have a target figure they aim to extract from each customer every month. The actual makeup of the services and products they offer is virtually meaningless. So they have no one but themselves to blame when customers eagerly jump ship in favour of unlimited data access, real web access and a semblance of choice in what you do with your mobile.

Mobile phone operators have repeatedly show themselves to be unimaginative, exploitative, moneygrabbing corporate behemoths with no real understanding the mobile data world. I don’t see that changing any time soon.

In bed with CSSEmbed – using data URIs and mhtml to reduce HTTP requests

Friday, February 12th, 2010

My team at LBi have added a pimped version of CSSEmbed into the continuous integration process of the project we’re working on. CSSEmbed is a tool by Yahoo’s Nicholas Zakas which takes CSS files as input, searches them for usages of image URLs, locates each image file being used via its URL or filepath, converts it into base64-encoded data and embeds that data back into the stylesheet in the form of a data URI, replacing the original URL. This has the effect of reducing the number of HTTP requests your browser has to make to load all of the images a page requires — everything in the stylesheet is downloaded in a single request. Because the number of concurrent HTTP requests your browser can make is limited, using CSSEmbed increases the browser’s chances of downloading all assets as quickly as possible.

Image by jek in the box

Performance

In practice getting the best performance out of cssembed is something of a balancing act, and takes some trial and error to get right. The size of the CSS file bloats pretty quickly once you start embedding even relatively small images in it. My first attempt increased one CSS file from around 80KB to almost 900KB! Now, while this file only takes one HTTP request to fetch, it also blocks downloads in that request channel for the entirety of the time it takes to come down the wire. After some experimentation we found the sweet spot was to embed images that were less than 1k in size, as the increased CSS file size was in a much better ratio to the number of HTTP requests saved.

We’ve tested this using PNG, GIF and JPEG in Firefox, Safari, Chrome and IE versions 6, 7 and 8.

Comparison

Here’s a comparison of a page I picked at random from our current project, showing data for using data URIs, both with and without Gzipping of the transfers:

Page size (KB) Number of
HTTP requests
Avg cold cache download time (ms) Avg warm cache download time (ms)
Using data URIs 540 45 792.30 726.60
Using data URIs, gzipped 235 45 802.00 748.30
Using original css/images, gzipped 194 56 882.60 771.00
Using original css/images 483 56 849.00 773.60

That’s a 19% fewer HTTP requests. Pretty impressive!

Pimping CSSEmbed

With some great help from my fellow LBi-ers Andrew and Raul, we amended CSSEmbed to take more arguments than the standard version, including a maximum image size to convert to data, a .txt file listing the CSS files to convert, a file extension to append to the outputted files and a directory to output to.

Continuous integration

Here’s a quick list of the steps we took to automate our CSSEmbed process:

1. Create a manifest of the stylesheets you want to embed image data into
2. Add an ant target that executes the cssembed jar twice, once to create files containing datauris (for modern browsers) and then again to create mhtml files (for older version of IE). Supply argument lines as necessary. Our ant targets do the following:
a. Supply root URLs to be prepended to relative URLs in the stylesheet, so the cssembed jar can locate the image files
b. Specify output directories: /datauri and /mhtml, respectively
c. Specify a file extension for the mhtml files: mht
d. Specify the location of the manifest .txt file
3. Amend your page’s head to serve the appropriate stylesheets to each browser

Here’s one of our ant targets:

<java jar="tools/cssembed-0.3.5.jar" fork="true">
     <arg line="--root '${deploy.home}/${project}-build/'" />
     <arg line="-o '${deploy.home}/${project}-build/assets/css/mhtml'" />
     <arg line="--maxsize 1"/>
     <arg line="--mhtml --mhtmlroot http://${project}.build/assets/css/mhtml/" />
     <arg line="--outextension mht" />
     <arg line="'tools/css-manifest.txt'"/>
</java>

A brief word on mhtml

The reason we output the mhtml files with a different file extension and put them in their own directory is to work around some presentation bugs in IE7, specifically IE7 on Windows Vista. IE7 on Vista doesn’t seem to like parsing PNG files as mhtml data unless the data is hosted in a separate file and served up as Content-Type: text/plain. Putting the mhtml files in a seperate directory with the ir own file extension means our client can use Apache directives either to serve either files with that extension as plain/text, or to serve any files from a particular directory as plain/text. It gives them the option and adds no performance overhead at all.

Serving it all up

Finally, with some clever use of conditional comments, we can produce valid markup that will let each browser download just those files it needs to. Here’s how to do it:

* Stylesheets for all browsers go here
<link rel="stylesheet" href="/path/to/styles.css" />

<!--[if gt IE 7]>-->
 * The datauri stylesheets go here.
 * Modern browsers and IE > 7 will see them.
 *
 * Note that the opening tag of the
 * conditional comment above is fully
 * enclosed in an HTML comment.
 *
 * All IE browsers read the conditional
 * comment and act on it, so only IE 8
 * sees these links

<link rel="stylesheet" href="/datauri/styles.css />

 * Note again that the closing tag of the
 * conditional comment below is fully
 * enclosed in an HTML comment

<!--<![endif]-->

<!--[if lt IE 8]>
 * The mhtml stylesheets go here.
 *
 * Only IE < 8 will see them.
 *
 * This whole section is inside an HTML
 * comment, so only IE browsers read the
 * conditional comment and act on it.
 *
 * Only IE < 8 sees these links

<link rel="stylesheet" href="/mhtml/styles.mht" />

 * Note the closing HTML comment tag below.

<![endif]-->

<!--[if lt IE 7]>
 * Here's an example of another conditional
 * comment, for supplying IE 6 stylesheets
 *
 * This is nothing to do with cssembed, it's
 * just here for illustration

<link rel="stylesheet" href="/ie6styles.css" />

<![endif]-->

Now whenever our build script runs, our CSS files are checked for image URLs and new CSS files are created with image data embedded. At runtime each browser downloads the appropriate stylesheets using almost 20% fewer requests than before. This means the page is ready faster than it was before we used data URIs, and we've freed up bandwidth for other activity to take place.

Why front-end developers are so important to the future of businesses on the web

Thursday, September 24th, 2009

or How traditional businesses who have moved to the web regularly undervalue their front-end web developers, and are worse off because of that

Distinction
The roles of web developers and web designers have been around for over 15 years now, and the role of a client-side or front-end web developer started to mature into a distinct entity around 10 years ago, as the content-presentation-behaviour layer paradigm became embedded in people’s working methodologies (and, with the introduction of Google’s then-new search algorithm, when the need for cleanly structured, easily indexable pages became, for businesses, not just an aspiration but a necessity). Unfortunately the perception of the front-end developer’s role remains somewhat coloured by an early association in observers’ minds with the other, loosely related role of the web designer. The role of web designer is an extremely important and valid one, but it is very different to that of the web developer, and the lack of a clear distinction between the two, in some people’s perception, is unhelpful and does both roles an injustice.

Skill set
The web developer (sometimes also called a client-side developer, front-end developer, web architect or front-end engineer) has a huge skill set and a job description to match. They are often expected and required to excel in many disciplines, and have good working knowledge of many others. They exist at the point where art, design, interaction, programming and behavioural and performance analysis intersect. Given the time, support and ambition of a good business, being a web developer can be an extremely fulfilling job. However, the role of a web developer is often misunderstood within even the most progressive and well-meaning of businesses.

Perception
The danger can be that front-end developers, working in a user-focused area, are seen as performing a superficial function — applying a polish to the heavy lifting done by another developer, say, or that dread comment, “making things look nice”. Let’s be clear, making things look nice is the sole responsibility of the designer. When front-end developers spend much of their time deploying underlying data received from a backend database into their views, or pages, they might mistakenly be thought of as merely translators or interpreters, transferring a graphical image — the Photoshop-ed design — into markup and style rules, purveyors of what is sometimes almost mockingly referred to as a ‘black art’ of making pixels lay out correctly onscreen. While this perception is perhaps unfortunate, it is understandable. It is a particular problem where a development workflow is — some might say artificially — segregated into database infrastructure/domain modeling/server side workflows/front-end workflows. In smaller organisations a front-end developer has the opportunity, if she wishes, to input into any of these areas. In larger organisations, the increased granularity of functional areas means those opportunities are greatly reduced, and as you can see from the segregation model above, the front-end development work comes at the end of a long chain of events and decisions which essentially shape and restrict the front-end developer’s choices.

Frustration
In such cases the development workflow is one-way, negates the developer’s architectural, organisational and behavioural skills and occurs late in the development process. This chronology minimizes the opportunity for the front-end developer to have effective input into, and feedback from, the interaction design they are now expected to code. This is a sad state of affairs and undoubtedly leads to frustration, feelings of being undervalued or ignored, and an extreme cases disenfranchisement and resignation, either in the figurative or practical sense. A good business will understand how highly-nuanced user behaviour is, and value skilled interpretation and shaping of that behaviour in the interests of improving their digital offering.

Value
The modern web developer has huge amounts of value to offer a business. Indeed the type of professional you often find in this role encapsulates the very best the web has to offer:

  • up-to-date knowledge of available and emerging technologies
  • extensive experience of implementing de facto web standards and programming patterns
  • database configuration and data manipulation
  • implementation across multiple platforms and legacy software applications
  • provisioning for mobile devices
  • data aggregation
  • graphics sourcing and creation
  • search engine optimisation (SEO)
  • a thorough understanding of the aesthetics and parameters of designing for the web

Further, the best web developers have wealth of knowledge and understanding around interaction design, user needs, hierarchies of data, navigation systems, user journeys, wireframing, design brief interpretation, focus group and usability testing and the art of a finely polished product. Largely gone are the days of HTML-monkeys, spending days on end converting Photoshop comps to pixel-perfect layouts. A web developer’s role is broad: from developing in what Yahoo!’s Nate Koechley calls ‘the world’s most hostile development environment’ — the browser — and ensuring cross-platform and cross-browser consistency, to working with art directors and designers and remaining true to their vision, to considerations and implementations of accessibility, usability and the overall user experience. A web developer is responsible for everything that sits on the client side of the web stack — the content, presentation and behaviour layers. Few other roles touch so many other key aspects of a business as does a web developer’s. Good businesses realise what an asset they have in their front-end web development team, and welcome their input into the product development process. Even better businesses have a User Experience team which encapsulates all those values, skills and judgements necessary to make great websites. Members of those teams are part of a feedback loop that results in great products, not just acceptable implementations of the first good idea that came up.

Slow
Large businesses and organisations move slowly. They may find it hard to understand they have developers whose skills and interests cross the boundaries their job descriptions impose on them. In addition, large businesses like to modularise their development teams into clearly segmented areas for planning and accountability. End-to-end developers don’t really fit this business model. Web development, certainly rapid prototyping at least, is moving away from monolithic relational database installs and towards schema-free, fluid data repositories like CouchDB and MongoDB. Many other layers of the application stack are now capable of being managed by a web developer. Most developers of this ilk, who are able to own the whole process from from domain and model definition, through to server infrastructure and on to a useful and appealing user experience, are running their own consultancies or are employed by the more enlightened web properties. Some examples of this type of person are Jeff Croft, Dan Rubin, John Resig, Jeffrey Zeldman, among many others. Functionality, data storage and interaction are increasingly moving to the client side (HTML5, Gears, RIAs, iPhone and Android web apps). The web stack sits on top of any technology, making the web developer one of the most versatile members of any business, let alone the technology department.

Undervalued
Unfortunately my experience has been that most large businesses massively undervalue their web developers, employing them in narrowly defined roles as the guys who make the site look nice, or fix the Javascript bugs that make the page break in IE. Larger business and the public sector have made moves towards working seriously on accessibility and usability, but the thinking behind such strategies remains superficial, those conceptual areas misunderstood. Too many people think of them in terms of awards and rating levels, not in ongoing process of improvement. Most of this work is also likely to be outsourced to a specialist third-party.

Career progression
One further, worrying, complication is the lack of clear career progression for front-end developers. Once you’ve spent a year or ten working in every nook and cranny of every browser out there until you can code progressively enhanced web pages blindfolded, what next? Economics and management structures mean there’s only so many architect or senior engineer roles to go round. The other option is to specialize in something and move forward from there. Current trends would seem to be leaning towards a big need for performance specialists in the next 5 years, as the client side moves ever further towards accommodating distributed, complex web applications. Page views will also continue their inexorable rise, placing stress and demand on infrastructure, databases and hardware, and thus even greater stress on a fast, responsive user experience.

Know your value
What does this mean if you’re a front-end or client-side web developer? Know your value. What are your skills? Are you a developer, an engineer, a User Experience architect or an Interaction Designer? Advertise your value. Shout about it. Don’t be knowingly undermined or ignored. Create a User Experience team within your business. If they’ve already got one, join it! Your job is incredibly important, and your present employer needs to realise that, as they stand to benefit.

Updated, 29th September 2009: just fixed a couple of typos

Wallpaper* Open House map is a prime example of awful Flash interface design

Saturday, September 19th, 2009

Further to my previous post regarding the Adobe-Omniture deal, and how it might improve the consistently awful Flash user experience people regularly have to deal with, this morning I found a very good example of bad Flash interface design to illustrate my point with.

The weekend of 19th and 20th September is London Open House weekend, when the public gets wide and varied access to the architectural pleasures of the City. Unfortunately, while Open House do publish a printed programme, the only way to discover locations of Open House participants online is through a fairly complex and ugly search form at http://www.londonopenhouse.org/public/london/find/. This method is not an intuitive way to find Open House locations, and does not encourage discovery or serendipity.

However, there is a map at Wallpaper magazine’s website. While this could have been a useful alternative to the official listings, it is instead problematic and discouraging. The Wallpaper site features a Google map, overlaid with a Flash map. There are several problems:

  • The Wallpaper Flash map features only a selection of the full listings. They call this the Wallpaper Edit. For the full listing you must still consult the printed programme or use the official search form.
  • The Wallpaper Flash map is extremely simplified, and lacks context, scale and travel information, as well as several other navigation aids.
  • The Wallpaper Flash map uses an entirely new and different navigation method. There is no drag control, and no zoom. Instead, the user needs to click on buttons near the top of the map which signify arbitrary areas of London. There is no option to show partial areas or the whole map at once.
  • The Wallpaper Flash map uses an entirely different scale to the accompanying standard image-based Google map. Changes in the orientation of one map are not reflected in the other, making switching between the two extremely disorienting.
  • The Wallpaper Flash map uses bespoke markers and a bespoke popup system, which opens a popup at the top left of the map, completely out of context of the click event. The Google map operates in an expected way, much as other web standards-based online maps do, by opening the popup at the location of the click event, where the user is focused.

The frustrating thing about Wallpaper using a custom Flash interface is that the Google map view already provides all the functionality the user needs, in an established, expected way, and in many ways in offers a superior user experience. Control of the Google map is more granular and direct, and the user benefits from the extra context and transport directions functionality. In addition, in case of the popup, the Google map provides more information, in the form of opening times. Even simple things, like the detail text being copyable, make the Google map a far more suitable implementation.

The Wallpaper Open House Flash map interface is a perfect example of the continued redundancy of Flash interface design, it’s ability to encourage design abuse, the lack of understanding of user behavioural patterns and the skewed importance of brand over usability.

The Flash user experience is regularly awful

Friday, September 18th, 2009

If the Adobe-Omniture deal means anything it’s that with feedback from behaviour tracking Flash developers might finally get around to addressing their applications’ regularly awful user experience.

Designers of Flash apps and widgets seem to go back to the drawing board every single time they make something. They start from scratch each time, creating new and confusing user interfaces to reflect whatever the fashionable paradigm of the time is — whizzy, futuristic dashboard? No problem! Over-elaborate transitions? Easy-peasy! Flash intro on your splash page? No sooner said than done (but is it still 1999?).

The problem with this approach is that with each new app and its innovative bespoke interface, users are regularly forced to learn a new method of navigation. The web has well-established, native patterns of interaction that do everything you need them to. Discoverability, when implemented correctly, is a finely nuanced mechanism for delivering complex systems to the user. But Flash apps are too often clunky, unnavigable, hastily put together bits of software designed to capitalise on trend and satisfy business interests keen on some thing ‘jazzy’ or ‘funky’. Way back in 2000 Jakob Nielsen said that Flash was 99% bad, Flash encouraged design abuse, and made bad design more likely, and not much has changed since he made those statements.

Flash applications avoid many established design patterns, are often explicitly designed to be impermanent, and regularly lack fallback content in the instance of the correct plugin not being installed, but these issues are rarely considered by a business pushing for a snazzy new feature they can parade before their board.

Hopefully the Adobe-Omniture deal will set off a new era of closely-monitored stats which might prove useful in finding where Flash applications and interfaces could be improved, and where using web standards to build a long-term solution might prove a more fruitful endeavour. Of course, Flash analytics can currently be recorded by other analytics tools such as Google Analytics, but Omniture is a far bigger hitter on large scale websites, and where influential Flash design is needed the most in order for good practice to trickle down into common usage.

A collection of HTML5 links, documents and discussions

Tuesday, September 15th, 2009

The movements around discussing, designing and implementing HTML5 are gaining real momentum now, and it was definitely one of the big buzzwords at the summer web conferences this year. I’ve started implementing one or two small pieces of the spec on production sites, and am looking forward to using some of the long-awaited functionality like the form-validation work that’s in there, as well as the offline storage that’s already been relatively widely deployed (well, by Google, at least: Cache pattern for offline HTML5 web applications). So I’ve been monitoring the specification documents a little more closely than usual, as one of the important things about working in some of the more developmental areas is knowing that the spec is likely to change.

As part of that, I’ve rounded up here some of the recent linkage around the HTML5 spec:

Zeldman
Zeldman, publisher of A List Apart, founder of Happy Cog Studios and “King of Web Standards” (according to Business Week), approves of the direction HTML5 is taking, although he does have a few issues with the spec. But, he says, “…many things we had previously considered serious problems were fixable issues related to language.”

“Half of standards making is minutia; the other half is politics.”

http://www.zeldman.com/2009/08/31/loving-html5/

HTML5 Super Friends
Zeldman’s friends like the direction of HTML5 too, and are happy to publicly say so, as part of the HTML5 Super Friends group:
http://www.zeldman.com/superfriends/

Although some people have griped that it would have been more conformant to use the proper channels to air some of these views, I agree with the group that the more public support and high-profile discussion we can get around the spec the better — it’ll eventually all filter down to the WHATWG mailing list anyway.

“We, the undersigned, wish to declare our support for the direction in which the HTML5 specification is heading. Its introduction of a limited set of additional semantic elements, its instructions on how to handle failure, and its integration of application development tools hold the promise of richer and more consistent user experiences, faster prototyping, and increased human and machine semantics.”

The HTML5 Super Friends have a list of technical hiccups and problems in the spec. Positively, they have also proposed solutions:
http://www.zeldman.com/superfriends/guide/

The Spec
Just the markup parts of the spec, omiting much of the browser implementation stuff
http://dev.w3.org/html5/markup/
and a one-page version
http://dev.w3.org/html5/markup/spec.html

Nicole Sullivan/Yahoo!
Nicole Sullivan of Yahoo!, and also a Super Friend, gives her thoughts on the HTML5 spec, with particular focus on bringing Standardistas and JavaScripters together

“It should be possible to do more with less javascript. I’d like to see browser support for common aspects of web pages as well as web applications. Practically every site in the known universe has toggle blocks, tabs, carousels, or accordion menus. I’d like to seen native browser support and CSS styling, so that these element incur no particular performance cost.”

http://www.stubbornella.org/content/2009/08/31/html5-who-is-bad-enough-to-take-on-canvas/

Technology can, and will, save the newspaper publishing industry

Tuesday, August 25th, 2009

Nik,

This started out as a comment on Nik Silver’s post but quickly turned into a post of its own.

I won’t get into the details of the extent to which the newspaper industry is dying, or why, because there are plainly more than enough commentators doing that already, many of them ill-informed and/or self-important. It’s pretty easy to stand on the sidelines of an industry experiencing a decline, in circulation though not necessarily in power or influence, and shout out well-meaning advice, much as one might stand on a river bank and shout “try boogie boarding” to a swimmer experiencing some trouble in the choppy water. But for all their chutzpah, some of these commentator’s suggestions do contain elements of truth which the newspaper industry would do well to acknowledge, and indeed sections of which have already done so.

Act like a startup

The point which irritates you so, Nik, understandably, the instruction to ‘act more like a startup’, is a naive one, and I’m not altogether sure the people shouting it from the river bank really understand what they themselves mean. But in its naivety and potential for misunderstanding there are some interesting concepts. ‘Startup’ is a useful shorthand amongst the never knowingly buzzword-averse self-congratulatory internet community for the kind of dynamic, vibrant, go-getting bunches of technically savvy kids that bubble up on a regular basis from sun-kissed California or, at a stretch, from the environs of a grey roundabout in London, to invent a cure-all solution to everyone’s online needs that will change your life, alter your children’s futures and also enable your fridge to tell you when you’re out of milk, and whose businesses seemingly run on smiles and laughter instead of cold hard cash.

People in other industries don’t run startups. They run ’small businesses’ — decidedly less glamourous — or, gasp, they are self-employed — about as far from glamour as it’s possible to get (all that paperwork…) But most people have never worked for a startup, and so don’t understand how they operate. Most startups do actually have investors, whether they are parents, friends, extended family or even plain old bank loans. Further, any startup you’re ever likely to hear about through TechCrunch or similar has proper, big investment from other businesses, and a smaller portion still have insulation from market forces and major investment from venture capitalists, themselves usually branches of multinational investment banks. So, how to reconcile these two views of startup culture? The answer is that startup culture is a concept based on a perception of the startup ecosystem as a whole. That’s where the idealism bordering on the fantasy originates. But while the distinction is important, it’s the perception that we’re interested in, and that could be helpful to the newspaper industry.

Culture and DNA

Simon’s point, that startups and pre-web established businesses have vastly different origins and thus business models — different corporate DNA, as he says — is a good one, but even he would have to admit that that is a weakness of the newspaper industry, which will need to adapt to survive.

My point, which I’ll illustrate here, is that when people tell newspapers, or any other industry, to act more like startups, the are invoking the visible perception of the startup culture as a whole, not any single element of it. And of that whole, people see only the successes. Of course, there is one benefit of this view — your sample is self-selecting — we only need to look at those startups which became successful and ‘made it’ (got bought, IPO’d) because they must have been doing something right. But if ‘traditional’ businesses can harness just a little of the spirit of that culture, the — yes — idealism of it, then they’ll be well on their way to changing themselves for the better.

Readiness to fail

To outsiders, startup culture shows a willingness to experiment, and a readiness to fail. Never mind that startups themselves are atomised to a massive extent — there are a dozen startups devoted just to monetizing the upload of pictures to Twitter, itself still, ostensibly, a startup — to an observer the startup ecosystem is willing to try any possible avenue of revenue, no matter how niche, and if it doesn’t work to try another one. Low overheads and low headcounts means it’s relatively pain- and friction-free for a startup to close down and it’s proprietor to try again another day. Many entrepeneurs are famously serial startup start-uppers. Newspapers, indeed most big businesses, for good or bad, can’t afford to try their hand at every passing opportunity that comes along. They have overheads, high staff levels, investors and strong unions to consider. But those same factors also mean they have the infrastructure and the backing to take well-considered risks, and they should take risks more often. One hindrance to a newspaper is the line of editorial policy that runs through everything they do. Newspapers, to some people’s fury, sometimes represent more than just business interests. They represent a political ideology, a social responsibility. They also represent decades or centuries old brand names. But the industry is surely mature enough that a newspaper can diversify its voice enough to allow some of its ventures to fail admirably? What is a brand name if it’s not strong enough to absorb an under-performing product.

Relentless innovation

To outsiders, startup culture looks relentlessly innovative. And it is. They’re often solving problems you never even knew you had. Usually small, driven by a strong independent spirit and urge to be the first to market, they are unusually open, agile and responsive to customer requests — having ’strategic agility’, in Simon’s words. They literally cannot afford to lost their market niche due to bad software or bad will. To this end, one of the most important points is that startups are not afraid to be in beta. Having your software released in beta status has become such a fetish it’s almost a parody. But the meaning is plain to see — the software is a work in progress, and will progress based on customer feedback. To come out of beta signifies an end to progress. I’m sure we’ll grow out of this silly fad, but for now it’s a useful shorthand that your software vendor is listening to you.

Technology can save you

The last point I’ll make, probably the most important one, is that startup culture — the culture everyone in the media is referring to when they talk about startup culture, not small business culture and your local council’s small business forum — startup culture is technology-led. That is, technology is the driving force behind the products, the innovation, the business itself. Technology is the fabric of the new economy, it pushes at what’s possible, and an industry has sprung up to exploit that. Three of the biggest failing industries right now are the music industry, the film industry and newspaper publishing. Three content-led industries right there. Run for years, decades, by content people. Content people and lawyers. While content producers excel at producing what they produce, and lawyers will always be necessary, neither of these groups is in any way qualified to predict where technology could take them, or how it threatens them. The music industry in particular buried it’s head in the sand and hoped technology would go away. Hollywood got a clue, a little too late after spending most of it’s technology energy on DVD copy-protection, ignorant to the fact that for every developer writing encryption software there were 10 more able to write decryption software. The music industry is pulling out of its nosedive. So how did these two behemoths manage it? Technology. The film industry, led by technology, is deploying digital cinema distribution, special 3d camera’s and projectors, and, led by sony, a technology company, Blu-Ray discs for home viewing. The music industry has been pulled out of its hole by, among others, Apple, and I’m sure Amazon and Nokia are helping out too. That leaves newspapers. While certain players are making headway based on technology platforms which were designed and built this century, many, many more are struggling to even comprehend how technology can not only help them, but save them.

I have the luxury of speaking from the trenches, working daily on the nuts and bolts of news on the web in the software development of a national newspaper, rather than on the board of that same newspaper having to make massive strategic decisions about the future of the company. But, if I had one piece of advice for anyone working at a newspaper and wondering where the future was going to happen, or how it might help them, or how they might help themselves, I’d point them in the direction of their technology department, if they have one. Go grab a software developer and ask them what they could build for you. If it’s one the web, it starts here, in ideas and code…

IE6 state of play linkage

Friday, August 14th, 2009

A few links to recent IE6 coverage.

Digg the Blog: Much Ado About IE6
“This goes directly to why most folks use IE6: they don’t have a choice. Three out of four IE6 users on Digg said they can’t upgrade due to some technical or workplace reason…Giving them a message saying, ‘Hey! Upgrade!’ in this case is not only pointless; it’s sadistic.”

BBC News | Technology: Microsoft backs long life for IE6 [support will continue until 2014, four years beyond their original deadline]
“‘Friends do not let friends use IE6,’ said Amy Barzdukas, Microsoft’s general manager for Internet Explorer.
‘If you are in my social set and I have been to your house for dinner, you are not using IE6,’ she said. ‘But it is much more complicated when you move into a business setting.’”

blogs.msdn.com | IEBlog: The Engineering Point Of View
“The engineering point of view on IE6 starts as an operating systems supplier. Dropping support for IE6 is not an option because we committed to supporting the IE included with Windows for the lifespan of the product. We keep our commitments.”

MoD to stick with IE6 despite security concerns
“According to parliamentary written answers received by Labour MP Tom Watson, the majority of [government] departments still require staff to use IE6. Most have plans to upgrade to the more secure IE7, and some to IE8, but the Ministry of Defence (MoD) has no plans to change.”

Conclusion: IE6, at least in corporate environments, isn’t going away for a few years yet.

Using Greasemonkey to Pimp the Telegraph’s MPs’ Expenses Data

Monday, July 13th, 2009

You may have noticed that the Telegraph have some useful MPs’ expenses data on their site. To go with it they’ve also got some nice graphs, which show an MP’s historical expense claim data. The trend of those claims is, as you might expect, mostly upwards. No surprises so far. But, until recently, inflation has always been a positive number, so you would be probably be right to expect expense claims to rise in tandem with the prices of everyday goods and services.

Unpimped Telegraph MPs' expense data charts
Above: the unpimped image of Michael Ancram’s expenses data chart

But I don’t find the graphs useful enough. I can easily see the trend of the claims just by looking at the accompanying data table. What I really want to know is how closely the rises in MPs’ expenses matched the standard inflation rate. My hunch was that the rise in value of the MPs’ claims would be ahead of the inflation rate, so how best to check this? And how to check it for a wide number of claimants? In addition, I wanted to visualise a breakdown of the claim types – for travel, stationary, 2nd home etc.

Pimping the Telegraph's expense data charts
Above: an image of Michael Ancram’s expenses data chart following the addition of my Greasemonkey script.

I was interested enough to ask the Telegraph, via Twitter, if they had any plans to add some economic context to the data. They actually responded to let me know that the data was available in a spreadsheet, so why not go ahead and mash it up? Great! But on checking the spreadsheet I found that the data only went back to 2004, while the data on the Telegraph’s website went back to 2001. I’m aware that expenses data going back further are available from the Houses of Parliament website, but I didn’t really have to time to go searching for them, converting the format to what I needed etc. So I decided to write a quick Greasemonkey script to operate on the data already published by the Telegraph, on their website, which happened to be good enough markup to extract data from and use.

Greasemonkey is a Firefox plugin which enables you to write JavaScript scripts which can operate on the contents of a web page solely within your browser. Anything you do to the data does not alter the original in any way, and so it’s a great way of changing the way websites work, customising them to look how you like and generally mucking about with them.

Here’s some technical info on what I did:

First I checked the Office of National Statistics for the annual inflation data from the Retail Price Index, the broadest measure of inflation, which includes mortgage interest payments. I grabbed data for the years covered by the Telegraph data — 2001/02 to 2007/08 and added it to my script. Then I loaded in a couple of extra scripts which help to grab the data from the page, draw a new graph add it to the page in place of the original.

I extract the data from the MP’s expenses table, which is actually the same way the Telegraph are drawing the graph in the first place. The difference is that I extract each row of data rather than just the totals. I draw all the data to the graph, then add an extra line which extrapolates the MP’s first year total claim across the whole term of the graph, based on the inflation data for the subsequent years. Finally, I added a series of checkboxes to isolate different data sets within the graph, so you can compare, say, just an MP’s claims for food against her claims for stationery.

The script was fairly quickly put together, so if you’re handy with a bit of JavaScript you might be able to think of something I haven’t. For example I’d like to see a comparison of MPs’ claims against nursing or teaching salaries. Even if you just want to hack around you can download it and amend it as much as you like without any danger. But here’s the really great thing — I’ve actually maintained most of the structure of the Telegraph’s original graph code, so if they want to take my script back and use it on their page they’re free to do so, and it should work perfectly!

Was this hack useful? Yes. For example, it allowed me to see that Ian Austin, Labour MP for Dudley North, made below-inflation claims rises for 2006/07 and 2007/08. And that Labour MP for Leeds North East Fabian Hamilton’s 2007/08 total claim (£170,794.00) was almost double what it would have been if his first recorded claim (£75,437.00) from 2001/02 had risen in line with inflation.

You can download my Greasemonkey script here. If you don’t want to do that (or your IT department won’t let you) you can see from the images above how the script operates on the original website.

Some notes about using this script in Greasemonkey:

I wrote this script using Greasemonkey 0.8.2. As of version 0.8 you can use the @require metadata key to load JavaScript libraries into your user-script. Greasemonkey will download those scripts to your local directory, meaning your browser isn’t increasing network traffic every time you run the script. If for any reason you need to use an earlier version of the plugin you’ll find some useful information here on loading in scripts without the @require key.

We don’t want the script to execute except when we’re looking at a Telegraph MPs’ expenses page, so for best performance you should set this script to run on the following URLs:

http://parliament.telegraph.co.uk/mpsexpenses/expenses/*
http://parliament.telegraph.co.uk/mpsexpenses/expense-microsite/expenses/*

Some good links to try with the script installed: