Development at paulcarvill.com, the home of Paul Carvill on the web

link: paulcarvill at flickr

paulcarvill.com

Hi, I'm Paul Carvill and I'm a web developer. I am Head of Interface Development at LBi, Europe's largest digital agency.

I also like walking, cooking, Bollywood and rock 'n' roll.

Archive for the ‘Development’ Category

Newer Entries »

Why front-end developers are so important to the future of businesses on the web

Thursday, September 24th, 2009

or How traditional businesses who have moved to the web regularly undervalue their front-end web developers, and are worse off because of that

Distinction
The roles of web developers and web designers have been around for over 15 years now, and the role of a client-side or front-end web developer started to mature into a distinct entity around 10 years ago, as the content-presentation-behaviour layer paradigm became embedded in people’s working methodologies (and, with the introduction of Google’s then-new search algorithm, when the need for cleanly structured, easily indexable pages became, for businesses, not just an aspiration but a necessity). Unfortunately the perception of the front-end developer’s role remains somewhat coloured by an early association in observers’ minds with the other, loosely related role of the web designer. The role of web designer is an extremely important and valid one, but it is very different to that of the web developer, and the lack of a clear distinction between the two, in some people’s perception, is unhelpful and does both roles an injustice.

Skill set
The web developer (sometimes also called a client-side developer, front-end developer, web architect or front-end engineer) has a huge skill set and a job description to match. They are often expected and required to excel in many disciplines, and have good working knowledge of many others. They exist at the point where art, design, interaction, programming and behavioural and performance analysis intersect. Given the time, support and ambition of a good business, being a web developer can be an extremely fulfilling job. However, the role of a web developer is often misunderstood within even the most progressive and well-meaning of businesses.

Perception
The danger can be that front-end developers, working in a user-focused area, are seen as performing a superficial function — applying a polish to the heavy lifting done by another developer, say, or that dread comment, “making things look nice”. Let’s be clear, making things look nice is the sole responsibility of the designer. When front-end developers spend much of their time deploying underlying data received from a backend database into their views, or pages, they might mistakenly be thought of as merely translators or interpreters, transferring a graphical image — the Photoshop-ed design — into markup and style rules, purveyors of what is sometimes almost mockingly referred to as a ‘black art’ of making pixels lay out correctly onscreen. While this perception is perhaps unfortunate, it is understandable. It is a particular problem where a development workflow is — some might say artificially — segregated into database infrastructure/domain modeling/server side workflows/front-end workflows. In smaller organisations a front-end developer has the opportunity, if she wishes, to input into any of these areas. In larger organisations, the increased granularity of functional areas means those opportunities are greatly reduced, and as you can see from the segregation model above, the front-end development work comes at the end of a long chain of events and decisions which essentially shape and restrict the front-end developer’s choices.

Frustration
In such cases the development workflow is one-way, negates the developer’s architectural, organisational and behavioural skills and occurs late in the development process. This chronology minimizes the opportunity for the front-end developer to have effective input into, and feedback from, the interaction design they are now expected to code. This is a sad state of affairs and undoubtedly leads to frustration, feelings of being undervalued or ignored, and an extreme cases disenfranchisement and resignation, either in the figurative or practical sense. A good business will understand how highly-nuanced user behaviour is, and value skilled interpretation and shaping of that behaviour in the interests of improving their digital offering.

Value
The modern web developer has huge amounts of value to offer a business. Indeed the type of professional you often find in this role encapsulates the very best the web has to offer:

  • up-to-date knowledge of available and emerging technologies
  • extensive experience of implementing de facto web standards and programming patterns
  • database configuration and data manipulation
  • implementation across multiple platforms and legacy software applications
  • provisioning for mobile devices
  • data aggregation
  • graphics sourcing and creation
  • search engine optimisation (SEO)
  • a thorough understanding of the aesthetics and parameters of designing for the web

Further, the best web developers have wealth of knowledge and understanding around interaction design, user needs, hierarchies of data, navigation systems, user journeys, wireframing, design brief interpretation, focus group and usability testing and the art of a finely polished product. Largely gone are the days of HTML-monkeys, spending days on end converting Photoshop comps to pixel-perfect layouts. A web developer’s role is broad: from developing in what Yahoo!’s Nate Koechley calls ‘the world’s most hostile development environment’ — the browser — and ensuring cross-platform and cross-browser consistency, to working with art directors and designers and remaining true to their vision, to considerations and implementations of accessibility, usability and the overall user experience. A web developer is responsible for everything that sits on the client side of the web stack — the content, presentation and behaviour layers. Few other roles touch so many other key aspects of a business as does a web developer’s. Good businesses realise what an asset they have in their front-end web development team, and welcome their input into the product development process. Even better businesses have a User Experience team which encapsulates all those values, skills and judgements necessary to make great websites. Members of those teams are part of a feedback loop that results in great products, not just acceptable implementations of the first good idea that came up.

Slow
Large businesses and organisations move slowly. They may find it hard to understand they have developers whose skills and interests cross the boundaries their job descriptions impose on them. In addition, large businesses like to modularise their development teams into clearly segmented areas for planning and accountability. End-to-end developers don’t really fit this business model. Web development, certainly rapid prototyping at least, is moving away from monolithic relational database installs and towards schema-free, fluid data repositories like CouchDB and MongoDB. Many other layers of the application stack are now capable of being managed by a web developer. Most developers of this ilk, who are able to own the whole process from from domain and model definition, through to server infrastructure and on to a useful and appealing user experience, are running their own consultancies or are employed by the more enlightened web properties. Some examples of this type of person are Jeff Croft, Dan Rubin, John Resig, Jeffrey Zeldman, among many others. Functionality, data storage and interaction are increasingly moving to the client side (HTML5, Gears, RIAs, iPhone and Android web apps). The web stack sits on top of any technology, making the web developer one of the most versatile members of any business, let alone the technology department.

Undervalued
Unfortunately my experience has been that most large businesses massively undervalue their web developers, employing them in narrowly defined roles as the guys who make the site look nice, or fix the Javascript bugs that make the page break in IE. Larger business and the public sector have made moves towards working seriously on accessibility and usability, but the thinking behind such strategies remains superficial, those conceptual areas misunderstood. Too many people think of them in terms of awards and rating levels, not in ongoing process of improvement. Most of this work is also likely to be outsourced to a specialist third-party.

Career progression
One further, worrying, complication is the lack of clear career progression for front-end developers. Once you’ve spent a year or ten working in every nook and cranny of every browser out there until you can code progressively enhanced web pages blindfolded, what next? Economics and management structures mean there’s only so many architect or senior engineer roles to go round. The other option is to specialize in something and move forward from there. Current trends would seem to be leaning towards a big need for performance specialists in the next 5 years, as the client side moves ever further towards accommodating distributed, complex web applications. Page views will also continue their inexorable rise, placing stress and demand on infrastructure, databases and hardware, and thus even greater stress on a fast, responsive user experience.

Know your value
What does this mean if you’re a front-end or client-side web developer? Know your value. What are your skills? Are you a developer, an engineer, a User Experience architect or an Interaction Designer? Advertise your value. Shout about it. Don’t be knowingly undermined or ignored. Create a User Experience team within your business. If they’ve already got one, join it! Your job is incredibly important, and your present employer needs to realise that, as they stand to benefit.

Updated, 29th September 2009: just fixed a couple of typos

Posted in Browsers, Design, Development, Internet, Usability, User Experience, Web design, Work | 55 Comments »

Wallpaper* Open House map is a prime example of awful Flash interface design

Saturday, September 19th, 2009

Further to my previous post regarding the Adobe-Omniture deal, and how it might improve the consistently awful Flash user experience people regularly have to deal with, this morning I found a very good example of bad Flash interface design to illustrate my point with.

The weekend of 19th and 20th September is London Open House weekend, when the public gets wide and varied access to the architectural pleasures of the City. Unfortunately, while Open House do publish a printed programme, the only way to discover locations of Open House participants online is through a fairly complex and ugly search form at http://www.londonopenhouse.org/public/london/find/. This method is not an intuitive way to find Open House locations, and does not encourage discovery or serendipity.

However, there is a map at Wallpaper magazine’s website. While this could have been a useful alternative to the official listings, it is instead problematic and discouraging. The Wallpaper site features a Google map, overlaid with a Flash map. There are several problems:

  • The Wallpaper Flash map features only a selection of the full listings. They call this the Wallpaper Edit. For the full listing you must still consult the printed programme or use the official search form.
  • The Wallpaper Flash map is extremely simplified, and lacks context, scale and travel information, as well as several other navigation aids.
  • The Wallpaper Flash map uses an entirely new and different navigation method. There is no drag control, and no zoom. Instead, the user needs to click on buttons near the top of the map which signify arbitrary areas of London. There is no option to show partial areas or the whole map at once.
  • The Wallpaper Flash map uses an entirely different scale to the accompanying standard image-based Google map. Changes in the orientation of one map are not reflected in the other, making switching between the two extremely disorienting.
  • The Wallpaper Flash map uses bespoke markers and a bespoke popup system, which opens a popup at the top left of the map, completely out of context of the click event. The Google map operates in an expected way, much as other web standards-based online maps do, by opening the popup at the location of the click event, where the user is focused.

The frustrating thing about Wallpaper using a custom Flash interface is that the Google map view already provides all the functionality the user needs, in an established, expected way, and in many ways in offers a superior user experience. Control of the Google map is more granular and direct, and the user benefits from the extra context and transport directions functionality. In addition, in case of the popup, the Google map provides more information, in the form of opening times. Even simple things, like the detail text being copyable, make the Google map a far more suitable implementation.

The Wallpaper Open House Flash map interface is a perfect example of the continued redundancy of Flash interface design, it’s ability to encourage design abuse, the lack of understanding of user behavioural patterns and the skewed importance of brand over usability.

Posted in Development, Internet, Usability, User Experience, Web design, flash | Comments Off

Technology can, and will, save the newspaper publishing industry

Tuesday, August 25th, 2009

Nik,

This started out as a comment on Nik Silver’s post but quickly turned into a post of its own.

I won’t get into the details of the extent to which the newspaper industry is dying, or why, because there are plainly more than enough commentators doing that already, many of them ill-informed and/or self-important. It’s pretty easy to stand on the sidelines of an industry experiencing a decline, in circulation though not necessarily in power or influence, and shout out well-meaning advice, much as one might stand on a river bank and shout “try boogie boarding” to a swimmer experiencing some trouble in the choppy water. But for all their chutzpah, some of these commentator’s suggestions do contain elements of truth which the newspaper industry would do well to acknowledge, and indeed sections of which have already done so.

Act like a startup

The point which irritates you so, Nik, understandably, the instruction to ‘act more like a startup’, is a naive one, and I’m not altogether sure the people shouting it from the river bank really understand what they themselves mean. But in its naivety and potential for misunderstanding there are some interesting concepts. ‘Startup’ is a useful shorthand amongst the never knowingly buzzword-averse self-congratulatory internet community for the kind of dynamic, vibrant, go-getting bunches of technically savvy kids that bubble up on a regular basis from sun-kissed California or, at a stretch, from the environs of a grey roundabout in London, to invent a cure-all solution to everyone’s online needs that will change your life, alter your children’s futures and also enable your fridge to tell you when you’re out of milk, and whose businesses seemingly run on smiles and laughter instead of cold hard cash.

People in other industries don’t run startups. They run ’small businesses’ — decidedly less glamourous — or, gasp, they are self-employed — about as far from glamour as it’s possible to get (all that paperwork…) But most people have never worked for a startup, and so don’t understand how they operate. Most startups do actually have investors, whether they are parents, friends, extended family or even plain old bank loans. Further, any startup you’re ever likely to hear about through TechCrunch or similar has proper, big investment from other businesses, and a smaller portion still have insulation from market forces and major investment from venture capitalists, themselves usually branches of multinational investment banks. So, how to reconcile these two views of startup culture? The answer is that startup culture is a concept based on a perception of the startup ecosystem as a whole. That’s where the idealism bordering on the fantasy originates. But while the distinction is important, it’s the perception that we’re interested in, and that could be helpful to the newspaper industry.

Culture and DNA

Simon’s point, that startups and pre-web established businesses have vastly different origins and thus business models — different corporate DNA, as he says — is a good one, but even he would have to admit that that is a weakness of the newspaper industry, which will need to adapt to survive.

My point, which I’ll illustrate here, is that when people tell newspapers, or any other industry, to act more like startups, the are invoking the visible perception of the startup culture as a whole, not any single element of it. And of that whole, people see only the successes. Of course, there is one benefit of this view — your sample is self-selecting — we only need to look at those startups which became successful and ‘made it’ (got bought, IPO’d) because they must have been doing something right. But if ‘traditional’ businesses can harness just a little of the spirit of that culture, the — yes — idealism of it, then they’ll be well on their way to changing themselves for the better.

Readiness to fail

To outsiders, startup culture shows a willingness to experiment, and a readiness to fail. Never mind that startups themselves are atomised to a massive extent — there are a dozen startups devoted just to monetizing the upload of pictures to Twitter, itself still, ostensibly, a startup — to an observer the startup ecosystem is willing to try any possible avenue of revenue, no matter how niche, and if it doesn’t work to try another one. Low overheads and low headcounts means it’s relatively pain- and friction-free for a startup to close down and it’s proprietor to try again another day. Many entrepeneurs are famously serial startup start-uppers. Newspapers, indeed most big businesses, for good or bad, can’t afford to try their hand at every passing opportunity that comes along. They have overheads, high staff levels, investors and strong unions to consider. But those same factors also mean they have the infrastructure and the backing to take well-considered risks, and they should take risks more often. One hindrance to a newspaper is the line of editorial policy that runs through everything they do. Newspapers, to some people’s fury, sometimes represent more than just business interests. They represent a political ideology, a social responsibility. They also represent decades or centuries old brand names. But the industry is surely mature enough that a newspaper can diversify its voice enough to allow some of its ventures to fail admirably? What is a brand name if it’s not strong enough to absorb an under-performing product.

Relentless innovation

To outsiders, startup culture looks relentlessly innovative. And it is. They’re often solving problems you never even knew you had. Usually small, driven by a strong independent spirit and urge to be the first to market, they are unusually open, agile and responsive to customer requests — having ’strategic agility’, in Simon’s words. They literally cannot afford to lost their market niche due to bad software or bad will. To this end, one of the most important points is that startups are not afraid to be in beta. Having your software released in beta status has become such a fetish it’s almost a parody. But the meaning is plain to see — the software is a work in progress, and will progress based on customer feedback. To come out of beta signifies an end to progress. I’m sure we’ll grow out of this silly fad, but for now it’s a useful shorthand that your software vendor is listening to you.

Technology can save you

The last point I’ll make, probably the most important one, is that startup culture — the culture everyone in the media is referring to when they talk about startup culture, not small business culture and your local council’s small business forum — startup culture is technology-led. That is, technology is the driving force behind the products, the innovation, the business itself. Technology is the fabric of the new economy, it pushes at what’s possible, and an industry has sprung up to exploit that. Three of the biggest failing industries right now are the music industry, the film industry and newspaper publishing. Three content-led industries right there. Run for years, decades, by content people. Content people and lawyers. While content producers excel at producing what they produce, and lawyers will always be necessary, neither of these groups is in any way qualified to predict where technology could take them, or how it threatens them. The music industry in particular buried it’s head in the sand and hoped technology would go away. Hollywood got a clue, a little too late after spending most of it’s technology energy on DVD copy-protection, ignorant to the fact that for every developer writing encryption software there were 10 more able to write decryption software. The music industry is pulling out of its nosedive. So how did these two behemoths manage it? Technology. The film industry, led by technology, is deploying digital cinema distribution, special 3d camera’s and projectors, and, led by sony, a technology company, Blu-Ray discs for home viewing. The music industry has been pulled out of its hole by, among others, Apple, and I’m sure Amazon and Nokia are helping out too. That leaves newspapers. While certain players are making headway based on technology platforms which were designed and built this century, many, many more are struggling to even comprehend how technology can not only help them, but save them.

I have the luxury of speaking from the trenches, working daily on the nuts and bolts of news on the web in the software development of a national newspaper, rather than on the board of that same newspaper having to make massive strategic decisions about the future of the company. But, if I had one piece of advice for anyone working at a newspaper and wondering where the future was going to happen, or how it might help them, or how they might help themselves, I’d point them in the direction of their technology department, if they have one. Go grab a software developer and ask them what they could build for you. If it’s one the web, it starts here, in ideas and code…

Tags: , ,
Posted in Development, Internet, Newspapers, Work | Comments Off

A collection of links about JavaScript and the MVC development pattern

Wednesday, August 19th, 2009

I’ve been doing most of my JavaScript development over the past 12 months in a Model-View-Controller pattern. This separation of function has a multitude of benefits for a programmer — it enables writing of clearer, more readable code, can result in more modular, reusable code and, perhaps most importantly, it enables a Test-driven approach to development.

Now A List Apart has published their own JavaScript MVC tutorial. It’s very useful, and very easy to follow. I recommend it as a starting point if you’re interested in this method of development.

Developing JavaScript in an MVC pattern is an especially good fit for web developers familiar with the now widespread, if not quite ubiquitous, separation of form and content in their programming methodology. Something like the paradigm which you apply to delivery of web content using HTML to provide data and CSS to provide the look and feel can be applied to JavaScript in an MVC pattern, where the Model provides the data, as might a server-side database, the Controller plays the part of the web server delivering that data and the View is the browser itself, rendering content to the page.

I’d also encourage developers to roll their own MVC framework rather than dive right in and using one of the existing ones, at least initially. It will give you a much better understanding of the pattern, it’s benefits and probably pitfalls, and will let you comfortably learn to extend your programming prowess at your own pace.

Note: you could also implement JavaScript on the server side using MVC, with the caveat that your View function is possibly less conceptually decoupled, but once you’re happy with the pattern this shouldn’t hamper you too much.

A List Apart: JavaScript MVC by Jonathan Snook
“At this stage of JavaScript and Ajax development and adoption, we need to consider separating our code’s components—MVC-style. Such separation may not be necessary in every situation—in some cases, it may even make things needlessly verbose. As our applications get more complex and require JavaScript interactions across multiple parts of our sites, however, separating JavaScript into Model, View, and Controller parts can produce more modular, reusable code.”

Ben Godfrey: Client-side MVC is maturing
“MVC is the dominant model for UI development in the desktop world. A modified form is very popular in web development. It makes a lot of sense to stick to MVC for RIA development. RIAs running in the browser (or Flash or AIR) bear close resemblance to desktop applications.”

A list of JavaScript MVC libraries and associated stuff:

I don’t use any of these, but they might be helpful for developers in a hurry. To be honest, the vast majority of my stuff is really VCS (View, Controller, Service), where the Service layer may be a remote API or HTML fragment which you needn’t model, but adding a Service layer isn’t a great leap once you’ve made the conceptual jump from more basic procedural JavaScript programming to MVC.

Tags:
Posted in Development, JavaScript, Work | 2 Comments »

IE6 state of play linkage

Friday, August 14th, 2009

A few links to recent IE6 coverage.

Digg the Blog: Much Ado About IE6
“This goes directly to why most folks use IE6: they don’t have a choice. Three out of four IE6 users on Digg said they can’t upgrade due to some technical or workplace reason…Giving them a message saying, ‘Hey! Upgrade!’ in this case is not only pointless; it’s sadistic.”

BBC News | Technology: Microsoft backs long life for IE6 [support will continue until 2014, four years beyond their original deadline]
“‘Friends do not let friends use IE6,’ said Amy Barzdukas, Microsoft’s general manager for Internet Explorer.
‘If you are in my social set and I have been to your house for dinner, you are not using IE6,’ she said. ‘But it is much more complicated when you move into a business setting.’”

blogs.msdn.com | IEBlog: The Engineering Point Of View
“The engineering point of view on IE6 starts as an operating systems supplier. Dropping support for IE6 is not an option because we committed to supporting the IE included with Windows for the lifespan of the product. We keep our commitments.”

MoD to stick with IE6 despite security concerns
“According to parliamentary written answers received by Labour MP Tom Watson, the majority of [government] departments still require staff to use IE6. Most have plans to upgrade to the more secure IE7, and some to IE8, but the Ministry of Defence (MoD) has no plans to change.”

Conclusion: IE6, at least in corporate environments, isn’t going away for a few years yet.

Tags: ,
Posted in Browsers, Development, Internet, Uncategorized, Web design, Work | Comments Off

HTML5 HTML text semantics granularity

Thursday, June 18th, 2009

Wow, HTML5 HTML semantic text description options are so granular I had to spend several minutes pondering whether my previous code snippet about relaxing Apache permissions warranted <code>, <kbd>, or <samp> elements, or a combination of all three.

In the end I settled on a <kbd> element for the bit I want you to type in (opening a file in vi from the command line), as that’s the bit you’re going to type. For the contents of the file I chose a <samp> element, as the text shown in the file is a sample of the output of my file, rather than a chunk of code you need to enter.

The difference between <code> and <samp> is very small, but it’s great that we actually now have this level of specificity, which should help ensure that HTML5 HTML is robust enough to last well into the future.

Tags:
Posted in Code snippets, Computing, Development, Web design, html5 | 2 Comments »

Geocoding location data in a Google spreadsheet

Wednesday, June 3rd, 2009

The problem: I have a spreadsheet full of locations, addresses and place names that I want to publish, along with a map, for at least tens of thousands of people to view.

A solution: Easy — I can put it in a Google spreadsheet, publish it, add a Google map to a page, download the data, geocode the locations and display them on the map.

Another problem: While this is ok in most cases, with a large spreadsheet the geocoding can take a very long time, making my page appear unresponsive and slow. In addition, I have no way of checking that the location data is good enough to map with.

Another solution: Download the data, geocode it using Yahoo!’s Placemaker service, generate a new spreadsheet containing accurate latitude/longitde data and use that in place of the original. The client then does no geocoding their side, it’s all supplied along with the data. Everybody’s happy!

— Go straight to the spreadsheet geocoder! —

I’ve done just that with this PHP script. It takes a Google spreadsheet key, and you must tell it what columns your location data is in. It will download the spreadsheet data, concatenate those location columns, make a request to Placemaker to geocode each location, and return a new CSV file with the geodata columns appended on the end.

I’ve detailed here the various bits that make up the script. The workflow is as follows:

Capture spreadsheet data from user > Load in spreadsheet from Google > For each line in spreadsheet make a Placemaker request > Append geolocation data columns to spreadsheet > Output all results into a CSV file

The script is set to not autodisambiguate, meaning that if it’s not sure what location you’ve supplied, it will return all likely candidates, in order of likelihood. I should mention that Yahoo!’s Placemaker is utterly awesome in find out the ‘whereness‘ of things.

To build your own version of my script will need a Placemaker API key. Other than that, please feel free to copy and paste the code, fix it, amend it and let me know if it’s useful, or if it needs more commenting, or how I could improve it. I wrote this code to fix a particular problem I was encountering, but I’m sure it could work in a few more cases too.

Something to note before I start: the script doesn’t much like having commas in the location data in your spreadsheet. Because Google only output CSV with a comma delimiter, this upsets my CSV parsing. Any suggestions welcome.

This function gets some CSV data from a published Google spreadsheet using a supplied key:


<?php

function getCsvDataFromGoogle($spreadsheetKey) {
	$key = $spreadsheetKey;
	$output = 'csv';
	$apiendpoint = 'http://spreadsheets.google.com/pub?key='.$key.'&output='.$output;
	$ch = curl_init();
	$options = array(CURLOPT_URL => $apiendpoint,
	                 CURLOPT_HEADER => false,
	                 CURLOPT_RETURNTRANSFER => true
	                );
	curl_setopt_array($ch, $options);
	$r = curl_exec($ch);
       curl_close($ch);
	return $r;
}

This function makes a Placemaker geocode request:


function getPlacemakerGeodata ($location) {
	$key = 'MY_PLACEMAKER_API_KEY';
	$apiendpoint = 'http://wherein.yahooapis.com/v1/document';
	$inputType = 'text/plain';
	$outputType = 'xml';
	$focus = '28298150'; // sets focus to Great Britain, not sure how effective this is yet
	$autoDisambiguate = 'false'; // returns the 1 most-likely place, else returns many likely places
	$post = 'appid='.$key.'&documentContent='.$location.'&documentType='.$inputType.'&outputType='.$outputType.'&focusWoeid='.$focus.'&autoDisambiguate='.$autoDisambiguate;
	$ch = curl_init($apiendpoint);
	curl_setopt($ch, CURLOPT_POST, 1);
	curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	$results = curl_exec($ch);
	return $results;
}

This function does the bulk of the work, and makes calls to all the other functions:


function parseCsvData($googleSpreadsheetKey) {
	$lines=split( "\n", getCsvDataFromGoogle($googleSpreadsheetKey) );
	if($_POST['format'] == 'csv') {
		if($_POST['locationColumns'] == '' || $_POST['key'] == '') {
			echo "please go back and specify both your google spreadsheet key and which columns contain your location data (in comma separated format, zero-indexed e.g. 0,1,9)";
			exit();
		}
		else {
			// get location columns from url
			$locations = $_POST['locationColumns'];
			$splitLocations = split(',', $locations);
			// set headers to 'csv'
			header("Content-type: application/csv;");
			header("Content-Disposition: attachment; filename=yourgeodata.csv");
			$out = fopen('php://output', 'w');
			for($i=1;$i

This function parses the XML which gets returned from Yahoo! Placemaker:


function parsePlacemakerXML($results, $delineator) {
	if($delineator == 'comma') { $delStart = ''; $delEnd = ','; }
	else { $delStart = '<td>'; $delEnd = '</td>'; }

	$places = simplexml_load_string($results, 'SimpleXMLElement', LIBXML_NOCDATA);
	$locarr = array();
	if($places->document->placeDetails) {
		foreach($places->document->placeDetails as $p) {
			if($delineator == 'comma') {
				$locarr[] = $p->place->name;
				$locarr[] = $p->place->centroid->latitude;
				$locarr[] = $p->place->centroid->longitude;
				return $locarr;
			}
			else {
				echo $delStart.$p->place->name.$delEnd;
				echo $delStart.$p->place->centroid->latitude.$delEnd;
				echo $delStart.$p->place->centroid->longitude.$delEnd;
			}
		}
	}
}

This bit runs when you load the page and works out if you're submitting some data or just viewing the page. If you've submitted data, it runs the main function:

if(ISSET($_POST['submit'])) {
	parseCsvData($_POST['key']);
}

Or if you're viewing the page for 1st time, you get a form to fill out:

else {
	echo "<html><head><title></title></head>";
	echo "<body>";
	echo "<p>Please enter your spreadsheet key and specify which columns contain your location data (use comma separated list e.g. 9,10,11):</p>";
	echo "<form method=POST><p><label>Key:<input type='text' name='key' /></label></p><p><label>Location columns: <input type='text' name='locationColumns' /></label></p><p><label>Format: <select name='format'><option value='csv'>csv</option><option value='table'>table</option></select></label></p><p><input type='submit' name='submit' /></p></form>";
	echo "</body></html>";
}
?>

Tags: , , , , ,
Posted in Computing, Development, Internet, Web design | Comments Off

Why people should build their own URL shorteners

Sunday, April 5th, 2009

Lots of blogposts regarding the evil that URL shortening services do appeared this weekend, from Jason Kottke (“URL shorteners suck”), Joshua Schachter, creator of Delicious (“on url shorteners”) and Dave Winer (“Josh is right, URL shorteners are risky”).

None of them are happy, with concerns, variously, about spam, speed, efficiency, transparency, longevity and what Joshua calls “the great linkrot apocalypse”.

I think the one idea that we should take from all of these arguments is that if you are using a URL shortening service you have no control over your links, now or in the future. One such service may get bought up by a nasty commercial entity who redirects all your existing short URLs to its own ends. Your URLs pointing to all your lovingly curated content will, effectively, become spam.

One solution to this quandry, which no one has mentioned, and one which large media organisations should pay attention to, is that building a URL shortener is really, really easy. I spoke to Simon Willison about it and he thinks a URL shortener will soon be the example app that someone builds to learn their way around a new language or web framework, like they currently do with a creating a blog. That way you get to solve many problems at once: control over your own links’ destiny and complete consumer confidence that your own-brand short URLs (gu.com/abcde, nyt.com/vwxyz) won’t take them someone nasty.

And the ability to shorten your own URLs isn’t necessarily restricted to large companies with lots of resources. Many people who want to use this sort of service already have all the tools they need — their blogging software. All that Movable Type or Wordpress, among others, need to do is add an extra database lookup table and pretty soon all their users can take care of their own URL shortening needs.

UPDATE: A useful comparison of existing URL shortening services (even the method of redirection matters: 301 is a permanent redirect, 302 a temporary one, with implications for SEO and link credit).

Tags:
Posted in Development, Internet | 1 Comment »

Using Python with MySQL on Mac OSX 10.5 Leopard

Wednesday, March 18th, 2009

If you try to build the MySQL Python library on Mac OS X 10.5 (Leopard) you’ll get an error similar to this:

/usr/include/sys/types.h:92: error: duplicate ‘unsigned’
/usr/include/sys/types.h:92: error: two or more data types in declaration specifiers
error: command 'gcc' failed with exit status 1

I found the fix for this error here: http://www.keningle.com/?p=11, via a comment on this site: http://dotnet.org.za/ncode/archive/2007/01/31/setting-up-mysql-for-python-mysqldb-on-mac-os-x-2.aspx. It’s just a couple of lines in Terminal, adding a symlink so the library knows where to look to find the files.

Some background: Unless you’re using SQLite, you need to install a Python library to interface with your chosen database (PostgreSQL, MySQL or Oracle).

The MySQL library can be downloaded here: http://www.djangoproject.com/r/python-mysql/

But it seems there are a few problems running this library on Mac OS X 10.5 (Leopard), hence the above fix.

Tags:
Posted in Development, Python | Comments Off

You’ll miss me when I’m gone: IE6, cross-browser consistency and device independence

Saturday, February 14th, 2009

A flurry of IE6 related activity on the web this week coincided with a discussion we are having at The Guardian on the same subject. We have been talking about the relative benefits of keeping website performance in IE6 consistent with that of other browsers, and the disproportionate amount of work this requires on the parts of developers and the QA team. We’ve been trying to figure out better processes to reduce the number of styling bugs in IE6, while not compromising the user experience or the hard work put in by our design team.

It turns out people have surprisingly strong views on cross-browser consistency. For some, IE6 represents much more than just ‘a browser’. It also represents, variously: a large market share; an important group of corporate users; a user’s freedom to choose whichever device she wishes to browse the web. Once you start dropping a browser for technological reasons, the argument goes, you might as well arbitrarily drop support for anything which you consider below par – mobile browsers, text browsers, people with small monitors.

The opposing view says that IE6 is many years old and two versions out of date, a huge security risk and a drain on resources. We shouldn’t be pandering to slow or paranoid IT departments who refuse to upgrade their systems. Anyway no one chooses to use IE6, it is forced upon them by said IT departments.

I’m loath to branch the code to produce a separate version of the site for any reason, be it a device or a browser. But I also see the amount of pain IE6 causes developers, especially when they’re trying to do something fancy with JavaScript, and even more especially trying to do so without using a standard library which might easily provide you with cross-browser methods for doing stuff.

I support IE because I have to. But I do also believe strongly in wide accessibility, through as many devices as possible. We should assume nothing — nothing — about how our users access the web. But I don’t think this is the point here. The point here is adhering web standards, which apply to both code and content. Remember, the content itself — the information — usually isn’t broken. It’s what you’re trying to do with it that’s broken. The CSS and the JavaScript. Go back to Tim Berners-Lee’s 2002 document on universality and device independence for a lesson in what putting stuff on the web is all about. Work with the web, not against it. It’s really good at presenting and sharing text and pictures. But it’s not a magazine layout. Berners-Lee once said,

“Anyone who slaps a ‘this page is best viewed with Browser X’ label on a Web page appears to be yearning for the bad old days, before the Web, when you had very little chance of reading a document written on another computer, another word processor, or another network.”

We can infer from this that a site isn’t ‘best viewed in’ anything: it’s just ‘viewed’, however it might end up. So, yes, your site might look lovely, but if getting it there is so complex that it breaks browsers, or takes up 50% of your development time, then you’re plainly doing something wrong.

Try taking your page back to basics, get rid of the awful advertisement JavaScript and the three different kinds of page tracking, and start paying more than just lip-service to web standards and accessibility. That XHTML doctype declaration you’re using, trying adhering to it. There, it probably works a lot better now, yes?

But ultimately, and as usual, I think the whole issue comes down to a business decision: how much time/money are we spending on development versus how much money that development brings in. It’s a brave person who decides to cut off 25% of their users.

Some points that came up as part of our ongoing discussion:

  1. Should the design be 100% consistent across all browsers, or would our designers be happy to sacrifice certain style elements? We currently stop a code release if something looks bad in IE6, although we have already made one or two decisions to remove an element from IE6 in order to expedite a code release. In both cases we ran things past the Guardian’s Creative Editor, Mark Porter, before doing so.
  2. If you want to drop suport for IE6, you have to completely and utterly drop support for it. And in all likelihood never look at it again. Because the next time you do, it will be horrifically broken. Stopping development on that browser doesn’t just mean it won’t get cool new features. It still gets the features, but they won’t be tailored to it, and will break it. That smart Javascript widget you just wrote? That breaks the page in IE6. Some new element you put in with a fixed width and margins? That breaks the page in IE6. You have to cut the cord. Be strong, give it a firm handshake and say goodbye.
  3. Turns out Microsoft haven’t quite cut the cord yet, though. Microsoft support Windows XP Service Pack 3 as a current product (it shipped in April 2008), and will retire support for it 2 years after the next service pack is released, or at the end of the Windows XP product lifecycle, whichever comes first. IE6, which shipped as a component of XPSP3, continues to have Mainstream Support as part of that product:
  4. Our current browser usage figures look like this:
    • IE 7: 35%
    • IE 6: 25%
    • Firefox 3: 25%
    • Safari: 7%
    • Firefox 2: 3%
    • Google Chrome: 1.5%
    • Opera: 0.5%
  5. We currently have a problem even testing in IE6, because the corporate build on the PCs we use doesn’t contain it, it has IE7 as standard. And you can’t run IE7 and IE6 concurrently. Ironically, our technical infrastructure is sufficiently advanced that we have difficulty supporting old technology.

That flurry of activity in full:

Tags:
Posted in Computing, Development, Web design, Work | Comments Off

Newer Entries »