Tag Technology

For San Francisco, and governments everywhere, technology startups are the perfect partners

A recent article in the New York Times describes SMART Muni, “an Apple iPad app that uses Global Positioning System technology to track all of the city’s buses in real time, allowing transit managers and passengers to monitor problems and delays”. It sounds like the perfect success story: civic coders taking open data (Muni tracks its buses and trains with NextBus, which provides an XML data feed) and using that data to improve operations and create real value for the agency.

Unfortunately, it’s not a success story: the app has never been used in production. As the article explains, “Muni hopes to put the app to good use some day, but the agency is $29 million over budget and cannot afford to buy the iPads required to run the software…[nor] is the city willing to invest $100,000 to run a pilot program.”

The costs involved here—a few hundred dollars each for some iPads, perhaps a few thousand dollars to fund a stipend for a civic coder, even $100,000 for a pilot—pale in comparison to the costs associated with the big-name IT consulting firms that governments are used to dealing with.

In addition, startups, teams of civic coders, and open source projects can often deliver a working prototype or even a completed project much faster than conventional development teams. As the New York Times describes, “a small team of volunteers took just 10 days last summer to create [the app].”

Unfortunately, the City of San Francisco is out of touch with the realities of technology: “‘Start-ups fail at a high rate,’ said Jay Nath, chief innovation officer of San Francisco. ‘As stewards of taxpayer dollars, we need to be thoughtful of using that money wisely and not absorbing too much risk.’” Nath is right about one thing: start-ups do fail at an alarming rate. But that’s not the risk you might think it is, because startups aren’t like conventional development projects.

Unlike conventional projects, startups fail fast. Instead of wasting years and millions of dollars, when a startup has an idea that isn’t going anywhere, it winds up quickly. Maybe it was a bad (or even outright infeasible) idea to begin with, or the startup had the wrong team, or they tried to do too much at once. Maybe their idea’s been superseded by a newer, even better technology. Whatever the reason may be, the startup doesn’t just grind away for years, running up a million-dollar bill. Instead, they admit that they can’t deliver, and get out gracefully.

Consider, for example, the FBI’s Virtual Case File, a five-year, $170-million development effort that never actually delivered any working software. Imagine if the VCF project had failed after three or six months, not five years. Imagine if it had spent less than a million dollars before failing, not $170 million. Of course, the project still wouldn’t be done—but we’d have known that something was wrong up front, instead of finding out five years later, after millions of taxpayer dollars had been wasted on a doomed development effort.

More importantly, startups do have the agility necessary to keep up with the ever-changing technology marketplace. A development effort that takes five or ten years is bound to deliver a product that is obsolete as soon as it arrives, unless major changes are made along the way.

The conventional development practices used by many government agencies and their contractors don’t incorporate that kind of agility. Specifications and requirements are written early in the project’s life, perhaps even before a development team has been selected (if the project must be put out for bids). Even if the requirements are found to be lacking—or flat-out wrong—development marches on. In the end, the team will deliver a product that meets the requirements (thus satisfying the bean-counters) but which is already out-of-date and which doesn’t actually do what users need it to do.

I alluded to these problems in my recent coverage of WMATA’s initiative to install real-time information displays at bus stops. By only considering bids from vendors with “standard, proven products” and “successful existing and fully operational implementations, in multiple transit agencies”, they potentially shut out innovative startups (or even teams of civic coders, like the Mobility Lab).

It’s entirely possible that the first team to tackle a thorny problem may fail—but rather than casting them as “failures that burn holes in the city’s budget”, we’ve got to communicate to governments and taxpayers alike that not all failures are the same. There’s a big difference between a project that runs for years, spends millions of dollars, and has nothing to show for it in the end, and a project that fails after just a few months, has spent well less than a million dollars, and can identify what went wrong, so the next project will be more successful.

When it comes to technology, the best way for governments to be good ‘stewards of taxpayer dollars’ is to adopt successful development practices: small, agile, competent teams, that build inexpensive, flexible products, and fail quickly if they can’t get the job done. The old way—forking over millions and millions to high-priced contractors until they finally declare defeat, then taking it up in a years-long legal battle—just doesn’t look like good stewardship anymore. Sure, established companies may have a long track record that startups don’t, but what’s it a record of? We don’t need any more million-dollar failures. We need smart civic coders developing next-generation solutions like SMART Muni, and we need governments to accept, embrace, and support them.

Google Apps Script: like AppleScript for the Web

I’ve always been a huge fan of AppleScript for automating tasks in scriptable appplications and (more importantly) gluing scriptable applications together. Particularly when working with applications which are designed to take full advantage of AppleScript, like BibDesk, Delicious Library, and XTension, AppleScript makes even complex tasks easy. Unlike macros which are confined to a single application, AppleScript is based on top of Apple Events, making it easy to target any scriptable application, even on a remote Mac over the network. More importantly, AppleScripts aren’t macros; they don’t just play back keyboard and mouse events; you get a real object-oriented view of the data being manipulated. But really good scriptable applications are hard to come by, and of course AppleScript does you no good if you’re using cloud-based applications like Google Docs.

Browser automation tools, like Selenium, and libraries like mechanize help fill the gap somewhat, but they’re far from providing the same rich environment that AppleScript does. To give a concrete example, I was recently working on a spreadsheet listing Twitter accounts for the top 50 transit agencies in the US (more on that project here). In the spreadsheet, I’d listed agencies’ accounts by username (that is, @username). But what I really wanted was a link to each account on Twitter (that is, http://twitter.com/username). I could have entered the links manually, but that would have required needless manual work. If I were using a conventional spreadsheet application on the desktop, I could have used whatever macro or scripting facility it provided, or I could have exported the file to CSV and used sed and awk to get the job done. But I was working in the cloud; I knew there had to be a better way.

Enter Google Apps Script. Google Apps Script provides for Google’s cloud-based applications the same scriptability that AppleScript provides for desktop applications on the Mac. In only a few minutes, after studying the documentation, I was able to produce a script which achieved the desired effect.

Buoyed by my quick success, I decided to try going a step further: what if I could use the Twitter API to automatically set each cell’s comment to the most recent Tweet? Doing so would give viewers a quick preview of the Twitter account’s content, without leaving the spreadsheet. Working off of some sample code from Google, I quickly wrote another script to do the job. I ran into trouble for a while until I found that the “Callback URL” in the Twitter application settings must be set to https://spreadsheets.google.com/macros; once that was done, everything worked perfectly. (Incidentally, the error message given in that case, “unexpected error”, is completely useless, and gives no clue as to the actual problem.) From there, all I had to do was set up a time-based trigger to run the script automatically so the Tweets would update periodically, and I was done.

For me, the real point—and the power of Google Apps Script—is how quickly and easily I was able to not only automate otherwise-tedious processes, but draw in data from disparate sources and display it automatically. I’ve only scratched the surface of what can be done with Google Apps Script; the technology can be made to do a lot more.

Designers: please don’t design usability out of your sites

Over at Raschke on Transport, I recently wrote about a design competition called Life at the Speed of Rail. I had points to make about some of the winning entries, but was I able to link to them in the gallery on the LASR site? No. Why not? The LASR site is heavily JavaScript-dependent; it’s all one HTML page with a whole bunch of JavaScript-driven fake hyperlinks. Browse with JavaScript disabled? Too bad.

This isn’t meant solely as a criticism of the LASR site; I’ve noticed this trend on a lot of ‘design’ sites. Lots of flash (even if they’re not built with Flash), and not a lot of usability. These sites break the Web in a variety of ways; one of the more egregious ones is not using real hyperlinks. You can’t link to the individual pages within these ‘sites’, and they break the Back button, too. Broadly speaking, these are practices which break the Web. Unfortunately, it’s started to infect even mainstream Web sites, like the new Gawker and Twitter. This is, needless to say, a bad idea.

Fancy transitions between pages are neat, but there is no need for a static Web site to require JavaScript. Doing so goes completely against the notion of progressive enhancement. And while it’s (somewhat) reasonable for a Web application like Twitter or (especially) Gmail to require JavaScript, a static site—whether it’s the news or the results of a design competition—should not depend on JavaScript, and it should have real hyperlinks that actually work. That’s how the Web works. Fail to get that right, and you’ve broken the Web.

Updates to tph.py

I have made some updates to tph.py, my tool for generating plots of transit service levels from GTFS feeds. Most importantly, these updates fix compatibility issues which kept it from working with certain agencies’ GTFS feeds. To start, here are two plots generated from the BART GTFS feed, using the new version of tph.py.

The first plot’s not actually all that useful, since it mixes up rail service and the AirBART bus shuttle, but it does demonstrate a point, which will be explained later. The second plot is actually useful; it readily demonstrates that the BART system was not designed to provide frequent service to the extremities of the network—while service through the core peaks at 22 trains per hour, most branches on average get four trains per hour.

Anyway, supporting the BART GTFS feed required two major changes: supporting feeds which do not use the direction_id field, and supporting feeds which use the frequencies.txt file (which is used in the BART GTFS feed for AirBART, hence its inclusion above) rather than explicit stoptimes for every trip. As a result of these changes, tph.py should now support any GTFS feed. However, feeds which do not use the direction_id field do require additional configuration to assign directions to routes and trips. This is all documented in the new documentation on the configuration file format.

In addition, tph.py‘s innards have had an overhaul; it no longer uses Google’s transitfeed module for parsing GTFS feeds. Instead, it uses a fork of the gtfs module, originally developed by Brandon Martin-Anderson and William Lachance. gtfs imports the feed and stores it in a SQLite database, using SQLAlchemy. This takes time upfront, but makes tph.py a lot faster to run. It also makes the code cleaner; some operations which previously required several nested for loops can now be done with a single SQL query.

Kermit: more than a file transfer program, a tangible link to the past

Thirty years ago today, the first file transfer was made using the Kermit protocol, the brainchild of the Kermit Project. Originally designed to link workstations to mainframes, the Kermit Project grew to encompass software for dozens of platforms. The main versions of Kermit today, C-Kermit and Kermit 95, do more than just transfer files; they also function as terminal emulators with advanced scripting capabilities. But the Kermit Project will not see the end of its thirty-first year, for it has been terminated by Columbia University, effective July 1, 2011.

Some may consider this to be a small loss—after all, the Kermit software may seem like a relic of days past, when TCP/IP was not the common denominator it is today, and SSH did not yet even exist. But I think the real value in Kermit is not just the software itself, but in what it stands for. In an industry that seems to forget its history and tries to reinvent everything every few years, the Kermit Project has managed to keep the software up-to-date and relevant without losing touch with its roots. Kermit today will happily communicate with older systems, a fact taken advantage of by retrocomputing enthusiasts and technologists needing a way to migrate data off of dinosaurs. You may have a hard time making the physical connection—after all, serial ports have become awfully hard to find lately—but if you can, you’ll find that Kermit will happily communicate with a decades-old computer just as well as one from today.

At the same time, modern Kermit clients are also perfectly capable of working in today’s networked environment; Kermit clients today support network links over Telnet, FTP, HTTP, and SSH. And all of these features can be scripted using Kermit’s built-in scripting language. The scripting language takes some getting used to, but once you get the hang of it, it’s really quite useful.